[fossil-users] Update performance for large working copies
Hi all, in case someone has time to fix this, let me write up the most annoying performance issue for larger repositories. When running fossil update, it will rewrite vfile table in .fslckout from scratch, even though most entries should not change on merges. If the working copy contains a few thousand files, that starts to take a very long time... Joerg ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Update performance for large working copies
On Tue, Jul 16, 2013 at 8:37 AM, Joerg Sonnenberger jo...@britannica.bec.de wrote: Hi all, in case someone has time to fix this, let me write up the most annoying performance issue for larger repositories. When running fossil update, it will rewrite vfile table in .fslckout from scratch, even though most entries should not change on merges. If the working copy contains a few thousand files, that starts to take a very long time... Should that say few hundred thousand instead of few thousand? -- D. Richard Hipp d...@sqlite.org ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Update performance for large working copies
On Tue, Jul 16, 2013 at 08:46:06AM -0400, Richard Hipp wrote: On Tue, Jul 16, 2013 at 8:37 AM, Joerg Sonnenberger jo...@britannica.bec.de wrote: Hi all, in case someone has time to fix this, let me write up the most annoying performance issue for larger repositories. When running fossil update, it will rewrite vfile table in .fslckout from scratch, even though most entries should not change on merges. If the working copy contains a few thousand files, that starts to take a very long time... Should that say few hundred thousand instead of few thousand? Just a matter of scale :) Essentially, the problem is that t(fossil update) = O(files in the working copy), when it should be O(files changed). Joerg ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Update performance for large working copies
On Tue, Jul 16, 2013 at 2:51 PM, Joerg Sonnenberger jo...@britannica.bec.de wrote: Just a matter of scale :) Essentially, the problem is that t(fossil update) = O(files in the working copy), when it should be O(files changed). Just out of curiosity - is this with the mtime-changes setting enabled or disabled? It seems to me (perhaps naively) that fossil must first understand what has changed, which means (at least in principal) checking all current entries in the repo, so i would expect O(N), N==number of files in current branch (or thereabouts). -- - stephan beal http://wanderinghorse.net/home/stephan/ http://gplus.to/sgbeal ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Update performance for large working copies
On Tue, Jul 16, 2013 at 03:17:10PM +0200, Stephan Beal wrote: On Tue, Jul 16, 2013 at 2:51 PM, Joerg Sonnenberger jo...@britannica.bec.de wrote: Just a matter of scale :) Essentially, the problem is that t(fossil update) = O(files in the working copy), when it should be O(files changed). Just out of curiosity - is this with the mtime-changes setting enabled or disabled? Enabled. It seems to me (perhaps naively) that fossil must first understand what has changed, which means (at least in principal) checking all current entries in the repo, so i would expect O(N), N==number of files in current branch (or thereabouts). No, it builds a change list by comparing old and new manifest. That part is fast. The problem is that it is completely rewriting the table after that. Joerg ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
[fossil-users] By week-of-year statistics
Hi, all, i've just added week-of-year (a.k.a. calendar week[1]) support to /stats_report and /timeline: http://fossil.wanderinghorse.net/repos/cwal/index.cgi/stats_report?view=byyear and i'm looking for input on the following: a) When showing by-week links for the combined year/month view it's way too slow, so i've disabled that: http://fossil-scm.org/index.html/info/0019550a03 (see the last part of that diff) If someone can suggest a way to speed that up (it's currently taking 4.5s on my machine with the main fossil repo) i would be much obliged. b) If someone with a better eye for design (that'd be just about anyone!) can suggest a less ugly/cramped presentation for how the week numbers are listed (see the first link above), i'd likewise be much obliged. Thank you in advance! [1]: it uses whatever definition sqlite3 uses for week of year, and that's non-negotiable ;). -- - stephan beal http://wanderinghorse.net/home/stephan/ http://gplus.to/sgbeal ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] By week-of-year statistics
On Tue, Jul 16, 2013 at 5:57 PM, Stephan Beal sgb...@googlemail.com wrote: b) If someone with a better eye for design (that'd be just about anyone!) can suggest a less ugly/cramped presentation for how the week numbers are listed (see the first link above), i'd likewise be much obliged. Here's a slightly more-filled-out one: http://fossil.wanderinghorse.net/repos/whio/index.cgi/stats_report?view=byyear but the main fossil repo is the best place to get an idea of how it looks with all weeks of a year being represented. -- - stephan beal http://wanderinghorse.net/home/stephan/ http://gplus.to/sgbeal ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] By week-of-year statistics
This looks cool. Will this data be available via the json interface? Being able to aggregate the stats over a lot of different fossils would be useful to some of us. On Tue, Jul 16, 2013 at 9:10 AM, Stephan Beal sgb...@googlemail.com wrote: On Tue, Jul 16, 2013 at 5:57 PM, Stephan Beal sgb...@googlemail.comwrote: b) If someone with a better eye for design (that'd be just about anyone!) can suggest a less ugly/cramped presentation for how the week numbers are listed (see the first link above), i'd likewise be much obliged. Here's a slightly more-filled-out one: http://fossil.wanderinghorse.net/repos/whio/index.cgi/stats_report?view=byyear but the main fossil repo is the best place to get an idea of how it looks with all weeks of a year being represented. -- - stephan beal http://wanderinghorse.net/home/stephan/ http://gplus.to/sgbeal ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users -- Matt -=- 90% of the nations wealth is held by 2% of the people. Bummer to be in the majority... ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] By week-of-year statistics
On Tue, Jul 16, 2013 at 6:49 PM, Matt Welland estifo...@gmail.com wrote: This looks cool. Will this data be available via the json interface? Being able to aggregate the stats over a lot of different fossils would be useful to some of us. The thought has crossed my mind, but i haven't decided upon an interface for it. /stats_report is only a couple months old and i haven't hacked on the JSON bits in that time. Do you have a particular JSON interface in mind for such reports? -- - stephan beal http://wanderinghorse.net/home/stephan/ http://gplus.to/sgbeal ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] By week-of-year statistics
On Tue, Jul 16, 2013 at 6:49 PM, Matt Welland estifo...@gmail.com wrote: This looks cool. Will this data be available via the json interface? Being able to aggregate the stats over a lot of different fossils would be useful to some of us. New version: http://fossil.wanderinghorse.net/repos/cwal/index.cgi/stats_report then select the By week link, or just click this: http://fossil.wanderinghorse.net/repos/cwal/index.cgi/stats_report?view=byweek @Richard: as i recall, that's a report you specifically asked for? -- - stephan beal http://wanderinghorse.net/home/stephan/ http://gplus.to/sgbeal ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] By week-of-year statistics
Very cool, Stephan! On Jul 16, 2013 1:30 PM, Stephan Beal sgb...@googlemail.com wrote: On Tue, Jul 16, 2013 at 6:49 PM, Matt Welland estifo...@gmail.com wrote: This looks cool. Will this data be available via the json interface? Being able to aggregate the stats over a lot of different fossils would be useful to some of us. New version: http://fossil.wanderinghorse.net/repos/cwal/index.cgi/stats_report then select the By week link, or just click this: http://fossil.wanderinghorse.net/repos/cwal/index.cgi/stats_report?view=byweek @Richard: as i recall, that's a report you specifically asked for? -- - stephan beal http://wanderinghorse.net/home/stephan/ http://gplus.to/sgbeal ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] By week-of-year statistics
On Tue, Jul 16, 2013 at 5:57 PM, Stephan Beal sgb...@googlemail.com wrote: b) If someone with a better eye for design (that'd be just about anyone!) can suggest a less ugly/cramped presentation for how the week numbers are listed (see the first link above), i'd likewise be much obliged. Ignore that - that view was scrapped and replaced with a separate page: http://fossil.wanderinghorse.net/repos/cwal/index.cgi/stats_report?view=byweek @Brad/Matt: improvement suggestions are welcomed. -- - stephan beal http://wanderinghorse.net/home/stephan/ http://gplus.to/sgbeal ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] By week-of-year statistics
Of top of head, reflexive position is that week numbers are running in wrong order, but if it's reversed, latest would be at bottom of page. That made me wonder though, what of vertical bars vs. horizontal? Just tossing ideas... On Jul 16, 2013 1:50 PM, Stephan Beal sgb...@googlemail.com wrote: On Tue, Jul 16, 2013 at 5:57 PM, Stephan Beal sgb...@googlemail.comwrote: b) If someone with a better eye for design (that'd be just about anyone!) can suggest a less ugly/cramped presentation for how the week numbers are listed (see the first link above), i'd likewise be much obliged. Ignore that - that view was scrapped and replaced with a separate page: http://fossil.wanderinghorse.net/repos/cwal/index.cgi/stats_report?view=byweek @Brad/Matt: improvement suggestions are welcomed. -- - stephan beal http://wanderinghorse.net/home/stephan/ http://gplus.to/sgbeal ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
[fossil-users] Shared SSH account consideration.
Hello, After giving a little though to handling shared SSH accounts, it might be as simple as the following change: https://www.fossil-scm.org/index.html/info/7a10b79a2c Basically, if the specified SSH command already has an '@' in it, don't use the username@host found in the URL, but use the username@ to provide Fossil authentication token once the SSH connection is established: local$ fossil clone -c 'ssh -e none -T guest@remote' ssh://amb@remote//tmp/test.fossil /tmp/test.fossil password for amb: remember password (Y/n)? y Round-trips: 1 Artifacts sent: 0 received: 0 ssh -e none -T guest@remote fossil http /tmp/test.fossil guest@remote's password: Round-trips: 2 Artifacts sent: 0 received: 1 ssh -e none -T guest@remote fossil http /tmp/test.fossil guest@remote's password: Round-trips: 2 Artifacts sent: 0 received: 3 Clone finished with 547 bytes sent, 1158 bytes received Rebuilding repository meta-data... 100.0% complete... project-id: 125ab947e5a9aff39560a0d60dddccac37f575fa admin-user: amb (password is 85cf0e) Then, after opening and making a change: local$ fossil commit -m test Autosync: ssh://amb:*@remote//tmp/test.fossil Round-trips: 1 Artifacts sent: 0 received: 0 ssh -e none -T guest@remote fossil http /tmp/test.fossil guest@remote's password: Round-trips: 1 Artifacts sent: 0 received: 0 Pull finished with 349 bytes sent, 286 bytes received New_Version: 6d982f5dfbfcf4d283bbb073fc04dfd7c50f5b32 Autosync: ssh://amb:*@remote//tmp/test.fossil Round-trips: 1 Artifacts sent: 2 received: 0 ssh -e none -T guest@remote fossil http /tmp/test.fossil guest@remote's password: Round-trips: 1 Artifacts sent: 2 received: 0 Sync finished with 542 bytes sent, 338 bytes received The info stored with the commit on the remote host shows that the changes were made by amb (not guest) coming from the local host's IP. So, it uses the SSH connection established by the guest@remote account, but still relies on the fossil credentials to be able to control commit. Of course this probably really only makes sense when using forced command SSH keys, but it does seem like a pretty direct way to be able to support a shared SSH account configuration. Thoughts? Thanks, Andy -- TAI64 timestamp: 400051e61a05 ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
