[fossil-users] slow commit
Hello, I've Fossil repo mostly containing uncompressed Gnucash XML files and commits even for trivial changes are quite slow: $ time fossil ci -m update logs You need a passphrase to unlock the secret key for user: Gour-Gadadhara Dasa g...@atmarama.net 4096-bit RSA key, ID 52B5C810, created 2011-04-02 New_Version: 6d606bba0322d91e38a5e36541264ff3e9f0036b 24.30user 0.55system 0:27.02elapsed 91%CPU (0avgtext+0avgdata 40804maxresident)k 0inputs+7304outputs (0major+39916minor)pagefaults 0swaps The diff with the previous version (long lines are wrapped) was just adding small *.log file: - mod trans_guid split_guid time_now date_entered date_posted acc_guid acc_name num description notes memo action reconciled amount value date_reconciled - = START = END = START = END = START = END = START = END = START = END = START D 3d73c5e0791e2f4e6ae85c728efc9702 5b5cecdd6785944a87d6beefae4d0746 2014-07-20 13:41:32.00 +0200 2014-07-20 13:41:32.00 +0200 2014-07-20 00:00:00.00 +0200 n 0/1 0/100 1970-01-01 01:00:00.00 +0100 = END = START D 13941b1b66123e0d0a487c21022dc2cd df0c47007052f8b1cddb6abc846eceb4 2014-07-20 13:41:52.00 +0200 2014-07-20 13:41:52.00 +0200 2014-07-20 00:00:00.00 +0200 n 0/1 0/100 1970-01-01 01:00:00.00 +0100 = END = START D 540642d9cc3486fa01d3c72cf16800b3 13bdd42ca95ba03a36abe86d320099ce 2014-07-20 13:41:52.00 +0200 2014-07-20 13:41:52.00 +0200 2014-07-20 00:00:00.00 +0200 n 0/1 0/100 1970-01-01 01:00:00.00 +0100 = END = START D df7895bb0f1855cef57cd747e522ee25 c8a5eda78157df75ef17d93a416985c6 2014-07-20 13:41:52.00 +0200 2014-07-20 13:41:52.00 +0200 2014-07-20 00:00:00.00 +0200 n 0/1 0/100 1970-01-01 01:00:00.00 +0100 = END = START D 92528876b4567dad73a7f16e15052207 359d5603f0b281007d91f46bcd56731b 2014-07-20 13:41:52.00 +0200 2014-07-20 13:41:52.00 +0200 2014-07-20 00:00:00.00 +0200 n 0/1 0/100 1970-01-01 01:00:00.00 +0100 = END - I wonder if you have any idea what to do in order to speed up commit time? Here is the dbstat output: repository-size: 406535168 bytes (406.5MB) artifact-count:7071 (stored as 5191 full text and 1880 delta blobs) artifact-sizes:850325 average, 4800459 max, 6011800801 bytes (6.0GB) total compression-ratio: 14:1 checkins: 973 files: 5843 across all branches wikipages: 0 (0 changes) tickets: 0 (0 changes) events:0 tagchanges:18 project-age: 888 days or approximately 2.43 years. project-id:de5578af6cd61b7f042995c2d074bc2667eb2500 server-id: dbd6c972edfd3f6539db54b92adfa4c65e269197 fossil-version:2014-07-19 19:20:25 [619fa857c9] [1.30] (gcc-4.9.0) sqlite-version:2014-07-01 11:54:02 [21981e3506] (3.8.6) database-stats:397007 pages, 1024 bytes/pg, 13 free pages, UTF-8, delete mode Any hint? -- A person who is not disturbed by the incessant flow of desires — that enter like rivers into the ocean, which is ever being filled but is always still — can alone achieve peace, and not the man who strives to satisfy such desires. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] slow commit
On Sun, Jul 20, 2014 at 1:52 PM, Gour g...@atmarama.net wrote: New_Version: 6d606bba0322d91e38a5e36541264ff3e9f0036b 24.30user 0.55system 0:27.02elapsed 91%CPU (0avgtext+0avgdata 40804maxresident)k 0inputs+7304outputs (0major+39916minor)pagefaults 0swaps ... checkins: 973 files: 5843 across all branches ... i'm just shooting in the dark here, but here's what i'd try first: fossil set mtime-changes 1 then, on the next commit: fossil commit -m ... --delta i'm assuming that your repo does not already use delta manifests, but the above will allow it to. Once fossil has ever created a delta for a given repo, it will continue to do so when it makes sense to do so (it falls back to a non-delta every so-many checkins, depending on how many total files have changed between the current version and the last baseline manifest). Can you try that out? -- - stephan beal http://wanderinghorse.net/home/stephan/ http://gplus.to/sgbeal Freedom is sloppy. But since tyranny's the only guaranteed byproduct of those who insist on a perfect world, freedom will have to do. -- Bigby Wolf ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] slow commit
Thus said Stephan Beal on Sun, 20 Jul 2014 14:03:12 +0200: fossil commit -m ... --delta Excellent! I always wondered how to turn on delta manifests. Does this turn it on for the entire repo, or just the files in the commit? Thanks, Andy -- TAI64 timestamp: 400053cbde68 ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] slow commit
On Sun, Jul 20, 2014 at 5:20 PM, Andy Bradford amb-fos...@bradfords.org wrote: Thus said Stephan Beal on Sun, 20 Jul 2014 14:03:12 +0200: fossil commit -m ... --delta Excellent! I always wondered how to turn on delta manifests. Does this turn it on for the entire repo, or just the files in the commit? Short answer: the given repository and (if i'm not sorely mistaken) clones derived from it. Details: by default fossil won't create any delta manifests, but once a given repo has seen any single delta (because you did --delta locally or your pulled from someone who did) then fossil says, okay, we'll consider deltas as an option for future commits. Whether or not a given checkin will be a delta is not knowable unless you know the algorithm with which it calculates the cost of a delta (basically a ratio of the number of files in the baseline vs the number of files which have changed since then). If the cost is too high, fossil tosses out the delta (which it had to generate in order to know its cost) and generates a baseline manifest instead. Looking now, the core fossil repo still has no delta manifests - nobody has been brave enough to do it, i guess ;). libfossil does, though: [stephan@host:~/cvs/fossil/libfossil]$ f-acat tip | head B 210ba2cf235e153771617d233da417c5f2433d21 C upstream\ss2\sfor\ss2.ob.pop()\ssemantics\schange. D 2014-07-20T11:28:19.480 F auto.def 420dbacf2fa9676ca1b57c7feefaba6ae5612091 F config.make.in 47e414a0c535712c590ec5c76cbac1249768aabf ... That first line (the B-card) tells us that this is a delta manifest derived from the B-card's checkin (a.k.a. manifest). The difference is this: [stephan@host:~/cvs/fossil/libfossil]$ f-acat tip | grep -c '^F ' 21 [stephan@host:~/cvs/fossil/libfossil]$ f-acat 210ba2cf235e153771617d233da417c5f2433d21 | grep -c '^F ' 279 Those are the number of files (F-cards) in the tip resp. its baseline manifest version (not its direct parent, which is a different artifact altogether). Once that number 21 grows too high (i _think_ it's around 30-something, but would have to go back and look at the math), the checkin which would have made it too high (==costly) will become a new baseline. -- - stephan beal http://wanderinghorse.net/home/stephan/ http://gplus.to/sgbeal Freedom is sloppy. But since tyranny's the only guaranteed byproduct of those who insist on a perfect world, freedom will have to do. -- Bigby Wolf ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] slow commit
Stephan Beal sgb...@googlemail.com writes: fossil set mtime-changes 1 Strange that it was 'off', although default should be 'on'. If not set, Fossil checks hashes, right? then, on the next commit: fossil commit -m ... --delta Did that, but not much improvement so far. Can you try that out? I also tried to disable gpg-signing, but not much influence. It seems that I need to wait some time or more commits to see how it goes. Sincerely, Gour -- He is a perfect yogī who, by comparison to his own self, sees the true equality of all beings, in both their happiness and their distress, O Arjuna! ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] slow commit
On Sun, Jul 20, 2014 at 5:58 PM, Gour g...@atmarama.net wrote: Stephan Beal sgb...@googlemail.com writes: fossil set mtime-changes 1 Strange that it was 'off', although default should be 'on'. If not set, Fossil checks hashes, right? Correct about the hashing, but i don't recall what the default is/should be. i suspect that hashing is the default because it's guaranteed to be correct, whereas mtime checks are a close (and much faster) approximation littered with corner cases. then, on the next commit: fossil commit -m ... --delta Did that, but not much improvement so far. Bummer :(. Can you try that out? I also tried to disable gpg-signing, but not much influence. It seems that I need to wait some time or more commits to see how it goes. i don't have any more ideas off-hand, but i've never worked with repos having anywhere near that many files. Maybe a list-member who has can suggest something. Maybe it's something as simple as changing the sqlite3 write mode (and maybe it's not). Your repo isn't on a network mount, is it? If so, all bets are off - network drives are generally problematic for i/o-heavy apps. -- - stephan beal http://wanderinghorse.net/home/stephan/ http://gplus.to/sgbeal Freedom is sloppy. But since tyranny's the only guaranteed byproduct of those who insist on a perfect world, freedom will have to do. -- Bigby Wolf ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] slow commit
Stephan Beal sgb...@googlemail.com writes: i don't have any more ideas off-hand, but i've never worked with repos having anywhere near that many files. Maybe a list-member who has can suggest something. Maybe it's something as simple as changing the sqlite3 write mode (and maybe it's not). OK. Your repo isn't on a network mount, is it? If so, all bets are off - network drives are generally problematic for i/o-heavy apps. Nope...just a local filesystem. Sincerely, Gour -- Never was there a time when I did not exist, nor you, nor all these kings; nor in the future shall any of us cease to be. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] slow commit
Gour wrote: Stephan Beal sgb...@googlemail.com writes: i don't have any more ideas off-hand, but i've never worked with repos having anywhere near that many files. Maybe a list-member who has can suggest something. Maybe it's something as simple as changing the sqlite3 write mode (and maybe it's not). OK. Maybe try this? fossil setting repo-cksum off Then re-test. -- Eric A. Rubin-Smith ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
[fossil-users] Seeking comments on new comment formatting...
I've added a new comment printing algorithm on the 'experimental' branch. By default, it uses the existing algorithm. The --comfmtflags option can be used to enable the new algorithm. The goals of the new algorithm are to preserve the existing formatting already present within the comment itself while honoring line length limitations. The new algorithm does not try to be overly smart; however, it is capable of performing line breaks on word boundaries (when that flag is enabled). I'm looking for some feedback and/or suggestions on how to improve it. With the --comfmtflags command line option, there are three flag values that may be combined (from the comments in the comformat.c file): COMMENT_PRINT_LEGACY (1): Forces use of the legacy comment printing algorithm. For backward compatibility, this is the default. COMMENT_PRINT_TRIM_SPACE (2): Trims leading and trailing spaces where they do not materially impact formatting (i.e. at the start of the comment string -AND- right before each line indentation). This flag does not apply to the legacy comment printing algorithm. COMMENT_PRINT_WORD_BREAK (4): Attempts to break lines on word boundaries while honoring the logical line length. If this flag is not specified, honoring the logical line length may result in breaking lines in the middle of words. This flag does not apply to the legacy comment printing algorithm. -- Joe Mistachkin ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] slow commit
Eric Rubin-Smith eas@gmail.com writes: Maybe try this? fossil setting repo-cksum off Then re-test. That had drastic effect: $ time fossil ci -m update logs You need a passphrase to unlock the secret key for user: Gour-Gadadhara Dasa g...@atmarama.net 4096-bit RSA key, ID 52B5C810, created 2011-04-02 New_Version: ddf259d88381967e3c19c73141f59193326a1c8f 0.32user 0.04system 0:02.55elapsed 14%CPU (0avgtext+0avgdata 7688maxresident)k 72inputs+7976outputs (0major+3087minor)pagefaults 0swaps but I wonder how safe is to operate Fossil repo with checksum checking off? Sincerely, Gour -- The senses are so strong and impetuous, O Arjuna, that they forcibly carry away the mind even of a man of discrimination who is endeavoring to control them. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] slow commit
Stephan Beal sgb...@googlemail.com writes: i don't have any more ideas off-hand, but i've never worked with repos having anywhere near that many files. After seeing that Eric's advice to turn checksumming off helps, I wonder if there is something which can be done to make Fossil operate better in this use-case? $du -h financije 789Mfinancije/ while the main file in the repo is 4.6M. The rest of the repo is populate with many files which are often very small (few K) and very short-lived, iow. those are so called *.log files of backup data files which are handy when/if one has to restore some old transactions in case something goes wrong (I had to use it several times), but the point is that, according to docs, Gnucash will remove them after a configurable amount of time (Preferences/General/Days to retain log files). So, the whole repo/project is not Linux kernel in size, but those small file accumulated in the course of time making the size of the whole repo 6G (uncompressed) with very long commit time when checksum checking is on? Let me say that when I was using this repo with Git I never had any performance problem, but it's understandable considering its different storage design and use of GC periodically. Sincerely, Gour -- Thus the wise living entity's pure consciousness becomes covered by his eternal enemy in the form of lust, which is never satisfied and which burns like fire. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
