On Mon, Jul 21, 2014 at 7:53 AM, Gour <g...@atmarama.net> wrote: > Stephan Beal <sgb...@googlemail.com> writes: > > > i don't have any more ideas off-hand, but i've never worked with repos > > having anywhere near that many files. > > After seeing that Eric's advice to turn checksumming off helps, I wonder > if there is something which can be done to make Fossil operate better in > this use-case? >
Thanks @Eric - i had forgotten about the R-card! > So, the whole repo/project is not Linux kernel in size, but those small > file accumulated in the course of time making the size of the whole repo > 6G (uncompressed) with very long commit time when checksum checking is > on? > Checksumming, in this sense, is the generation of the so-called R-card. The R-card is the 3rd or 4th line of defense against corruption, is _very_ costly to calculate, and is very possibly overkill. In libfossil i have been wavering back and forth for months about whether to disable it by default (more due to memory costs than speed). Let me say that when I was using this repo with Git I never had any > performance problem, but it's understandable considering its different > storage design and use of GC periodically. > The problem here is how the R-card card is calculated. It has to perform an md5 against ALL files in the repo (all 5000+ of them) for EVERY commit, no matter how small the change. To get the md5, it has to extract each full file from the db, which itself is very memory-hungry and may require traversing/undeltifying/decompressing an arbitrary number of versions of each file. We might want to consider a heuristic which automatically disables repo-cksum once a repo reaches a certain file count. Or disabling it by default (AFAIK the R-card has never once revealed any corruption which has slipped through other checks). -- ----- stephan beal http://wanderinghorse.net/home/stephan/ http://gplus.to/sgbeal "Freedom is sloppy. But since tyranny's the only guaranteed byproduct of those who insist on a perfect world, freedom will have to do." -- Bigby Wolf
_______________________________________________ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
