Re: [fossil-users] Bug report: Terrible Performance, when Checking in LLVM Source
On Dec 4, 2016, at 12:50 PM, Karel Gardas wrote: > > On Sun, Dec 4, 2016 at 5:28 AM, Martin Vahi wrote: >> It is about 4.4GiB, over 100k files, over 6k folders, >> but it should not be that bad. After all, that's what >> many projects look like in 2016. > > This statement is IMHO a bit unfair. You basically grab *4* > subversions trees (if I count well) and you smash that to one big tree > and commit into fossil repo. So if you expect speed of svn, then > please compare fairly independent subversion trees with independent > fossil trees. Actually, you’d have to compare Fossil against checking out every single revision from each separate Subversion repository: $ for r in $(seq 1 $maxrev) ; do svn co -r $r … ; done A Subversion checkout just gets you the tip of trunk by default, and you have to go back to the server in order to walk back through history. Fossil grabs everything. There are plans laid out on the mailing list for making Fossil do both shallow and narrow checkouts. As it stands, Fossil always gives you a 100% wide and 100% deep checkout. Subversion allows both: it gives shallow checkouts always, and you can ask for a narrow checkout by specifying a subdirectory within the repo, grabbing only that slice. It’s a good idea. Someone just has to write it. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Bug report: Terrible Performance, when Checking in LLVM Source
On Sun, Dec 4, 2016 at 9:55 PM, Joerg Sonnenberger wrote: > No. What repo checksum does is compute a separate checksum over the > concatenation of all files. Uf, so I've been completely off the picture. Thanks for correction! ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Bug report: Terrible Performance, when Checking in LLVM Source
Hi, @Martin Vahi said : > « [...] CPU is AMD 64bit [...] » I don't know what AMD 64 means for you : it could be a dual core or a Quad core, and who knows an optocore... The amount of RAM could let the calculation of the Checksum be faster ... The bus speed can help a lot etc. > « [...] The uplink is ~1MiB/s (~10Mbps) and the ping from my local machine to the remote machine is about 20ms [...] » 1MiB : it's theorical I guess. You've said that files uploaded ONE by ONE should take 1 hour when a block of the equivalent data is 2 hours ... ?? No, when you process ONCE a copy it is faster than MANY little ones ... @Jungle said : > « Doesn't this depend on the CPU and available RAM? Search the archives for importing large repos » It is not importing it is exporting : "commit" Martin said. BTW, I agree with your FIRST question (RAM). @Karel said : > « [...] subversions trees [...] » So Martin should migrate the SVN trees to a git ones first and then import the git ones into a fossil one ... No ? > « [...] If however I'm still off, I would appreciate reference to some material explaining repo/chksums business in fossil. [...] » So do I :-) @Joerg said : > « [...] What repo checksum does is compute a separate checksum over the concatenation of all files [...] » Hmmm... You should say that EACH files are checksumed AND the repo itself is checksumed. This should explain why it takes so long for a large repo... No ? Best Regards K. De : Joerg Sonnenberger À : fossil-users@lists.fossil-scm.org Envoyé le : Dimanche 4 décembre 2016 20h55 Objet : Re: [fossil-users] Bug report: Terrible Performance, when Checking in LLVM Source On Sun, Dec 04, 2016 at 09:23:37PM +0100, Karel Gardas wrote: > On Sun, Dec 4, 2016 at 8:57 PM, Joerg Sonnenberger wrote: > > On Sun, Dec 04, 2016 at 08:50:44PM +0100, Karel Gardas wrote: > >> Otherwise as Nikita recommended, switching off repo checksums helps a > >> lot, but then make sure you are on the filesystem like ZFS/btrfs which > >> does that for you transparently and you do not need to do that on the > >> fossil side. > > > > Eh, no. You do not need a file system with automatic hashing. Every > > single file is still recorded by checksum in Fossil. It is not what the > > repo checksum option does. > > Errhmm, thanks for correction. Am I right that repo checksum switched > on means that modified files will be those where checksum stored and > checksum computed from the file on fs is different? And once you > switch that off, you rely purely on comparison of modification time on > file in fs and I guess stored modif time in repo db? If so, then > indeed I've been completely mistaken and thank you very much for your > kind correction. If however I'm still off, I would appreciate > reference to some material explaining repo/chksums business in fossil. No. What repo checksum does is compute a separate checksum over the concatenation of all files. Joerg ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Bug report: Terrible Performance, when Checking in LLVM Source
On Sun, Dec 04, 2016 at 09:23:37PM +0100, Karel Gardas wrote: > On Sun, Dec 4, 2016 at 8:57 PM, Joerg Sonnenberger wrote: > > On Sun, Dec 04, 2016 at 08:50:44PM +0100, Karel Gardas wrote: > >> Otherwise as Nikita recommended, switching off repo checksums helps a > >> lot, but then make sure you are on the filesystem like ZFS/btrfs which > >> does that for you transparently and you do not need to do that on the > >> fossil side. > > > > Eh, no. You do not need a file system with automatic hashing. Every > > single file is still recorded by checksum in Fossil. It is not what the > > repo checksum option does. > > Errhmm, thanks for correction. Am I right that repo checksum switched > on means that modified files will be those where checksum stored and > checksum computed from the file on fs is different? And once you > switch that off, you rely purely on comparison of modification time on > file in fs and I guess stored modif time in repo db? If so, then > indeed I've been completely mistaken and thank you very much for your > kind correction. If however I'm still off, I would appreciate > reference to some material explaining repo/chksums business in fossil. No. What repo checksum does is compute a separate checksum over the concatenation of all files. Joerg ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Bug report: Terrible Performance, when Checking in LLVM Source
On Sun, Dec 4, 2016 at 8:57 PM, Joerg Sonnenberger wrote: > On Sun, Dec 04, 2016 at 08:50:44PM +0100, Karel Gardas wrote: >> Otherwise as Nikita recommended, switching off repo checksums helps a >> lot, but then make sure you are on the filesystem like ZFS/btrfs which >> does that for you transparently and you do not need to do that on the >> fossil side. > > Eh, no. You do not need a file system with automatic hashing. Every > single file is still recorded by checksum in Fossil. It is not what the > repo checksum option does. Errhmm, thanks for correction. Am I right that repo checksum switched on means that modified files will be those where checksum stored and checksum computed from the file on fs is different? And once you switch that off, you rely purely on comparison of modification time on file in fs and I guess stored modif time in repo db? If so, then indeed I've been completely mistaken and thank you very much for your kind correction. If however I'm still off, I would appreciate reference to some material explaining repo/chksums business in fossil. Thanks! Karel ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Bug report: Terrible Performance, when Checking in LLVM Source
On Sun, Dec 04, 2016 at 08:50:44PM +0100, Karel Gardas wrote: > Otherwise as Nikita recommended, switching off repo checksums helps a > lot, but then make sure you are on the filesystem like ZFS/btrfs which > does that for you transparently and you do not need to do that on the > fossil side. Eh, no. You do not need a file system with automatic hashing. Every single file is still recorded by checksum in Fossil. It is not what the repo checksum option does. Joerg ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Bug report: Terrible Performance, when Checking in LLVM Source
On Sun, Dec 4, 2016 at 5:28 AM, Martin Vahi wrote: > It is about 4.4GiB, over 100k files, over 6k folders, > but it should not be that bad. After all, that's what > many projects look like in 2016. This statement is IMHO a bit unfair. You basically grab *4* subversions trees (if I count well) and you smash that to one big tree and commit into fossil repo. So if you expect speed of svn, then please compare fairly independent subversion trees with independent fossil trees. Otherwise as Nikita recommended, switching off repo checksums helps a lot, but then make sure you are on the filesystem like ZFS/btrfs which does that for you transparently and you do not need to do that on the fossil side. ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Bug report: Terrible Performance, when Checking in LLVM Source
Try setting repo-cksum property off, it should help a lot. By default Fossil double-checks all the files on disk to make sure their contents is not changed. That's fine for a small project, but for large ones Nikita On Sat, Dec 3, 2016 at 8:29 PM Martin Vahi wrote: > Reproduction: > > 1) Download LLVM source. >It might be done by executing the bash script from > > > http://www.softf1.com/cgi-bin/tree1/technology/flaws/mmmv_parasail_projects.bash/artifact/3a28f1fb67a5d860 > > 2) Commit the source. > > It is about 4.4GiB, over 100k files, over 6k folders, > but it should not be that bad. After all, that's what > many projects look like in 2016. > > > Regards, > martin.v...@softf1.com > > ___ > fossil-users mailing list > fossil-users@lists.fossil-scm.org > http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users > ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Bug report: Terrible Performance, when Checking in LLVM Source
On 12/03/2016 08:28 PM, Martin Vahi wrote: Reproduction: 1) Download LLVM source. It might be done by executing the bash script from http://www.softf1.com/cgi-bin/tree1/technology/flaws/mmmv_parasail_projects.bash/artifact/3a28f1fb67a5d860 2) Commit the source. It is about 4.4GiB, over 100k files, over 6k folders, but it should not be that bad. After all, that's what many projects look like in 2016. Doesn't this depend on the CPU and available RAM? Search the archives for importing large repos. This very likely is not an issue with your network connection. Regards, martin.v...@softf1.com ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
[fossil-users] Bug report: Terrible Performance, when Checking in LLVM Source
Reproduction: 1) Download LLVM source. It might be done by executing the bash script from http://www.softf1.com/cgi-bin/tree1/technology/flaws/mmmv_parasail_projects.bash/artifact/3a28f1fb67a5d860 2) Commit the source. It is about 4.4GiB, over 100k files, over 6k folders, but it should not be that bad. After all, that's what many projects look like in 2016. Regards, martin.v...@softf1.com ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users