Re: [fossil-users] Working with large repository
On Mon, 16 Jun 2014 23:08:22 +0200 Baptiste Daroussin baptiste.darous...@gmail.com wrote: Hi, I just want to share some feedback, for fun I tried to convert some of the FreeBSD repositories to fossil to see how it performs I first tried the FreeBSD Documentation repository, to go the easy way I took the FreeBSD git mirror (https://github.com/freebsd/freebsd-doc and run git fast-export --full-tree --all | fossil import --git ../doc.fossil The worked pretty well, and I manage to convert in a few hour the repo, the result it very usable (that wasn't the case a few years ago, some operations where slow) Result is available here: https://fossil.etoilebsd.net/doc if some want to play. At the same time I decided to try with ports and base repository which I would have expected to be a bit more complicated I cloned: https://github.com/freebsd/freebsd and https://github.com/freebsd/freebsd-ports I run the same command for the conversion git fast-export --full-tree --all | fossil import --git ../ports.fossil both are now running for more than 70H and still not finished!!! consuming 100% CPU This is running on pretty descent hardware. hw.model: Intel(R) Xeon(R) CPU E3-1225 V2 @ 3.20GHz I guess I would have to run a svndump imported manually if I want a slightly faster conversion to happen, and given how the incremental import works with fossil import --git I cannot imagine running a fossil mirror using that tool. If anyone has ideas on how I can maintain a mirror more easily, tips welcome I tried the same in 2012 an early 2013 without --full-tree, the worse problem I had wasn't the time expended but the size of fossil file while importing. I tweaked fossil import to use 512MB-2GB of sqlite3 cache making it faster. I didn't import from git directly, I splitted it in 2 phases: git fast-export xz -vv --lzma2=preset3,dict=256MiB git.xz xzcat -vv git.xz fossil import git.fossil This way I'm less IO and CPU bounded, Check my email on this list 1st of July 2013. Bapt --- --- Eduardo Morras emorr...@yahoo.es ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Working with large repository
On Tue, Jun 17, 2014 at 5:30 PM, Eduardo Morras emorr...@yahoo.es wrote: git fast-export xz -vv --lzma2=preset3,dict=256MiB git.xz xzcat -vv git.xz fossil import git.fossil That is a _very_ interesting trick. Question to you both: do you know if fossil has generated any delta manifests in the import? By default fossil does not generate delta manifests, but they can potentially save space for large repos. Once fossil has generated _one_ delta, it will allow itself to generate more, but by default it will never generate any. A delta can be forced at commit time with --delta, but i don't know if the git import feature (never needed it) supports something like that. -- - stephan beal http://wanderinghorse.net/home/stephan/ http://gplus.to/sgbeal Freedom is sloppy. But since tyranny's the only guaranteed byproduct of those who insist on a perfect world, freedom will have to do. -- Bigby Wolf ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Working with large repository
I'll be able to tell you if ever this conversion finishes :) right now 88H and still running @eduardo at the time iirc the git export did not contain the full history but a small part of it, right now for sure it contains the full history 2014-06-17 17:39 GMT+02:00 Stephan Beal sgb...@googlemail.com: On Tue, Jun 17, 2014 at 5:30 PM, Eduardo Morras emorr...@yahoo.es wrote: git fast-export xz -vv --lzma2=preset3,dict=256MiB git.xz xzcat -vv git.xz fossil import git.fossil That is a _very_ interesting trick. Question to you both: do you know if fossil has generated any delta manifests in the import? By default fossil does not generate delta manifests, but they can potentially save space for large repos. Once fossil has generated _one_ delta, it will allow itself to generate more, but by default it will never generate any. A delta can be forced at commit time with --delta, but i don't know if the git import feature (never needed it) supports something like that. -- - stephan beal http://wanderinghorse.net/home/stephan/ http://gplus.to/sgbeal Freedom is sloppy. But since tyranny's the only guaranteed byproduct of those who insist on a perfect world, freedom will have to do. -- Bigby Wolf ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
Re: [fossil-users] Working with large repository
On Tue, 17 Jun 2014 17:39:45 +0200 Stephan Beal sgb...@googlemail.com wrote: On Tue, Jun 17, 2014 at 5:30 PM, Eduardo Morras emorr...@yahoo.es wrote: git fast-export xz -vv --lzma2=preset3,dict=256MiB git.xz xzcat -vv git.xz fossil import git.fossil That is a _very_ interesting trick. You can pause it, view % of work done, etc... Question to you both: do you know if fossil has generated any delta manifests in the import? By default fossil does not generate delta manifests, but they can potentially save space for large repos. Once fossil has generated _one_ delta, it will allow itself to generate more, but by default it will never generate any. A delta can be forced at commit time with --delta, but i don't know if the git import feature (never needed it) supports something like that. Fossil rebuilds repository after the import and file size shrinks a lot, don't know if it's because delta encoding or something else. The openbsd import to fossil file has +300GB, and shrinks to 1.5GB after rebuild. Didn't have more time to dive in it. A final note, the import is one big transaction. -- - stephan beal --- --- Eduardo Morras emorr...@yahoo.es ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
[fossil-users] Working with large repository
Hi, I just want to share some feedback, for fun I tried to convert some of the FreeBSD repositories to fossil to see how it performs I first tried the FreeBSD Documentation repository, to go the easy way I took the FreeBSD git mirror (https://github.com/freebsd/freebsd-doc and run git fast-export --full-tree --all | fossil import --git ../doc.fossil The worked pretty well, and I manage to convert in a few hour the repo, the result it very usable (that wasn't the case a few years ago, some operations where slow) Result is available here: https://fossil.etoilebsd.net/doc if some want to play. At the same time I decided to try with ports and base repository which I would have expected to be a bit more complicated I cloned: https://github.com/freebsd/freebsd and https://github.com/freebsd/freebsd-ports I run the same command for the conversion git fast-export --full-tree --all | fossil import --git ../ports.fossil both are now running for more than 70H and still not finished!!! consuming 100% CPU This is running on pretty descent hardware. hw.model: Intel(R) Xeon(R) CPU E3-1225 V2 @ 3.20GHz I guess I would have to run a svndump imported manually if I want a slightly faster conversion to happen, and given how the incremental import works with fossil import --git I cannot imagine running a fossil mirror using that tool. If anyone has ideas on how I can maintain a mirror more easily, tips welcome Bapt ___ fossil-users mailing list fossil-users@lists.fossil-scm.org http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
