Re: [fossil-users] Bug report: Terrible Performance, when Checking in LLVM Source

[Warren Young]

On Dec 4, 2016, at 12:50 PM, Karel Gardas  wrote:
> 
> On Sun, Dec 4, 2016 at 5:28 AM, Martin Vahi  wrote:
>> It is about 4.4GiB, over 100k files, over 6k folders,
>> but it should not be that bad. After all, that's what
>> many projects look like in 2016.
> 
> This statement is IMHO a bit unfair. You basically grab *4*
> subversions trees (if I count well) and you smash that to one big tree
> and commit into fossil repo. So if you expect speed of svn, then
> please compare fairly independent subversion trees with independent
> fossil trees.

Actually, you’d have to compare Fossil against checking out every single 
revision from each separate Subversion repository:

$ for r in $(seq 1 $maxrev) ; do svn co -r $r … ; done

A Subversion checkout just gets you the tip of trunk by default, and you have 
to go back to the server in order to walk back through history.  Fossil grabs 
everything.

There are plans laid out on the mailing list for making Fossil do both shallow 
and narrow checkouts.  As it stands, Fossil always gives you a 100% wide and 
100% deep checkout.

Subversion allows both: it gives shallow checkouts always, and you can ask for 
a narrow checkout by specifying a subdirectory within the repo, grabbing only 
that slice.

It’s a good idea.  Someone just has to write it.
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Bug report: Terrible Performance, when Checking in LLVM Source

[Karel Gardas]

On Sun, Dec 4, 2016 at 9:55 PM, Joerg Sonnenberger  wrote:
> No. What repo checksum does is compute a separate checksum over the
> concatenation of all files.

Uf, so I've been completely off the picture. Thanks for correction!
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Bug report: Terrible Performance, when Checking in LLVM Source

[K. Fossil user]

Hi,
@Martin Vahi said :
> « [...] CPU is AMD 64bit [...] »

I don't know what AMD 64 means for you : it could be a dual core or a Quad 
core, and who knows an optocore...

The amount of RAM could let the calculation of the Checksum be faster ...
The bus speed can help a lot
etc.

> « [...] The uplink is ~1MiB/s (~10Mbps) and the ping
from my local machine to the remote machine is about 20ms [...] »

1MiB : it's theorical I guess.

You've said that files uploaded ONE by ONE should take 1 hour when a block of 
the equivalent data is 2 hours ... ??

No, when you process ONCE a copy it is faster than MANY little ones ...

@Jungle said :
> « Doesn't this depend on the CPU and available RAM? Search the archives
for importing large repos »

It is not importing it is exporting : "commit" Martin said.
BTW, I agree with your FIRST question (RAM).
@Karel said :
> « [...] subversions trees [...] »

So Martin should migrate the SVN trees to a git ones first and then import the 
git ones into a fossil one ... No ?

> « [...] If however I'm still off, I would appreciate
reference to some material explaining repo/chksums business in fossil. [...] »

So do I :-)

@Joerg said :
> « [...] What repo checksum does is compute a separate checksum over the
concatenation of all files [...] »

Hmmm...

You should say that EACH files are checksumed AND the repo itself is checksumed.

This should explain why it takes so long for a large repo... No ?
  
Best Regards

K.

  De : Joerg Sonnenberger <jo...@bec.de>
 À : fossil-users@lists.fossil-scm.org 
 Envoyé le : Dimanche 4 décembre 2016 20h55
 Objet : Re: [fossil-users] Bug report: Terrible Performance, when Checking in 
LLVM Source
   
On Sun, Dec 04, 2016 at 09:23:37PM +0100, Karel Gardas wrote:
> On Sun, Dec 4, 2016 at 8:57 PM, Joerg Sonnenberger <jo...@bec.de> wrote:
> > On Sun, Dec 04, 2016 at 08:50:44PM +0100, Karel Gardas wrote:
> >> Otherwise as Nikita recommended, switching off repo checksums helps a
> >> lot, but then make sure you are on the filesystem like ZFS/btrfs which
> >> does that for you transparently and you do not need to do that on the
> >> fossil side.
> >
> > Eh, no. You do not need a file system with automatic hashing. Every
> > single file is still recorded by checksum in Fossil. It is not what the
> > repo checksum option does.
> 
> Errhmm, thanks for correction. Am I right that repo checksum switched
> on means that modified files will be those where checksum stored and
> checksum computed from the file on fs is different? And once you
> switch that off, you rely purely on comparison of modification time on
> file in fs and I guess stored modif time in repo db? If so, then
> indeed I've been completely mistaken and thank you very much for your
> kind correction. If however I'm still off, I would appreciate
> reference to some material explaining repo/chksums business in fossil.

No. What repo checksum does is compute a separate checksum over the
concatenation of all files.

Joerg
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users


   ___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Bug report: Terrible Performance, when Checking in LLVM Source

[Joerg Sonnenberger]

On Sun, Dec 04, 2016 at 09:23:37PM +0100, Karel Gardas wrote:
> On Sun, Dec 4, 2016 at 8:57 PM, Joerg Sonnenberger  wrote:
> > On Sun, Dec 04, 2016 at 08:50:44PM +0100, Karel Gardas wrote:
> >> Otherwise as Nikita recommended, switching off repo checksums helps a
> >> lot, but then make sure you are on the filesystem like ZFS/btrfs which
> >> does that for you transparently and you do not need to do that on the
> >> fossil side.
> >
> > Eh, no. You do not need a file system with automatic hashing. Every
> > single file is still recorded by checksum in Fossil. It is not what the
> > repo checksum option does.
> 
> Errhmm, thanks for correction. Am I right that repo checksum switched
> on means that modified files will be those where checksum stored and
> checksum computed from the file on fs is different? And once you
> switch that off, you rely purely on comparison of modification time on
> file in fs and I guess stored modif time in repo db? If so, then
> indeed I've been completely mistaken and thank you very much for your
> kind correction. If however I'm still off, I would appreciate
> reference to some material explaining repo/chksums business in fossil.

No. What repo checksum does is compute a separate checksum over the
concatenation of all files.

Joerg
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Bug report: Terrible Performance, when Checking in LLVM Source

[Karel Gardas]

On Sun, Dec 4, 2016 at 8:57 PM, Joerg Sonnenberger  wrote:
> On Sun, Dec 04, 2016 at 08:50:44PM +0100, Karel Gardas wrote:
>> Otherwise as Nikita recommended, switching off repo checksums helps a
>> lot, but then make sure you are on the filesystem like ZFS/btrfs which
>> does that for you transparently and you do not need to do that on the
>> fossil side.
>
> Eh, no. You do not need a file system with automatic hashing. Every
> single file is still recorded by checksum in Fossil. It is not what the
> repo checksum option does.

Errhmm, thanks for correction. Am I right that repo checksum switched
on means that modified files will be those where checksum stored and
checksum computed from the file on fs is different? And once you
switch that off, you rely purely on comparison of modification time on
file in fs and I guess stored modif time in repo db? If so, then
indeed I've been completely mistaken and thank you very much for your
kind correction. If however I'm still off, I would appreciate
reference to some material explaining repo/chksums business in fossil.
Thanks! Karel
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Bug report: Terrible Performance, when Checking in LLVM Source

[Joerg Sonnenberger]

On Sun, Dec 04, 2016 at 08:50:44PM +0100, Karel Gardas wrote:
> Otherwise as Nikita recommended, switching off repo checksums helps a
> lot, but then make sure you are on the filesystem like ZFS/btrfs which
> does that for you transparently and you do not need to do that on the
> fossil side.

Eh, no. You do not need a file system with automatic hashing. Every
single file is still recorded by checksum in Fossil. It is not what the
repo checksum option does.

Joerg
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Bug report: Terrible Performance, when Checking in LLVM Source

[Nikita Borodikhin]

Try setting repo-cksum property off, it should help a lot.
By default Fossil double-checks all the files on disk to make sure their
contents is not changed.  That's fine for a small project, but for large
ones

Nikita

On Sat, Dec 3, 2016 at 8:29 PM Martin Vahi  wrote:

> Reproduction:
>
> 1) Download LLVM source.
>It might be done by executing the bash script from
>
>
> http://www.softf1.com/cgi-bin/tree1/technology/flaws/mmmv_parasail_projects.bash/artifact/3a28f1fb67a5d860
>
> 2) Commit the source.
>
> It is about 4.4GiB, over 100k files, over 6k folders,
> but it should not be that bad. After all, that's what
> many projects look like in 2016.
>
>
> Regards,
> martin.v...@softf1.com
>
> ___
> fossil-users mailing list
> fossil-users@lists.fossil-scm.org
> http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
>
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Bug report: Terrible Performance, when Checking in LLVM Source

[jungle boogie]

On 12/03/2016 08:28 PM, Martin Vahi wrote:

Reproduction:

1) Download LLVM source.
   It might be done by executing the bash script from

http://www.softf1.com/cgi-bin/tree1/technology/flaws/mmmv_parasail_projects.bash/artifact/3a28f1fb67a5d860

2) Commit the source.

It is about 4.4GiB, over 100k files, over 6k folders,
but it should not be that bad. After all, that's what
many projects look like in 2016.



Doesn't this depend on the CPU and available RAM? Search the archives 
for importing large repos. This very likely is not an issue with your 
network connection.




Regards,
martin.v...@softf1.com



___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

[fossil-users] Bug report: Terrible Performance, when Checking in LLVM Source

[Martin Vahi]

Reproduction:

1) Download LLVM source.
   It might be done by executing the bash script from

http://www.softf1.com/cgi-bin/tree1/technology/flaws/mmmv_parasail_projects.bash/artifact/3a28f1fb67a5d860

2) Commit the source.

It is about 4.4GiB, over 100k files, over 6k folders,
but it should not be that bad. After all, that's what
many projects look like in 2016.


Regards,
martin.v...@softf1.com

___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users