Re: [fossil-users] Bug report: Terrible Performance, when Checking in LLVM Source

[Karel Gardas]

On Sun, Dec 4, 2016 at 9:55 PM, Joerg Sonnenberger  wrote:
> No. What repo checksum does is compute a separate checksum over the
> concatenation of all files.

Uf, so I've been completely off the picture. Thanks for correction!
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Bug report: Terrible Performance, when Checking in LLVM Source

[K. Fossil user]

Hi,
@Martin Vahi said :
> « [...] CPU is AMD 64bit [...] »

I don't know what AMD 64 means for you : it could be a dual core or a Quad 
core, and who knows an optocore...

The amount of RAM could let the calculation of the Checksum be faster ...
The bus speed can help a lot
etc.

> « [...] The uplink is ~1MiB/s (~10Mbps) and the ping
from my local machine to the remote machine is about 20ms [...] »

1MiB : it's theorical I guess.

You've said that files uploaded ONE by ONE should take 1 hour when a block of 
the equivalent data is 2 hours ... ??

No, when you process ONCE a copy it is faster than MANY little ones ...

@Jungle said :
> « Doesn't this depend on the CPU and available RAM? Search the archives
for importing large repos »

It is not importing it is exporting : "commit" Martin said.
BTW, I agree with your FIRST question (RAM).
@Karel said :
> « [...] subversions trees [...] »

So Martin should migrate the SVN trees to a git ones first and then import the 
git ones into a fossil one ... No ?

> « [...] If however I'm still off, I would appreciate
reference to some material explaining repo/chksums business in fossil. [...] »

So do I :-)

@Joerg said :
> « [...] What repo checksum does is compute a separate checksum over the
concatenation of all files [...] »

Hmmm...

You should say that EACH files are checksumed AND the repo itself is checksumed.

This should explain why it takes so long for a large repo... No ?
  
Best Regards

K.

  De : Joerg Sonnenberger 
 À : fossil-users@lists.fossil-scm.org 
 Envoyé le : Dimanche 4 décembre 2016 20h55
 Objet : Re: [fossil-users] Bug report: Terrible Performance, when Checking in 
LLVM Source
   
On Sun, Dec 04, 2016 at 09:23:37PM +0100, Karel Gardas wrote:
> On Sun, Dec 4, 2016 at 8:57 PM, Joerg Sonnenberger  wrote:
> > On Sun, Dec 04, 2016 at 08:50:44PM +0100, Karel Gardas wrote:
> >> Otherwise as Nikita recommended, switching off repo checksums helps a
> >> lot, but then make sure you are on the filesystem like ZFS/btrfs which
> >> does that for you transparently and you do not need to do that on the
> >> fossil side.
> >
> > Eh, no. You do not need a file system with automatic hashing. Every
> > single file is still recorded by checksum in Fossil. It is not what the
> > repo checksum option does.
> 
> Errhmm, thanks for correction. Am I right that repo checksum switched
> on means that modified files will be those where checksum stored and
> checksum computed from the file on fs is different? And once you
> switch that off, you rely purely on comparison of modification time on
> file in fs and I guess stored modif time in repo db? If so, then
> indeed I've been completely mistaken and thank you very much for your
> kind correction. If however I'm still off, I would appreciate
> reference to some material explaining repo/chksums business in fossil.

No. What repo checksum does is compute a separate checksum over the
concatenation of all files.

Joerg
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users


   ___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Bug report: Terrible Performance, when Checking in LLVM Source

[Joerg Sonnenberger]

On Sun, Dec 04, 2016 at 09:23:37PM +0100, Karel Gardas wrote:
> On Sun, Dec 4, 2016 at 8:57 PM, Joerg Sonnenberger  wrote:
> > On Sun, Dec 04, 2016 at 08:50:44PM +0100, Karel Gardas wrote:
> >> Otherwise as Nikita recommended, switching off repo checksums helps a
> >> lot, but then make sure you are on the filesystem like ZFS/btrfs which
> >> does that for you transparently and you do not need to do that on the
> >> fossil side.
> >
> > Eh, no. You do not need a file system with automatic hashing. Every
> > single file is still recorded by checksum in Fossil. It is not what the
> > repo checksum option does.
> 
> Errhmm, thanks for correction. Am I right that repo checksum switched
> on means that modified files will be those where checksum stored and
> checksum computed from the file on fs is different? And once you
> switch that off, you rely purely on comparison of modification time on
> file in fs and I guess stored modif time in repo db? If so, then
> indeed I've been completely mistaken and thank you very much for your
> kind correction. If however I'm still off, I would appreciate
> reference to some material explaining repo/chksums business in fossil.

No. What repo checksum does is compute a separate checksum over the
concatenation of all files.

Joerg
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Bug report: Terrible Performance, when Checking in LLVM Source

[Karel Gardas]

On Sun, Dec 4, 2016 at 8:57 PM, Joerg Sonnenberger  wrote:
> On Sun, Dec 04, 2016 at 08:50:44PM +0100, Karel Gardas wrote:
>> Otherwise as Nikita recommended, switching off repo checksums helps a
>> lot, but then make sure you are on the filesystem like ZFS/btrfs which
>> does that for you transparently and you do not need to do that on the
>> fossil side.
>
> Eh, no. You do not need a file system with automatic hashing. Every
> single file is still recorded by checksum in Fossil. It is not what the
> repo checksum option does.

Errhmm, thanks for correction. Am I right that repo checksum switched
on means that modified files will be those where checksum stored and
checksum computed from the file on fs is different? And once you
switch that off, you rely purely on comparison of modification time on
file in fs and I guess stored modif time in repo db? If so, then
indeed I've been completely mistaken and thank you very much for your
kind correction. If however I'm still off, I would appreciate
reference to some material explaining repo/chksums business in fossil.
Thanks! Karel
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Bug report: Terrible Performance, when Checking in LLVM Source

[Joerg Sonnenberger]

On Sun, Dec 04, 2016 at 08:50:44PM +0100, Karel Gardas wrote:
> Otherwise as Nikita recommended, switching off repo checksums helps a
> lot, but then make sure you are on the filesystem like ZFS/btrfs which
> does that for you transparently and you do not need to do that on the
> fossil side.

Eh, no. You do not need a file system with automatic hashing. Every
single file is still recorded by checksum in Fossil. It is not what the
repo checksum option does.

Joerg
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users

Re: [fossil-users] Bug report: Terrible Performance, when Checking in LLVM Source

[Nikita Borodikhin]

Try setting repo-cksum property off, it should help a lot.
By default Fossil double-checks all the files on disk to make sure their
contents is not changed.  That's fine for a small project, but for large
ones

Nikita

On Sat, Dec 3, 2016 at 8:29 PM Martin Vahi  wrote:

> Reproduction:
>
> 1) Download LLVM source.
>It might be done by executing the bash script from
>
>
> http://www.softf1.com/cgi-bin/tree1/technology/flaws/mmmv_parasail_projects.bash/artifact/3a28f1fb67a5d860
>
> 2) Commit the source.
>
> It is about 4.4GiB, over 100k files, over 6k folders,
> but it should not be that bad. After all, that's what
> many projects look like in 2016.
>
>
> Regards,
> martin.v...@softf1.com
>
> ___
> fossil-users mailing list
> fossil-users@lists.fossil-scm.org
> http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users
>
___
fossil-users mailing list
fossil-users@lists.fossil-scm.org
http://lists.fossil-scm.org:8080/cgi-bin/mailman/listinfo/fossil-users