On Tue, Jun 30, 2020, at 1:20 AM, Fabian Groffen wrote:
> Hi,
> On 29-06-2020 21:13:43 -0500, Sid Spry wrote:
> > Hello,
> > 
> > I have some runnable pseudocode outlining a faster tree verification 
> > algorithm.
> > Before I create patches I'd like to see if there is any guidance on making 
> > the
> > changes as unobtrusive as possible. If the radical change in algorithm is
> > acceptable I can work on adding the changes.
> > 
> > Instead of composing any kind of structured data out of the portage tree my
> > algorithm just lists all files and then optionally batches them out to 
> > threads.
> > There is a noticeable speedup by eliding the tree traversal operations which
> > can be seen when running the algorithm with a single thread and comparing 
> > it to
> > the current algorithm in gemato (which should still be discussed here?).
> I remember something that gemato used to use multiple threads, but
> because it totally saturated disk-IO, it was brought back to a single
> thread.  People were complaining about unusable systems.

I think this is an argument for cgroups limits support on the portage process or
account as opposed to an argument against picking a better algorithm. That is
something I have been working towards, but I am only one man.

> In any case, can you share your performance results?  What speedup did
> you see, on warm and hot FS caches?  Which type of disk do you use?

I ran all tests multiple times to make them warm off of a Samsung SSD, but
nothing very precise yet.

% gemato verify --openpgp-key signkey.asc /var/db/repos/gentoo
INFO:root:Verifying /var/db/repos/gentoo...
INFO:root:/var/db/repos/gentoo verified in 16.45 seconds

sometimes going higher, closer to 18s, vs.

% ./veriftree.py

So roughly an order of magnitude speedup without batching to threads.

> You could compare against qmanifest, which uses OpenMP-based
> paralllelism while verifying the tree.  On SSDs this does help.

I lost my notes -- how do I specify to either gemato or qmanifest the GnuPG
directory? My code is partially structured as it is because I had problems doing
this. I rediscovered -K/--openpgp-key in gemato but am unsure for qmanifest.

Reply via email to