Bug#852106: [pkg-gnupg-maint] Bug#852106: gpgme1.0: Build allocates 200 GB as a normal thing

[Santiago Vila]

On Mon, Jan 23, 2017 at 05:41:29PM -0500, Daniel Kahn Gillmor wrote:

> I'm open for suggestions for how to address this.  We could just drop
> THREAD_COUNT from 100 to 10 to reduce the RAM consumption, but that
> would mean minimizing some of the concurrency that the test suite is
> (rightly) trying to judge itself against.  I'd rather not remove those
> kinds of checks if we don't need to.
> 
> Any suggestions about what the right thing to do is?  

Just try to be more "normal", i.e. try reducing the memory
requirements to, say, 4 GB, or 8 GB if you really need it.
I would consider 16 GB to be already too much by today's standards,
considering the memory requirements of all the other packages.

I have put the current memory requirements for the packages
that may be built with "dpkg-buildpackage -A" here:

https://people.debian.org/~sanvila/A.memory.txt

Try "sort -k2 -n" on that file and you will see not only that gpgme1.0
is at the end but also how far this package is from the average, the
median, or any other significant statistic measure that you can
imagine.

Of course, there is not a policy anywhere saying "packages must not
use more than X GB of memory", so just apply common sense.

Thanks.

Bug#852106: [pkg-gnupg-maint] Bug#852106: gpgme1.0: Build allocates 200 GB as a normal thing

[Daniel Kahn Gillmor]

Hi Santiago--

On Sat 2017-01-21 12:40:06 -0500, Santiago Vila wrote:
> I have a cron job monitoring "Committed_AS" in /proc/meminfo every time
> my autobuilders are running, that way I know how much memory each
> package requires.
>
> Well, building gpgme1.0 allocates 100 GB, 200 GB and sometimes 500 GB.

yikes!  I've never noticed that, but i can replicate it now that i'm
looking for it.  I'm a bit surprised a cronjob would have caught it,
because i only see the effect for a few seconds at most.  but it's
definitely a significant spike in Committed_AS.

> Then try to build this package in such machine, and it will fail.
>
> I have detected only one other package with a problem like this.
> If you are curious:
>
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=848589
>
> Every other package which I tried builds fine with only 20 GB of RAM,
> so this is clearly an anomaly.

I would have guessed that this happens in gpgme due to the C++ library
build which is now part of libgpgme -- i know that g++ and the linker
can both be pretty beastly.  and snap-align is also C++.

however, i did a bit more inspection, and i see these commitments only
during the test suite.  in particular, on a system with 8GiB of RAM, my
RAM commitments moved from a normal level of:

Committed_AS: 7360592 kB

to:

Committed_AS:   108890120 kB

during the execution of tests/gpg/t-thread-keylist-verify.


this test makes 200 threads, 100 of which do a gpg verification
operation, and the other 100 do a keylist operation; that's when the RAM
allocation spikes.

Interestingly, just before that happens is tests/gpg/t-thread-keylist,
which doesn't seem to allocate nearly as much RAM despite being of the
same general form (it just does keylist without the simultaneous verify,
though).

Each thread does spawn an additional gpg process, so there's certainly
memory committments there.  But even with 200 concurrent processes, that
amounts to 0.5GiB of RAM commitments per process, which seems large to
me.

I'm open for suggestions for how to address this.  We could just drop
THREAD_COUNT from 100 to 10 to reduce the RAM consumption, but that
would mean minimizing some of the concurrency that the test suite is
(rightly) trying to judge itself against.  I'd rather not remove those
kinds of checks if we don't need to.

Any suggestions about what the right thing to do is?  

--dkg


signature.asc
Description: PGP signature