Re: Efficent batch fetching with verification?

2017-11-04 Thread Werner Koch 
On Sat,  4 Nov 2017 06:06, robb...@gentoo.org said:

> Yes, the older versions do perform much worse, but even with gnupg2.2,
> each exec of gpg is still at least 100ms, which adds up over time.

I doubt that, let's see:

  $ time sh -c 'seq 1 1 | xargs -n 1 gpg --version >/dev/null'
  
  real0m0.010s
  user0m0.004s
  sys 0m0.004s
  $ time sh -c 'seq 1 100 | xargs -n 1 gpg --version >/dev/null'
  
  real0m0.361s
  user0m0.068s
  sys 0m0.024s

This is less than 4ms per exec.  So you problem is for sure not the
fork/exec.


Salam-Shalom,

   Werner

-- 
Die Gedanken sind frei.  Ausnahmen regelt ein Bundesgesetz.


pgpgiYrylJG41.pgp
Description: PGP signature
___
Gnupg-users mailing list
Gnupg-users@gnupg.org
http://lists.gnupg.org/mailman/listinfo/gnupg-users

Re: Efficent batch fetching with verification?

2017-11-04 Thread Peter Lebbing 
On 03/11/17 21:06, Robin H. Johnson wrote:
> You missed xargs itself,

Actually, I did not :-).

> this mostly centers around the command-line
> length limit. I can get in about ~3200 fingerprints per GPG call.

I asked "what is exec'ing much". I don't see one exec every 3200
fingerprints as overhead at all.

In your other reply, you say the 100 ms exec overhead for these 3200
keyserver fetches is significant. But I see a lot of round trips to the
keyserver; I didn't check the docs, but it must not be fetching many
keys in every HKP request. Perhaps even just a single key per request.
That is 3200 round trips to a remote server. And then the data will be
checked: this means running expensive asymmetric crypto.

So how long does this one gpg with 3200 key fetches run for you, as wall
time, and as cpu time? TBH, I'm having a hard time believing the
starting up of gpg.exe is relevant.

Cheers,

Peter.

-- 
I use the GNU Privacy Guard (GnuPG) in combination with Enigmail.
You can send me encrypted mail if you want some privacy.
My key is available at 



signature.asc
Description: OpenPGP digital signature
___
Gnupg-users mailing list
Gnupg-users@gnupg.org
http://lists.gnupg.org/mailman/listinfo/gnupg-users

Re: Efficent batch fetching with verification?

2017-11-03 Thread Robin H. Johnson 
On Fri, Nov 03, 2017 at 08:17:38PM +0100, Werner Koch wrote:
> On Fri,  3 Nov 2017 06:20, robb...@gentoo.org said:
> 
> > Presently, the code is effectively this:
> > ...cat-list-of-fingerprints... | xargs gpg --recv
> >
> > This has the downside of causing many execs.
> 
> Right after a clean startup of your user session you will
> see these execs:
> 
>   1. xargs execs gpg
>   2. gpg execs gpg-agent
>   3. gpg execs dirmngr
> 
> If xargs needs to exec another gpg you won't see new execs for gpg-agent
> or dirmngr.  And the startup time of gpg can be neglecated compared to
> the latency of the keyservers.
> 
> Or may it be that you are using gpg 1.4 or 2.0?  Those invoke keyserver
> helpers and that may very well be one exec per supplied fingerprint.
Yes, the older versions do perform much worse, but even with gnupg2.2,
each exec of gpg is still at least 100ms, which adds up over time.

Part this may be having a huge keyring present (50k+ keys).

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Asst. Treasurer
E-Mail   : robb...@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136


signature.asc
Description: Digital signature
___
Gnupg-users mailing list
Gnupg-users@gnupg.org
http://lists.gnupg.org/mailman/listinfo/gnupg-users

Re: Efficent batch fetching with verification?

2017-11-03 Thread Robin H. Johnson 
On Fri, Nov 03, 2017 at 12:50:06PM +0100, Peter Lebbing wrote:
> On 03/11/17 06:20, Robin H. Johnson wrote:
> > Presently, the code is effectively this:
> > ...cat-list-of-fingerprints... | xargs gpg --recv
> > 
> > This has the downside of causing many exec
...
> Anyway, I didn't look any further, but what is exec'ing much here then?
> Which version of GnuPG are you using? I'm using the Debian stretch
> provided 2.1.18 with a systemd supervised dirmngr. I can't readily think
> of which process would be starting often here... am I completey
> forgetting about something? :-)
You missed xargs itself, this mostly centers around the command-line
length limit. I can get in about ~3200 fingerprints per GPG call.

GnuPG 2.2.1, findutils/xargs 4.6.0.

Thanks for the idea of --no-auto-check-trustdb, I did miss that and it
helps for speedups.

-- 
Robin Hugh Johnson
Gentoo Linux: Dev, Infra Lead, Foundation Asst. Treasurer
E-Mail   : robb...@gentoo.org
GnuPG FP : 11ACBA4F 4778E3F6 E4EDF38E B27B944E 34884E85
GnuPG FP : 7D0B3CEB E9B85B1F 825BCECF EE05E6F6 A48F6136


signature.asc
Description: Digital signature
___
Gnupg-users mailing list
Gnupg-users@gnupg.org
http://lists.gnupg.org/mailman/listinfo/gnupg-users

Re: Efficent batch fetching with verification?

2017-11-03 Thread Werner Koch 
On Fri,  3 Nov 2017 06:20, robb...@gentoo.org said:

> Presently, the code is effectively this:
> ...cat-list-of-fingerprints... | xargs gpg --recv
>
> This has the downside of causing many execs.

Right after a clean startup of your user session you will
see these execs:

  1. xargs execs gpg
  2. gpg execs gpg-agent
  3. gpg execs dirmngr

If xargs needs to exec another gpg you won't see new execs for gpg-agent
or dirmngr.  And the startup time of gpg can be neglecated compared to
the latency of the keyservers.

Or may it be that you are using gpg 1.4 or 2.0?  Those invoke keyserver
helpers and that may very well be one exec per supplied fingerprint.


Shalom-Salam,

   Werner

-- 
Die Gedanken sind frei.  Ausnahmen regelt ein Bundesgesetz.


pgpuQqL_bOgma.pgp
Description: PGP signature
___
Gnupg-users mailing list
Gnupg-users@gnupg.org
http://lists.gnupg.org/mailman/listinfo/gnupg-users

Re: Efficent batch fetching with verification?

2017-11-03 Thread Peter Lebbing 
On 03/11/17 06:20, Robin H. Johnson wrote:
> Presently, the code is effectively this:
> ...cat-list-of-fingerprints... | xargs gpg --recv
> 
> This has the downside of causing many exec
I just tried this and a list of 1319 fingerprints caused one single call
to "gpg --recv FPR1 FPR2 FPR3 ... FPR1319". I don't understand why my
gpg is then doing trust database calculations every so many keys, so
what I ended up doing was:

$ cat list-of-fingerprints | xargs strace -ff -o gpgtrace -e
trace=process gpg --no-auto-check-trustdb --recv

And this ran happily until killed by me, fetching and updating keys,
with just a single execve, no spawns.

Anyway, I didn't look any further, but what is exec'ing much here then?
Which version of GnuPG are you using? I'm using the Debian stretch
provided 2.1.18 with a systemd supervised dirmngr. I can't readily think
of which process would be starting often here... am I completey
forgetting about something? :-)

Peter.

-- 
I use the GNU Privacy Guard (GnuPG) in combination with Enigmail.
You can send me encrypted mail if you want some privacy.
My key is available at 



signature.asc
Description: OpenPGP digital signature
___
Gnupg-users mailing list
Gnupg-users@gnupg.org
http://lists.gnupg.org/mailman/listinfo/gnupg-users