Re: Analyzing dumps (Was: 6 million)

2020-05-04 Thread Ángel
On 2020-05-03 at 11:15 +0200, Kiss Gabor (Bitman) wrote:
> On Sat, 2 May 2020, Wiktor Kwapisiewicz wrote:
> 
> > On 02.05.2020 07:55, Gabor Kiss wrote:
> > > I would create such a programs from the scratch but I cannot
> > > find even the format description of the dump file. :-(
> > 
> > Last time I checked dumps where just packet piles so any OpenPGP tool
> > could read it.
> 
> Thanks again for the hint.
> 
> I wrote a small Perl script to see what is in dump files
> at http://keys.niif.hu/keydump/. (Server is managed by me.)
> 
> I found broken dumps. Certain RFC-4880 packets are truncated. For example
> let's see signatures of key 0x7cec0e7c93115f7e:
> 
> 00483ad0 89 01 22 04 10 01 02  00 0c 05 02 44 cf db 85  |..."D...|
> 00483ae0  05 03 00 93 89 01 22 04  10 01 02 00 0c 05 02 4d  |.."M|
> 
> We can see a signature packet starting at 00483ad1.
> (89 01 22 is a typical old style packet header.) Its length should be
> 0x122 octets however it breaks in middle of the second subpacket starting
> at 00483ae0. A new packet starts at 00483ae4 but my simple parser cannot
> detect this and gets confused.
> (Unfortunately such a truncated packet may block the import procedure
> also on a newly set up key server, I guess.)
> 
> I cannot imagine how this dump could be created.
> Could the attacker upload broken packets or is it "sks dump"
> who garbled the dump file? Or file became bad during
> compression/decompression?

My own script can handle them fine. 
There is always a container key around the packets, so it's not a
problem. But yes, sometimes there is a container that claims to have
more data than it has available, particularly around subpackets. Seems
that part is not verified by sks.

I see exactly one error on your first file (), on this key of
'Florian Streibelt':
Public Key v4 DSA - Key 3B8EA41F82F61240 - Created 2004-10-22 19:25:43
#ERROR: INVALID PACKET: Packet of 70 bytes but at least 1165 bytes needed
# Creation ERROR: INVALID PACKET: Packet of 70 bytes but at least 1165 
bytes needed
- ERROR: INVALID PACKET: Packet of 70 bytes but at least 1165 bytes needed
ERROR: INVALID PACKET: Packet of 70 bytes but at least 1165 bytes needed


While the whole keys are manageable, I do see a large number of errors than I 
used to.

So I took a mattrude dump from 80 days ago. That had 5978288 keys vs 6008063 of 
your dump 
(an increase of 0.5% keys). Then I compared the number of keys with an error in 
both cases.
I considered a key to have an error if it detected a problem anywhere,
regardless of the number of signatures it had (the top one was a key
with 110 diagnostics).

On the February dump, there were 135 "broken keys" out of 5978288
(0,002258%)
On your dump, there are 134 "broken keys" out of 6008063 keys (0,00223%)


So not a problem of your server/dump, and not an increase of garbled
keys, either.

As for the 0x7cec0e7c93115f7e key (Terry Lansdown) I can import it from
your keyserver and use fine, also exporting with no error.
That key lives on keydump-sks-0054.pgp at 5956981 with a length of 29111
bytes.

I can find a funny signature there:
  - Signature v4 sig RSA SHA1
# Creation 2006-08-02 00:53:57
# SigExpiration 111d21h47m45s
ERROR: INVALID PACKET: Packet of 290 bytes but at least 8710 bytes
needed

This seems the opposite to your problem, though.

that is followed by 
  - Signature v4 sig RSA SHA1
# Creation 2006-08-15 12:17:53
# SigExpiration 14d
- Issuer 9710B89BCA57AD7C


Interestingly, pgpdump does get confused by it

> Old: Signature Packet(tag 2)(290 bytes)
> Ver 4 - new
> Sig type - Generic certification of a User ID and Public Key 
> packet(0x10).
> Pub alg - RSA Encrypt or Sign(pub 1)
> Hash alg - SHA1(hash 2)
> Hashed Sub: signature creation time(sub 2)(4 bytes)
> Time - Wed Aug  2 00:53:57 CEST 2006
> Hashed Sub: signature expiration time(sub 3)(4 bytes)
> Time - Tue Nov 21 21:41:42 CET 2006
> Sub: reserved(sub 1)(15 bytes)
> Sub: additional decryption key(sub 10) WARNING: see CA-2000-18!!!(-1 
> bytes)
> Class - Unknown class(09)
> Pub alg - ElGamal Encrypt-Only(pub 16)
> Fingerprint - 97 10 b8 9b ca 57 bf 6b 5b 5b 4f 74 8d 81 87 63 
> 79 34 9b 83 
> Sub: unknown(sub 46, critical)(13301 bytes)
> Hash left 2 bytes - c6 6e 
> RSA m^d mod n(13079 bits) - ...
> -> PKCS-1
> New: unknown(tag 56)(11 bytes)
> Old: Signature Packet(tag 2)(until eof)
> Ver 218 - unknown


gpg --list-packets view is slightly different:


> :signature packet: algo 1, keyid 9710B89BCA57AD7C
> version 4, created 1154472837, md5len 0, sigclass 0x10
> digest algo 2, begin of digest 47 45
> hashed subpkt 2 len 4 (sig created 2006-08-01)
> hashed subpkt 3 len 4 (sig expires after 14d0h0m)
> subpkt 16 len 8 (issuer key ID 

Re: Analyzing dumps (Was: 6 million)

2020-05-04 Thread William Hay
Thus spake "Kiss Gabor (Bitman)":
> I cannot imagine how this dump could be created.
> Could the attacker upload broken packets or is it "sks dump"
> who garbled the dump file? Or file became bad during
> compression/decompression?

I don't think sks does much validation of packets so anything could
be uploaded. I've found in the past with broken dumps that hkt from 
hopenpgp-tools works fairly well as a filter.  It manages to skip 
broken keys and carry on while GnuPG just aborts at the first badly 
broken key.

William