Re: [Sks-devel] disk full, keys.niif.hu crashed
> "t" == tiker writes: t> Here's a (temporary) link to an image of what I see: t> http://www.funkymonkey.org/tmp/bigkey.jpg It is hard to check w/o knowing the key hash, but can iconv(1) decode that uid into utf8? Perhaps it is in one of the legacy 16bit encodings? Can you get that uid (just the uid) into a file so that it can be checked? -JimC -- James Cloos OpenPGP: 0x997A9F17ED7DAEA6 ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] disk full, keys.niif.hu crashed
Well, it turns out that the cause of our issues, the method to re-create these keys and make things worse is already posted publicly. Take a look at the recently reported issues on the SKS bitbucket site. I don't think my SKS node has enough storage space to survive long enough for this issue to be fixed. I may have to shut it down. Rob D On 2018-06-15 16:01, tiker wrote: > I don't think so but I could be wrong. (I'm no expert here.) > > Binary attachments (like images) are marked as "uat [contents > ommited]". In this case, it's a "uid" row that starts the binary data > instead of a text line showing a name. > > Here's a (temporary) link to an image of what I see: > http://www.funkymonkey.org/tmp/bigkey.jpg > > I'll send an email to Kristian F. with the details about this key to > review and comment on. > > Thanks. > Rob D > > > On 2018-06-15 15:24, Phil Pennock wrote: >> On 2018-06-15 at 12:40 -0400, tiker wrote: >>> The problems seem to be caused by a large key. There's at least 2 >>> different hash values for this key (so probably recently updated) and >>> one of the versions of the key is 22mb. The size is causing timeouts on >>> some reverse proxies and the constant retries is causing the .log files >>> to be created and growing in the DB directory. >> The current advice over at >> https://bitbucket.org/skskeyserver/sks-keyserver/wiki/Peering is to set >> client_max_body_size to 8 MiB. >> >>> I don't think I want to post the key ID here because it's hard on the >>> servers grabbing this key but someone should look at it and figure out >>> what to do with this. My node only seems to sync with about 10% of its >>> peers. >> Is this something with a binary image attribute? :( >> >> -Phil > > > > ___ > Sks-devel mailing list > Sks-devel@nongnu.org > https://lists.nongnu.org/mailman/listinfo/sks-devel signature.asc Description: OpenPGP digital signature ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] disk full, keys.niif.hu crashed
I don't think so but I could be wrong. (I'm no expert here.) Binary attachments (like images) are marked as "uat [contents ommited]". In this case, it's a "uid" row that starts the binary data instead of a text line showing a name. Here's a (temporary) link to an image of what I see: http://www.funkymonkey.org/tmp/bigkey.jpg I'll send an email to Kristian F. with the details about this key to review and comment on. Thanks. Rob D On 2018-06-15 15:24, Phil Pennock wrote: > On 2018-06-15 at 12:40 -0400, tiker wrote: >> The problems seem to be caused by a large key. There's at least 2 >> different hash values for this key (so probably recently updated) and >> one of the versions of the key is 22mb. The size is causing timeouts on >> some reverse proxies and the constant retries is causing the .log files >> to be created and growing in the DB directory. > The current advice over at > https://bitbucket.org/skskeyserver/sks-keyserver/wiki/Peering is to set > client_max_body_size to 8 MiB. > >> I don't think I want to post the key ID here because it's hard on the >> servers grabbing this key but someone should look at it and figure out >> what to do with this. My node only seems to sync with about 10% of its >> peers. > Is this something with a binary image attribute? :( > > -Phil signature.asc Description: OpenPGP digital signature ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] disk full, keys.niif.hu crashed
On 2018-06-15 at 12:40 -0400, tiker wrote: > The problems seem to be caused by a large key. There's at least 2 > different hash values for this key (so probably recently updated) and > one of the versions of the key is 22mb. The size is causing timeouts on > some reverse proxies and the constant retries is causing the .log files > to be created and growing in the DB directory. The current advice over at https://bitbucket.org/skskeyserver/sks-keyserver/wiki/Peering is to set client_max_body_size to 8 MiB. > I don't think I want to post the key ID here because it's hard on the > servers grabbing this key but someone should look at it and figure out > what to do with this. My node only seems to sync with about 10% of its > peers. Is this something with a binary image attribute? :( -Phil ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] disk full, keys.niif.hu crashed
On 2018-06-15 at 09:40 +0200, André Keller wrote: > On 15.06.2018 05:54, Kiss Gabor (Bitman) wrote: > > Yesterday at 18:15 (CEST) keys.niif.hu started to produce tons > > of logs in /var/lib/sks/DB. In less than 2 hours the 40 GB filesystem > > got fulfilled. > > Deleting files and restarting processes did not help: > > keys.communityrack.org shares the same fate. Trying to get it online > again... sks-peer.spodhuis.org saw a spike at the same time, AWS CloudWatch metrics show that the dedicated EBS volume used for /var/sks hit 175,000 write operations per minute, when it's usually around 22,000 peaking around 56,000. The write _bytes_ is peaking around the same as normal, so throughput is probably capping out. I actually used some of the burst credits I had. I'm in the middle of migrating OS-view metrics monitoring, in part to handle having moved SKS into AWS, and don't currently have graphs showing change in used capacity. I'm currently at 30GB in use. I see no change in rate of new keys or updated keys. I do see 21GiB in use for the DB directory. -Phil ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] disk full, keys.niif.hu crashed
The problems seem to be caused by a large key. There's at least 2 different hash values for this key (so probably recently updated) and one of the versions of the key is 22mb. The size is causing timeouts on some reverse proxies and the constant retries is causing the .log files to be created and growing in the DB directory. When viewing the key through the web interface (both hash versions so far) one of the UID packets turns into a binary blob of garbage on the screen. But does seem to end correctly but after the 22mb of junk on the screen, the sub keys appear to be ok at the end. This might be the cause of the error I posted with my previous message. I've checked a couple SKS servers for this key and so far, they all seem to have issues with this key. This key was also appears to have been created yesterday which may explain your two crashes. I don't think I want to post the key ID here because it's hard on the servers grabbing this key but someone should look at it and figure out what to do with this. My node only seems to sync with about 10% of its peers. Thanks. Rob D On 2018-06-15 11:53, Keith Erekson wrote: > This has happened to my keyserver twice in the last two days. I assumed > it was some sort of malicious behavior, because it happened quite > suddenly both times and had the effect of a DoS. ;-) > > For example, I have over 1700 binary log files like "log.002014", > each 10MB, created in the last 24 hours. (It would have kept going, but > the filesystem filled up.) > > The timestamps show that often 30 or 40 of them are created in the same > minute. > > ~Keith > > > On 06/14/2018 11:54 PM, Kiss Gabor (Bitman) wrote: >> Yesterday at 18:15 (CEST) keys.niif.hu started to produce tons >> of logs in /var/lib/sks/DB. In less than 2 hours the 40 GB filesystem >> got fulfilled. >> Deleting files and restarting processes did not help: >> >> recon.log: >> 2018-06-15 05:50:09 Opening log >> 2018-06-15 05:50:09 sks_recon, SKS version 1.1.6 >> 2018-06-15 05:50:09 Using BerkelyDB version 5.3.28 >> 2018-06-15 05:50:09 Copyright Yaron Minsky 2002-2013 >> 2018-06-15 05:50:09 Licensed under GPL. See LICENSE file for details >> 2018-06-15 05:50:09 recon port: 11370 >> 2018-06-15 05:50:09 Opening PTree database >> 2018-06-15 05:50:09 Setting up PTree data structure >> 2018-06-15 05:50:09 PTree setup complete >> 2018-06-15 05:50:09 Initiating catchup >> 2018-06-15 05:50:10 DB closed >> >> db.log: >> 2018-06-15 05:50:09 Opening log >> 2018-06-15 05:50:09 sks_db, SKS version 1.1.6 >> 2018-06-15 05:50:09 Using BerkelyDB version 5.3.28 >> 2018-06-15 05:50:09 Copyright Yaron Minsky 2002, 2003, 2004 >> 2018-06-15 05:50:09 Licensed under GPL. See LICENSE file for details >> 2018-06-15 05:50:09 http port: 11371 >> 2018-06-15 05:50:09 Membership: (zimmermann.mayfirst.org 11370)[], ... >> (keys.jpbe.de 11370)[] >> 2018-06-15 05:50:09 address for zimmermann.mayfirst.org:11370 changed from >> [] to >> [, ] >> ... >> 2018-06-15 05:50:10 address for keys.jpbe.de:11370 changed from [] to >> [, > [185.120.22.22]:11370>] >> 2018-06-15 05:50:10 Opening KeyDB database >> 2018-06-15 05:50:10 Shutting down database >> >> Unfortunately I cannot work on restoration till Sunday evening. >> >> Gabor >> >> ___ >> Sks-devel mailing list >> Sks-devel@nongnu.org >> https://lists.nongnu.org/mailman/listinfo/sks-devel > > > > ___ > Sks-devel mailing list > Sks-devel@nongnu.org > https://lists.nongnu.org/mailman/listinfo/sks-devel signature.asc Description: OpenPGP digital signature ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] disk full, keys.niif.hu crashed
This has happened to my keyserver twice in the last two days. I assumed it was some sort of malicious behavior, because it happened quite suddenly both times and had the effect of a DoS. ;-) For example, I have over 1700 binary log files like "log.002014", each 10MB, created in the last 24 hours. (It would have kept going, but the filesystem filled up.) The timestamps show that often 30 or 40 of them are created in the same minute. ~Keith On 06/14/2018 11:54 PM, Kiss Gabor (Bitman) wrote: > Yesterday at 18:15 (CEST) keys.niif.hu started to produce tons > of logs in /var/lib/sks/DB. In less than 2 hours the 40 GB filesystem > got fulfilled. > Deleting files and restarting processes did not help: > > recon.log: > 2018-06-15 05:50:09 Opening log > 2018-06-15 05:50:09 sks_recon, SKS version 1.1.6 > 2018-06-15 05:50:09 Using BerkelyDB version 5.3.28 > 2018-06-15 05:50:09 Copyright Yaron Minsky 2002-2013 > 2018-06-15 05:50:09 Licensed under GPL. See LICENSE file for details > 2018-06-15 05:50:09 recon port: 11370 > 2018-06-15 05:50:09 Opening PTree database > 2018-06-15 05:50:09 Setting up PTree data structure > 2018-06-15 05:50:09 PTree setup complete > 2018-06-15 05:50:09 Initiating catchup > 2018-06-15 05:50:10 DB closed > > db.log: > 2018-06-15 05:50:09 Opening log > 2018-06-15 05:50:09 sks_db, SKS version 1.1.6 > 2018-06-15 05:50:09 Using BerkelyDB version 5.3.28 > 2018-06-15 05:50:09 Copyright Yaron Minsky 2002, 2003, 2004 > 2018-06-15 05:50:09 Licensed under GPL. See LICENSE file for details > 2018-06-15 05:50:09 http port: 11371 > 2018-06-15 05:50:09 Membership: (zimmermann.mayfirst.org 11370)[], ... > (keys.jpbe.de 11370)[] > 2018-06-15 05:50:09 address for zimmermann.mayfirst.org:11370 changed from [] > to > [, ] > ... > 2018-06-15 05:50:10 address for keys.jpbe.de:11370 changed from [] to > [, ] > 2018-06-15 05:50:10 Opening KeyDB database > 2018-06-15 05:50:10 Shutting down database > > Unfortunately I cannot work on restoration till Sunday evening. > > Gabor > > ___ > Sks-devel mailing list > Sks-devel@nongnu.org > https://lists.nongnu.org/mailman/listinfo/sks-devel signature.asc Description: OpenPGP digital signature ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] disk full, keys.niif.hu crashed
My little Raspberry Pi node is still online but its file system is also
filling up.
It's trying to get updated keys from its peers but is constantly failing
with:
2018-06-15 08:39:53 Error getting missing keys:
Invalid_argument("String.create")
All of my peers have a different number of keys (one peer has 77,
another peer has 30, etc.) so I think all of the nodes are having an issue.
Rob D
On 2018-06-15 08:27, Paul M Furley wrote:
> Glad I wasn't the only one :) keyserver.paulfurley.com also got
> destroyed, rebuilt this morning.
>
> I've been getting a lot of traffic alerts from my host lately (>200MB
> per hour), anyone know if there's a reason there's been a lot more
> traffic lately?
>
> I haven't yet managed to investigate if it's peering traffic traffic
> from the pool.
>
> Kind regards,
>
> Paul
>
> On 15/06/18 08:40, André Keller wrote:
>> Hi,
>>
>> On 15.06.2018 05:54, Kiss Gabor (Bitman) wrote:
>>> Yesterday at 18:15 (CEST) keys.niif.hu started to produce tons
>>> of logs in /var/lib/sks/DB. In less than 2 hours the 40 GB filesystem
>>> got fulfilled.
>>> Deleting files and restarting processes did not help:
>> keys.communityrack.org shares the same fate. Trying to get it online
>> again...
>>
>>
>> Regards
>>
>> André
>>
>>
>>
>>
>> ___
>> Sks-devel mailing list
>> Sks-devel@nongnu.org
>> https://lists.nongnu.org/mailman/listinfo/sks-devel
>>
>
>
> ___
> Sks-devel mailing list
> Sks-devel@nongnu.org
> https://lists.nongnu.org/mailman/listinfo/sks-devel
signature.asc
Description: OpenPGP digital signature
___
Sks-devel mailing list
Sks-devel@nongnu.org
https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] disk full, keys.niif.hu crashed
some nodes have the db cleanup, some nodes have loggging; Graph of disk space There was definitely an injection of keys, will perform some clean up ops later. Kind Regards, Mike On 15/06/18 13:27, Paul M Furley wrote: > Glad I wasn't the only one :) keyserver.paulfurley.com also got > destroyed, rebuilt this morning. > > I've been getting a lot of traffic alerts from my host lately (>200MB > per hour), anyone know if there's a reason there's been a lot more > traffic lately? > > I haven't yet managed to investigate if it's peering traffic traffic > from the pool. > > Kind regards, > > Paul > > On 15/06/18 08:40, André Keller wrote: >> Hi, >> >> On 15.06.2018 05:54, Kiss Gabor (Bitman) wrote: >>> Yesterday at 18:15 (CEST) keys.niif.hu started to produce tons >>> of logs in /var/lib/sks/DB. In less than 2 hours the 40 GB filesystem >>> got fulfilled. >>> Deleting files and restarting processes did not help: >> keys.communityrack.org shares the same fate. Trying to get it online >> again... >> >> >> Regards >> >> André >> >> >> >> >> ___ >> Sks-devel mailing list >> Sks-devel@nongnu.org >> https://lists.nongnu.org/mailman/listinfo/sks-devel >> > > > ___ > Sks-devel mailing list > Sks-devel@nongnu.org > https://lists.nongnu.org/mailman/listinfo/sks-devel signature.asc Description: OpenPGP digital signature ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] disk full, keys.niif.hu crashed
Glad I wasn't the only one :) keyserver.paulfurley.com also got destroyed, rebuilt this morning. I've been getting a lot of traffic alerts from my host lately (>200MB per hour), anyone know if there's a reason there's been a lot more traffic lately? I haven't yet managed to investigate if it's peering traffic traffic from the pool. Kind regards, Paul On 15/06/18 08:40, André Keller wrote: > Hi, > > On 15.06.2018 05:54, Kiss Gabor (Bitman) wrote: >> Yesterday at 18:15 (CEST) keys.niif.hu started to produce tons >> of logs in /var/lib/sks/DB. In less than 2 hours the 40 GB filesystem >> got fulfilled. >> Deleting files and restarting processes did not help: > > keys.communityrack.org shares the same fate. Trying to get it online > again... > > > Regards > > André > > > > > ___ > Sks-devel mailing list > Sks-devel@nongnu.org > https://lists.nongnu.org/mailman/listinfo/sks-devel > signature.asc Description: OpenPGP digital signature ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] disk full, keys.niif.hu crashed
FWIW, you can set the DB_LOG_AUTOREMOVE flag for the database - the logs should be removed automatically [root@instance-4 ~]# cat /var/lib/sks/KDB/DB_CONFIG set_flags DB_LOG_AUTOREMOVE Best regards, Am 15.06.18 um 09:40 schrieb André Keller: > Hi, > > On 15.06.2018 05:54, Kiss Gabor (Bitman) wrote: >> Yesterday at 18:15 (CEST) keys.niif.hu started to produce tons >> of logs in /var/lib/sks/DB. In less than 2 hours the 40 GB filesystem >> got fulfilled. >> Deleting files and restarting processes did not help: > keys.communityrack.org shares the same fate. Trying to get it online > again... > > > Regards > > André > > > > > ___ > Sks-devel mailing list > Sks-devel@nongnu.org > https://lists.nongnu.org/mailman/listinfo/sks-devel signature.asc Description: OpenPGP digital signature ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
Re: [Sks-devel] disk full, keys.niif.hu crashed
Hi, On 15.06.2018 05:54, Kiss Gabor (Bitman) wrote: > Yesterday at 18:15 (CEST) keys.niif.hu started to produce tons > of logs in /var/lib/sks/DB. In less than 2 hours the 40 GB filesystem > got fulfilled. > Deleting files and restarting processes did not help: keys.communityrack.org shares the same fate. Trying to get it online again... Regards André ___ Sks-devel mailing list Sks-devel@nongnu.org https://lists.nongnu.org/mailman/listinfo/sks-devel
