fyi, Arnold
-------- Weitergeleitete Nachricht -------- Betreff: Re: [PDB Tech] new rate limiting mechanism is too strict Datum: Tue, 17 May 2022 16:54:59 +0000 (UTC) Von: Chris Caputo <[email protected]> Antwort an: Chris Caputo <[email protected]> An: [email protected] Kopie (CC): Theo de Raadt <[email protected]> All,I am behind the throttling rollout in the last 24 hours, and have worked with Theo to loosen things up for now. I've also reached out to pierky re changes requested for arouteserver and will endeavor to delay resumption of the same throttling until after arouteserver users have had reasonable time to upgrade.
Highlights for all client developers:
- Implement support for PeeringDB API keys:
https://docs.peeringdb.com/howto/api_keys/
The idea being that we will throttle users using API keys less.
- Add a delay in between queries that is randomly between 2 and 2.5
seconds, to reduce thundering herd. This delay will mean a client
queries PeeringDB at most 30 hits per minute, which will be unthrottled
if they are authenticated using API keys and not making identical
requests.
- Highly preferred over separate queries: If you don't need non-public contact info from PeeringDB, is that you implement peeringdb-py (peeringdb-py: http://peeringdb.github.io/peeringdb-py/) client-side caching. Doing so enables you to locally query the heck out of a local sqlite (or whatever) database. The start time of a peeringdb-py run should be randomized per the docs (http://peeringdb.github.io/peeringdb-py/cli/). At the SeattleIX we use peeringdb-py and here is what the once per hour update looks like for all of PeeringDB:
[17/May/2022:15:40:09 +0000] "GET /api/org?since=1652794724&depth=0 HTTP/1.1" 200 392 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.423 [17/May/2022:15:40:10 +0000] "GET /api/fac?since=1652773361&depth=0 HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.409 [17/May/2022:15:40:10 +0000] "GET /api/net?since=1652796557&depth=0 HTTP/1.1" 200 1695 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.426 [17/May/2022:15:40:11 +0000] "GET /api/ix?since=1652397370&depth=0 HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.397 [17/May/2022:15:40:11 +0000] "GET /api/ixfac?since=1652763759&depth=0 HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.414 [17/May/2022:15:40:12 +0000] "GET /api/ixlan?since=1652781160&depth=0 HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.399 [17/May/2022:15:40:12 +0000] "GET /api/ixpfx?since=1652429334&depth=0 HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.408 [17/May/2022:15:40:13 +0000] "GET /api/netfac?since=1652790428&depth=0 HTTP/1.1" 200 318 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.553 [17/May/2022:15:40:14 +0000] "GET /api/netixlan?since=1652796556&depth=0 HTTP/1.1" 200 399 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.590 [17/May/2022:15:40:14 +0000] "GET /api/poc?since=1652785835&depth=0 HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.640
It is fast because, as I understand it, django serializes PeeringDB changes, and the timestamp (since last update) results in only the changes being delivered.
Finally: My apology to those disrupted by this. Please feel free to reach out to me with any questions or concerns.
Thanks, Chris _______________________________________________ Pdb-tech mailing list [email protected] https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech -- Arnold Nipper Chief Technology Evangelist and Co-Founder DE-CIX Management GmbH Lindleystraße 12 | 60314 Frankfurt a.M. | Germany Phone +49 69 1730902 22 | Mobile +49 172 2650958 [email protected] | www.de-cix.net Geschaeftsfuehrer Ivaylo Ivanov und Sebastian Seifert Registergericht AG Koeln HRB 51135 Want to work at DE-CIX: https://de-cix.net/en/about-de-cix/careers
OpenPGP_signature.asc
Description: OpenPGP digital signature
_______________________________________________ Pdb-tech mailing list [email protected] https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
