Re: [PDB Tech] Question about API ratelimits
On Thu, 11 Apr 2024, Tom Strickx wrote: > Hey folks, > > We're wondering how the ratelimits are enforced these days. > Specifically, authenticated (so with API key) requests. Are these enforced > by API-key, by AccountID, by IP, by subnet, by star sign, ...? > Let me know if there's some piece of documentation documenting all of this. > > Thanks! > -- > Tom Strickx > Principal Network Engineer > AS13335 - Cloudflare Hi Tom, Per: https://docs.peeringdb.com/howto/work_within_peeringdbs_query_limits/ - Anonymous queries limited to 20/minute per IP address - Authenticated queries limited to 40/minute per user or organization (when an organizational API key is used) This comes from a set of HOWTOs that may be of interest to others, including one on the caching software peeringdb-py: https://docs.peeringdb.com/howtos/ The Seattle IX uses peeringdb-py to perform many queries of PeeringDB per day to inform its web site and route servers, with nil impact to PeeringDB itself, since the queries hit a local database instead. There are also query limits for repeated identical from unauthenticated queries. These are per IP address and per /24 or /64 address block. If you want to dig into the Django code for throttling, check out: https://github.com/peeringdb/peeringdb/blob/master/peeringdb_server/rest_throttles.py Let me know if you have other questions. I volunteer on PDB Ops. Chris ___ Pdb-tech mailing list Pdb-tech@lists.peeringdb.com https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
[PDB Tech] Fwd: new rate limiting mechanism is too strict
fyi, Arnold Weitergeleitete Nachricht Betreff: Re: [PDB Tech] new rate limiting mechanism is too strict Datum: Tue, 17 May 2022 16:54:59 + (UTC) Von: Chris Caputo Antwort an: Chris Caputo An: pdb-tech@lists.peeringdb.com Kopie (CC): Theo de Raadt All, I am behind the throttling rollout in the last 24 hours, and have worked with Theo to loosen things up for now. I've also reached out to pierky re changes requested for arouteserver and will endeavor to delay resumption of the same throttling until after arouteserver users have had reasonable time to upgrade. Highlights for all client developers: - Implement support for PeeringDB API keys: https://docs.peeringdb.com/howto/api_keys/ The idea being that we will throttle users using API keys less. - Add a delay in between queries that is randomly between 2 and 2.5 seconds, to reduce thundering herd. This delay will mean a client queries PeeringDB at most 30 hits per minute, which will be unthrottled if they are authenticated using API keys and not making identical requests. - Highly preferred over separate queries: If you don't need non-public contact info from PeeringDB, is that you implement peeringdb-py (peeringdb-py: http://peeringdb.github.io/peeringdb-py/) client-side caching. Doing so enables you to locally query the heck out of a local sqlite (or whatever) database. The start time of a peeringdb-py run should be randomized per the docs (http://peeringdb.github.io/peeringdb-py/cli/). At the SeattleIX we use peeringdb-py and here is what the once per hour update looks like for all of PeeringDB: [17/May/2022:15:40:09 +] "GET /api/org?since=1652794724=0 HTTP/1.1" 200 392 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.423 [17/May/2022:15:40:10 +] "GET /api/fac?since=1652773361=0 HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.409 [17/May/2022:15:40:10 +] "GET /api/net?since=1652796557=0 HTTP/1.1" 200 1695 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.426 [17/May/2022:15:40:11 +] "GET /api/ix?since=1652397370=0 HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.397 [17/May/2022:15:40:11 +] "GET /api/ixfac?since=1652763759=0 HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.414 [17/May/2022:15:40:12 +] "GET /api/ixlan?since=1652781160=0 HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.399 [17/May/2022:15:40:12 +] "GET /api/ixpfx?since=1652429334=0 HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.408 [17/May/2022:15:40:13 +] "GET /api/netfac?since=1652790428=0 HTTP/1.1" 200 318 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.553 [17/May/2022:15:40:14 +] "GET /api/netixlan?since=1652796556=0 HTTP/1.1" 200 399 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.590 [17/May/2022:15:40:14 +] "GET /api/poc?since=1652785835=0 HTTP/1.1" 200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.640 It is fast because, as I understand it, django serializes PeeringDB changes, and the timestamp (since last update) results in only the changes being delivered. Finally: My apology to those disrupted by this. Please feel free to reach out to me with any questions or concerns. Thanks, Chris ___ Pdb-tech mailing list Pdb-tech@lists.peeringdb.com https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech -- Arnold Nipper Chief Technology Evangelist and Co-Founder DE-CIX Management GmbH Lindleystraße 12 | 60314 Frankfurt a.M. | Germany Phone +49 69 1730902 22 | Mobile +49 172 2650958 arnold.nip...@de-cix.net | www.de-cix.net Geschaeftsfuehrer Ivaylo Ivanov und Sebastian Seifert Registergericht AG Koeln HRB 51135 Want to work at DE-CIX: https://de-cix.net/en/about-de-cix/careers OpenPGP_signature.asc Description: OpenPGP digital signature ___ Pdb-tech mailing list Pdb-tech@lists.peeringdb.com https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
[PDB Tech] Question about API ratelimits
Hey folks, We're wondering how the ratelimits are enforced these days. Specifically, authenticated (so with API key) requests. Are these enforced by API-key, by AccountID, by IP, by subnet, by star sign, ...? Let me know if there's some piece of documentation documenting all of this. Thanks! -- Tom Strickx Principal Network Engineer AS13335 - Cloudflare ___ Pdb-tech mailing list Pdb-tech@lists.peeringdb.com https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
