Re: [PDB Tech] PeeringDB API throttling status and schedule
Based on: https://github.com/peeringdb/peeringdb/issues/427#issuecomment-1218382411 the anonymous query limit has been reverted to 20/min. Current qpm limits: - anonymous queries limited to 20/minute per IP address - authenticated queries limited to 40/minute per user/org Thanks, Chris On Mon, 15 Aug 2022, Chris Caputo wrote: > Per the below plan, this change was just implemented: > > --- > On August 15th, adjust and watch for feedback from the community: > > - anonymous queries limited to 10/minute per IP address > - authenticated queries limited to 40/minute per user/org > --- > > Please advise if you run into any issues. > > Thank you, > Chris > > On Sun, 14 Aug 2022, Chris Caputo wrote: > > Sorry - I totally jumped the gun on what UTC day it is! Reverted back to > > the August 8th settings: > > > > - anonymous queries limited to 20/minute per IP address > > - authenticated queries limited to 60/minute per user/org > > > > Chris > > > > On Sun, 14 Aug 2022, Chris Caputo wrote: > > > Per the below plan, this change was just implemented: > > > > > > --- > > > On August 15th, adjust and watch for feedback from the community: > > > > > > - anonymous queries limited to 10/minute per IP address > > > - authenticated queries limited to 40/minute per user/org > > > --- > > > > > > Please advise if you run into any issues. > > > > > > Thank you, > > > Chris > > > > > > On Tue, 31 May 2022, Chris Caputo wrote: > > > > After the initial introduction of PeeringDB API throttling, some > > > > software > > > > both open source and private, has been identified and updated. (open > > > > source details are below; please upgrade and encourage others to do so) > > > > > > > > This API throttling is being implemented to control costs by > > > > encouraging > > > > efficient software design while making sure the PeeringDB resource is > > > > shared well. The use of API keys is being encouraged so that admins can > > > > reach out to users/orgs with runaway or inefficient software, and > > > > because > > > > it is more secure than user/pass. In addition, org API keys ease > > > > employee > > > > transitions. > > > > > > > > Some tips for coders is below. > > > > > > > > API throttling in place today: > > > > > > > > - repeated anonymous identical requests with a response size above > > > > 100k > > > > are being limited to 1/hour > > > > - repeated anonymous identical requests of any size are being limited > > > > to > > > > 2/minute > > > > - anonymous queries are being limited to 400/minute per IP address > > > > - authenticated queries are being limited to 500/minute per user/org > > > > > > > > Here is the current schedule of throttling changes. The schedule may > > > > adjust as needed as new packages that need update are discovered, so as > > > > to > > > > minimize disruption to the community... > > > > > > > > On June 27th, adjust and watch for feedback from the community: > > > > > > > > - anonymous queries limited to 300/minute per IP address > > > > - authenticated queries limited to 400/minute per user/org > > > > > > > > On July 11th, adjust and watch for feedback from the community: > > > > > > > > - anonymous queries limited to 200/minute per IP address > > > > - authenticated queries limited to 300/minute per user/org > > > > > > > > On July 18th, adjust and watch for feedback from the community: > > > > > > > > - anonymous queries limited to 100/minute per IP address > > > > - authenticated queries limited to 200/minute per user/org > > > > > > > > On July 25th, adjust and watch for feedback from the community: > > > > > > > > - anonymous queries limited to 50/minute per IP address > > > > - authenticated queries limited to 100/minute per user/org > > > > > > > > On August 1st, adjust and watch for feedback from the community: > > > > > > > > - anonymous queries limited to 30/minute per IP address > > > > - authenticated queries limited to 80/minute per user/org > > > > > > > > On August 8th, adjust and watch for feedback from the community: > > > > > > > > - anonymous queries limited to 20/minute per IP address > > > > - authenticated queries limited to 60/minute per user/org > > > > > > > > On August 15th, adjust and watch for feedback from the community: > > > > > > > > - anonymous queries limited to 10/minute per IP address > > > > - authenticated queries limited to 40/minute per user/org > > > > > > > > Feedback/questions/concerns welcome. > > > > > > > > Thanks, > > > > Chris > > > > > > > > Software: > > > > > > > > - arouteserver v1.16.0: has many updates including API key support > > > > along > > > > with more efficient querying. > > > > > > > > - PeerFinder: API key & efficient querying patches at > > > > https://github.com/rucarrol/PeerFinder/pull/17 will hopefully be > > > > integrated. > > > > > > > > Coding tips: > > > > > > > > - Begin using a
Re: [PDB Tech] PeeringDB API throttling status and schedule
Per the below plan, this change was just implemented: --- On August 15th, adjust and watch for feedback from the community: - anonymous queries limited to 10/minute per IP address - authenticated queries limited to 40/minute per user/org --- Please advise if you run into any issues. Thank you, Chris On Sun, 14 Aug 2022, Chris Caputo wrote: > Sorry - I totally jumped the gun on what UTC day it is! Reverted back to > the August 8th settings: > > - anonymous queries limited to 20/minute per IP address > - authenticated queries limited to 60/minute per user/org > > Chris > > On Sun, 14 Aug 2022, Chris Caputo wrote: > > Per the below plan, this change was just implemented: > > > > --- > > On August 15th, adjust and watch for feedback from the community: > > > > - anonymous queries limited to 10/minute per IP address > > - authenticated queries limited to 40/minute per user/org > > --- > > > > Please advise if you run into any issues. > > > > Thank you, > > Chris > > > > On Tue, 31 May 2022, Chris Caputo wrote: > > > After the initial introduction of PeeringDB API throttling, some software > > > both open source and private, has been identified and updated. (open > > > source details are below; please upgrade and encourage others to do so) > > > > > > This API throttling is being implemented to control costs by encouraging > > > efficient software design while making sure the PeeringDB resource is > > > shared well. The use of API keys is being encouraged so that admins can > > > reach out to users/orgs with runaway or inefficient software, and because > > > it is more secure than user/pass. In addition, org API keys ease employee > > > transitions. > > > > > > Some tips for coders is below. > > > > > > API throttling in place today: > > > > > > - repeated anonymous identical requests with a response size above 100k > > > are being limited to 1/hour > > > - repeated anonymous identical requests of any size are being limited > > > to > > > 2/minute > > > - anonymous queries are being limited to 400/minute per IP address > > > - authenticated queries are being limited to 500/minute per user/org > > > > > > Here is the current schedule of throttling changes. The schedule may > > > adjust as needed as new packages that need update are discovered, so as > > > to > > > minimize disruption to the community... > > > > > > On June 27th, adjust and watch for feedback from the community: > > > > > > - anonymous queries limited to 300/minute per IP address > > > - authenticated queries limited to 400/minute per user/org > > > > > > On July 11th, adjust and watch for feedback from the community: > > > > > > - anonymous queries limited to 200/minute per IP address > > > - authenticated queries limited to 300/minute per user/org > > > > > > On July 18th, adjust and watch for feedback from the community: > > > > > > - anonymous queries limited to 100/minute per IP address > > > - authenticated queries limited to 200/minute per user/org > > > > > > On July 25th, adjust and watch for feedback from the community: > > > > > > - anonymous queries limited to 50/minute per IP address > > > - authenticated queries limited to 100/minute per user/org > > > > > > On August 1st, adjust and watch for feedback from the community: > > > > > > - anonymous queries limited to 30/minute per IP address > > > - authenticated queries limited to 80/minute per user/org > > > > > > On August 8th, adjust and watch for feedback from the community: > > > > > > - anonymous queries limited to 20/minute per IP address > > > - authenticated queries limited to 60/minute per user/org > > > > > > On August 15th, adjust and watch for feedback from the community: > > > > > > - anonymous queries limited to 10/minute per IP address > > > - authenticated queries limited to 40/minute per user/org > > > > > > Feedback/questions/concerns welcome. > > > > > > Thanks, > > > Chris > > > > > > Software: > > > > > > - arouteserver v1.16.0: has many updates including API key support along > > > with more efficient querying. > > > > > > - PeerFinder: API key & efficient querying patches at > > > https://github.com/rucarrol/PeerFinder/pull/17 will hopefully be > > > integrated. > > > > > > Coding tips: > > > > > > - Begin using a PeeringDB API key for all requests: > > > > > > https://docs.peeringdb.com/howto/api_keys/ > > > > > > - Begin performing actual caching, such as by using peeringdb-py. > > > > > > http://peeringdb.github.io/peeringdb-py/ > > > > > > - If unable to use a caching agent such as peeringdb-py: > > > > > >- Use an API key. > > > > > >- Set a User-Agent: header. > > > > > >- Use bulk queries (asn__in=$list_of_ASN_separated_by_comma) by > > > querying 30 to 150 ASNs at a time (tune as appropriate). > > > > > >- Add a delay in between queries that is randomly between 2 and 2.5 > > > seconds,
Re: [PDB Tech] PeeringDB API throttling status and schedule
Sorry - I totally jumped the gun on what UTC day it is! Reverted back to the August 8th settings: - anonymous queries limited to 20/minute per IP address - authenticated queries limited to 60/minute per user/org Chris On Sun, 14 Aug 2022, Chris Caputo wrote: > Per the below plan, this change was just implemented: > > --- > On August 15th, adjust and watch for feedback from the community: > > - anonymous queries limited to 10/minute per IP address > - authenticated queries limited to 40/minute per user/org > --- > > Please advise if you run into any issues. > > Thank you, > Chris > > On Tue, 31 May 2022, Chris Caputo wrote: > > After the initial introduction of PeeringDB API throttling, some software > > both open source and private, has been identified and updated. (open > > source details are below; please upgrade and encourage others to do so) > > > > This API throttling is being implemented to control costs by encouraging > > efficient software design while making sure the PeeringDB resource is > > shared well. The use of API keys is being encouraged so that admins can > > reach out to users/orgs with runaway or inefficient software, and because > > it is more secure than user/pass. In addition, org API keys ease employee > > transitions. > > > > Some tips for coders is below. > > > > API throttling in place today: > > > > - repeated anonymous identical requests with a response size above 100k > > are being limited to 1/hour > > - repeated anonymous identical requests of any size are being limited to > > 2/minute > > - anonymous queries are being limited to 400/minute per IP address > > - authenticated queries are being limited to 500/minute per user/org > > > > Here is the current schedule of throttling changes. The schedule may > > adjust as needed as new packages that need update are discovered, so as to > > minimize disruption to the community... > > > > On June 27th, adjust and watch for feedback from the community: > > > > - anonymous queries limited to 300/minute per IP address > > - authenticated queries limited to 400/minute per user/org > > > > On July 11th, adjust and watch for feedback from the community: > > > > - anonymous queries limited to 200/minute per IP address > > - authenticated queries limited to 300/minute per user/org > > > > On July 18th, adjust and watch for feedback from the community: > > > > - anonymous queries limited to 100/minute per IP address > > - authenticated queries limited to 200/minute per user/org > > > > On July 25th, adjust and watch for feedback from the community: > > > > - anonymous queries limited to 50/minute per IP address > > - authenticated queries limited to 100/minute per user/org > > > > On August 1st, adjust and watch for feedback from the community: > > > > - anonymous queries limited to 30/minute per IP address > > - authenticated queries limited to 80/minute per user/org > > > > On August 8th, adjust and watch for feedback from the community: > > > > - anonymous queries limited to 20/minute per IP address > > - authenticated queries limited to 60/minute per user/org > > > > On August 15th, adjust and watch for feedback from the community: > > > > - anonymous queries limited to 10/minute per IP address > > - authenticated queries limited to 40/minute per user/org > > > > Feedback/questions/concerns welcome. > > > > Thanks, > > Chris > > > > Software: > > > > - arouteserver v1.16.0: has many updates including API key support along > > with more efficient querying. > > > > - PeerFinder: API key & efficient querying patches at > > https://github.com/rucarrol/PeerFinder/pull/17 will hopefully be > > integrated. > > > > Coding tips: > > > > - Begin using a PeeringDB API key for all requests: > > > > https://docs.peeringdb.com/howto/api_keys/ > > > > - Begin performing actual caching, such as by using peeringdb-py. > > > > http://peeringdb.github.io/peeringdb-py/ > > > > - If unable to use a caching agent such as peeringdb-py: > > > >- Use an API key. > > > >- Set a User-Agent: header. > > > >- Use bulk queries (asn__in=$list_of_ASN_separated_by_comma) by > > querying 30 to 150 ASNs at a time (tune as appropriate). > > > >- Add a delay in between queries that is randomly between 2 and 2.5 > > seconds, to reduce thundering herd. > ___ > Pdb-tech mailing list > Pdb-tech@lists.peeringdb.com > https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech > ___ Pdb-tech mailing list Pdb-tech@lists.peeringdb.com https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
Re: [PDB Tech] PeeringDB API throttling status and schedule
Per the below plan, this change was just implemented: --- On August 15th, adjust and watch for feedback from the community: - anonymous queries limited to 10/minute per IP address - authenticated queries limited to 40/minute per user/org --- Please advise if you run into any issues. Thank you, Chris On Tue, 31 May 2022, Chris Caputo wrote: > After the initial introduction of PeeringDB API throttling, some software > both open source and private, has been identified and updated. (open > source details are below; please upgrade and encourage others to do so) > > This API throttling is being implemented to control costs by encouraging > efficient software design while making sure the PeeringDB resource is > shared well. The use of API keys is being encouraged so that admins can > reach out to users/orgs with runaway or inefficient software, and because > it is more secure than user/pass. In addition, org API keys ease employee > transitions. > > Some tips for coders is below. > > API throttling in place today: > > - repeated anonymous identical requests with a response size above 100k > are being limited to 1/hour > - repeated anonymous identical requests of any size are being limited to > 2/minute > - anonymous queries are being limited to 400/minute per IP address > - authenticated queries are being limited to 500/minute per user/org > > Here is the current schedule of throttling changes. The schedule may > adjust as needed as new packages that need update are discovered, so as to > minimize disruption to the community... > > On June 27th, adjust and watch for feedback from the community: > > - anonymous queries limited to 300/minute per IP address > - authenticated queries limited to 400/minute per user/org > > On July 11th, adjust and watch for feedback from the community: > > - anonymous queries limited to 200/minute per IP address > - authenticated queries limited to 300/minute per user/org > > On July 18th, adjust and watch for feedback from the community: > > - anonymous queries limited to 100/minute per IP address > - authenticated queries limited to 200/minute per user/org > > On July 25th, adjust and watch for feedback from the community: > > - anonymous queries limited to 50/minute per IP address > - authenticated queries limited to 100/minute per user/org > > On August 1st, adjust and watch for feedback from the community: > > - anonymous queries limited to 30/minute per IP address > - authenticated queries limited to 80/minute per user/org > > On August 8th, adjust and watch for feedback from the community: > > - anonymous queries limited to 20/minute per IP address > - authenticated queries limited to 60/minute per user/org > > On August 15th, adjust and watch for feedback from the community: > > - anonymous queries limited to 10/minute per IP address > - authenticated queries limited to 40/minute per user/org > > Feedback/questions/concerns welcome. > > Thanks, > Chris > > Software: > > - arouteserver v1.16.0: has many updates including API key support along > with more efficient querying. > > - PeerFinder: API key & efficient querying patches at > https://github.com/rucarrol/PeerFinder/pull/17 will hopefully be > integrated. > > Coding tips: > > - Begin using a PeeringDB API key for all requests: > > https://docs.peeringdb.com/howto/api_keys/ > > - Begin performing actual caching, such as by using peeringdb-py. > > http://peeringdb.github.io/peeringdb-py/ > > - If unable to use a caching agent such as peeringdb-py: > >- Use an API key. > >- Set a User-Agent: header. > >- Use bulk queries (asn__in=$list_of_ASN_separated_by_comma) by > querying 30 to 150 ASNs at a time (tune as appropriate). > >- Add a delay in between queries that is randomly between 2 and 2.5 > seconds, to reduce thundering herd. ___ Pdb-tech mailing list Pdb-tech@lists.peeringdb.com https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
Re: [PDB Tech] PeeringDB API throttling status and schedule (fwd)
Thus spake Chris Caputo (ccap...@alt.net) on Tue, Aug 09, 2022 at 07:11:02PM +: > Dale, if you are getting 200 for an obviously bad api-key, then the > authentication format is not correct. Examples/details at: > > https://github.com/peeringdb/peeringdb/issues/1220#issuecomment-1209763911 > > With a correctly formated auth request, 401 (unauth) will be returned for > a bad key. > > Please reach out to me privately with your source IP if you'd like me to > review how the server sees your requests, or for efficiency efforts, or if > you need any help getting api-key authentication working. Will do! I really appreciate it. Like many things, hopefully it's it's mostly PEBCAK. Dale > On Tue, 9 Aug 2022, Stephen McManus wrote: > > > However, for a read-only API key, how does one know if it's working? > > > I set 'Authorization': 'Api-Key foo-bar-1234-4312' for a GET, and I > > > got results back vs a 4xx error code. So from an error handling > > > perspective it seems hard to gauge if I am using a valid api key > > > getting premium service vs an invalid api key quietly lumped into > > > the anonymous rate-limit bucket. > > > > This is something we should fix. I've filed > > https://github.com/peeringdb/peeringdb/issues/1220 to get it addressed > > > > -Steve > > > > > > > > > > > On Aug 9, 2022, at 1:56 PM, Dale W. Carder wrote: > > > > > > Thus spake Chris Caputo (ccap...@alt.net) on Mon, Aug 08, 2022 at > > > 04:41:17PM +: > > >> Per the below plan, this change was just implemented: > > >> > > >> --- > > >> On August 8th, adjust and watch for feedback from the community: > > >> > > >> - anonymous queries limited to 20/minute per IP address > > >> - authenticated queries limited to 60/minute per user/org > > >> --- > > >> > > >> Please advise if you run into any issues. > > > > > > This is about where I start to get concerned. First off, I'm not > > > sure how well communicated this was. I'd like to think that I'm > > > generally aware of what's happening in our ecosystem, but someone > > > (thankfully) had to point this out to me. > > > > > > So, our provisioning code is perhaps naive... jobs are dispatched > > > into a task queue where they are run to completion, one per ASN. > > > At present it would be non-trivial to implement a bulk query to > > > cache ahead of time (making peeringdb lookups asynchronous), but > > > that absolutely is on our longer-term roadmap. It's also not the > > > easiest to rate-limit the queue as only some of them actually need > > > a peeringdb lookup (a huge amount of our peers are private asn > > > and/or in a non-dfz l3vpn's), but we have limited the concurrency > > > and can count on the general case that our code is reassuringly > > > slow. > > > > > > Luckily, some of the other things suggested below are easy, and I > > > was testing it out today. We'll set a custom user-agent, limit > > > our query to only the fields we care about, and use an api key. > > > > > > However, for a read-only API key, how does one know if it's working? > > > I set 'Authorization': 'Api-Key foo-bar-1234-4312' for a GET, and I > > > got results back vs a 4xx error code. So from an error handling > > > perspective it seems hard to gauge if I am using a valid api key > > > getting premium service vs an invalid api key quietly lumped into > > > the anonymous rate-limit bucket. > > > > > > Dale > > > > > > > > >> On Tue, 31 May 2022, Chris Caputo wrote: > > >>> After the initial introduction of PeeringDB API throttling, some > > >>> software > > >>> both open source and private, has been identified and updated. (open > > >>> source details are below; please upgrade and encourage others to do so) > > >>> > > >>> This API throttling is being implemented to control costs by > > >>> encouraging > > >>> efficient software design while making sure the PeeringDB resource is > > >>> shared well. The use of API keys is being encouraged so that admins can > > >>> reach out to users/orgs with runaway or inefficient software, and > > >>> because > > >>> it is more secure than user/pass. In addition, org API keys ease > > >>> employee > > >>> transitions. > > >>> > > >>> Some tips for coders is below. > > >>> > > >>> API throttling in place today: > > >>> > > >>> - repeated anonymous identical requests with a response size above > > >>> 100k > > >>>are being limited to 1/hour > > >>> - repeated anonymous identical requests of any size are being limited > > >>> to > > >>>2/minute > > >>> - anonymous queries are being limited to 400/minute per IP address > > >>> - authenticated queries are being limited to 500/minute per user/org > > >>> > > >>> Here is the current schedule of throttling changes. The schedule may > > >>> adjust as needed as new packages that need update are discovered, so as > > >>> to > > >>> minimize disruption to the community... > > >>> > > >>> On June 27th, adjust and watch for feedback from
Re: [PDB Tech] PeeringDB API throttling status and schedule (fwd)
Dale, if you are getting 200 for an obviously bad api-key, then the authentication format is not correct. Examples/details at: https://github.com/peeringdb/peeringdb/issues/1220#issuecomment-1209763911 With a correctly formated auth request, 401 (unauth) will be returned for a bad key. Please reach out to me privately with your source IP if you'd like me to review how the server sees your requests, or for efficiency efforts, or if you need any help getting api-key authentication working. Thanks! Chris On Tue, 9 Aug 2022, Stephen McManus wrote: > > However, for a read-only API key, how does one know if it's working? > > I set 'Authorization': 'Api-Key foo-bar-1234-4312' for a GET, and I > > got results back vs a 4xx error code. So from an error handling > > perspective it seems hard to gauge if I am using a valid api key > > getting premium service vs an invalid api key quietly lumped into > > the anonymous rate-limit bucket. > > This is something we should fix. I've filed > https://github.com/peeringdb/peeringdb/issues/1220 to get it addressed > > -Steve > > > > > > On Aug 9, 2022, at 1:56 PM, Dale W. Carder wrote: > > > > Thus spake Chris Caputo (ccap...@alt.net) on Mon, Aug 08, 2022 at > > 04:41:17PM +: > >> Per the below plan, this change was just implemented: > >> > >> --- > >> On August 8th, adjust and watch for feedback from the community: > >> > >> - anonymous queries limited to 20/minute per IP address > >> - authenticated queries limited to 60/minute per user/org > >> --- > >> > >> Please advise if you run into any issues. > > > > This is about where I start to get concerned. First off, I'm not > > sure how well communicated this was. I'd like to think that I'm > > generally aware of what's happening in our ecosystem, but someone > > (thankfully) had to point this out to me. > > > > So, our provisioning code is perhaps naive... jobs are dispatched > > into a task queue where they are run to completion, one per ASN. > > At present it would be non-trivial to implement a bulk query to > > cache ahead of time (making peeringdb lookups asynchronous), but > > that absolutely is on our longer-term roadmap. It's also not the > > easiest to rate-limit the queue as only some of them actually need > > a peeringdb lookup (a huge amount of our peers are private asn > > and/or in a non-dfz l3vpn's), but we have limited the concurrency > > and can count on the general case that our code is reassuringly > > slow. > > > > Luckily, some of the other things suggested below are easy, and I > > was testing it out today. We'll set a custom user-agent, limit > > our query to only the fields we care about, and use an api key. > > > > However, for a read-only API key, how does one know if it's working? > > I set 'Authorization': 'Api-Key foo-bar-1234-4312' for a GET, and I > > got results back vs a 4xx error code. So from an error handling > > perspective it seems hard to gauge if I am using a valid api key > > getting premium service vs an invalid api key quietly lumped into > > the anonymous rate-limit bucket. > > > > Dale > > > > > >> On Tue, 31 May 2022, Chris Caputo wrote: > >>> After the initial introduction of PeeringDB API throttling, some software > >>> both open source and private, has been identified and updated. (open > >>> source details are below; please upgrade and encourage others to do so) > >>> > >>> This API throttling is being implemented to control costs by encouraging > >>> efficient software design while making sure the PeeringDB resource is > >>> shared well. The use of API keys is being encouraged so that admins can > >>> reach out to users/orgs with runaway or inefficient software, and because > >>> it is more secure than user/pass. In addition, org API keys ease employee > >>> transitions. > >>> > >>> Some tips for coders is below. > >>> > >>> API throttling in place today: > >>> > >>> - repeated anonymous identical requests with a response size above 100k > >>>are being limited to 1/hour > >>> - repeated anonymous identical requests of any size are being limited to > >>>2/minute > >>> - anonymous queries are being limited to 400/minute per IP address > >>> - authenticated queries are being limited to 500/minute per user/org > >>> > >>> Here is the current schedule of throttling changes. The schedule may > >>> adjust as needed as new packages that need update are discovered, so as > >>> to > >>> minimize disruption to the community... > >>> > >>> On June 27th, adjust and watch for feedback from the community: > >>> > >>> - anonymous queries limited to 300/minute per IP address > >>> - authenticated queries limited to 400/minute per user/org > >>> > >>> On July 11th, adjust and watch for feedback from the community: > >>> > >>> - anonymous queries limited to 200/minute per IP address > >>> - authenticated queries limited to 300/minute per user/org > >>> > >>> On July 18th, adjust and watch for
Re: [PDB Tech] PeeringDB API throttling status and schedule (fwd)
> However, for a read-only API key, how does one know if it's working? > I set 'Authorization': 'Api-Key foo-bar-1234-4312' for a GET, and I > got results back vs a 4xx error code. So from an error handling > perspective it seems hard to gauge if I am using a valid api key > getting premium service vs an invalid api key quietly lumped into > the anonymous rate-limit bucket. This is something we should fix. I've filed https://github.com/peeringdb/peeringdb/issues/1220 to get it addressed -Steve > On Aug 9, 2022, at 1:56 PM, Dale W. Carder wrote: > > Thus spake Chris Caputo (ccap...@alt.net) on Mon, Aug 08, 2022 at 04:41:17PM > +: >> Per the below plan, this change was just implemented: >> >> --- >> On August 8th, adjust and watch for feedback from the community: >> >> - anonymous queries limited to 20/minute per IP address >> - authenticated queries limited to 60/minute per user/org >> --- >> >> Please advise if you run into any issues. > > This is about where I start to get concerned. First off, I'm not > sure how well communicated this was. I'd like to think that I'm > generally aware of what's happening in our ecosystem, but someone > (thankfully) had to point this out to me. > > So, our provisioning code is perhaps naive... jobs are dispatched > into a task queue where they are run to completion, one per ASN. > At present it would be non-trivial to implement a bulk query to > cache ahead of time (making peeringdb lookups asynchronous), but > that absolutely is on our longer-term roadmap. It's also not the > easiest to rate-limit the queue as only some of them actually need > a peeringdb lookup (a huge amount of our peers are private asn > and/or in a non-dfz l3vpn's), but we have limited the concurrency > and can count on the general case that our code is reassuringly > slow. > > Luckily, some of the other things suggested below are easy, and I > was testing it out today. We'll set a custom user-agent, limit > our query to only the fields we care about, and use an api key. > > However, for a read-only API key, how does one know if it's working? > I set 'Authorization': 'Api-Key foo-bar-1234-4312' for a GET, and I > got results back vs a 4xx error code. So from an error handling > perspective it seems hard to gauge if I am using a valid api key > getting premium service vs an invalid api key quietly lumped into > the anonymous rate-limit bucket. > > Dale > > >> On Tue, 31 May 2022, Chris Caputo wrote: >>> After the initial introduction of PeeringDB API throttling, some software >>> both open source and private, has been identified and updated. (open >>> source details are below; please upgrade and encourage others to do so) >>> >>> This API throttling is being implemented to control costs by encouraging >>> efficient software design while making sure the PeeringDB resource is >>> shared well. The use of API keys is being encouraged so that admins can >>> reach out to users/orgs with runaway or inefficient software, and because >>> it is more secure than user/pass. In addition, org API keys ease employee >>> transitions. >>> >>> Some tips for coders is below. >>> >>> API throttling in place today: >>> >>> - repeated anonymous identical requests with a response size above 100k >>>are being limited to 1/hour >>> - repeated anonymous identical requests of any size are being limited to >>>2/minute >>> - anonymous queries are being limited to 400/minute per IP address >>> - authenticated queries are being limited to 500/minute per user/org >>> >>> Here is the current schedule of throttling changes. The schedule may >>> adjust as needed as new packages that need update are discovered, so as to >>> minimize disruption to the community... >>> >>> On June 27th, adjust and watch for feedback from the community: >>> >>> - anonymous queries limited to 300/minute per IP address >>> - authenticated queries limited to 400/minute per user/org >>> >>> On July 11th, adjust and watch for feedback from the community: >>> >>> - anonymous queries limited to 200/minute per IP address >>> - authenticated queries limited to 300/minute per user/org >>> >>> On July 18th, adjust and watch for feedback from the community: >>> >>> - anonymous queries limited to 100/minute per IP address >>> - authenticated queries limited to 200/minute per user/org >>> >>> On July 25th, adjust and watch for feedback from the community: >>> >>> - anonymous queries limited to 50/minute per IP address >>> - authenticated queries limited to 100/minute per user/org >>> >>> On August 1st, adjust and watch for feedback from the community: >>> >>> - anonymous queries limited to 30/minute per IP address >>> - authenticated queries limited to 80/minute per user/org >>> >>> On August 8th, adjust and watch for feedback from the community: >>> >>> - anonymous queries limited to 20/minute per IP address >>> - authenticated queries limited to 60/minute per user/org >>>
Re: [PDB Tech] PeeringDB API throttling status and schedule (fwd)
Thus spake Chris Caputo (ccap...@alt.net) on Mon, Aug 08, 2022 at 04:41:17PM +: > Per the below plan, this change was just implemented: > > --- > On August 8th, adjust and watch for feedback from the community: > > - anonymous queries limited to 20/minute per IP address > - authenticated queries limited to 60/minute per user/org > --- > > Please advise if you run into any issues. This is about where I start to get concerned. First off, I'm not sure how well communicated this was. I'd like to think that I'm generally aware of what's happening in our ecosystem, but someone (thankfully) had to point this out to me. So, our provisioning code is perhaps naive... jobs are dispatched into a task queue where they are run to completion, one per ASN. At present it would be non-trivial to implement a bulk query to cache ahead of time (making peeringdb lookups asynchronous), but that absolutely is on our longer-term roadmap. It's also not the easiest to rate-limit the queue as only some of them actually need a peeringdb lookup (a huge amount of our peers are private asn and/or in a non-dfz l3vpn's), but we have limited the concurrency and can count on the general case that our code is reassuringly slow. Luckily, some of the other things suggested below are easy, and I was testing it out today. We'll set a custom user-agent, limit our query to only the fields we care about, and use an api key. However, for a read-only API key, how does one know if it's working? I set 'Authorization': 'Api-Key foo-bar-1234-4312' for a GET, and I got results back vs a 4xx error code. So from an error handling perspective it seems hard to gauge if I am using a valid api key getting premium service vs an invalid api key quietly lumped into the anonymous rate-limit bucket. Dale > On Tue, 31 May 2022, Chris Caputo wrote: > > After the initial introduction of PeeringDB API throttling, some software > > both open source and private, has been identified and updated. (open > > source details are below; please upgrade and encourage others to do so) > > > > This API throttling is being implemented to control costs by encouraging > > efficient software design while making sure the PeeringDB resource is > > shared well. The use of API keys is being encouraged so that admins can > > reach out to users/orgs with runaway or inefficient software, and because > > it is more secure than user/pass. In addition, org API keys ease employee > > transitions. > > > > Some tips for coders is below. > > > > API throttling in place today: > > > > - repeated anonymous identical requests with a response size above 100k > > are being limited to 1/hour > > - repeated anonymous identical requests of any size are being limited to > > 2/minute > > - anonymous queries are being limited to 400/minute per IP address > > - authenticated queries are being limited to 500/minute per user/org > > > > Here is the current schedule of throttling changes. The schedule may > > adjust as needed as new packages that need update are discovered, so as to > > minimize disruption to the community... > > > > On June 27th, adjust and watch for feedback from the community: > > > > - anonymous queries limited to 300/minute per IP address > > - authenticated queries limited to 400/minute per user/org > > > > On July 11th, adjust and watch for feedback from the community: > > > > - anonymous queries limited to 200/minute per IP address > > - authenticated queries limited to 300/minute per user/org > > > > On July 18th, adjust and watch for feedback from the community: > > > > - anonymous queries limited to 100/minute per IP address > > - authenticated queries limited to 200/minute per user/org > > > > On July 25th, adjust and watch for feedback from the community: > > > > - anonymous queries limited to 50/minute per IP address > > - authenticated queries limited to 100/minute per user/org > > > > On August 1st, adjust and watch for feedback from the community: > > > > - anonymous queries limited to 30/minute per IP address > > - authenticated queries limited to 80/minute per user/org > > > > On August 8th, adjust and watch for feedback from the community: > > > > - anonymous queries limited to 20/minute per IP address > > - authenticated queries limited to 60/minute per user/org > > > > On August 15th, adjust and watch for feedback from the community: > > > > - anonymous queries limited to 10/minute per IP address > > - authenticated queries limited to 40/minute per user/org > > > > Feedback/questions/concerns welcome. > > > > Thanks, > > Chris > > > > Software: > > > > - arouteserver v1.16.0: has many updates including API key support along > > with more efficient querying. > > > > - PeerFinder: API key & efficient querying patches at > > https://github.com/rucarrol/PeerFinder/pull/17 will hopefully be > > integrated. > > > > Coding tips: > > > > - Begin using a
Re: [PDB Tech] PeeringDB API throttling status and schedule
Bill Norton has posted the following on Twitter: - https://twitter.com/DrPeering/status/1556692279682682880 - To quote William Shakespeare's King Lear: “That way madness lies”. Here’s todays @PeeringDB rate-limiting update that prompts that quotation: - anonymous queries limited to 20 (was 30)/minute per IP address - authenticated queries limited to 60 (was 80)/minute per user/org Bill, it is hard to know if you are simply being cute/provocative or actually seeing a serious issue. Bill/All, please advise if you think today's change is impacting your code and/or if you think it should be reverted temporarily or otherwise. I am happy to work with you to improve code as able. The purpose of this throttling ramp has been to give folks a heads-up & time to improve poor designs. Interactive queries tend to fall within these guardrails without issue while scraping systems get a heads-up that they are using the resource inefficiently. Next week's planned change is to: - anonymous queries limited to 10/minute per IP address - authenticated queries limited to 40/minute per user/org and then I don't see any more reductions needed unless conditions & feedback warrant otherwise. Thanks, Chris On Mon, 8 Aug 2022, Chris Caputo wrote: > Per the below plan, this change was just implemented: > > --- > On August 8th, adjust and watch for feedback from the community: > > - anonymous queries limited to 20/minute per IP address > - authenticated queries limited to 60/minute per user/org > --- > > Please advise if you run into any issues. > > Thank you, > Chris > > On Tue, 31 May 2022, Chris Caputo wrote: > > After the initial introduction of PeeringDB API throttling, some software > > both open source and private, has been identified and updated. (open > > source details are below; please upgrade and encourage others to do so) > > > > This API throttling is being implemented to control costs by encouraging > > efficient software design while making sure the PeeringDB resource is > > shared well. The use of API keys is being encouraged so that admins can > > reach out to users/orgs with runaway or inefficient software, and because > > it is more secure than user/pass. In addition, org API keys ease employee > > transitions. > > > > Some tips for coders is below. > > > > API throttling in place today: > > > > - repeated anonymous identical requests with a response size above 100k > > are being limited to 1/hour > > - repeated anonymous identical requests of any size are being limited to > > 2/minute > > - anonymous queries are being limited to 400/minute per IP address > > - authenticated queries are being limited to 500/minute per user/org > > > > Here is the current schedule of throttling changes. The schedule may > > adjust as needed as new packages that need update are discovered, so as to > > minimize disruption to the community... > > > > On June 27th, adjust and watch for feedback from the community: > > > > - anonymous queries limited to 300/minute per IP address > > - authenticated queries limited to 400/minute per user/org > > > > On July 11th, adjust and watch for feedback from the community: > > > > - anonymous queries limited to 200/minute per IP address > > - authenticated queries limited to 300/minute per user/org > > > > On July 18th, adjust and watch for feedback from the community: > > > > - anonymous queries limited to 100/minute per IP address > > - authenticated queries limited to 200/minute per user/org > > > > On July 25th, adjust and watch for feedback from the community: > > > > - anonymous queries limited to 50/minute per IP address > > - authenticated queries limited to 100/minute per user/org > > > > On August 1st, adjust and watch for feedback from the community: > > > > - anonymous queries limited to 30/minute per IP address > > - authenticated queries limited to 80/minute per user/org > > > > On August 8th, adjust and watch for feedback from the community: > > > > - anonymous queries limited to 20/minute per IP address > > - authenticated queries limited to 60/minute per user/org > > > > On August 15th, adjust and watch for feedback from the community: > > > > - anonymous queries limited to 10/minute per IP address > > - authenticated queries limited to 40/minute per user/org > > > > Feedback/questions/concerns welcome. > > > > Thanks, > > Chris > > > > Software: > > > > - arouteserver v1.16.0: has many updates including API key support along > > with more efficient querying. > > > > - PeerFinder: API key & efficient querying patches at > > https://github.com/rucarrol/PeerFinder/pull/17 will hopefully be > > integrated. > > > > Coding tips: > > > > - Begin using a PeeringDB API key for all requests: > > > > https://docs.peeringdb.com/howto/api_keys/ > > > > - Begin performing actual caching, such as by using peeringdb-py. > > > >
Re: [PDB Tech] PeeringDB API throttling status and schedule
[resend] Per the below plan, this change was just implemented: --- On August 8th, adjust and watch for feedback from the community: - anonymous queries limited to 20/minute per IP address - authenticated queries limited to 60/minute per user/org --- Please advise if you run into any issues. Thank you, Chris On Tue, 31 May 2022, Chris Caputo wrote: > After the initial introduction of PeeringDB API throttling, some software > both open source and private, has been identified and updated. (open > source details are below; please upgrade and encourage others to do so) > > This API throttling is being implemented to control costs by encouraging > efficient software design while making sure the PeeringDB resource is > shared well. The use of API keys is being encouraged so that admins can > reach out to users/orgs with runaway or inefficient software, and because > it is more secure than user/pass. In addition, org API keys ease employee > transitions. > > Some tips for coders is below. > > API throttling in place today: > > - repeated anonymous identical requests with a response size above 100k > are being limited to 1/hour > - repeated anonymous identical requests of any size are being limited to > 2/minute > - anonymous queries are being limited to 400/minute per IP address > - authenticated queries are being limited to 500/minute per user/org > > Here is the current schedule of throttling changes. The schedule may > adjust as needed as new packages that need update are discovered, so as to > minimize disruption to the community... > > On June 27th, adjust and watch for feedback from the community: > > - anonymous queries limited to 300/minute per IP address > - authenticated queries limited to 400/minute per user/org > > On July 11th, adjust and watch for feedback from the community: > > - anonymous queries limited to 200/minute per IP address > - authenticated queries limited to 300/minute per user/org > > On July 18th, adjust and watch for feedback from the community: > > - anonymous queries limited to 100/minute per IP address > - authenticated queries limited to 200/minute per user/org > > On July 25th, adjust and watch for feedback from the community: > > - anonymous queries limited to 50/minute per IP address > - authenticated queries limited to 100/minute per user/org > > On August 1st, adjust and watch for feedback from the community: > > - anonymous queries limited to 30/minute per IP address > - authenticated queries limited to 80/minute per user/org > > On August 8th, adjust and watch for feedback from the community: > > - anonymous queries limited to 20/minute per IP address > - authenticated queries limited to 60/minute per user/org > > On August 15th, adjust and watch for feedback from the community: > > - anonymous queries limited to 10/minute per IP address > - authenticated queries limited to 40/minute per user/org > > Feedback/questions/concerns welcome. > > Thanks, > Chris > > Software: > > - arouteserver v1.16.0: has many updates including API key support along > with more efficient querying. > > - PeerFinder: API key & efficient querying patches at > https://github.com/rucarrol/PeerFinder/pull/17 will hopefully be > integrated. > > Coding tips: > > - Begin using a PeeringDB API key for all requests: > > https://docs.peeringdb.com/howto/api_keys/ > > - Begin performing actual caching, such as by using peeringdb-py. > > http://peeringdb.github.io/peeringdb-py/ > > - If unable to use a caching agent such as peeringdb-py: > >- Use an API key. > >- Set a User-Agent: header. > >- Use bulk queries (asn__in=$list_of_ASN_separated_by_comma) by > querying 30 to 150 ASNs at a time (tune as appropriate). > >- Add a delay in between queries that is randomly between 2 and 2.5 > seconds, to reduce thundering herd. ___ Pdb-tech mailing list Pdb-tech@lists.peeringdb.com https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
Re: [PDB Tech] PeeringDB API throttling status and schedule (fwd)
Per the below plan, this change was just implemented: --- On August 8th, adjust and watch for feedback from the community: - anonymous queries limited to 20/minute per IP address - authenticated queries limited to 60/minute per user/org --- Please advise if you run into any issues. Thank you, Chris On Tue, 31 May 2022, Chris Caputo wrote: > After the initial introduction of PeeringDB API throttling, some software > both open source and private, has been identified and updated. (open > source details are below; please upgrade and encourage others to do so) > > This API throttling is being implemented to control costs by encouraging > efficient software design while making sure the PeeringDB resource is > shared well. The use of API keys is being encouraged so that admins can > reach out to users/orgs with runaway or inefficient software, and because > it is more secure than user/pass. In addition, org API keys ease employee > transitions. > > Some tips for coders is below. > > API throttling in place today: > > - repeated anonymous identical requests with a response size above 100k > are being limited to 1/hour > - repeated anonymous identical requests of any size are being limited to > 2/minute > - anonymous queries are being limited to 400/minute per IP address > - authenticated queries are being limited to 500/minute per user/org > > Here is the current schedule of throttling changes. The schedule may > adjust as needed as new packages that need update are discovered, so as to > minimize disruption to the community... > > On June 27th, adjust and watch for feedback from the community: > > - anonymous queries limited to 300/minute per IP address > - authenticated queries limited to 400/minute per user/org > > On July 11th, adjust and watch for feedback from the community: > > - anonymous queries limited to 200/minute per IP address > - authenticated queries limited to 300/minute per user/org > > On July 18th, adjust and watch for feedback from the community: > > - anonymous queries limited to 100/minute per IP address > - authenticated queries limited to 200/minute per user/org > > On July 25th, adjust and watch for feedback from the community: > > - anonymous queries limited to 50/minute per IP address > - authenticated queries limited to 100/minute per user/org > > On August 1st, adjust and watch for feedback from the community: > > - anonymous queries limited to 30/minute per IP address > - authenticated queries limited to 80/minute per user/org > > On August 8th, adjust and watch for feedback from the community: > > - anonymous queries limited to 20/minute per IP address > - authenticated queries limited to 60/minute per user/org > > On August 15th, adjust and watch for feedback from the community: > > - anonymous queries limited to 10/minute per IP address > - authenticated queries limited to 40/minute per user/org > > Feedback/questions/concerns welcome. > > Thanks, > Chris > > Software: > > - arouteserver v1.16.0: has many updates including API key support along > with more efficient querying. > > - PeerFinder: API key & efficient querying patches at > https://github.com/rucarrol/PeerFinder/pull/17 will hopefully be > integrated. > > Coding tips: > > - Begin using a PeeringDB API key for all requests: > > https://docs.peeringdb.com/howto/api_keys/ > > - Begin performing actual caching, such as by using peeringdb-py. > > http://peeringdb.github.io/peeringdb-py/ > > - If unable to use a caching agent such as peeringdb-py: > >- Use an API key. > >- Set a User-Agent: header. > >- Use bulk queries (asn__in=$list_of_ASN_separated_by_comma) by > querying 30 to 150 ASNs at a time (tune as appropriate). > >- Add a delay in between queries that is randomly between 2 and 2.5 > seconds, to reduce thundering herd. ___ Pdb-tech mailing list Pdb-tech@lists.peeringdb.com https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
Re: [PDB Tech] PeeringDB API throttling status and schedule
Per the below plan, this change was just implemented: --- On August 1st, adjust and watch for feedback from the community: - anonymous queries limited to 30/minute per IP address - authenticated queries limited to 80/minute per user/org --- Please advise if you run into any issues. Thank you, Chris On Tue, 31 May 2022, Chris Caputo wrote: > After the initial introduction of PeeringDB API throttling, some software > both open source and private, has been identified and updated. (open > source details are below; please upgrade and encourage others to do so) > > This API throttling is being implemented to control costs by encouraging > efficient software design while making sure the PeeringDB resource is > shared well. The use of API keys is being encouraged so that admins can > reach out to users/orgs with runaway or inefficient software, and because > it is more secure than user/pass. In addition, org API keys ease employee > transitions. > > Some tips for coders is below. > > API throttling in place today: > > - repeated anonymous identical requests with a response size above 100k > are being limited to 1/hour > - repeated anonymous identical requests of any size are being limited to > 2/minute > - anonymous queries are being limited to 400/minute per IP address > - authenticated queries are being limited to 500/minute per user/org > > Here is the current schedule of throttling changes. The schedule may > adjust as needed as new packages that need update are discovered, so as to > minimize disruption to the community... > > On June 27th, adjust and watch for feedback from the community: > > - anonymous queries limited to 300/minute per IP address > - authenticated queries limited to 400/minute per user/org > > On July 11th, adjust and watch for feedback from the community: > > - anonymous queries limited to 200/minute per IP address > - authenticated queries limited to 300/minute per user/org > > On July 18th, adjust and watch for feedback from the community: > > - anonymous queries limited to 100/minute per IP address > - authenticated queries limited to 200/minute per user/org > > On July 25th, adjust and watch for feedback from the community: > > - anonymous queries limited to 50/minute per IP address > - authenticated queries limited to 100/minute per user/org > > On August 1st, adjust and watch for feedback from the community: > > - anonymous queries limited to 30/minute per IP address > - authenticated queries limited to 80/minute per user/org > > On August 8th, adjust and watch for feedback from the community: > > - anonymous queries limited to 20/minute per IP address > - authenticated queries limited to 60/minute per user/org > > On August 15th, adjust and watch for feedback from the community: > > - anonymous queries limited to 10/minute per IP address > - authenticated queries limited to 40/minute per user/org > > Feedback/questions/concerns welcome. > > Thanks, > Chris > > Software: > > - arouteserver v1.16.0: has many updates including API key support along > with more efficient querying. > > - PeerFinder: API key & efficient querying patches at > https://github.com/rucarrol/PeerFinder/pull/17 will hopefully be > integrated. > > Coding tips: > > - Begin using a PeeringDB API key for all requests: > > https://docs.peeringdb.com/howto/api_keys/ > > - Begin performing actual caching, such as by using peeringdb-py. > > http://peeringdb.github.io/peeringdb-py/ > > - If unable to use a caching agent such as peeringdb-py: > >- Use an API key. > >- Set a User-Agent: header. > >- Use bulk queries (asn__in=$list_of_ASN_separated_by_comma) by > querying 30 to 150 ASNs at a time (tune as appropriate). > >- Add a delay in between queries that is randomly between 2 and 2.5 > seconds, to reduce thundering herd. ___ Pdb-tech mailing list Pdb-tech@lists.peeringdb.com https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
Re: [PDB Tech] PeeringDB API throttling status and schedule
Per the below plan, this change was just implemented: --- On July 25th, adjust and watch for feedback from the community: - anonymous queries limited to 50/minute per IP address - authenticated queries limited to 100/minute per user/org --- Please advise if you run into any issues. Thank you, Chris On Tue, 31 May 2022, Chris Caputo wrote: > After the initial introduction of PeeringDB API throttling, some software > both open source and private, has been identified and updated. (open > source details are below; please upgrade and encourage others to do so) > > This API throttling is being implemented to control costs by encouraging > efficient software design while making sure the PeeringDB resource is > shared well. The use of API keys is being encouraged so that admins can > reach out to users/orgs with runaway or inefficient software, and because > it is more secure than user/pass. In addition, org API keys ease employee > transitions. > > Some tips for coders is below. > > API throttling in place today: > > - repeated anonymous identical requests with a response size above 100k > are being limited to 1/hour > - repeated anonymous identical requests of any size are being limited to > 2/minute > - anonymous queries are being limited to 400/minute per IP address > - authenticated queries are being limited to 500/minute per user/org > > Here is the current schedule of throttling changes. The schedule may > adjust as needed as new packages that need update are discovered, so as to > minimize disruption to the community... > > On June 27th, adjust and watch for feedback from the community: > > - anonymous queries limited to 300/minute per IP address > - authenticated queries limited to 400/minute per user/org > > On July 11th, adjust and watch for feedback from the community: > > - anonymous queries limited to 200/minute per IP address > - authenticated queries limited to 300/minute per user/org > > On July 18th, adjust and watch for feedback from the community: > > - anonymous queries limited to 100/minute per IP address > - authenticated queries limited to 200/minute per user/org > > On July 25th, adjust and watch for feedback from the community: > > - anonymous queries limited to 50/minute per IP address > - authenticated queries limited to 100/minute per user/org > > On August 1st, adjust and watch for feedback from the community: > > - anonymous queries limited to 30/minute per IP address > - authenticated queries limited to 80/minute per user/org > > On August 8th, adjust and watch for feedback from the community: > > - anonymous queries limited to 20/minute per IP address > - authenticated queries limited to 60/minute per user/org > > On August 15th, adjust and watch for feedback from the community: > > - anonymous queries limited to 10/minute per IP address > - authenticated queries limited to 40/minute per user/org > > Feedback/questions/concerns welcome. > > Thanks, > Chris > > Software: > > - arouteserver v1.16.0: has many updates including API key support along > with more efficient querying. > > - PeerFinder: API key & efficient querying patches at > https://github.com/rucarrol/PeerFinder/pull/17 will hopefully be > integrated. > > Coding tips: > > - Begin using a PeeringDB API key for all requests: > > https://docs.peeringdb.com/howto/api_keys/ > > - Begin performing actual caching, such as by using peeringdb-py. > > http://peeringdb.github.io/peeringdb-py/ > > - If unable to use a caching agent such as peeringdb-py: > >- Use an API key. > >- Set a User-Agent: header. > >- Use bulk queries (asn__in=$list_of_ASN_separated_by_comma) by > querying 30 to 150 ASNs at a time (tune as appropriate). > >- Add a delay in between queries that is randomly between 2 and 2.5 > seconds, to reduce thundering herd. ___ Pdb-tech mailing list Pdb-tech@lists.peeringdb.com https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
Re: [PDB Tech] PeeringDB API throttling status and schedule
Per the below plan, this change was just implemented: --- On July 18th, adjust and watch for feedback from the community: - anonymous queries limited to 100/minute per IP address - authenticated queries limited to 200/minute per user/org --- Please advise if you run into any issues. Thank you, Chris On Tue, 31 May 2022, Chris Caputo wrote: > After the initial introduction of PeeringDB API throttling, some software > both open source and private, has been identified and updated. (open > source details are below; please upgrade and encourage others to do so) > > This API throttling is being implemented to control costs by encouraging > efficient software design while making sure the PeeringDB resource is > shared well. The use of API keys is being encouraged so that admins can > reach out to users/orgs with runaway or inefficient software, and because > it is more secure than user/pass. In addition, org API keys ease employee > transitions. > > Some tips for coders is below. > > API throttling in place today: > > - repeated anonymous identical requests with a response size above 100k > are being limited to 1/hour > - repeated anonymous identical requests of any size are being limited to > 2/minute > - anonymous queries are being limited to 400/minute per IP address > - authenticated queries are being limited to 500/minute per user/org > > Here is the current schedule of throttling changes. The schedule may > adjust as needed as new packages that need update are discovered, so as to > minimize disruption to the community... > > On June 27th, adjust and watch for feedback from the community: > > - anonymous queries limited to 300/minute per IP address > - authenticated queries limited to 400/minute per user/org > > On July 11th, adjust and watch for feedback from the community: > > - anonymous queries limited to 200/minute per IP address > - authenticated queries limited to 300/minute per user/org > > On July 18th, adjust and watch for feedback from the community: > > - anonymous queries limited to 100/minute per IP address > - authenticated queries limited to 200/minute per user/org > > On July 25th, adjust and watch for feedback from the community: > > - anonymous queries limited to 50/minute per IP address > - authenticated queries limited to 100/minute per user/org > > On August 1st, adjust and watch for feedback from the community: > > - anonymous queries limited to 30/minute per IP address > - authenticated queries limited to 80/minute per user/org > > On August 8th, adjust and watch for feedback from the community: > > - anonymous queries limited to 20/minute per IP address > - authenticated queries limited to 60/minute per user/org > > On August 15th, adjust and watch for feedback from the community: > > - anonymous queries limited to 10/minute per IP address > - authenticated queries limited to 40/minute per user/org > > Feedback/questions/concerns welcome. > > Thanks, > Chris > > Software: > > - arouteserver v1.16.0: has many updates including API key support along > with more efficient querying. > > - PeerFinder: API key & efficient querying patches at > https://github.com/rucarrol/PeerFinder/pull/17 will hopefully be > integrated. > > Coding tips: > > - Begin using a PeeringDB API key for all requests: > > https://docs.peeringdb.com/howto/api_keys/ > > - Begin performing actual caching, such as by using peeringdb-py. > > http://peeringdb.github.io/peeringdb-py/ > > - If unable to use a caching agent such as peeringdb-py: > >- Use an API key. > >- Set a User-Agent: header. > >- Use bulk queries (asn__in=$list_of_ASN_separated_by_comma) by > querying 30 to 150 ASNs at a time (tune as appropriate). > >- Add a delay in between queries that is randomly between 2 and 2.5 > seconds, to reduce thundering herd. ___ Pdb-tech mailing list Pdb-tech@lists.peeringdb.com https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
Re: [PDB Tech] PeeringDB API throttling status and schedule
Per the below plan, this change was just implemented: --- On July 11th, adjust and watch for feedback from the community: - anonymous queries limited to 200/minute per IP address - authenticated queries limited to 300/minute per user/org --- Please advise if you run into any issues. Thank you, Chris On Tue, 31 May 2022, Chris Caputo wrote: > After the initial introduction of PeeringDB API throttling, some software > both open source and private, has been identified and updated. (open > source details are below; please upgrade and encourage others to do so) > > This API throttling is being implemented to control costs by encouraging > efficient software design while making sure the PeeringDB resource is > shared well. The use of API keys is being encouraged so that admins can > reach out to users/orgs with runaway or inefficient software, and because > it is more secure than user/pass. In addition, org API keys ease employee > transitions. > > Some tips for coders is below. > > API throttling in place today: > > - repeated anonymous identical requests with a response size above 100k > are being limited to 1/hour > - repeated anonymous identical requests of any size are being limited to > 2/minute > - anonymous queries are being limited to 400/minute per IP address > - authenticated queries are being limited to 500/minute per user/org > > Here is the current schedule of throttling changes. The schedule may > adjust as needed as new packages that need update are discovered, so as to > minimize disruption to the community... > > On June 27th, adjust and watch for feedback from the community: > > - anonymous queries limited to 300/minute per IP address > - authenticated queries limited to 400/minute per user/org > > On July 11th, adjust and watch for feedback from the community: > > - anonymous queries limited to 200/minute per IP address > - authenticated queries limited to 300/minute per user/org > > On July 18th, adjust and watch for feedback from the community: > > - anonymous queries limited to 100/minute per IP address > - authenticated queries limited to 200/minute per user/org > > On July 25th, adjust and watch for feedback from the community: > > - anonymous queries limited to 50/minute per IP address > - authenticated queries limited to 100/minute per user/org > > On August 1st, adjust and watch for feedback from the community: > > - anonymous queries limited to 30/minute per IP address > - authenticated queries limited to 80/minute per user/org > > On August 8th, adjust and watch for feedback from the community: > > - anonymous queries limited to 20/minute per IP address > - authenticated queries limited to 60/minute per user/org > > On August 15th, adjust and watch for feedback from the community: > > - anonymous queries limited to 10/minute per IP address > - authenticated queries limited to 40/minute per user/org > > Feedback/questions/concerns welcome. > > Thanks, > Chris > > Software: > > - arouteserver v1.16.0: has many updates including API key support along > with more efficient querying. > > - PeerFinder: API key & efficient querying patches at > https://github.com/rucarrol/PeerFinder/pull/17 will hopefully be > integrated. > > Coding tips: > > - Begin using a PeeringDB API key for all requests: > > https://docs.peeringdb.com/howto/api_keys/ > > - Begin performing actual caching, such as by using peeringdb-py. > > http://peeringdb.github.io/peeringdb-py/ > > - If unable to use a caching agent such as peeringdb-py: > >- Use an API key. > >- Set a User-Agent: header. > >- Use bulk queries (asn__in=$list_of_ASN_separated_by_comma) by > querying 30 to 150 ASNs at a time (tune as appropriate). > >- Add a delay in between queries that is randomly between 2 and 2.5 > seconds, to reduce thundering herd. ___ Pdb-tech mailing list Pdb-tech@lists.peeringdb.com https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
Re: [PDB Tech] PeeringDB API throttling status and schedule
Per the below plan, this change was just implemented: --- On June 27th, adjust and watch for feedback from the community: - anonymous queries limited to 300/minute per IP address - authenticated queries limited to 400/minute per user/org --- Please advise if you run into any issues. Thank you, Chris On Tue, 31 May 2022, Chris Caputo wrote: > After the initial introduction of PeeringDB API throttling, some software > both open source and private, has been identified and updated. (open > source details are below; please upgrade and encourage others to do so) > > This API throttling is being implemented to control costs by encouraging > efficient software design while making sure the PeeringDB resource is > shared well. The use of API keys is being encouraged so that admins can > reach out to users/orgs with runaway or inefficient software, and because > it is more secure than user/pass. In addition, org API keys ease employee > transitions. > > Some tips for coders is below. > > API throttling in place today: > > - repeated anonymous identical requests with a response size above 100k > are being limited to 1/hour > - repeated anonymous identical requests of any size are being limited to > 2/minute > - anonymous queries are being limited to 400/minute per IP address > - authenticated queries are being limited to 500/minute per user/org > > Here is the current schedule of throttling changes. The schedule may > adjust as needed as new packages that need update are discovered, so as to > minimize disruption to the community... > > On June 27th, adjust and watch for feedback from the community: > > - anonymous queries limited to 300/minute per IP address > - authenticated queries limited to 400/minute per user/org > > On July 11th, adjust and watch for feedback from the community: > > - anonymous queries limited to 200/minute per IP address > - authenticated queries limited to 300/minute per user/org > > On July 18th, adjust and watch for feedback from the community: > > - anonymous queries limited to 100/minute per IP address > - authenticated queries limited to 200/minute per user/org > > On July 25th, adjust and watch for feedback from the community: > > - anonymous queries limited to 50/minute per IP address > - authenticated queries limited to 100/minute per user/org > > On August 1st, adjust and watch for feedback from the community: > > - anonymous queries limited to 30/minute per IP address > - authenticated queries limited to 80/minute per user/org > > On August 8th, adjust and watch for feedback from the community: > > - anonymous queries limited to 20/minute per IP address > - authenticated queries limited to 60/minute per user/org > > On August 15th, adjust and watch for feedback from the community: > > - anonymous queries limited to 10/minute per IP address > - authenticated queries limited to 40/minute per user/org > > Feedback/questions/concerns welcome. > > Thanks, > Chris > > Software: > > - arouteserver v1.16.0: has many updates including API key support along > with more efficient querying. > > - PeerFinder: API key & efficient querying patches at > https://github.com/rucarrol/PeerFinder/pull/17 will hopefully be > integrated. > > Coding tips: > > - Begin using a PeeringDB API key for all requests: > > https://docs.peeringdb.com/howto/api_keys/ > > - Begin performing actual caching, such as by using peeringdb-py. > > http://peeringdb.github.io/peeringdb-py/ > > - If unable to use a caching agent such as peeringdb-py: > >- Use an API key. > >- Set a User-Agent: header. > >- Use bulk queries (asn__in=$list_of_ASN_separated_by_comma) by > querying 30 to 150 ASNs at a time (tune as appropriate). > >- Add a delay in between queries that is randomly between 2 and 2.5 > seconds, to reduce thundering herd. ___ Pdb-tech mailing list Pdb-tech@lists.peeringdb.com https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
Re: [PDB Tech] PeeringDB API throttling status and schedule
Hi Jörg, The 40 requests per minute is based on the limited server resources and how long operations take to be handled by the servers. We have situations where the load skyrockets and single requests are delayed by minutes. We can adjust it down the line, but I think an API request every 1.5 seconds is plenty unless client software is inefficiently designed. Clients hoping to provide user-interactive response times based on large amounts of data, should implement local caching, such as via peeringdb-py. Peeringdb-py employs incremental updates which are super fast. The maximum size of query strings is around 12k. It is based on AWS load balancer limitations last I tested. Check out: https://github.com/peeringdb/peeringdb/issues/362#issuecomment-782891886 for details. A quick check of the logs did not show any queries from you with large numbers of ASNs. But I may have had the wrong IP. Feel free to reach out to me directly if you would like me to share the server-side perspective for your queries as you tune things. Thanks! Chris On Thu, 2 Jun 2022, Jörg Kost wrote: > Hi Chris, > > Is there a basis for calculating why there should only be 40 requests for > authorized participants at the end? Also, is the Query_String limited to some > maximum size? > > When I benchmark it, even with the maximum utilization of 150 ASN numbers in > the query list for a large IX like DE-CIX, I see about ten queries with > ASN_LIST, including the IX and NetIX queries. With that, we would have already > exhausted 25% of the volume. > > My general suggestion would be that we leave a bit more headroom for requests > in the same period without a self-throttling penalty. The target value should > conclude at 10% of the queries for the largest IX as a variable; therefore, > in 2022, at least 100 ~ 120 requests per minute shall be allowed. > > I wrote https://github.com/ipcjk/ixgen half a decade ago (god, how time > flies). I patched in the API keys yesterday; ASN_LIST will also be included in > the next release. However, there is another significant advantage; the thing > works with a local cache of the JSON files from PeeringDB. It can be used as a > simple API server directly as a binary with compatible queries. So you can > quickly get rid of 1000+ queries in a few seconds without SQL, other > dependencies, and bugging the original peeringDB-source. > > BR Jörg > > On 31 May 2022, at 21:31, Chris Caputo wrote: > > > After the initial introduction of PeeringDB API throttling, some software > > both open source and private, has been identified and updated. (open > > source details are below; please upgrade and encourage others to do so) > > > > This API throttling is being implemented to control costs by encouraging > > efficient software design while making sure the PeeringDB resource is > > shared well. The use of API keys is being encouraged so that admins can > > reach out to users/orgs with runaway or inefficient software, and because > > it is more secure than user/pass. In addition, org API keys ease employee > > transitions. > > > > Some tips for coders is below. > > > > API throttling in place today: > > > > - repeated anonymous identical requests with a response size above 100k > > are being limited to 1/hour > > - repeated anonymous identical requests of any size are being limited to > > 2/minute > > - anonymous queries are being limited to 400/minute per IP address > > - authenticated queries are being limited to 500/minute per user/org > > > > Here is the current schedule of throttling changes. The schedule may > > adjust as needed as new packages that need update are discovered, so as to > > minimize disruption to the community... > > > > On June 27th, adjust and watch for feedback from the community: > > > > - anonymous queries limited to 300/minute per IP address > > - authenticated queries limited to 400/minute per user/org > > > > On July 11th, adjust and watch for feedback from the community: > > > > - anonymous queries limited to 200/minute per IP address > > - authenticated queries limited to 300/minute per user/org > > > > On July 18th, adjust and watch for feedback from the community: > > > > - anonymous queries limited to 100/minute per IP address > > - authenticated queries limited to 200/minute per user/org > > > > On July 25th, adjust and watch for feedback from the community: > > > > - anonymous queries limited to 50/minute per IP address > > - authenticated queries limited to 100/minute per user/org > > > > On August 1st, adjust and watch for feedback from the community: > > > > - anonymous queries limited to 30/minute per IP address > > - authenticated queries limited to 80/minute per user/org > > > > On August 8th, adjust and watch for feedback from the community: > > > > - anonymous queries limited to 20/minute per IP address > > - authenticated queries limited to 60/minute per user/org > > > > On August 15th, adjust and watch for
Re: [PDB Tech] PeeringDB API throttling status and schedule
Hi Chris, Is there a basis for calculating why there should only be 40 requests for authorized participants at the end? Also, is the Query_String limited to some maximum size? When I benchmark it, even with the maximum utilization of 150 ASN numbers in the query list for a large IX like DE-CIX, I see about ten queries with ASN_LIST, including the IX and NetIX queries. With that, we would have already exhausted 25% of the volume. My general suggestion would be that we leave a bit more headroom for requests in the same period without a self-throttling penalty. The target value should conclude at 10% of the queries for the largest IX as a variable; therefore, in 2022, at least 100 ~ 120 requests per minute shall be allowed. I wrote https://github.com/ipcjk/ixgen half a decade ago (god, how time flies). I patched in the API keys yesterday; ASN_LIST will also be included in the next release. However, there is another significant advantage; the thing works with a local cache of the JSON files from PeeringDB. It can be used as a simple API server directly as a binary with compatible queries. So you can quickly get rid of 1000+ queries in a few seconds without SQL, other dependencies, and bugging the original peeringDB-source. BR Jörg On 31 May 2022, at 21:31, Chris Caputo wrote: After the initial introduction of PeeringDB API throttling, some software both open source and private, has been identified and updated. (open source details are below; please upgrade and encourage others to do so) This API throttling is being implemented to control costs by encouraging efficient software design while making sure the PeeringDB resource is shared well. The use of API keys is being encouraged so that admins can reach out to users/orgs with runaway or inefficient software, and because it is more secure than user/pass. In addition, org API keys ease employee transitions. Some tips for coders is below. API throttling in place today: - repeated anonymous identical requests with a response size above 100k are being limited to 1/hour - repeated anonymous identical requests of any size are being limited to 2/minute - anonymous queries are being limited to 400/minute per IP address - authenticated queries are being limited to 500/minute per user/org Here is the current schedule of throttling changes. The schedule may adjust as needed as new packages that need update are discovered, so as to minimize disruption to the community... On June 27th, adjust and watch for feedback from the community: - anonymous queries limited to 300/minute per IP address - authenticated queries limited to 400/minute per user/org On July 11th, adjust and watch for feedback from the community: - anonymous queries limited to 200/minute per IP address - authenticated queries limited to 300/minute per user/org On July 18th, adjust and watch for feedback from the community: - anonymous queries limited to 100/minute per IP address - authenticated queries limited to 200/minute per user/org On July 25th, adjust and watch for feedback from the community: - anonymous queries limited to 50/minute per IP address - authenticated queries limited to 100/minute per user/org On August 1st, adjust and watch for feedback from the community: - anonymous queries limited to 30/minute per IP address - authenticated queries limited to 80/minute per user/org On August 8th, adjust and watch for feedback from the community: - anonymous queries limited to 20/minute per IP address - authenticated queries limited to 60/minute per user/org On August 15th, adjust and watch for feedback from the community: - anonymous queries limited to 10/minute per IP address - authenticated queries limited to 40/minute per user/org Feedback/questions/concerns welcome. Thanks, Chris Software: - arouteserver v1.16.0: has many updates including API key support along with more efficient querying. - PeerFinder: API key & efficient querying patches at https://github.com/rucarrol/PeerFinder/pull/17 will hopefully be integrated. Coding tips: - Begin using a PeeringDB API key for all requests: https://docs.peeringdb.com/howto/api_keys/ - Begin performing actual caching, such as by using peeringdb-py. http://peeringdb.github.io/peeringdb-py/ - If unable to use a caching agent such as peeringdb-py: - Use an API key. - Set a User-Agent: header. - Use bulk queries (asn__in=$list_of_ASN_separated_by_comma) by querying 30 to 150 ASNs at a time (tune as appropriate). - Add a delay in between queries that is randomly between 2 and 2.5 seconds, to reduce thundering herd. ___ Pdb-tech mailing list Pdb-tech@lists.peeringdb.com https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech ___ Pdb-tech mailing list Pdb-tech@lists.peeringdb.com
[PDB Tech] PeeringDB API throttling status and schedule
After the initial introduction of PeeringDB API throttling, some software both open source and private, has been identified and updated. (open source details are below; please upgrade and encourage others to do so) This API throttling is being implemented to control costs by encouraging efficient software design while making sure the PeeringDB resource is shared well. The use of API keys is being encouraged so that admins can reach out to users/orgs with runaway or inefficient software, and because it is more secure than user/pass. In addition, org API keys ease employee transitions. Some tips for coders is below. API throttling in place today: - repeated anonymous identical requests with a response size above 100k are being limited to 1/hour - repeated anonymous identical requests of any size are being limited to 2/minute - anonymous queries are being limited to 400/minute per IP address - authenticated queries are being limited to 500/minute per user/org Here is the current schedule of throttling changes. The schedule may adjust as needed as new packages that need update are discovered, so as to minimize disruption to the community... On June 27th, adjust and watch for feedback from the community: - anonymous queries limited to 300/minute per IP address - authenticated queries limited to 400/minute per user/org On July 11th, adjust and watch for feedback from the community: - anonymous queries limited to 200/minute per IP address - authenticated queries limited to 300/minute per user/org On July 18th, adjust and watch for feedback from the community: - anonymous queries limited to 100/minute per IP address - authenticated queries limited to 200/minute per user/org On July 25th, adjust and watch for feedback from the community: - anonymous queries limited to 50/minute per IP address - authenticated queries limited to 100/minute per user/org On August 1st, adjust and watch for feedback from the community: - anonymous queries limited to 30/minute per IP address - authenticated queries limited to 80/minute per user/org On August 8th, adjust and watch for feedback from the community: - anonymous queries limited to 20/minute per IP address - authenticated queries limited to 60/minute per user/org On August 15th, adjust and watch for feedback from the community: - anonymous queries limited to 10/minute per IP address - authenticated queries limited to 40/minute per user/org Feedback/questions/concerns welcome. Thanks, Chris Software: - arouteserver v1.16.0: has many updates including API key support along with more efficient querying. - PeerFinder: API key & efficient querying patches at https://github.com/rucarrol/PeerFinder/pull/17 will hopefully be integrated. Coding tips: - Begin using a PeeringDB API key for all requests: https://docs.peeringdb.com/howto/api_keys/ - Begin performing actual caching, such as by using peeringdb-py. http://peeringdb.github.io/peeringdb-py/ - If unable to use a caching agent such as peeringdb-py: - Use an API key. - Set a User-Agent: header. - Use bulk queries (asn__in=$list_of_ASN_separated_by_comma) by querying 30 to 150 ASNs at a time (tune as appropriate). - Add a delay in between queries that is randomly between 2 and 2.5 seconds, to reduce thundering herd. ___ Pdb-tech mailing list Pdb-tech@lists.peeringdb.com https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
