from:"Chris Caputo"

Re: [PDB Tech] PeeringDB API Discrepency

2016-10-04 Thread Chris Caputo

On Tue, 4 Oct 2016, Arnold Nipper wrote:
> On 04.10.2016 18:53, James Bensley wrote:
> > On 13 September 2016 at 11:00, Stefan Pratter  wrote:
> >> Hi,
> >>
> >> Thank you for bringing this to our attention, i have opened a github ticket
> >> for it here:
> >>
> >> https://github.com/peeringdb/peeringdb/issues/68
> >>
> >> Stefan
> > 
> > 
> > Whilst playing the the PDB API some more I have found another "issue".
> > It's not an issue with the API as such but perhaps a feature request;
> > there seems to be no duplication detection.
> > 
> > I was trying to automate some stuff at this IX: 
> > https://www.peeringdb.com/ix/745
> > 
> > If you look there are two peers listed at that exchange with the same
> > IPv4 address (AS 8468 and AS 60688, both with IP 195.66.246.11). This
> > kind of thing should be easily detectable. Any scope for this sort of
> > function to be added, where the AS number and company are different
> > there shouldn't be a matching IP, the IP can only be announced by one
> > AS number at an exchange.
> 
> Woow ... how could this happen? Afaik it is not possible to add an IP
> twice or even more often.
> 
> Thanks for spotting. We will follow up with LINX to sort out the issue.

It is possible.  We just had someone do this at the SIX in the past week.  
They added our route server addresses, which were registered already in 
PeeringDB.

Chris
___
Pdb-tech mailing list
Pdb-tech@lists.peeringdb.com
http://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech

Re: [PDB Tech] PeeringDB API Discrepency

2016-10-04 Thread Chris Caputo

Slightly related: It would be nice if in addition to the Peer Name column, 
the IPv4/IPv6 column and the Speed column could have natural sorts 
available.  (Natural sort by IPv4 addr, rather than IPv6 addr, unless IPv6 
is put into a separate column.)

Chris

On Tue, 4 Oct 2016, Matt Griswold wrote:
> Agreed, not sure how that got in, made 
> https://github.com/peeringdb/peeringdb/issues/71 and will check it out / 
> add better tests.
> 
> 
> On Oct 4, 2016 3:46 PM, "Arnold Nipper" <arnold.nip...@de-cix.net> wrote:
>   On 04.10.2016 22:29, Chris Caputo wrote:
>   > On Tue, 4 Oct 2016, Arnold Nipper wrote:
>   >> On 04.10.2016 18:53, James Bensley wrote:
>   >>> On 13 September 2016 at 11:00, Stefan Pratter <ste...@20c.com> 
> wrote:
>   >>>> Hi,
>   >>>>
>   >>>> Thank you for bringing this to our attention, i have opened a 
> github ticket
>   >>>> for it here:
>   >>>>
>   >>>> https://github.com/peeringdb/peeringdb/issues/68
>   >>>>
>   >>>> Stefan
>   >>>
>   >>>
>   >>> Whilst playing the the PDB API some more I have found another 
> "issue".
>   >>> It's not an issue with the API as such but perhaps a feature 
> request;
>   >>> there seems to be no duplication detection.
>   >>>
>   >>> I was trying to automate some stuff at this IX: 
> https://www.peeringdb.com/ix/745
>   >>>
>   >>> If you look there are two peers listed at that exchange with the 
> same
>   >>> IPv4 address (AS 8468 and AS 60688, both with IP 195.66.246.11). 
> This
>   >>> kind of thing should be easily detectable. Any scope for this sort 
> of
>   >>> function to be added, where the AS number and company are different
>   >>> there shouldn't be a matching IP, the IP can only be announced by 
> one
>   >>> AS number at an exchange.
>   >>
>   >> Woow ... how could this happen? Afaik it is not possible to add an IP
>   >> twice or even more often.
>   >>
>   >> Thanks for spotting. We will follow up with LINX to sort out the 
> issue.
>   >
>   > It is possible.  We just had someone do this at the SIX in the past 
> week.
>   > They added our route server addresses, which were registered already 
> in
>   > PeeringDB.
>   >
> 
>   Definitely a severe bug imho and should be fixed immediately. I'm really
>   astonished that this pops up only now.
> 
> 
>   Arnold
>   --
>   Arnold Nipper
>   Chief Technology Evangelist and Co-Founder
> 
>   DE-CIX Management GmbH | Lindleystrasse 12 | 60314 Frankfurt am Main |
>   Germany | www.de-cix.net | Phone +49 69 1730902 22 |
>   Mobile +49 172 2650958 | Fax +49 69 4056 2716 |
>   arnold.nip...@de-cix.net | Geschaeftsfuehrer Harald A. Summa |
>   Registergericht AG Koeln HRB 51135
> 
> 
>   ___
>   Pdb-tech mailing list
>   Pdb-tech@lists.peeringdb.com
>   http://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
> 
> 
> ___
Pdb-tech mailing list
Pdb-tech@lists.peeringdb.com
http://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech

Re: [PDB Tech] allow empty IP field or not?

2016-12-27 Thread Chris Caputo

Sometimes folks have the IPv6 flag enabled, but they aren't yet doing IPv6 
at a particular IXP.  If they fill out the IPv6 address, we've seen 
participants with automated session configuration based on PeeringDB try 
to ND for a router that isn't yet configured to handle the ND.  This 
results in broadcasts/multicasts on the fabric which our janitor must then 
chase down to get quashed.  Members then tell the janitor that, hey we saw 
it in PeeringDB, and then the janitor has to explain that in the 
particular case they are not yet doing IPv6...

When this happens with IPv4 it is no big deal since the arpsponge handles 
it, but for IPv6 it is inconvenient.

Chris/SIX
___
Pdb-tech mailing list
Pdb-tech@lists.peeringdb.com
http://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech

Re: [PDB Tech] peeringDB Python Module Issues with Python 3?

2020-02-11 Thread Chris Caputo

Same here:

  https://github.com/peeringdb/django-peeringdb/issues/37
  "errors with django 3 so limit to django 2 or fix them #37"

Bill, if your issue is unique after review of:

  https://github.com/peeringdb/django-peeringdb/issues

consider a new issue with details.  The django 3 issue manifests as:

  - django.core.exceptions.SynchronousOnlyOperation: You cannot call this 
from an async context - use a thread or sync_to_async.

Thanks,
Chris

On Tue, 11 Feb 2020, Milton Ngan wrote:
> The only issue I had was with django-peeringdb needing to use Django < 
> 3.x. A whole bunch of things don’t work with the latest version of the 
> framework. I had to put this requirement into my python projects that 
> used the django-peeringdb module and PY3.
> 
> Django<=2.2.9
> 
> 
> On Feb 11, 2020, at 8:38 AM, William Marantz  wrote:
> > Hi All,
> > 
> > Has anyone seen issues with the peeringDB python module when 
> > converting to Python 3.x? I was unable to get the module working 
> > during my python 3 conversion as it seemed to require a localDB and no 
> > longer performed remote calls properly.  The documents seem a bit 
> > light.
> > 
> > This is not a current issue for me, I just wanted to bring it up in 
> > case others were aware of the issue and/or a fix. I rewrote my code to 
> > make API calls instead of using the python module and all is working 
> > perfectly.
> > 
> > Best Regards,
> > 
> >   -Bill
> > ___
> > Pdb-tech mailing list
> > Pdb-tech@lists.peeringdb.com
> > https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
> 
> ___
> Pdb-tech mailing list
> Pdb-tech@lists.peeringdb.com
> https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech___
Pdb-tech mailing list
Pdb-tech@lists.peeringdb.com
https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech

Re: [PDB Tech] Problem doing initial 'peeringdb sync', foreign key constraint fails?

2020-05-15 Thread Chris Caputo

Hi.  Does restarting with a fresh/empty local database help at all?

Also, what does the following show?:

peeringdb --version
django-admin --version
pip freeze

Thanks,
Chris

On Fri, 15 May 2020, Brian Dickson wrote:
> Arnold Nipper said to send to this list.
> 
> (This is kind of urgent for me, my goal is to get a snapshot of the data, so 
> any alternative workaround would be helpful in the
> meantime.)
> 
> I'm in the process of setting up a local copy, and have run into a problem 
> after getting the peeringdb-py stuff set up.
> 
> When I do peeringdb sync (with the correct mysql database UTF8 stuff and 
> everything,), I am currently getting a persistent error
> which aborts the sync with no data in the tables:
> 
> (Apologies in advance for the long error output text.)
> Syncing to https://www.peeringdb.com/api
> Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan poc
> Fetching & updating all: org
> Updates to be processed: 5
> Ignoring object updated after sync began: (org-26015)
> Ignoring object updated after sync began: (org-26052)
> Ignoring object updated after sync began: (org-17918)
> Ignoring object updated after sync began: (org-26053)
> Fetching & updating all: fac
> Updates to be processed: 0
> Fetching & updating all: net
> Updates to be processed: 18800
> Ignoring object updated after sync began: (net-1356)
> Ignoring object updated after sync began: (net-3684)
> Ignoring object updated after sync began: (net-7924)
> Ignoring object updated after sync began: (net-10733)
> Ignoring object updated after sync began: (net-13084)
> Ignoring object updated after sync began: (net-14581)
> Ignoring object updated after sync began: (net-15702)
> Traceback (most recent call last):
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/utils.py",
> line 84, in _execute
>     return self.cursor.execute(sql, params)
>   File
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/backends/mysql/base.py",
>  line
> 71, in execute
>     return self.cursor.execute(query, args)
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py",
>  line 209, in
> execute
>     res = self._query(query)
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/cursors.py",
>  line 315, in
> _query
>     db.query(q)
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/MySQLdb/connections.py",
>  line
> 226, in query
>     _mysql.connection.query(self, query)
> MySQLdb._exceptions.IntegrityError: (1452, 'Cannot add or update a child row: 
> a foreign key constraint fails
> (`peeringdb`.`peeringdb_network`, CONSTRAINT 
> `peeringdb_network_org_id_404d6106_fk_peeringdb_organization_id` FOREIGN KEY
> (`org_id`) REFERENCES `peeringdb_organization` (`id`))')
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/bin/peeringdb", 
> line 8, in 
>     sys.exit(main())
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/cli.py",
>  line 62, in
> main
>     return handler(config=cfg, **vars(options))
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py",
>  line 20,
> in _wrapped
>     r = func(*a, **k)
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py",
>  line 219,
> in handle
>     client.update_all(rs)
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py",
>  line 66,
> in update_all
>     self._atomic_update(lambda: ctx.sync_resource(r, since=since))
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py",
>  line 78,
> in _atomic_update
>     sync_func()
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py",
>  line 66,
> in 
>     self._atomic_update(lambda: ctx.sync_resource(r, since=since))
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_tasks_async.py",
>  line
> 65, in _wrapped
>     return loop.run_until_complete(func(*a, **k))
>   File 
> "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py",
>  line 488, in
> run_until_complete
>     return future.result()
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_tasks_async.py",
>  line
> 41, in _wrapped
>     item = gen.send(r)
>   File 
>

Re: [PDB Tech] Problem doing initial 'peeringdb sync', foreign key constraint fails?

2020-05-15 Thread Chris Caputo

s/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/cli.py",
>  line 62, in main
> 
>     return handler(config=cfg, **vars(options))
> 
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py",
>  line 20, in _wrapped
> 
>     r = func(*a, **k)
> 
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/commands.py",
>  line 219, in handle
> 
>     client.update_all(rs)
> 
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py",
>  line 66, in update_all
> 
>     self._atomic_update(lambda: ctx.sync_resource(r, since=since))
> 
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py",
>  line 78, in _atomic_update
> 
>     sync_func()
> 
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py",
>  line 66, in 
> 
>     self._atomic_update(lambda: ctx.sync_resource(r, since=since))
> 
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_tasks_async.py",
>  line 65, in _wrapped
> 
>     return loop.run_until_complete(func(*a, **k))
> 
>   File 
> "/opt/local/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py",
>  line 488, in run_until_complete
> 
>     return future.result()
> 
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_tasks_async.py",
>  line 41, in _wrapped
> 
>     item = gen.send(r)
> 
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/peeringdb/_update.py",
>  line 287, in sync_row
> 
>     B.clean(obj)
> 
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django_peeringdb/client_adaptor/backend.py",
>  line 145, in clean
> 
>     obj.full_clean()
> 
>   File 
> "/Users/bdickson1/Documents/projects/site-selection/pdbvenv/lib/python3.6/site-packages/django/db/models/base.py",
>  line 1203, in full_clean
> 
>     raise ValidationError(errors)
> 
> django.core.exceptions.ValidationError: {'org': ['organization instance with 
> id 13687 does not exist.']}
> 
>  
> 
>   On Fri, May 15, 2020 at 10:55 AM Chris Caputo  wrote:
>   Hi.  Does restarting with a fresh/empty local database help at all?
> 
>   Also, what does the following show?:
> 
>   peeringdb --version
>   django-admin --version
>   pip freeze
> 
>   Thanks,
>   Chris
> 
>   On Fri, 15 May 2020, Brian Dickson wrote:
>   > Arnold Nipper said to send to this list.
>   >
>   > (This is kind of urgent for me, my goal is to get a snapshot of the 
> data, so any alternative workaround would be helpful in the
>   > meantime.)
>   >
>   > I'm in the process of setting up a local copy, and have run into a 
> problem after getting the peeringdb-py stuff set up.
>   >
>   > When I do peeringdb sync (with the correct mysql database UTF8 stuff 
> and everything,), I am currently getting a persistent error
>   > which aborts the sync with no data in the tables:
>   >
>   > (Apologies in advance for the long error output text.)
>   > Syncing to https://www.peeringdb.com/api
>   > Updating resources: org fac net ix ixfac ixlan ixpfx netfac netixlan 
> poc
>   > Fetching & updating all: org
>   > Updates to be processed: 5
>   > Ignoring object updated after sync began: (org-26015)
>   > Ignoring object updated after sync began: (org-26052)
>   > Ignoring object updated after sync began: (org-17918)
>   > Ignoring object updated after sync began: (org-26053)
>   > Fetching & updating all: fac
>   > Updates to be processed: 0
>   > Fetching & updating all: net
>   > Updates to be processed: 18800
>   > Ignoring object updated after sync began: (net-1356)
>   > Ignoring object updated after sync began: (net-3684)
>   > Ignoring object updated after sync began: (net-7924)
>   > Ignoring object updated after sync began: (net-10733)
>   > Ignoring object updated after sync began: (net-13084)
>   > Ignoring object updated after sync began: (net-14581)
>   &

[PDB Tech] PeeringDB development container

2020-12-08 Thread Chris Caputo

For people wanting to contribute code to PeeringDB, I think this week is a 
big deal.

In case you missed the social media, check out:

  https://docs.peeringdb.com/blog/contributing_code/

  https://github.com/peeringdb/peeringdb/blob/master/docs/container.md

That later link contains step by step instructions for setting up a dev 
environment with the basic tools of GitHub/git and Docker.

If anyone has any trouble setting things up, or has questions, feel free 
to send them to this list.

Chris
(in my user & volunteer role)
___
Pdb-tech mailing list
Pdb-tech@lists.peeringdb.com
https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech

Re: [PDB Tech] new rate limiting mechanism is too strict

2022-05-17 Thread Chris Caputo

All,

I am behind the throttling rollout in the last 24 hours, and have worked 
with Theo to loosen things up for now. I've also reached out to pierky re 
changes requested for arouteserver and will endeavor to delay resumption 
of the same throttling until after arouteserver users have had reasonable 
time to upgrade.

Highlights for all client developers:

 - Implement support for PeeringDB API keys:

 https://docs.peeringdb.com/howto/api_keys/

   The idea being that we will throttle users using API keys less.

 - Add a delay in between queries that is randomly between 2 and 2.5 
   seconds, to reduce thundering herd. This delay will mean a client 
   queries PeeringDB at most 30 hits per minute, which will be unthrottled 
   if they are authenticated using API keys and not making identical 
   requests.

 - Highly preferred over separate queries: If you don't need non-public 
   contact info from PeeringDB, is that you implement peeringdb-py 
   (peeringdb-py: http://peeringdb.github.io/peeringdb-py/) client-side 
   caching. Doing so enables you to locally query the heck out of a local 
   sqlite (or whatever) database. The start time of a peeringdb-py run 
   should be randomized per the docs 
   (http://peeringdb.github.io/peeringdb-py/cli/). At the SeattleIX we use 
   peeringdb-py and here is what the once per hour update looks like for 
   all of PeeringDB:

[17/May/2022:15:40:09 +] "GET /api/org?since=1652794724=0 HTTP/1.1" 
200 392 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.423
[17/May/2022:15:40:10 +] "GET /api/fac?since=1652773361=0 HTTP/1.1" 
200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.409
[17/May/2022:15:40:10 +] "GET /api/net?since=1652796557=0 HTTP/1.1" 
200 1695 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.426
[17/May/2022:15:40:11 +] "GET /api/ix?since=1652397370=0 HTTP/1.1" 
200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.397
[17/May/2022:15:40:11 +] "GET /api/ixfac?since=1652763759=0 HTTP/1.1" 
200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.414
[17/May/2022:15:40:12 +] "GET /api/ixlan?since=1652781160=0 HTTP/1.1" 
200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.399
[17/May/2022:15:40:12 +] "GET /api/ixpfx?since=1652429334=0 HTTP/1.1" 
200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.408
[17/May/2022:15:40:13 +] "GET /api/netfac?since=1652790428=0 
HTTP/1.1" 200 318 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.553
[17/May/2022:15:40:14 +] "GET /api/netixlan?since=1652796556=0 
HTTP/1.1" 200 399 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.590
[17/May/2022:15:40:14 +] "GET /api/poc?since=1652785835=0 HTTP/1.1" 
200 24 "-" "PeeringDB/1.2.1.1 django_peeringdb/2.13.0" 0.640

   It is fast because, as I understand it, django serializes PeeringDB 
   changes, and the timestamp (since last update) results in only the 
   changes being delivered.

Finally: My apology to those disrupted by this. Please feel free to reach 
out to me with any questions or concerns.

Thanks,
Chris
___
Pdb-tech mailing list
Pdb-tech@lists.peeringdb.com
https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech

Re: [PDB Tech] new rate limiting mechanism is too strict

2022-05-17 Thread Chris Caputo

On Tue, 17 May 2022, Arnold Nipper wrote:
> On 17.05.2022 18:54, Chris Caputo wrote:
> > Highlights for all client developers:
> 
> I would add: instead of querying PDB for each ASN one by one, use the
> asn__in=$list_of_ASN_separated_by_commata feature.

Ooh 200 IQ maneuver there. Genius!

Thanks,
Chris
___
Pdb-tech mailing list
Pdb-tech@lists.peeringdb.com
https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech

[PDB Tech] PeeringDB API throttling status and schedule

2022-05-31 Thread Chris Caputo

After the initial introduction of PeeringDB API throttling, some software 
both open source and private, has been identified and updated. (open 
source details are below; please upgrade and encourage others to do so)

This API throttling is being implemented to control costs by encouraging 
efficient software design while making sure the PeeringDB resource is 
shared well. The use of API keys is being encouraged so that admins can 
reach out to users/orgs with runaway or inefficient software, and because 
it is more secure than user/pass. In addition, org API keys ease employee 
transitions.

Some tips for coders is below.

API throttling in place today:

  - repeated anonymous identical requests with a response size above 100k 
are being limited to 1/hour
  - repeated anonymous identical requests of any size are being limited to 
2/minute
  - anonymous queries are being limited to 400/minute per IP address
  - authenticated queries are being limited to 500/minute per user/org

Here is the current schedule of throttling changes. The schedule may 
adjust as needed as new packages that need update are discovered, so as to 
minimize disruption to the community...

On June 27th, adjust and watch for feedback from the community:

  - anonymous queries limited to 300/minute per IP address
  - authenticated queries limited to 400/minute per user/org

On July 11th, adjust and watch for feedback from the community:

  - anonymous queries limited to 200/minute per IP address
  - authenticated queries limited to 300/minute per user/org

On July 18th, adjust and watch for feedback from the community:

  - anonymous queries limited to 100/minute per IP address
  - authenticated queries limited to 200/minute per user/org

On July 25th, adjust and watch for feedback from the community:

  - anonymous queries limited to 50/minute per IP address
  - authenticated queries limited to 100/minute per user/org

On August 1st, adjust and watch for feedback from the community:

  - anonymous queries limited to 30/minute per IP address
  - authenticated queries limited to 80/minute per user/org

On August 8th, adjust and watch for feedback from the community:

  - anonymous queries limited to 20/minute per IP address
  - authenticated queries limited to 60/minute per user/org

On August 15th, adjust and watch for feedback from the community:

  - anonymous queries limited to 10/minute per IP address
  - authenticated queries limited to 40/minute per user/org

Feedback/questions/concerns welcome.

Thanks,
Chris

Software:

- arouteserver v1.16.0: has many updates including API key support along 
  with more efficient querying.

- PeerFinder: API key & efficient querying patches at 
  https://github.com/rucarrol/PeerFinder/pull/17 will hopefully be 
  integrated.

Coding tips:

- Begin using a PeeringDB API key for all requests:

https://docs.peeringdb.com/howto/api_keys/

- Begin performing actual caching, such as by using peeringdb-py.

http://peeringdb.github.io/peeringdb-py/

- If unable to use a caching agent such as peeringdb-py:

   - Use an API key.

   - Set a User-Agent: header.

   - Use bulk queries (asn__in=$list_of_ASN_separated_by_comma) by 
 querying 30 to 150 ASNs at a time (tune as appropriate).

   - Add a delay in between queries that is randomly between 2 and 2.5 
 seconds, to reduce thundering herd.
___
Pdb-tech mailing list
Pdb-tech@lists.peeringdb.com
https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech

[PDB Tech] PeerFinder support for API keys

2022-05-22 Thread Chris Caputo

Hi all,

For those using PeerFinder, I have submitted a PR to add support for API 
keys:

  https://github.com/rucarrol/PeerFinder/pull/17

Feel free to pull from my fork if you need support immediately:

  https://github.com/ccaputo/PeerFinder

Happy to answer questions about this.

Thanks,
Chris
___
Pdb-tech mailing list
Pdb-tech@lists.peeringdb.com
https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech

Re: [PDB Tech] PeeringDB API throttling status and schedule

2022-06-02 Thread Chris Caputo

Hi Jörg,

The 40 requests per minute is based on the limited server resources and 
how long operations take to be handled by the servers. We have situations 
where the load skyrockets and single requests are delayed by minutes. We 
can adjust it down the line, but I think an API request every 1.5 seconds 
is plenty unless client software is inefficiently designed. Clients hoping 
to provide user-interactive response times based on large amounts of data, 
should implement local caching, such as via peeringdb-py. Peeringdb-py 
employs incremental updates which are super fast.

The maximum size of query strings is around 12k. It is based on AWS load 
balancer limitations last I tested. Check out:

  https://github.com/peeringdb/peeringdb/issues/362#issuecomment-782891886

for details.

A quick check of the logs did not show any queries from you with large 
numbers of ASNs. But I may have had the wrong IP. Feel free to reach out 
to me directly if you would like me to share the server-side perspective 
for your queries as you tune things.

Thanks!

Chris

On Thu, 2 Jun 2022, Jörg Kost wrote:
> Hi Chris,
> 
> Is there a basis for calculating why there should only be 40 requests for
> authorized participants at the end? Also, is the Query_String limited to some
> maximum size?
> 
> When I benchmark it, even with the maximum utilization of 150 ASN numbers in
> the query list for a large IX like DE-CIX, I see about ten queries with
> ASN_LIST, including the IX and NetIX queries. With that, we would have already
> exhausted 25% of the volume.
> 
> My general suggestion would be that we leave a bit more headroom for requests
> in the same period without a self-throttling penalty. The target value should
> conclude at 10% of the queries for the largest IX as a variable;  therefore,
> in 2022, at least 100 ~ 120 requests per minute shall be allowed.
> 
> I wrote https://github.com/ipcjk/ixgen half a decade ago (god, how time
> flies). I patched in the API keys yesterday; ASN_LIST will also be included in
> the next release. However, there is another significant advantage; the thing
> works with a local cache of the JSON files from PeeringDB. It can be used as a
> simple API server directly as a binary with compatible queries. So you can
> quickly get rid of 1000+ queries in a few seconds without SQL, other
> dependencies, and bugging the original peeringDB-source.
> 
> BR Jörg
> 
> On 31 May 2022, at 21:31, Chris Caputo wrote:
> 
> > After the initial introduction of PeeringDB API throttling, some software
> > both open source and private, has been identified and updated. (open
> > source details are below; please upgrade and encourage others to do so)
> >
> > This API throttling is being implemented to control costs by encouraging
> > efficient software design while making sure the PeeringDB resource is
> > shared well. The use of API keys is being encouraged so that admins can
> > reach out to users/orgs with runaway or inefficient software, and because
> > it is more secure than user/pass. In addition, org API keys ease employee
> > transitions.
> >
> > Some tips for coders is below.
> >
> > API throttling in place today:
> >
> >   - repeated anonymous identical requests with a response size above 100k
> > are being limited to 1/hour
> >   - repeated anonymous identical requests of any size are being limited to
> > 2/minute
> >   - anonymous queries are being limited to 400/minute per IP address
> >   - authenticated queries are being limited to 500/minute per user/org
> >
> > Here is the current schedule of throttling changes. The schedule may
> > adjust as needed as new packages that need update are discovered, so as to
> > minimize disruption to the community...
> >
> > On June 27th, adjust and watch for feedback from the community:
> >
> >   - anonymous queries limited to 300/minute per IP address
> >   - authenticated queries limited to 400/minute per user/org
> >
> > On July 11th, adjust and watch for feedback from the community:
> >
> >   - anonymous queries limited to 200/minute per IP address
> >   - authenticated queries limited to 300/minute per user/org
> >
> > On July 18th, adjust and watch for feedback from the community:
> >
> >   - anonymous queries limited to 100/minute per IP address
> >   - authenticated queries limited to 200/minute per user/org
> >
> > On July 25th, adjust and watch for feedback from the community:
> >
> >   - anonymous queries limited to 50/minute per IP address
> >   - authenticated queries limited to 100/minute per user/org
> >
> > On August 1st, adjust and watch for feedback from the community:
> &g

Re: [PDB Tech] PeeringDB API throttling status and schedule

2022-06-27 Thread Chris Caputo

Per the below plan, this change was just implemented:

---
On June 27th, adjust and watch for feedback from the community:

  - anonymous queries limited to 300/minute per IP address
  - authenticated queries limited to 400/minute per user/org
---

Please advise if you run into any issues.

Thank you,
Chris

On Tue, 31 May 2022, Chris Caputo wrote:
> After the initial introduction of PeeringDB API throttling, some software 
> both open source and private, has been identified and updated. (open 
> source details are below; please upgrade and encourage others to do so)
> 
> This API throttling is being implemented to control costs by encouraging 
> efficient software design while making sure the PeeringDB resource is 
> shared well. The use of API keys is being encouraged so that admins can 
> reach out to users/orgs with runaway or inefficient software, and because 
> it is more secure than user/pass. In addition, org API keys ease employee 
> transitions.
> 
> Some tips for coders is below.
> 
> API throttling in place today:
> 
>   - repeated anonymous identical requests with a response size above 100k 
> are being limited to 1/hour
>   - repeated anonymous identical requests of any size are being limited to 
> 2/minute
>   - anonymous queries are being limited to 400/minute per IP address
>   - authenticated queries are being limited to 500/minute per user/org
> 
> Here is the current schedule of throttling changes. The schedule may 
> adjust as needed as new packages that need update are discovered, so as to 
> minimize disruption to the community...
> 
> On June 27th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 300/minute per IP address
>   - authenticated queries limited to 400/minute per user/org
> 
> On July 11th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 200/minute per IP address
>   - authenticated queries limited to 300/minute per user/org
> 
> On July 18th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 100/minute per IP address
>   - authenticated queries limited to 200/minute per user/org
> 
> On July 25th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 50/minute per IP address
>   - authenticated queries limited to 100/minute per user/org
> 
> On August 1st, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 30/minute per IP address
>   - authenticated queries limited to 80/minute per user/org
> 
> On August 8th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 20/minute per IP address
>   - authenticated queries limited to 60/minute per user/org
> 
> On August 15th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 10/minute per IP address
>   - authenticated queries limited to 40/minute per user/org
> 
> Feedback/questions/concerns welcome.
> 
> Thanks,
> Chris
> 
> Software:
> 
> - arouteserver v1.16.0: has many updates including API key support along 
>   with more efficient querying.
> 
> - PeerFinder: API key & efficient querying patches at 
>   https://github.com/rucarrol/PeerFinder/pull/17 will hopefully be 
>   integrated.
> 
> Coding tips:
> 
> - Begin using a PeeringDB API key for all requests:
> 
> https://docs.peeringdb.com/howto/api_keys/
> 
> - Begin performing actual caching, such as by using peeringdb-py.
> 
> http://peeringdb.github.io/peeringdb-py/
> 
> - If unable to use a caching agent such as peeringdb-py:
> 
>- Use an API key.
> 
>- Set a User-Agent: header.
> 
>- Use bulk queries (asn__in=$list_of_ASN_separated_by_comma) by 
>  querying 30 to 150 ASNs at a time (tune as appropriate).
> 
>- Add a delay in between queries that is randomly between 2 and 2.5 
>  seconds, to reduce thundering herd.
___
Pdb-tech mailing list
Pdb-tech@lists.peeringdb.com
https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech

Re: [PDB Tech] PeeringDB API throttling status and schedule

2022-07-11 Thread Chris Caputo

Per the below plan, this change was just implemented:

---
On July 11th, adjust and watch for feedback from the community:

  - anonymous queries limited to 200/minute per IP address
  - authenticated queries limited to 300/minute per user/org
---

Please advise if you run into any issues.

Thank you,
Chris

On Tue, 31 May 2022, Chris Caputo wrote:
> After the initial introduction of PeeringDB API throttling, some software 
> both open source and private, has been identified and updated. (open 
> source details are below; please upgrade and encourage others to do so)
> 
> This API throttling is being implemented to control costs by encouraging 
> efficient software design while making sure the PeeringDB resource is 
> shared well. The use of API keys is being encouraged so that admins can 
> reach out to users/orgs with runaway or inefficient software, and because 
> it is more secure than user/pass. In addition, org API keys ease employee 
> transitions.
> 
> Some tips for coders is below.
> 
> API throttling in place today:
> 
>   - repeated anonymous identical requests with a response size above 100k 
> are being limited to 1/hour
>   - repeated anonymous identical requests of any size are being limited to 
> 2/minute
>   - anonymous queries are being limited to 400/minute per IP address
>   - authenticated queries are being limited to 500/minute per user/org
> 
> Here is the current schedule of throttling changes. The schedule may 
> adjust as needed as new packages that need update are discovered, so as to 
> minimize disruption to the community...
> 
> On June 27th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 300/minute per IP address
>   - authenticated queries limited to 400/minute per user/org
> 
> On July 11th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 200/minute per IP address
>   - authenticated queries limited to 300/minute per user/org
> 
> On July 18th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 100/minute per IP address
>   - authenticated queries limited to 200/minute per user/org
> 
> On July 25th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 50/minute per IP address
>   - authenticated queries limited to 100/minute per user/org
> 
> On August 1st, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 30/minute per IP address
>   - authenticated queries limited to 80/minute per user/org
> 
> On August 8th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 20/minute per IP address
>   - authenticated queries limited to 60/minute per user/org
> 
> On August 15th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 10/minute per IP address
>   - authenticated queries limited to 40/minute per user/org
> 
> Feedback/questions/concerns welcome.
> 
> Thanks,
> Chris
> 
> Software:
> 
> - arouteserver v1.16.0: has many updates including API key support along 
>   with more efficient querying.
> 
> - PeerFinder: API key & efficient querying patches at 
>   https://github.com/rucarrol/PeerFinder/pull/17 will hopefully be 
>   integrated.
> 
> Coding tips:
> 
> - Begin using a PeeringDB API key for all requests:
> 
> https://docs.peeringdb.com/howto/api_keys/
> 
> - Begin performing actual caching, such as by using peeringdb-py.
> 
> http://peeringdb.github.io/peeringdb-py/
> 
> - If unable to use a caching agent such as peeringdb-py:
> 
>- Use an API key.
> 
>- Set a User-Agent: header.
> 
>- Use bulk queries (asn__in=$list_of_ASN_separated_by_comma) by 
>  querying 30 to 150 ASNs at a time (tune as appropriate).
> 
>- Add a delay in between queries that is randomly between 2 and 2.5 
>  seconds, to reduce thundering herd.
___
Pdb-tech mailing list
Pdb-tech@lists.peeringdb.com
https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech

Re: [PDB Tech] PeeringDB API throttling status and schedule

2022-07-18 Thread Chris Caputo

Per the below plan, this change was just implemented:

---
On July 18th, adjust and watch for feedback from the community:

  - anonymous queries limited to 100/minute per IP address
  - authenticated queries limited to 200/minute per user/org
---

Please advise if you run into any issues.

Thank you,
Chris

On Tue, 31 May 2022, Chris Caputo wrote:
> After the initial introduction of PeeringDB API throttling, some software 
> both open source and private, has been identified and updated. (open 
> source details are below; please upgrade and encourage others to do so)
> 
> This API throttling is being implemented to control costs by encouraging 
> efficient software design while making sure the PeeringDB resource is 
> shared well. The use of API keys is being encouraged so that admins can 
> reach out to users/orgs with runaway or inefficient software, and because 
> it is more secure than user/pass. In addition, org API keys ease employee 
> transitions.
> 
> Some tips for coders is below.
> 
> API throttling in place today:
> 
>   - repeated anonymous identical requests with a response size above 100k 
> are being limited to 1/hour
>   - repeated anonymous identical requests of any size are being limited to 
> 2/minute
>   - anonymous queries are being limited to 400/minute per IP address
>   - authenticated queries are being limited to 500/minute per user/org
> 
> Here is the current schedule of throttling changes. The schedule may 
> adjust as needed as new packages that need update are discovered, so as to 
> minimize disruption to the community...
> 
> On June 27th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 300/minute per IP address
>   - authenticated queries limited to 400/minute per user/org
> 
> On July 11th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 200/minute per IP address
>   - authenticated queries limited to 300/minute per user/org
> 
> On July 18th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 100/minute per IP address
>   - authenticated queries limited to 200/minute per user/org
> 
> On July 25th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 50/minute per IP address
>   - authenticated queries limited to 100/minute per user/org
> 
> On August 1st, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 30/minute per IP address
>   - authenticated queries limited to 80/minute per user/org
> 
> On August 8th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 20/minute per IP address
>   - authenticated queries limited to 60/minute per user/org
> 
> On August 15th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 10/minute per IP address
>   - authenticated queries limited to 40/minute per user/org
> 
> Feedback/questions/concerns welcome.
> 
> Thanks,
> Chris
> 
> Software:
> 
> - arouteserver v1.16.0: has many updates including API key support along 
>   with more efficient querying.
> 
> - PeerFinder: API key & efficient querying patches at 
>   https://github.com/rucarrol/PeerFinder/pull/17 will hopefully be 
>   integrated.
> 
> Coding tips:
> 
> - Begin using a PeeringDB API key for all requests:
> 
> https://docs.peeringdb.com/howto/api_keys/
> 
> - Begin performing actual caching, such as by using peeringdb-py.
> 
> http://peeringdb.github.io/peeringdb-py/
> 
> - If unable to use a caching agent such as peeringdb-py:
> 
>- Use an API key.
> 
>- Set a User-Agent: header.
> 
>- Use bulk queries (asn__in=$list_of_ASN_separated_by_comma) by 
>  querying 30 to 150 ASNs at a time (tune as appropriate).
> 
>- Add a delay in between queries that is randomly between 2 and 2.5 
>  seconds, to reduce thundering herd.
___
Pdb-tech mailing list
Pdb-tech@lists.peeringdb.com
https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech

Re: [PDB Tech] PeerFinder support for API keys

2022-07-18 Thread Chris Caputo

Hi. Just a quick note to say the below was merged into PeerFinder a few 
days ago.

Chris

On Mon, 23 May 2022, Chris Caputo wrote:
> Hi all,
> 
> For those using PeerFinder, I have submitted a PR to add support for API 
> keys:
> 
>   https://github.com/rucarrol/PeerFinder/pull/17
> 
> Feel free to pull from my fork if you need support immediately:
> 
>   https://github.com/ccaputo/PeerFinder
> 
> Happy to answer questions about this.
> 
> Thanks,
> Chris
___
Pdb-tech mailing list
Pdb-tech@lists.peeringdb.com
https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech

Re: [PDB Tech] PeeringDB API throttling status and schedule

2022-07-25 Thread Chris Caputo

Per the below plan, this change was just implemented:

---
On July 25th, adjust and watch for feedback from the community:

  - anonymous queries limited to 50/minute per IP address
  - authenticated queries limited to 100/minute per user/org
---

Please advise if you run into any issues.

Thank you,
Chris

On Tue, 31 May 2022, Chris Caputo wrote:
> After the initial introduction of PeeringDB API throttling, some software 
> both open source and private, has been identified and updated. (open 
> source details are below; please upgrade and encourage others to do so)
> 
> This API throttling is being implemented to control costs by encouraging 
> efficient software design while making sure the PeeringDB resource is 
> shared well. The use of API keys is being encouraged so that admins can 
> reach out to users/orgs with runaway or inefficient software, and because 
> it is more secure than user/pass. In addition, org API keys ease employee 
> transitions.
> 
> Some tips for coders is below.
> 
> API throttling in place today:
> 
>   - repeated anonymous identical requests with a response size above 100k 
> are being limited to 1/hour
>   - repeated anonymous identical requests of any size are being limited to 
> 2/minute
>   - anonymous queries are being limited to 400/minute per IP address
>   - authenticated queries are being limited to 500/minute per user/org
> 
> Here is the current schedule of throttling changes. The schedule may 
> adjust as needed as new packages that need update are discovered, so as to 
> minimize disruption to the community...
> 
> On June 27th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 300/minute per IP address
>   - authenticated queries limited to 400/minute per user/org
> 
> On July 11th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 200/minute per IP address
>   - authenticated queries limited to 300/minute per user/org
> 
> On July 18th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 100/minute per IP address
>   - authenticated queries limited to 200/minute per user/org
> 
> On July 25th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 50/minute per IP address
>   - authenticated queries limited to 100/minute per user/org
> 
> On August 1st, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 30/minute per IP address
>   - authenticated queries limited to 80/minute per user/org
> 
> On August 8th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 20/minute per IP address
>   - authenticated queries limited to 60/minute per user/org
> 
> On August 15th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 10/minute per IP address
>   - authenticated queries limited to 40/minute per user/org
> 
> Feedback/questions/concerns welcome.
> 
> Thanks,
> Chris
> 
> Software:
> 
> - arouteserver v1.16.0: has many updates including API key support along 
>   with more efficient querying.
> 
> - PeerFinder: API key & efficient querying patches at 
>   https://github.com/rucarrol/PeerFinder/pull/17 will hopefully be 
>   integrated.
> 
> Coding tips:
> 
> - Begin using a PeeringDB API key for all requests:
> 
> https://docs.peeringdb.com/howto/api_keys/
> 
> - Begin performing actual caching, such as by using peeringdb-py.
> 
> http://peeringdb.github.io/peeringdb-py/
> 
> - If unable to use a caching agent such as peeringdb-py:
> 
>- Use an API key.
> 
>- Set a User-Agent: header.
> 
>- Use bulk queries (asn__in=$list_of_ASN_separated_by_comma) by 
>  querying 30 to 150 ASNs at a time (tune as appropriate).
> 
>- Add a delay in between queries that is randomly between 2 and 2.5 
>  seconds, to reduce thundering herd.
___
Pdb-tech mailing list
Pdb-tech@lists.peeringdb.com
https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech

Re: [PDB Tech] PeeringDB API throttling status and schedule

2022-08-01 Thread Chris Caputo

Per the below plan, this change was just implemented:

---
On August 1st, adjust and watch for feedback from the community:

  - anonymous queries limited to 30/minute per IP address
  - authenticated queries limited to 80/minute per user/org
---

Please advise if you run into any issues.

Thank you,
Chris

On Tue, 31 May 2022, Chris Caputo wrote:
> After the initial introduction of PeeringDB API throttling, some software 
> both open source and private, has been identified and updated. (open 
> source details are below; please upgrade and encourage others to do so)
> 
> This API throttling is being implemented to control costs by encouraging 
> efficient software design while making sure the PeeringDB resource is 
> shared well. The use of API keys is being encouraged so that admins can 
> reach out to users/orgs with runaway or inefficient software, and because 
> it is more secure than user/pass. In addition, org API keys ease employee 
> transitions.
> 
> Some tips for coders is below.
> 
> API throttling in place today:
> 
>   - repeated anonymous identical requests with a response size above 100k 
> are being limited to 1/hour
>   - repeated anonymous identical requests of any size are being limited to 
> 2/minute
>   - anonymous queries are being limited to 400/minute per IP address
>   - authenticated queries are being limited to 500/minute per user/org
> 
> Here is the current schedule of throttling changes. The schedule may 
> adjust as needed as new packages that need update are discovered, so as to 
> minimize disruption to the community...
> 
> On June 27th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 300/minute per IP address
>   - authenticated queries limited to 400/minute per user/org
> 
> On July 11th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 200/minute per IP address
>   - authenticated queries limited to 300/minute per user/org
> 
> On July 18th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 100/minute per IP address
>   - authenticated queries limited to 200/minute per user/org
> 
> On July 25th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 50/minute per IP address
>   - authenticated queries limited to 100/minute per user/org
> 
> On August 1st, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 30/minute per IP address
>   - authenticated queries limited to 80/minute per user/org
> 
> On August 8th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 20/minute per IP address
>   - authenticated queries limited to 60/minute per user/org
> 
> On August 15th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 10/minute per IP address
>   - authenticated queries limited to 40/minute per user/org
> 
> Feedback/questions/concerns welcome.
> 
> Thanks,
> Chris
> 
> Software:
> 
> - arouteserver v1.16.0: has many updates including API key support along 
>   with more efficient querying.
> 
> - PeerFinder: API key & efficient querying patches at 
>   https://github.com/rucarrol/PeerFinder/pull/17 will hopefully be 
>   integrated.
> 
> Coding tips:
> 
> - Begin using a PeeringDB API key for all requests:
> 
> https://docs.peeringdb.com/howto/api_keys/
> 
> - Begin performing actual caching, such as by using peeringdb-py.
> 
> http://peeringdb.github.io/peeringdb-py/
> 
> - If unable to use a caching agent such as peeringdb-py:
> 
>- Use an API key.
> 
>- Set a User-Agent: header.
> 
>- Use bulk queries (asn__in=$list_of_ASN_separated_by_comma) by 
>  querying 30 to 150 ASNs at a time (tune as appropriate).
> 
>- Add a delay in between queries that is randomly between 2 and 2.5 
>  seconds, to reduce thundering herd.
___
Pdb-tech mailing list
Pdb-tech@lists.peeringdb.com
https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech

Re: [PDB Tech] PeeringDB API throttling status and schedule

2022-08-18 Thread Chris Caputo

Based on:

  https://github.com/peeringdb/peeringdb/issues/427#issuecomment-1218382411
  
the anonymous query limit has been reverted to 20/min. Current qpm limits:

  - anonymous queries limited to 20/minute per IP address  
  - authenticated queries limited to 40/minute per user/org

Thanks,
Chris

On Mon, 15 Aug 2022, Chris Caputo wrote:
> Per the below plan, this change was just implemented:
> 
> ---
> On August 15th, adjust and watch for feedback from the community:
>   
>   - anonymous queries limited to 10/minute per IP address  
>   - authenticated queries limited to 40/minute per user/org
> ---
> 
> Please advise if you run into any issues.
> 
> Thank you,
> Chris
> 
> On Sun, 14 Aug 2022, Chris Caputo wrote:
> > Sorry - I totally jumped the gun on what UTC day it is! Reverted back to 
> > the August 8th settings:
> >   
> >   - anonymous queries limited to 20/minute per IP address  
> >   - authenticated queries limited to 60/minute per user/org
> > 
> > Chris
> > 
> > On Sun, 14 Aug 2022, Chris Caputo wrote:
> > > Per the below plan, this change was just implemented:
> > > 
> > > ---
> > > On August 15th, adjust and watch for feedback from the community:
> > > 
> > >   - anonymous queries limited to 10/minute per IP address
> > >   - authenticated queries limited to 40/minute per user/org
> > > ---
> > > 
> > > Please advise if you run into any issues.
> > > 
> > > Thank you,
> > > Chris
> > > 
> > > On Tue, 31 May 2022, Chris Caputo wrote:
> > > > After the initial introduction of PeeringDB API throttling, some 
> > > > software 
> > > > both open source and private, has been identified and updated. (open 
> > > > source details are below; please upgrade and encourage others to do so)
> > > > 
> > > > This API throttling is being implemented to control costs by 
> > > > encouraging 
> > > > efficient software design while making sure the PeeringDB resource is 
> > > > shared well. The use of API keys is being encouraged so that admins can 
> > > > reach out to users/orgs with runaway or inefficient software, and 
> > > > because 
> > > > it is more secure than user/pass. In addition, org API keys ease 
> > > > employee 
> > > > transitions.
> > > > 
> > > > Some tips for coders is below.
> > > > 
> > > > API throttling in place today:
> > > > 
> > > >   - repeated anonymous identical requests with a response size above 
> > > > 100k 
> > > > are being limited to 1/hour
> > > >   - repeated anonymous identical requests of any size are being limited 
> > > > to 
> > > > 2/minute
> > > >   - anonymous queries are being limited to 400/minute per IP address
> > > >   - authenticated queries are being limited to 500/minute per user/org
> > > > 
> > > > Here is the current schedule of throttling changes. The schedule may 
> > > > adjust as needed as new packages that need update are discovered, so as 
> > > > to 
> > > > minimize disruption to the community...
> > > > 
> > > > On June 27th, adjust and watch for feedback from the community:
> > > > 
> > > >   - anonymous queries limited to 300/minute per IP address
> > > >   - authenticated queries limited to 400/minute per user/org
> > > > 
> > > > On July 11th, adjust and watch for feedback from the community:
> > > > 
> > > >   - anonymous queries limited to 200/minute per IP address
> > > >   - authenticated queries limited to 300/minute per user/org
> > > > 
> > > > On July 18th, adjust and watch for feedback from the community:
> > > > 
> > > >   - anonymous queries limited to 100/minute per IP address
> > > >   - authenticated queries limited to 200/minute per user/org
> > > > 
> > > > On July 25th, adjust and watch for feedback from the community:
> > > > 
> > > >   - anonymous queries limited to 50/minute per IP address
> > > >   - authenticated queries limited to 100/minute per user/org
> > > > 
> > > > On August 1st, adjust and watch for feedback from the community:
> > > > 
> > > >   - anonymous queries limited to 30/minute per IP address
> > > >   - authenticated queries limited to 80/minute per user/org
> > > > 
&g

[PDB Tech] peeringdb-py example setup

2022-09-19 Thread Chris Caputo

In case useful to others, below is how peeringdb-py is set up & upgraded 
at the SeattleIX.

Questions/improvements welcome.

Chris

---

INSTALL:

mkdir ~/peeringdb-py && cd ~/peeringdb-py

virtualenv --python=python3 pdbvenv

source pdbvenv/bin/activate

pip install --upgrade pip

pip install -U setuptools

pip install git+https://github.com/peeringdb/django-peeringdb.git

pip install git+https://github.com/peeringdb/peeringdb-py.git

pip install "django<3"

Write defaults: 
peeringdb config set -n

# Adjust [HOME] below to be the home directory.
edit ~/.peeringdb/config.yaml
change: name: /home/[HOME]/peeringdb-py/peeringdb.sqlite3
change: api_key: '[Read-only Org/User API Key]'

peeringdb config show | sed 's/  api_key:.*/  api_key: CENSORED/g'  
orm:
  backend: django_peeringdb
  database:
engine: sqlite3
host: ''
name: /home/[HOME]/peeringdb-py/peeringdb.sqlite3
password: ''
port: 0
user: ''
  migrate: true
  secret_key: ''
sync:
  api_key: CENSORED
  only: []
  password: ''
  strip_tz: 1
  timeout: 0
  url: https://www.peeringdb.com/api
  user: ''

peeringdb --version 
peeringdb 1.2.1.1

django-admin --version  
2.2.28

pip freeze | grep django-peeringdb  
django-peeringdb @ 
git+https://github.com/peeringdb/django-peeringdb.git@14eb990d6803e89f38027419349e14e1927b390f

time peeringdb sync 
real14m47.515s
user14m27.077s
sys 0m1.939s

time peeringdb sync 
real0m3.110s
user0m1.074s
sys 0m0.088s

edit ~/crontab
add: # Run hourly with a random 5 minute sleep at start, to reduce thundering 
herd load on PeeringDB servers
add: 00 * * * * sleep $[RANDOM\%300] ; cd /home/[HOME]/peeringdb-py ; touch 
peeringdb.sync.log ; date >> peeringdb.sync.log ; ./pdbvenv/bin/peeringdb sync 
>> peeringdb.sync.log 2>&1

crontab ~/crontab

crontab -l # confirm above setting of crontab

---

UPGRADE:

cd ~/peeringdb-py

tar cvzf pdbvenv.`date +\%Y\%m\%d\%H\%M\%S`.tgz pdbvenv # make a backup

source pdbvenv/bin/activate

pip install --upgrade pip

pip install -U setuptools

pip install git+https://github.com/peeringdb/django-peeringdb.git

pip install --upgrade git+https://github.com/peeringdb/peeringdb-py.git

pip list --outdated --format=freeze | grep -v '^\-e' | cut -d = -f 1  | xargs 
-n1 pip install -U

pip install "django<3"

peeringdb --version 
peeringdb 1.2.1.1

django-admin --version  
2.2.28

pip freeze | grep django-peeringdb  
django-peeringdb @ 
git+https://github.com/peeringdb/django-peeringdb.git@14eb990d6803e89f38027419349e14e1927b390f

time peeringdb sync 
real0m2.809s
user0m0.772s
sys 0m0.045s

---

EXAMPLE USAGE:

cd ~/peeringdb-py

sqlite3 peeringdb.sqlite3 .schema # dump schema

sqlite3 -header peeringdb.sqlite3 "SELECT * FROM peeringdb_network_ixlan WHERE 
asn = 6456;" # show AS6456
id|status|created|updated|version|asn|ipaddr4|ipaddr6|is_rs_peer|notes|speed|ixlan_id|net_id|operational
1534|ok|2010-07-29 00:00:00|2020-10-09 
20:54:19|0|6456|206.81.80.10|2001:504:16::1938|1||2|13|416|1
21681|ok|2015-02-06 00:00:00|2020-10-09 
20:54:20|0|6456|206.81.81.41|2001:504:16::297:0:1938|1||2|13|416|1
28528|ok|2016-04-22 03:19:39|2020-10-09 
20:54:19|0|6456|206.81.82.10|2001:504:16:1::1938|1||2|1285|416|1
28529|ok|2016-04-22 03:20:00|2020-10-09 
20:54:20|0|6456|206.81.83.41|2001:504:16:1:0:297:0:1938|1||2|1285|416|1

---
___
Pdb-tech mailing list
Pdb-tech@lists.peeringdb.com
https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech

Re: [PDB Tech] PeeringDB Account Verification

2022-09-18 Thread Chris Caputo

As I understand things, an account is needed in order to create/use an API 
key.

Noelle, please be aware of:

  https://github.com/peeringdb/peeringdb/issues/380

The apparent intention of that issue is that accounts without an 
organization affiliation will eventually be purged.

You may want to check out peeringdb-py if not already using:

  https://peeringdb.github.io/peeringdb-py/

That enables mirroring of the bulk of the PeeringDB database, for 
high-speed local querying, and lightweight incremental updates.

Chris

On Sun, 18 Sep 2022, Arnold Nipper wrote:
> Hi Matt and Chris,
> 
> afair the use of an API key is recommended. And iirc I'll need an account to
> register an API key. Or am I mistaken?
> 
> 
> Greetings
> Arnold
> 
> On 17.09.2022 20:46, Matt Griswold wrote:
> > The API is public and open, you should not need to create an account to
> > access it. The only thing authentication does is allow access to network
> > contacts.
> > 
> > Is there something specific you're trying to accomplish that's not
> > happening with the unauthenticated API?
> > 
> > Cheers
> > 
> > On Fri, Sep 16, 2022 at 12:09:22PM -0400, Noelle Kenny wrote:
> >> Hello,
> >>
> >> My name is Noelle Kenny with Inflect. I'm hoping to create a verified
> >> account with PeeringDB in order to gain access to the API. I don't have an
> >> organization in PeeringDB to affiliate with, which may be why I'm unable to
> >> be verified. Any insight would be greatly appreciated.
> >>
> >> Thanks,
> >> Noelle
> >>
> >> -- 
> >> *Noelle Kenny *| *Inflect, Inc. *
> >> Product Manager
> >> noe...@inflect.com | *818-645-0325*
> >> *LinkedIn* 
> >> *Discord: *Noelle | FAB#7684
> >> 
___
Pdb-tech mailing list
Pdb-tech@lists.peeringdb.com
https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech

Re: [PDB Tech] PeeringDB API throttling status and schedule

2022-08-14 Thread Chris Caputo

Per the below plan, this change was just implemented:

---
On August 15th, adjust and watch for feedback from the community:
  
  - anonymous queries limited to 10/minute per IP address  
  - authenticated queries limited to 40/minute per user/org
---

Please advise if you run into any issues.

Thank you,
Chris

On Sun, 14 Aug 2022, Chris Caputo wrote:
> Sorry - I totally jumped the gun on what UTC day it is! Reverted back to 
> the August 8th settings:
>   
>   - anonymous queries limited to 20/minute per IP address  
>   - authenticated queries limited to 60/minute per user/org
> 
> Chris
> 
> On Sun, 14 Aug 2022, Chris Caputo wrote:
> > Per the below plan, this change was just implemented:
> > 
> > ---
> > On August 15th, adjust and watch for feedback from the community:
> > 
> >   - anonymous queries limited to 10/minute per IP address
> >   - authenticated queries limited to 40/minute per user/org
> > ---
> > 
> > Please advise if you run into any issues.
> > 
> > Thank you,
> > Chris
> > 
> > On Tue, 31 May 2022, Chris Caputo wrote:
> > > After the initial introduction of PeeringDB API throttling, some software 
> > > both open source and private, has been identified and updated. (open 
> > > source details are below; please upgrade and encourage others to do so)
> > > 
> > > This API throttling is being implemented to control costs by encouraging 
> > > efficient software design while making sure the PeeringDB resource is 
> > > shared well. The use of API keys is being encouraged so that admins can 
> > > reach out to users/orgs with runaway or inefficient software, and because 
> > > it is more secure than user/pass. In addition, org API keys ease employee 
> > > transitions.
> > > 
> > > Some tips for coders is below.
> > > 
> > > API throttling in place today:
> > > 
> > >   - repeated anonymous identical requests with a response size above 100k 
> > > are being limited to 1/hour
> > >   - repeated anonymous identical requests of any size are being limited 
> > > to 
> > > 2/minute
> > >   - anonymous queries are being limited to 400/minute per IP address
> > >   - authenticated queries are being limited to 500/minute per user/org
> > > 
> > > Here is the current schedule of throttling changes. The schedule may 
> > > adjust as needed as new packages that need update are discovered, so as 
> > > to 
> > > minimize disruption to the community...
> > > 
> > > On June 27th, adjust and watch for feedback from the community:
> > > 
> > >   - anonymous queries limited to 300/minute per IP address
> > >   - authenticated queries limited to 400/minute per user/org
> > > 
> > > On July 11th, adjust and watch for feedback from the community:
> > > 
> > >   - anonymous queries limited to 200/minute per IP address
> > >   - authenticated queries limited to 300/minute per user/org
> > > 
> > > On July 18th, adjust and watch for feedback from the community:
> > > 
> > >   - anonymous queries limited to 100/minute per IP address
> > >   - authenticated queries limited to 200/minute per user/org
> > > 
> > > On July 25th, adjust and watch for feedback from the community:
> > > 
> > >   - anonymous queries limited to 50/minute per IP address
> > >   - authenticated queries limited to 100/minute per user/org
> > > 
> > > On August 1st, adjust and watch for feedback from the community:
> > > 
> > >   - anonymous queries limited to 30/minute per IP address
> > >   - authenticated queries limited to 80/minute per user/org
> > > 
> > > On August 8th, adjust and watch for feedback from the community:
> > > 
> > >   - anonymous queries limited to 20/minute per IP address
> > >   - authenticated queries limited to 60/minute per user/org
> > > 
> > > On August 15th, adjust and watch for feedback from the community:
> > > 
> > >   - anonymous queries limited to 10/minute per IP address
> > >   - authenticated queries limited to 40/minute per user/org
> > > 
> > > Feedback/questions/concerns welcome.
> > > 
> > > Thanks,
> > > Chris
> > > 
> > > Software:
> > > 
> > > - arouteserver v1.16.0: has many updates including API key support along 
> > >   with more efficient querying.
> > > 
> > > - PeerFinder: API key & efficient querying patches

Re: [PDB Tech] PeeringDB API throttling status and schedule

2022-08-13 Thread Chris Caputo

Sorry - I totally jumped the gun on what UTC day it is! Reverted back to 
the August 8th settings:
  
  - anonymous queries limited to 20/minute per IP address  
  - authenticated queries limited to 60/minute per user/org

Chris

On Sun, 14 Aug 2022, Chris Caputo wrote:
> Per the below plan, this change was just implemented:
> 
> ---
> On August 15th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 10/minute per IP address
>   - authenticated queries limited to 40/minute per user/org
> ---
> 
> Please advise if you run into any issues.
> 
> Thank you,
> Chris
> 
> On Tue, 31 May 2022, Chris Caputo wrote:
> > After the initial introduction of PeeringDB API throttling, some software 
> > both open source and private, has been identified and updated. (open 
> > source details are below; please upgrade and encourage others to do so)
> > 
> > This API throttling is being implemented to control costs by encouraging 
> > efficient software design while making sure the PeeringDB resource is 
> > shared well. The use of API keys is being encouraged so that admins can 
> > reach out to users/orgs with runaway or inefficient software, and because 
> > it is more secure than user/pass. In addition, org API keys ease employee 
> > transitions.
> > 
> > Some tips for coders is below.
> > 
> > API throttling in place today:
> > 
> >   - repeated anonymous identical requests with a response size above 100k 
> > are being limited to 1/hour
> >   - repeated anonymous identical requests of any size are being limited to 
> > 2/minute
> >   - anonymous queries are being limited to 400/minute per IP address
> >   - authenticated queries are being limited to 500/minute per user/org
> > 
> > Here is the current schedule of throttling changes. The schedule may 
> > adjust as needed as new packages that need update are discovered, so as to 
> > minimize disruption to the community...
> > 
> > On June 27th, adjust and watch for feedback from the community:
> > 
> >   - anonymous queries limited to 300/minute per IP address
> >   - authenticated queries limited to 400/minute per user/org
> > 
> > On July 11th, adjust and watch for feedback from the community:
> > 
> >   - anonymous queries limited to 200/minute per IP address
> >   - authenticated queries limited to 300/minute per user/org
> > 
> > On July 18th, adjust and watch for feedback from the community:
> > 
> >   - anonymous queries limited to 100/minute per IP address
> >   - authenticated queries limited to 200/minute per user/org
> > 
> > On July 25th, adjust and watch for feedback from the community:
> > 
> >   - anonymous queries limited to 50/minute per IP address
> >   - authenticated queries limited to 100/minute per user/org
> > 
> > On August 1st, adjust and watch for feedback from the community:
> > 
> >   - anonymous queries limited to 30/minute per IP address
> >   - authenticated queries limited to 80/minute per user/org
> > 
> > On August 8th, adjust and watch for feedback from the community:
> > 
> >   - anonymous queries limited to 20/minute per IP address
> >   - authenticated queries limited to 60/minute per user/org
> > 
> > On August 15th, adjust and watch for feedback from the community:
> > 
> >   - anonymous queries limited to 10/minute per IP address
> >   - authenticated queries limited to 40/minute per user/org
> > 
> > Feedback/questions/concerns welcome.
> > 
> > Thanks,
> > Chris
> > 
> > Software:
> > 
> > - arouteserver v1.16.0: has many updates including API key support along 
> >   with more efficient querying.
> > 
> > - PeerFinder: API key & efficient querying patches at 
> >   https://github.com/rucarrol/PeerFinder/pull/17 will hopefully be 
> >   integrated.
> > 
> > Coding tips:
> > 
> > - Begin using a PeeringDB API key for all requests:
> > 
> > https://docs.peeringdb.com/howto/api_keys/
> > 
> > - Begin performing actual caching, such as by using peeringdb-py.
> > 
> > http://peeringdb.github.io/peeringdb-py/
> > 
> > - If unable to use a caching agent such as peeringdb-py:
> > 
> >- Use an API key.
> > 
> >- Set a User-Agent: header.
> > 
> >- Use bulk queries (asn__in=$list_of_ASN_separated_by_comma) by 
> >  querying 30 to 150 ASNs at a time (tune as appropriate).
> > 
> >- Add a delay in between queries that is randomly between 2 and 2.5 
> >  seconds, to reduce thundering herd.
> ___
> Pdb-tech mailing list
> Pdb-tech@lists.peeringdb.com
> https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech
> 
___
Pdb-tech mailing list
Pdb-tech@lists.peeringdb.com
https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech

Re: [PDB Tech] PeeringDB API throttling status and schedule

2022-08-13 Thread Chris Caputo

Per the below plan, this change was just implemented:

---
On August 15th, adjust and watch for feedback from the community:

  - anonymous queries limited to 10/minute per IP address
  - authenticated queries limited to 40/minute per user/org
---

Please advise if you run into any issues.

Thank you,
Chris

On Tue, 31 May 2022, Chris Caputo wrote:
> After the initial introduction of PeeringDB API throttling, some software 
> both open source and private, has been identified and updated. (open 
> source details are below; please upgrade and encourage others to do so)
> 
> This API throttling is being implemented to control costs by encouraging 
> efficient software design while making sure the PeeringDB resource is 
> shared well. The use of API keys is being encouraged so that admins can 
> reach out to users/orgs with runaway or inefficient software, and because 
> it is more secure than user/pass. In addition, org API keys ease employee 
> transitions.
> 
> Some tips for coders is below.
> 
> API throttling in place today:
> 
>   - repeated anonymous identical requests with a response size above 100k 
> are being limited to 1/hour
>   - repeated anonymous identical requests of any size are being limited to 
> 2/minute
>   - anonymous queries are being limited to 400/minute per IP address
>   - authenticated queries are being limited to 500/minute per user/org
> 
> Here is the current schedule of throttling changes. The schedule may 
> adjust as needed as new packages that need update are discovered, so as to 
> minimize disruption to the community...
> 
> On June 27th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 300/minute per IP address
>   - authenticated queries limited to 400/minute per user/org
> 
> On July 11th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 200/minute per IP address
>   - authenticated queries limited to 300/minute per user/org
> 
> On July 18th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 100/minute per IP address
>   - authenticated queries limited to 200/minute per user/org
> 
> On July 25th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 50/minute per IP address
>   - authenticated queries limited to 100/minute per user/org
> 
> On August 1st, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 30/minute per IP address
>   - authenticated queries limited to 80/minute per user/org
> 
> On August 8th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 20/minute per IP address
>   - authenticated queries limited to 60/minute per user/org
> 
> On August 15th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 10/minute per IP address
>   - authenticated queries limited to 40/minute per user/org
> 
> Feedback/questions/concerns welcome.
> 
> Thanks,
> Chris
> 
> Software:
> 
> - arouteserver v1.16.0: has many updates including API key support along 
>   with more efficient querying.
> 
> - PeerFinder: API key & efficient querying patches at 
>   https://github.com/rucarrol/PeerFinder/pull/17 will hopefully be 
>   integrated.
> 
> Coding tips:
> 
> - Begin using a PeeringDB API key for all requests:
> 
> https://docs.peeringdb.com/howto/api_keys/
> 
> - Begin performing actual caching, such as by using peeringdb-py.
> 
> http://peeringdb.github.io/peeringdb-py/
> 
> - If unable to use a caching agent such as peeringdb-py:
> 
>- Use an API key.
> 
>- Set a User-Agent: header.
> 
>- Use bulk queries (asn__in=$list_of_ASN_separated_by_comma) by 
>  querying 30 to 150 ASNs at a time (tune as appropriate).
> 
>- Add a delay in between queries that is randomly between 2 and 2.5 
>  seconds, to reduce thundering herd.
___
Pdb-tech mailing list
Pdb-tech@lists.peeringdb.com
https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech

Re: [PDB Tech] PeeringDB API throttling status and schedule (fwd)

2022-08-08 Thread Chris Caputo

Per the below plan, this change was just implemented:

---
On August 8th, adjust and watch for feedback from the community:

  - anonymous queries limited to 20/minute per IP address
  - authenticated queries limited to 60/minute per user/org
---

Please advise if you run into any issues.

Thank you,
Chris

On Tue, 31 May 2022, Chris Caputo wrote:
> After the initial introduction of PeeringDB API throttling, some software 
> both open source and private, has been identified and updated. (open 
> source details are below; please upgrade and encourage others to do so)
> 
> This API throttling is being implemented to control costs by encouraging 
> efficient software design while making sure the PeeringDB resource is 
> shared well. The use of API keys is being encouraged so that admins can 
> reach out to users/orgs with runaway or inefficient software, and because 
> it is more secure than user/pass. In addition, org API keys ease employee 
> transitions.
> 
> Some tips for coders is below.
> 
> API throttling in place today:
> 
>   - repeated anonymous identical requests with a response size above 100k 
> are being limited to 1/hour
>   - repeated anonymous identical requests of any size are being limited to 
> 2/minute
>   - anonymous queries are being limited to 400/minute per IP address
>   - authenticated queries are being limited to 500/minute per user/org
> 
> Here is the current schedule of throttling changes. The schedule may 
> adjust as needed as new packages that need update are discovered, so as to 
> minimize disruption to the community...
> 
> On June 27th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 300/minute per IP address
>   - authenticated queries limited to 400/minute per user/org
> 
> On July 11th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 200/minute per IP address
>   - authenticated queries limited to 300/minute per user/org
> 
> On July 18th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 100/minute per IP address
>   - authenticated queries limited to 200/minute per user/org
> 
> On July 25th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 50/minute per IP address
>   - authenticated queries limited to 100/minute per user/org
> 
> On August 1st, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 30/minute per IP address
>   - authenticated queries limited to 80/minute per user/org
> 
> On August 8th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 20/minute per IP address
>   - authenticated queries limited to 60/minute per user/org
> 
> On August 15th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 10/minute per IP address
>   - authenticated queries limited to 40/minute per user/org
> 
> Feedback/questions/concerns welcome.
> 
> Thanks,
> Chris
> 
> Software:
> 
> - arouteserver v1.16.0: has many updates including API key support along 
>   with more efficient querying.
> 
> - PeerFinder: API key & efficient querying patches at 
>   https://github.com/rucarrol/PeerFinder/pull/17 will hopefully be 
>   integrated.
> 
> Coding tips:
> 
> - Begin using a PeeringDB API key for all requests:
> 
> https://docs.peeringdb.com/howto/api_keys/
> 
> - Begin performing actual caching, such as by using peeringdb-py.
> 
> http://peeringdb.github.io/peeringdb-py/
> 
> - If unable to use a caching agent such as peeringdb-py:
> 
>- Use an API key.
> 
>- Set a User-Agent: header.
> 
>- Use bulk queries (asn__in=$list_of_ASN_separated_by_comma) by 
>  querying 30 to 150 ASNs at a time (tune as appropriate).
> 
>- Add a delay in between queries that is randomly between 2 and 2.5 
>  seconds, to reduce thundering herd.
___
Pdb-tech mailing list
Pdb-tech@lists.peeringdb.com
https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech

Re: [PDB Tech] PeeringDB API throttling status and schedule

2022-08-08 Thread Chris Caputo

Bill Norton has posted the following on Twitter:

  - https://twitter.com/DrPeering/status/1556692279682682880

- To quote William Shakespeare's King Lear: “That way madness lies”.  
  Here’s todays @PeeringDB rate-limiting update that prompts that 
  quotation:
  - anonymous queries limited to 20 (was 30)/minute per IP address
  - authenticated queries limited to 60 (was 80)/minute per user/org

Bill, it is hard to know if you are simply being cute/provocative or 
actually seeing a serious issue.

Bill/All, please advise if you think today's change is impacting your code 
and/or if you think it should be reverted temporarily or otherwise. I am 
happy to work with you to improve code as able. The purpose of this 
throttling ramp has been to give folks a heads-up & time to improve poor 
designs. Interactive queries tend to fall within these guardrails without 
issue while scraping systems get a heads-up that they are using the 
resource inefficiently.

Next week's planned change is to:

  - anonymous queries limited to 10/minute per IP address
  - authenticated queries limited to 40/minute per user/org

and then I don't see any more reductions needed unless conditions & 
feedback warrant otherwise.

Thanks,
Chris

On Mon, 8 Aug 2022, Chris Caputo wrote:
> Per the below plan, this change was just implemented:
> 
> ---
> On August 8th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 20/minute per IP address
>   - authenticated queries limited to 60/minute per user/org
> ---
> 
> Please advise if you run into any issues.
> 
> Thank you,
> Chris
> 
> On Tue, 31 May 2022, Chris Caputo wrote:
> > After the initial introduction of PeeringDB API throttling, some software 
> > both open source and private, has been identified and updated. (open 
> > source details are below; please upgrade and encourage others to do so)
> > 
> > This API throttling is being implemented to control costs by encouraging 
> > efficient software design while making sure the PeeringDB resource is 
> > shared well. The use of API keys is being encouraged so that admins can 
> > reach out to users/orgs with runaway or inefficient software, and because 
> > it is more secure than user/pass. In addition, org API keys ease employee 
> > transitions.
> > 
> > Some tips for coders is below.
> > 
> > API throttling in place today:
> > 
> >   - repeated anonymous identical requests with a response size above 100k 
> > are being limited to 1/hour
> >   - repeated anonymous identical requests of any size are being limited to 
> > 2/minute
> >   - anonymous queries are being limited to 400/minute per IP address
> >   - authenticated queries are being limited to 500/minute per user/org
> > 
> > Here is the current schedule of throttling changes. The schedule may 
> > adjust as needed as new packages that need update are discovered, so as to 
> > minimize disruption to the community...
> > 
> > On June 27th, adjust and watch for feedback from the community:
> > 
> >   - anonymous queries limited to 300/minute per IP address
> >   - authenticated queries limited to 400/minute per user/org
> > 
> > On July 11th, adjust and watch for feedback from the community:
> > 
> >   - anonymous queries limited to 200/minute per IP address
> >   - authenticated queries limited to 300/minute per user/org
> > 
> > On July 18th, adjust and watch for feedback from the community:
> > 
> >   - anonymous queries limited to 100/minute per IP address
> >   - authenticated queries limited to 200/minute per user/org
> > 
> > On July 25th, adjust and watch for feedback from the community:
> > 
> >   - anonymous queries limited to 50/minute per IP address
> >   - authenticated queries limited to 100/minute per user/org
> > 
> > On August 1st, adjust and watch for feedback from the community:
> > 
> >   - anonymous queries limited to 30/minute per IP address
> >   - authenticated queries limited to 80/minute per user/org
> > 
> > On August 8th, adjust and watch for feedback from the community:
> > 
> >   - anonymous queries limited to 20/minute per IP address
> >   - authenticated queries limited to 60/minute per user/org
> > 
> > On August 15th, adjust and watch for feedback from the community:
> > 
> >   - anonymous queries limited to 10/minute per IP address
> >   - authenticated queries limited to 40/minute per user/org
> > 
> > Feedback/questions/concerns welcome.
> > 
> > Thanks,
> > Chris
> > 
> > Software:
> > 
> > - arouteserver v1.16.0: has

Re: [PDB Tech] PeeringDB API throttling status and schedule

2022-08-08 Thread Chris Caputo

[resend]

Per the below plan, this change was just implemented:

---
On August 8th, adjust and watch for feedback from the community:

  - anonymous queries limited to 20/minute per IP address
  - authenticated queries limited to 60/minute per user/org
---

Please advise if you run into any issues.

Thank you,
Chris

On Tue, 31 May 2022, Chris Caputo wrote:
> After the initial introduction of PeeringDB API throttling, some software 
> both open source and private, has been identified and updated. (open 
> source details are below; please upgrade and encourage others to do so)
> 
> This API throttling is being implemented to control costs by encouraging 
> efficient software design while making sure the PeeringDB resource is 
> shared well. The use of API keys is being encouraged so that admins can 
> reach out to users/orgs with runaway or inefficient software, and because 
> it is more secure than user/pass. In addition, org API keys ease employee 
> transitions.
> 
> Some tips for coders is below.
> 
> API throttling in place today:
> 
>   - repeated anonymous identical requests with a response size above 100k 
> are being limited to 1/hour
>   - repeated anonymous identical requests of any size are being limited to 
> 2/minute
>   - anonymous queries are being limited to 400/minute per IP address
>   - authenticated queries are being limited to 500/minute per user/org
> 
> Here is the current schedule of throttling changes. The schedule may 
> adjust as needed as new packages that need update are discovered, so as to 
> minimize disruption to the community...
> 
> On June 27th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 300/minute per IP address
>   - authenticated queries limited to 400/minute per user/org
> 
> On July 11th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 200/minute per IP address
>   - authenticated queries limited to 300/minute per user/org
> 
> On July 18th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 100/minute per IP address
>   - authenticated queries limited to 200/minute per user/org
> 
> On July 25th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 50/minute per IP address
>   - authenticated queries limited to 100/minute per user/org
> 
> On August 1st, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 30/minute per IP address
>   - authenticated queries limited to 80/minute per user/org
> 
> On August 8th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 20/minute per IP address
>   - authenticated queries limited to 60/minute per user/org
> 
> On August 15th, adjust and watch for feedback from the community:
> 
>   - anonymous queries limited to 10/minute per IP address
>   - authenticated queries limited to 40/minute per user/org
> 
> Feedback/questions/concerns welcome.
> 
> Thanks,
> Chris
> 
> Software:
> 
> - arouteserver v1.16.0: has many updates including API key support along 
>   with more efficient querying.
> 
> - PeerFinder: API key & efficient querying patches at 
>   https://github.com/rucarrol/PeerFinder/pull/17 will hopefully be 
>   integrated.
> 
> Coding tips:
> 
> - Begin using a PeeringDB API key for all requests:
> 
> https://docs.peeringdb.com/howto/api_keys/
> 
> - Begin performing actual caching, such as by using peeringdb-py.
> 
> http://peeringdb.github.io/peeringdb-py/
> 
> - If unable to use a caching agent such as peeringdb-py:
> 
>- Use an API key.
> 
>- Set a User-Agent: header.
> 
>- Use bulk queries (asn__in=$list_of_ASN_separated_by_comma) by 
>  querying 30 to 150 ASNs at a time (tune as appropriate).
> 
>- Add a delay in between queries that is randomly between 2 and 2.5 
>  seconds, to reduce thundering herd.
___
Pdb-tech mailing list
Pdb-tech@lists.peeringdb.com
https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech

Re: [PDB Tech] PeeringDB API throttling status and schedule (fwd)

2022-08-09 Thread Chris Caputo

Dale, if you are getting 200 for an obviously bad api-key, then the 
authentication format is not correct. Examples/details at:

  https://github.com/peeringdb/peeringdb/issues/1220#issuecomment-1209763911

With a correctly formated auth request, 401 (unauth) will be returned for 
a bad key.

Please reach out to me privately with your source IP if you'd like me to 
review how the server sees your requests, or for efficiency efforts, or if 
you need any help getting api-key authentication working.

Thanks!

Chris

On Tue, 9 Aug 2022, Stephen McManus wrote:
> > However, for a read-only API key, how does one know if it's working?
> > I set 'Authorization': 'Api-Key foo-bar-1234-4312' for a GET, and I
> > got results back vs a 4xx error code.  So from an error handling 
> > perspective it seems hard to gauge if I am using a valid api key 
> > getting premium service vs an invalid api key quietly lumped into 
> > the anonymous rate-limit bucket.
> 
> This is something we should fix. I've filed 
> https://github.com/peeringdb/peeringdb/issues/1220 to get it addressed
> 
> -Steve
> 
> 
> 
> 
> > On Aug 9, 2022, at 1:56 PM, Dale W. Carder  wrote:
> > 
> > Thus spake Chris Caputo (ccap...@alt.net) on Mon, Aug 08, 2022 at 
> > 04:41:17PM +:
> >> Per the below plan, this change was just implemented:
> >> 
> >> ---
> >> On August 8th, adjust and watch for feedback from the community:
> >> 
> >>  - anonymous queries limited to 20/minute per IP address
> >>  - authenticated queries limited to 60/minute per user/org
> >> ---
> >> 
> >> Please advise if you run into any issues.
> > 
> > This is about where I start to get concerned.  First off, I'm not 
> > sure how well communicated this was.  I'd like to think that I'm
> > generally aware of what's happening in our ecosystem, but someone 
> > (thankfully) had to point this out to me.
> > 
> > So, our provisioning code is perhaps naive... jobs are dispatched 
> > into a task queue where they are run to completion, one per ASN.  
> > At present it would be non-trivial to implement a bulk query to 
> > cache ahead of time (making peeringdb lookups asynchronous), but 
> > that absolutely is on our longer-term roadmap.  It's also not the
> > easiest to rate-limit the queue as only some of them actually need
> > a peeringdb lookup (a huge amount of our peers are private asn
> > and/or in a non-dfz l3vpn's), but we have limited the concurrency
> > and can count on the general case that our code is reassuringly 
> > slow.
> > 
> > Luckily, some of the other things suggested below are easy, and I
> > was testing it out today.  We'll set a custom user-agent, limit
> > our query to only the fields we care about, and use an api key.
> > 
> > However, for a read-only API key, how does one know if it's working?
> > I set 'Authorization': 'Api-Key foo-bar-1234-4312' for a GET, and I
> > got results back vs a 4xx error code.  So from an error handling 
> > perspective it seems hard to gauge if I am using a valid api key 
> > getting premium service vs an invalid api key quietly lumped into 
> > the anonymous rate-limit bucket.
> > 
> > Dale
> > 
> > 
> >> On Tue, 31 May 2022, Chris Caputo wrote:
> >>> After the initial introduction of PeeringDB API throttling, some software 
> >>> both open source and private, has been identified and updated. (open 
> >>> source details are below; please upgrade and encourage others to do so)
> >>> 
> >>> This API throttling is being implemented to control costs by encouraging 
> >>> efficient software design while making sure the PeeringDB resource is 
> >>> shared well. The use of API keys is being encouraged so that admins can 
> >>> reach out to users/orgs with runaway or inefficient software, and because 
> >>> it is more secure than user/pass. In addition, org API keys ease employee 
> >>> transitions.
> >>> 
> >>> Some tips for coders is below.
> >>> 
> >>> API throttling in place today:
> >>> 
> >>>  - repeated anonymous identical requests with a response size above 100k 
> >>>are being limited to 1/hour
> >>>  - repeated anonymous identical requests of any size are being limited to 
> >>>2/minute
> >>>  - anonymous queries are being limited to 400/minute per IP address
> >>>  - authenticated queries are being limited to 500/minute per user/org
> >>> 
> >>> Here is the current schedule

[PDB Tech] non-canonical queries losing auth on redirect to canonical

2022-08-09 Thread Chris Caputo

It is important to query against the canonical "www.peeringdb.com".

Python and presumably other libraries will drop the authentication request 
header on a redirect, as a safety precaution:

  https://github.com/psf/requests/blob/177dd90f/requests/sessions.py#L284-L296

Unfortunately, when this happens is opaque because PeeringDB will try to 
satisfy what it thinks is simply an anonymous request.

If you have an opinion on whether it makes sense for PeeringDB to stop 
redirecting API requests, check out:

  https://github.com/peeringdb/peeringdb/issues/1139

Chris
___
Pdb-tech mailing list
Pdb-tech@lists.peeringdb.com
https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech

Re: [PDB Tech] Question about API ratelimits

2024-04-11 Thread Chris Caputo

On Thu, 11 Apr 2024, Tom Strickx wrote:
> Hey folks,
> 
> We're wondering how the ratelimits are enforced these days.
> Specifically, authenticated (so with API key) requests. Are these enforced
> by API-key, by AccountID, by IP, by subnet, by star sign, ...?
> Let me know if there's some piece of documentation documenting all of this.
> 
> Thanks!
> -- 
> Tom Strickx
> Principal Network Engineer
> AS13335 - Cloudflare

Hi Tom,

Per:

  https://docs.peeringdb.com/howto/work_within_peeringdbs_query_limits/

  - Anonymous queries limited to 20/minute per IP address

  - Authenticated queries limited to 40/minute per user or organization 
(when an organizational API key is used)

This comes from a set of HOWTOs that may be of interest to others, 
including one on the caching software peeringdb-py:

  https://docs.peeringdb.com/howtos/

The Seattle IX uses peeringdb-py to perform many queries of PeeringDB per 
day to inform its web site and route servers, with nil impact to PeeringDB 
itself, since the queries hit a local database instead.

There are also query limits for repeated identical from unauthenticated 
queries. These are per IP address and per /24 or /64 address block.

If you want to dig into the Django code for throttling, check out:

https://github.com/peeringdb/peeringdb/blob/master/peeringdb_server/rest_throttles.py

Let me know if you have other questions. I volunteer on PDB Ops.

Chris
___
Pdb-tech mailing list
Pdb-tech@lists.peeringdb.com
https://lists.peeringdb.com/cgi-bin/mailman/listinfo/pdb-tech

Re: [PDB Tech] PeeringDB API Discrepency

Re: [PDB Tech] PeeringDB API Discrepency

Re: [PDB Tech] allow empty IP field or not?

Re: [PDB Tech] peeringDB Python Module Issues with Python 3?

Re: [PDB Tech] Problem doing initial 'peeringdb sync', foreign key constraint fails?

Re: [PDB Tech] Problem doing initial 'peeringdb sync', foreign key constraint fails?

[PDB Tech] PeeringDB development container

Re: [PDB Tech] new rate limiting mechanism is too strict

Re: [PDB Tech] new rate limiting mechanism is too strict

[PDB Tech] PeeringDB API throttling status and schedule

[PDB Tech] PeerFinder support for API keys

Re: [PDB Tech] PeeringDB API throttling status and schedule

Re: [PDB Tech] PeeringDB API throttling status and schedule

Re: [PDB Tech] PeeringDB API throttling status and schedule

Re: [PDB Tech] PeeringDB API throttling status and schedule

Re: [PDB Tech] PeerFinder support for API keys

Re: [PDB Tech] PeeringDB API throttling status and schedule

Re: [PDB Tech] PeeringDB API throttling status and schedule

Re: [PDB Tech] PeeringDB API throttling status and schedule

[PDB Tech] peeringdb-py example setup

Re: [PDB Tech] PeeringDB Account Verification

Re: [PDB Tech] PeeringDB API throttling status and schedule

Re: [PDB Tech] PeeringDB API throttling status and schedule

Re: [PDB Tech] PeeringDB API throttling status and schedule

Re: [PDB Tech] PeeringDB API throttling status and schedule (fwd)

Re: [PDB Tech] PeeringDB API throttling status and schedule

Re: [PDB Tech] PeeringDB API throttling status and schedule

Re: [PDB Tech] PeeringDB API throttling status and schedule (fwd)

[PDB Tech] non-canonical queries losing auth on redirect to canonical

Re: [PDB Tech] Question about API ratelimits

30 matches

Site Navigation

Mail list logo

Footer information