Jason's comment about the YouTube API is spot on, I think. Unless you're doing some serious enterprise scale implementation, you can do this simply, and not worry much about fancy antics to mitigate database call impact.
Sounds like a fun project. -james On Fri, September 18, 2015 3:00 am, [email protected] wrote: > Send Portland mailing list submissions to > [email protected] > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/portland > or, via email, send a message with subject or body 'help' to > [email protected] > > You can reach the person managing the list at > [email protected] > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Portland digest..." > > > Today's Topics: > > > 1. Re: Creating an API with Metered Billing (Lennon Day-Reynolds) > 2. Re: Creating an API with Metered Billing (Jeff Schwaber) > 3. Re: Creating an API with Metered Billing (Jason Champion) > > > > ---------------------------------------------------------------------- > > > Message: 1 > Date: Thu, 17 Sep 2015 10:40:26 -0700 > From: Lennon Day-Reynolds <[email protected]> > To: "Python Users Group -- Portland, Oregon USA" <[email protected]> > Subject: Re: [portland] Creating an API with Metered Billing > Message-ID: > <CAPT-2MgA-jcfJPO9T3d2_J4CCjd2d-TK5ZqVPMV3q7cwQ=z...@mail.gmail.com> > Content-Type: text/plain; charset=UTF-8 > > > I've worked on these kinds of APIs before. Jeff's point about counts > lagging # of calls is valid, and totally an architectural and business > decision you need to think about before you sit down to build a > rate-limiting system. > > Think of accuracy vs. efficiency here as a slider you can move in > either direction. What's your biggest risk or downside? Would permitting > over-usage for a few of your users harm your service or the business in > some way? (E.g., a voting system should allow zero over-counting, but a > weather API can probably permit a few "free" over-quota reads.) What's > your expected nominal usage rate? Do you expect big bursts of activity? > > If you go the most strict route (e.g., update a counter in your main > app DB for every API request) you'll get very accurate gating, but > depending on your database access and application usage patterns you could > easily end up doing more work maintaining counts than you do actually > providing your service to customers. Worst-case, if the DB gets swamped > then usage counting could actually cause downtime for the rest of your > system. > > Log-based rollups are slower to converge, but potentially much cheaper > to maintain (esp. if you have any existing async job infrastructure in > place). Depending on your access rates and how "bursty" traffic can be > they might be a totally acceptable option. You also didn't mention your > underlying web stack, but assuming you're going to run on multiple hosts > you might have to work a bit to gather + order logs in one place to get > accurate rollups. > > There's also a middle ground where you use memcached or redis to store > your counts. In-place increments in those systems will be cheaper than a > transactional DB write, though you can't do a simple filter on event > timestamp as you can in SQL. I've commonly used a simple bucketed storage > model with TTLs on each bucket to store a sliding window of recent access > counts. > > I haven' vetted any of these implementations in production, but > several people have written libraries accomplishing exactly this pattern: > > > http://flask.pocoo.org/snippets/70/ > https://github.com/DomainTools/rate-limit > http://limits.readthedocs.org/en/stable/ <- (this one appears to have > both Flask and Django adapters available, too) > https://pypi.python.org/pypi/rratelimit/0.0.4 > > > (& etc., etc.) > > > On Wed, Sep 16, 2015 at 4:56 PM, Jeff Schwaber <[email protected]> wrote: > >> That's super interesting! >> >> >> I haven't done usage limits, but I've played with the throttling stuff >> of Django Rest Framework: >> http://www.django-rest-framework.org/api-guide/throttling/ and you could >> definitely put usage counting in there. Once you've got usage >> counting, limits seem like a simple step inside that framework. >> >> The big challenge is going to be that, naively, now all of your API >> requests are database updates, and from a scalability point of view, >> that sucks. Of course you could make them logging statements instead and >> then have a background process reading the log to generate the current >> counts, but then you'll be behind a bit, so users may bounce over the >> limits a bit. >> >> Jeff >> >> >> On Wed, Sep 16, 2015 at 4:44 PM, Jason Champion >> <[email protected]> >> wrote: >> >> >>> Question: >>> >>> >>> Does anyone have any experience with creating APIs that have usage >>> limits and metered billing? Can you suggest any good >>> articles/howtos/resources on the subject? >>> >>> I've created plenty of APIs (REST, XMLRPC, etc.), but they've always >>> been open to all with no auth or billing. >>> >>> Thank you, >>> Jason >>> _______________________________________________ >>> Portland mailing list >>> [email protected] >>> https://mail.python.org/mailman/listinfo/portland >>> >>> >> -------------- next part -------------- >> An HTML attachment was scrubbed... >> URL: >> <http://mail.python.org/pipermail/portland/attachments/20150916/d54c2d3 >> 8/attachment.html> >> _______________________________________________ >> Portland mailing list >> [email protected] >> https://mail.python.org/mailman/listinfo/portland >> > > > > -- > Lennon Day-Reynolds <[email protected]> > http://rcoder.net/ > > > > ------------------------------ > > > Message: 2 > Date: Thu, 17 Sep 2015 11:11:52 -0700 > From: Jeff Schwaber <[email protected]> > To: "Python Users Group -- Portland, Oregon USA" <[email protected]> > Subject: Re: [portland] Creating an API with Metered Billing > Message-ID: > <caaf6mjoqqdtj4bnesaqng98x-x+xxlfz1aomee5orceux29...@mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > > On Thu, Sep 17, 2015 at 10:40 AM, Lennon Day-Reynolds <[email protected]> > wrote: > > >> Log-based rollups are slower to converge, but potentially much cheaper >> to maintain (esp. if you have any existing async job infrastructure in >> place). Depending on your access rates and how "bursty" traffic can be >> they might be a totally acceptable option. You also didn't mention your >> underlying web stack, but assuming you're going to run on multiple hosts >> you might have to work a bit to gather + order logs in one place to get >> accurate rollups. >> > > Yeah, this is a fair point. There are services, for example Papertrail, > that aggregate logs these days, so it can certainly be worked around, but > you need to integrate with or build something to do this. Timeliness will > definitely vary. > > >> There's also a middle ground where you use memcached or redis to store >> your counts. In-place increments in those systems will be cheaper than a >> transactional DB write, though you can't do a simple filter on event >> timestamp as you can in SQL. I've commonly used a simple bucketed >> storage model with TTLs on each bucket to store a sliding window of >> recent access counts. >> > > +1 - totally valid option, too. > > > Jeff > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > <http://mail.python.org/pipermail/portland/attachments/20150917/dd2326fc/ > attachment.html> > > ------------------------------ > > > Message: 3 > Date: Thu, 17 Sep 2015 21:21:00 -0700 > From: Jason Champion <[email protected]> > To: "Python Users Group -- Portland, Oregon USA" <[email protected]> > Subject: Re: [portland] Creating an API with Metered Billing > Message-ID: <[email protected]> > Content-Type: text/plain; charset=windows-1252; format=flowed > > > Thank you Jeff and Lennon for the links+info. > > > As an example, the YouTube Analytics API takes 1-3 minutes to shut off > once you hit your maximum quota. In that time you can go over limit by up > to about 1%, which seems like a pretty reasonable margin of error. > > Plus they look like the "nice guy" for letting you have a few extra > calls now and then. :) > > > ------------------------------ > > > Subject: Digest Footer > > > _______________________________________________ > Portland mailing list > [email protected] > https://mail.python.org/mailman/listinfo/portland > > > > ------------------------------ > > > End of Portland Digest, Vol 100, Issue 4 > **************************************** > > _______________________________________________ Portland mailing list [email protected] https://mail.python.org/mailman/listinfo/portland
