Hey Jeff and Sylvain- Thanks for the info on the shard counter. I've been looking it over and I don't completely understand the code, I understand the idea.
However, it is hard for me to believe it is a contention issue because this is a very low traffic download. Something around 1 download every two hours. The shard solution degrades to my counting solution when the cache has expired. Anyway ... thanks for the notes, just seems a strange solution when simple aggregates would make it simple. On Fri, Oct 17, 2008 at 10:24 AM, Jeff S <[EMAIL PROTECTED]> wrote: > > The reason that you are seeing timeouts, is that you have too many put > transactions overlapping one another (contention). When a transaction > fails, it is automatically retried. The sample that Sylvain pointed to > is an excellent solution to this common design problem. The sharded > counter avoids write contention by splitting the writes over multiple > entities. The faster the counter is being incremented, the more shards > you will need. This technique is an example of "scaling horizontally". > Here's another great design for a sharded counter, written by Bill > Katz: > > http://oji.me/px > > Happy coding, > > Jeff > > P.S. The Google I/O presentation that Sylvain's sharded counter was > from can be found here: > http://sites.google.com/site/io/building-scalable-web-applications-with-google-app-engine > Counters are explained starting at around 19 minutes in. > > > > On Oct 17, 9:59 am, Sylvain <[EMAIL PROTECTED]> wrote: >> Use this :http://paste.blixt.org/1581 >> >> From Google I/O >> >> On 17 oct, 08:03, rjcarr <[EMAIL PROTECTED]> wrote: >> >> >> >> > I have a simple app engine app that offers a download. I catch the >> > download request and record the time of the download in the data store >> > before sending the file. I'd also like to keep track of the total >> > number of downloads. Obviously, the number of rows in the data store >> > represents the total number of downloads. >> >> > However, looking at the API, the count() method of Query return a >> > maximum of 1000. This won't work because my number will grow much >> > larger than that. >> >> > So as a fix I added a count field to my data model so that the count >> > is updated upon every request. It looks something like this: >> >> > class Counter(db.Model): >> > count = db.IntegerProperty() >> > date = db.DateTimeProperty(auto_now_add=True) >> >> > class DownloadHandler(webapp.RequestHandler): >> > def get(self): >> > query = Counter.all() >> > query.order("-count") >> >> > counter = query.fetch(1) >> > count = counter[0].count >> >> > counter = Counter() >> > counter.count = count + 1 >> > counter.put() >> >> > This works, but I'm seeing quite a few timeout errors in the logs. >> >> > So, what am I missing? Is there a better way to keep track of a >> > counter? I'd prefer to make the counter persistent and not simply a >> > member of the class as that seems volatile and likely to reset (say, >> > on a redeploy). > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
