On Mon, Apr 9, 2012 at 5:03 PM, Barry Hunter <[email protected]> wrote: > On Mon, Apr 9, 2012 at 9:39 PM, Wilson MacGyver <[email protected]> wrote: >> Hi, >> >> This is more of a design question, so I thought this would be more >> suited here instead of stackoverflow. >> >> I have a bunch of files in google storage. I want to track the access >> to those files. So one approach >> I came up with is, mark all files as private, gave GAE read only >> access. and the url pattern to access >> them is /{customerId}/filename >> >> this hits GAE, which check to make sure the customerId is valid, and >> then request the file from google sotrage, >> and return it. The part I'm not sure how to handle is to output a >> daily stats. I want to be able to produce >> a daily summary of customer X access Y number of files yesterday. >> >> I've only been able to come up with 3 approaches. >> >> 1: keep a counter in datastore, which is yucky, > > Curious, why is it yucky? > > If worried about performance, defer the updates using the task queue. > (so the actual increment is done by a task after the actual request > has finished to the user)
because the current system it's replacing, has at least 60 files access per sec during working hours. so that translates to 60 data counter increment in the datastore. > > >> so implement a sharded >> counter to track this and update >> the datastore. > > Are you really expecting these files to be read so much by each user > that a sharded counter is needed? > > Sounds like over thinking the problem :) yea, actually it's quite a bit of read. This is one reason I'm exploring GAE to solve this problem. because the traffic pattern is very busy during working hours but drops to nothing afterwards. Sometimes with certain events going on, it goes up to hundreds of files read per sec. > >> >> 2: use app_cfg to download daily raw http request log, and parse it myself > > There is also a LogService API for accessing the logs from within App > Engine itself > https://developers.google.com/appengine/docs/python/logservice/ this I didn't know, will look into this. Thanks -- Omnem crede diem tibi diluxisse supremum. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
