wait, ok sorry, I wasn't clear. I didn't mean "literally" like Google Analytics. What I'm saying is that each user has profile attributes and I want to be able to breakdown what type of users do what type of actions on my site. I only mentioned Google Analytics because I felt that was the thing that seemed to closest resemble what I am trying to do.
So let's say a user does an action on my site that I want to monitor, should I have a table that spits out all his profile attributes into another table? Then have a cron job every 5 minutes tally up all the different attributes of all the people who voted the last 5 minutes into a stored entity? Is this how Google Analytics works? It seems like it reading this segment from bigtable on Analytics: This program is invoked whenever a page is visited. It records various information about the request in Google Analytics, such as a user identi er and information about the page being fetched. Google Analytics summarizes this data and makes it available to webmasters. We briefl y describe two of the tables used by Google Analytics. The raw click table (.200 TB) maintains a row for each end-user session. The row name is a tuple containing the website's name and the time at which the session was created. This schema ensures that sessions that visit the same web site are contiguous, and that they are sorted chronologically. This table compresses to 14% of its original size. The summary table (.20 TB) contains various predefined summaries for each website. This table is generated from the raw click table by periodically scheduled MapReduce jobs. Each MapReduce job extracts recent session data from the raw click table. The overall system's throughput is limited by the throughput of GFS. This table compresses to 29% of its original size. This just seems like such a hack job to me, so trying to see if there's a better way to do what I want to do. On Dec 18, 7:26 pm, Jason Smith <[email protected]> wrote: > Analytics and reporting are time-consuming. You have to write custom > software for practically every question you have about the data. If > you modify your question, you must start over (at least the processing > job, and probably the code too, which must be debugged, again.) > > Analytics is not mission-critical, so I suggest an offline log- > processing technique. Regularly fetch (with --append) both your > access.log (no --severity flag in request_logs) and your debugging log > (--severity=0 or maybe 1). Then use traditional tools or write your > own for doing the analysis. > > You can either infer the information from the access log (e.g. an HTTP > PUT to a path that includes the user ID), or else you can add a log > message to your code: > > if user.is_administrator() and action in ('update', 'create', > 'delete'): > logging.info('TRACE: Admin %r performed %s', user, action) > > That will be very easy to spot in your debug log you downloaded. > > On Dec 19, 1:25 am, killer barney <[email protected]> wrote: > > > > > So here's a problem that I'm having a lot of trouble with. > > > I want to keep track of the statistical breakdown of my users who > > insert an entity in a certain table of mine. (Kind of like google > > analytics, when a user visits a webpage, how many are using browsers > > X, how many stay for how long, etc) How should I go about this? On > > every insert, should I list out all of the user's data into the entity > > and then have a cron job that goes and collects and calculates all the > > data so that it can store it in a culmultive property? It doesn't seem > > like a very scalable idea. What if I keep adding more stats, that > > table just gets bigger and bigger. > > > In a relational database, it seems pretty easy. Just join and the > > count. Anyone have any ideas on the datastore? -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
