Hi Nacho,
You've got a few options to choose from depending on your needs.
They will all, in one form or another, require you to maintain some
type of 'counter' entities. If you design your process and counter
entities well, you will be able to use key_names to generate reports
very efficiently.
If your reports are 'off-line,' or 'batch' type (ie not real-time)
processes you just need to periodically iterate over your new /
changed data to generate and update your counter entities. I would
suggest the mapper API, but last I knew it lacks support for queries;
it has to iterate over all entities of a particular kind -- so it may
be quite inefficient if you've got a lot of data.
I have handled similar cases by using timestamps and by using a
global 'batch' number, so that I can identify entities written since I
last aggregated. I prefer using batch-numbers over timestamps, since
the timestamps could be wrong (possibly by a lot). Then it is just a
case of inserting tasks to aggregate your data.
If you want more realtime reporting, Brett Slatkin's gave a talk,
Building high-throughput data pipelines with Google App Engine, that
covers some nice techniques.
http://www.google.com/events/io/2010/sessions/high-throughput-data-pipelines-appengine.html
I've been using that general type of technique for a year very
successfully. I've also written a little library that uses that
technique.
https://bitbucket.org/thebobert/slagg
If you adda new report type and you want it for historical data,
you'll need to generate the counter entities for the old data.
Robert
On Tue, Jan 4, 2011 at 16:57, nacho <[email protected]> wrote:
> I already know that in the datastore there is no aggregation
> functions, and i also know about the 1000 entites fetch limit in a
> query.
>
> But I would like to know how did you solve this problem.
>
> My application allows to webmasters to charge to their users by
> sending sms to access to their contents. Every webmaster can have many
> sites and every website can have many contents.
>
> So I would like to display some reports, for example:
>
> 1) how many sms did receive dayly from one date to another for every
> site and for one site in especific.
> 2) how many sms did receive from 0hs to 24hs for a especific date for
> every site and for one site in especific.
> 3) how many sms did receive from each country from one date to another
> date.
>
> 4) how many money did receive dayly from one date to another for every
> site and for one site in especific.
> 5) how many money did receive from 0hs to 24hs for a especific date
> for every site and for one site in especific.
> 6) how many money did receive from each country from one date to
> another date.
>
> 7) the same reports to the root user aggregating by webmaster / site /
> content and grouping by date, hour, etc.
>
> etc etc etc
>
> Do I need to create counters for every report that i need?
>
> Let's say that i do this, and in one month i think that could be nice
> to display another report and i dont have the counter for this, for
> example, how many sms the webmaster receive by minute. Here, what can
> i do?
>
> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To post to this group, send email to [email protected].
> To unsubscribe from this group, send email to
> [email protected].
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.
>
>
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/google-appengine?hl=en.
