Hi Ingo, The 1000 entity limit was removed some time ago. The datastore has no built-in aggregation functions. You may want to look into using the TaskQueue to break up server-side full-scan type aggregations into numerous smaller tasks. Brett Slakin's 2010 IO talk about high performance data pipelines may be useful to you.
Have you considered doing the aggregations up-front instead of on-the-fly? Or, running a periodic aggregate routine? Robert On Fri, Jul 16, 2010 at 6:10 AM, ingo <[email protected]> wrote: > hello everyone, > > i want to provide detailed reports in my app. those reports require > data aggregated across full tables. in the client i want to plot the > change of values over time. this then requires to select very huge > data sets. however, in the app engine there is the restriction to get > 1000 items in one request. so what is the best approach to do queries > with big result sets? > > i can imagine that the client could split the big request into 10 > smaller ones to collect, merge and display the data. but i think this > is complex, error prone and slow. i would rather like to do this on > the server side. but do i have to split the requests to work around > the 1000 items limit or is there another pattern that developers > should use to handle huge datasets. > > basically i want to know two things: > 1) how can i retrieve huge amounts of data regarding the get 1000 > items limit? > 2) is the app engine performance good enough to allow showing reports > on the client side i.e. can app engine retrieve and analyse > (avg,min,max,grouping) huge amounts of data (10k - 1million rows) fast > enough? > > one more thing: is it possible to reduce the amount of data that is > returned dramatically by doing all the aggregations (e.g. by grouping > and summing) completely within the datastore? currently i am > experimenting with doing full table scans (but i can only retrieve > 1000 items in one request..) and aggregating the data within the > application (on server side). > > kind regards, > ingo > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
