Hi David, There is no way you're going to return a million entities in a single request. You should look into cursors. Cursors allow you to fetch results from a query, then resume fetching more results where you left off. Cursors are nice because you can use them within the same request to efficiently fetch a larger number of entities in smaller batches, or you can use them across requests (by passing them between requests). You could also potentially use the new files API to dump your entities to the blobstore (maybe as CSV) then download the blob.
http://code.google.com/appengine/docs/python/datastore/queryclass.html#Query_cursor http://code.google.com/appengine/docs/python/blobstore/overview.html#Writing_Files_to_the_Blobstore Within a request, you can use a cursor something like this: query = YourKind.all().order('some_prop') cursor = None while True: if cursor: query.with_cursor(cursor) entities = query.fetch(500) # do stuff with entities if len(entities) < 500: break cursor = query.cursor() Across requests you'd just need to pass the cursor to the next request (perhaps in a form field, for example). Robert On Sat, May 28, 2011 at 04:06, David Stone <[email protected]> wrote: > Hello, > > Suppose i have 4,5 or even more kinds (tables) stored in GAE data > store and each table has quite a lot of data, let's say each table has > 5 million rows (entities) or even more. Now i am facing a big problem > with generating and returning a big size result for a query. The > reason is that a request handler will time out if takes more than 30 > second to generate the query result and i think there is a probably > 10MB limitation for a http response to return the query result. > > So i am wondering is there a way to send the query result as small > portion (data streaming) each time back to me. > > I have read from this link: > http://code.google.com/appengine/docs/python/runtime.html > > The link mentions "App Engine collects all of the data the request > handler script writes to the standard output stream, then waits for > the script to exit. When the script exits, all of the output data is > sent to the user. > > App Engine does not support sending data to the user's browser before > exiting the handler. Some web servers use this technique to "stream" > data to the user's browser over a period of time in response to a > single request. App Engine does not support this streaming technique." > > As far as i understand their information is that there is no way to > send the query result as many small portions by time and each time > sends out for example 5000 rows. > > At the same time, this link (http://arstechnica.com/web/news/2010/12/ > app-engine-gets-streaming-api-and-longer-background-tasks.ars) > mentions there is a streaming API from GAE. > > Now i am confused by these messages. Any idea or suggestion about this > issue would be well appreciated. > > Another question is that once i have a query generating a lot of > result might cause time out to the request handler due to the 30 > seconds time limitation. How could i overcome this problem? > > Thank you very much for all the answers from you. > > David > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
