Here's my scenario: every once in a while, I want to let certain users dump a large number of records from the datastore (e.g. 1000+) into a CSV file.
I've already figured out how to get around the 1000-record limit by using a time-based auto-increment field. I query the database multiple times, increasing the offset on that field until I've queried all the records. There's a good chance I'll have more data than can fit within the constraints of one GAE request-response cycle, so I'm using AJAX to spread the calls out over several requests. Here's where I'm stuck. I have a setup where I'm fetching 1000 records at a time and would like to concatenate all these fetches to generate my CSV file. On a traditional setup, I could just save a file to disk and keep appending stuff to it before returning it to the user. On GAE, I can't store anything larger than 1 MB in the datastore. I know the GAE team plans to offer large file storage at some point down the road, but I need this in the near future so I'd like to see what workarounds people are trying right now. Some options I'm considering ... 1) Use AJAX calls to download small chunks, one at a time to the client. Hard part is figuring out how to use client-side scripting to save data to the user's file system without running afoul of browser security restrictions. 2) Use AJAX calls to download small chunks, concatenate them, and then ask the user to paste the text into a text editor and save it as a CSV. Hard part is doing this in a way that's user friendly. 3) Force the user to download multiple smaller CSV files. Again, not user friendly. 4) Save to S3. Hard part is that S3 doesn't allow me to append to a file already in S3, something that would have been nice considering I'm spreading out small 1000-record writes to the same file over multiple requests. 5) Set up a separate box that GAE hits up when it needs to do an operation like this. Not hard to do but very annoying. Sorta defeats the purpose of App Engine. Am I missing any options here? Thoughts? -- Andrew --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
