Hi there,
Good question. I think you'll probably need to make the export an
asynchronous process that iterates through your dataset and writes out
a CSV blob to a temp Datastore entity, or to memcache in <1MB
chunks.
I've done some toy examples of this, but nothing in production. My
recipe involves 3 basic parts:
- AJAX progress-bar type control on the client UI (reusable)
- Main request handler that is response for queueing the export
request, polling for status, and fetching the completed results
(reusable)
- Async task queue handler that does the work of building the export
data (custom per-export type)
Pseudo-code might work like this:
- User clicks 'export' button, which fires off an AJAX request to your
GAE request handler URL to kick off an export
- Request handler is invoked in your app
- Creates an "ExportJob" entity in the datastore. ExportJob might
have these properties:
- Key key
- boolean isDone
- int entitiesProcessed
- List<String> memcacheKeys
- Queues up a TaskQueue invocation with params: offset=0,
exportJobKey=[key] (key of ExportJob entity in datastore)
- Returns the exportJobKey back to the client
- AJAX call completes - client now has a key to the job
- Client polls the server using that key, which returns the
"entitiesProcessed" and "isDone" fields, so you can tell the user
"Processed xxx entities"
- ExportJobTask handler gets invoked in background
- In loop, loads a window of entities from the DataStore using the
offset passed in
- Appends CSV data to a StringBuilder
- Loop exit when a timeout upper limit is reached (say, 15
seconds) OR when stringbuilder.size() is close to 1MB OR when all
entities are done
- Loads ExportJob with this key from DataStore
- Puts the CSV stringbuilder into a new memcache entry, using the
key + a numeric offset
- Appends the memcache key to ExportJob.memcacheKeys
- Sets ExportJob.entitiesProcessed += CSV line count
- If no more entities:
- Sets ExportJob.isDone = true
- Saves ExportJob back to DataStore
- Else
- Stores ExportJob back to DataStore
- Queues another taskqueue job back to itself with the same
key, and with offset = entitiesProcessed-1
- When job done, AJAX handler invokes initial request handler with key
and another flag "getData=1" or something
- Loads ExportJob from datastore
- Loops through ExportJob.memcacheKeys
- Loads CSV blob from memcache
- streams to servletresponse
- deletes memache entry
- Deletes ExportJob
- Done..
Maybe there's a simpler way?
-- James
On Nov 11, 2:12 am, alf <[email protected]> wrote:
> hi all,
>
> I need export a large dataset may be 25k entitys with 15string
> properties. I know limit of 1000 and 30 seconds. Any idea to can
> export data to csv, I can split in little chuck ok but how download?.
> I know appcfg but my feature is for end user and I would like do a
> simple page with a button.
>
> Can I create a datataset in memory and build and put in a datastore to
> dowload. What happend if download process spent more than 30 sec. ?
>
> many thanks.
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Google App Engine for Java" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/google-appengine-java?hl=en
-~----------~----~----~----~------~----~------~--~---
