Thanks for the tip, but why write a web app when Java Applets are required, that whouldn't be a good solution. Also, the uploaded file needs to be parsed in it's entirety (CRC check, value references, etc.), and it's not XML.
I think I have to parse the file server-side, populate (Java) Entity objects and serialize as many I can into 1 MB blobs. When that is done, start a task that put the de-serialized entities in batches of 500 into the datastore. The response for the file upload request will have to contain some unique task URL that the browser can (AJAX) poll to display the progress. Before I commit to such a elaborate solution, I'll have to test the batch-put performance to see if GAE is even suitable for this kind of app. http://groups.google.com/group/google-appengine/browse_thread/thread/41276f470e12037c# Users of an online apps shouldn't have to wait hours for a simple data import just because it's hosted at GAE. If the app where using an SQL database this would only take a minute. On Aug 14, 4:48 pm, Juraj Vitko <[email protected]> wrote: > I think you need to write your own Flash or Java Applet based chunked > uploader. Or use an existing one and let us know, so that we can use > it too. > > On Aug 12, 11:36 pm, Stakka <[email protected]> wrote: > > > I'am working on an browser based accounting app which has a feature to > > import ledger transactions through file uploads. I'am currently only > > running on the local dev server, but from what I've read datastore > > puts -- even batch -- is very slow and CPU (quota) intensive when > > deployed live. > > > How do I overcome this problem if the user uploads a large file with > > thousands transaction? > > > I've seen solutions where you batch put entities in chunks of 500. > > That only works if you run a custom upload tool on your computer, not > > from a browser since the request is limited to 30 seconds. Am I forced > > to use the Task Queue? But where do I store the raw uploaded file or > > the preferably parsed interim transaction entities when the task isn't > > executing? > > > Funny App Engine has a 10 megabyte request (file upload) size limit > > when storing 10 megabyte worth of entities seems to be so hard. > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en -~----------~----~----~----~------~----~------~--~---
