Ok.. well, after successfully doing a bulkloader.py --dump from my live app using the 1.3.1 SDK (it was slow [see my previous post] but it worked)... I got real excited to do a --restore into my Development Environment..
I issue my command: bulkloader.py --restore --url=http://localhost:8081/remote_api--filename=dumpfile.csv --app_id=myappid :( Then I became sad when I just started getting these errors and the --restore would fail after partially transferring some of the entities: File "C:\Python25\lib\urllib2.py", line 381, in open response = self._open(req, data) File "C:\Python25\lib\urllib2.py", line 399, in _open '_open', req) File "C:\Python25\lib\urllib2.py", line 360, in _call_chain result = func(*args) File "C:\Python25\lib\urllib2.py", line 1107, in http_open return self.do_open(httplib.HTTPConnection, req) File "C:\Python25\lib\urllib2.py", line 1082, in do_open raise URLError(err) URLError: <urlopen error (10061, 'Connection refused')> [DEBUG ] Waiting for progress_thread to terminate... [DEBUG ] [Thread-13] ProgressTrackerThread: exiting [DEBUG ] ... done. [INFO ] 1160 entites total, 0 previously transferred [INFO ] 300 entities (80771 bytes) transferred in 68.5 seconds [INFO ] Some entities not successfully transferred Being lazy and impatient.. I frantically googled around .. but all I get are these people going on and on about commenting out login: admin from app.yaml for the remote_api or passing in the --email param or passwords to bypass authentication (granted, maybe they were having an Authentication Error.. but I was not.. just vanilla Connection refused.. and some of the threads [remember that key word, it helps solve this in the end] did connect and upload some data). Being too lazy to read all the recommendations like "if you close one eye and count to ten after running the --restore.. it works!".. I sat still and thought.. 'Some of it works.. why not all? Maybe I can force it to put all of them at once before it gets a Connection Refused' Tried that with --batch_size=40000 and --bandwidth_limit=20000000 .. this was hopeful.. it sat and sat.. processing gobbling up all of my CPU.. :( ..before giving an: [ERROR ] Error in Thread-3: The request to API call datastore_v3.Put() was too large. But.. that was understandable. Then.. I noticed the --num_threads flag.. and then I thought about the fact that some of the threads succeeded.. and then I thought, "I bet first thread Connects".. So.. I issued this command: bulkloader.py --restore --num_threads=1 --url= http://localhost:8081/remote_api --filename=dumpfile.csv --app_id=myappid and it did: ...... and ...... and ........... Until finally: ....................... [INFO ] 42224 entites total, 0 previously transferred [INFO ] 42224 entities (12099243 bytes) transferred in 321.0 seconds [INFO ] All entities successfully transferred So, long story short.. there's some sort of bug with using more than one thread (the default) to --restore a lot of entities into your local Dev Environment. Forcing --num_threads=1 seems to make it work. I'd like to thank Tim Gunn for "make it work", and I would also like to thank the old fashioned art of human thinking. I hope this helps anyone who runs into issues doing a bulk restore to localhost. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
