I'm downloading data from my app's datastore and applying a simpletext connector to save keys and b64-encoded data from some fields of all entities of a particular type to disk on my local Windows system. The download itself succeeds, giving me a .sql3 database of 50GB, which is consistent with the total entity size reported by the AppEngine dashboard -- so I believe everything gets downloaded okay. However, the simpletext transform fails partway through unpacking the .sql3 database and writing the text file, with an IO error 0. A transcript of the session, including the Python interpreter's call stack, is at the bottom of this post. The command line includes the db_filename and result_db_filename arguments because this is an attempt to resume an earlier session that had failed in the exact same way.
Not sure if any of the following is significant: -- I'm doing the download to the E: drive of a VM; this VM has 150GB of storage, about 140GB of which were free before the start of the download -- so there should be enough disk space to store the transformed data. There's only about 1.2GB free on the VM's C: drive. -- The last 2 entries in the call stack refer to files on the C: drive that don't exist -- I had originally installed the AppEngine SDK on C: but moved it to E: before this run, in the belief that the download was failing because there wasn't enough space on C: for temporary storage during the download. The SDK is now on the E: drive, and both TMP and TEMP environment variables are set to e:\temp, which does exist. -- I'm using Python 2.5.4. -- The transfer fails partway through, writing several GB of transformed data to the output file but without completing the transforms. Any tips or clues would be appreciated -- thanks! ================ E:\work\Swapper>appcfg.py download_data --kind=SomeKind --url=http:// someapp.appspot.com/_ah/remote_api --filename=output.dat --config_file=config.yaml --db_filename=bulkloader- progress-20110620.103156.sql3 --result_db_filename=bulkloader- results-20 110620.103156.sql3 --http_limit=10 --bandwidth_limit=250000 Downloading data records. [INFO ] Logging to bulkloader-log-20110624.113729 [INFO ] Throttling transfers: [INFO ] Bandwidth: 250000 bytes/second [INFO ] HTTP connections: 10/second [INFO ] Entities inserted/fetched/modified: 20/second [INFO ] Batch Size: 10 2011-06-24 11:37:30,858 WARNING appengine_rpc.py:435 ssl module not found. Without the ssl module, the identity of the remote host cannot be verified, and connections may NOT be secure. To fix this, please install the ssl module from http://pypi.python.org/pypi/ssl . To learn more, see http://code.google.com/appengine/kb/general.html#rpcssl . Please enter login credentials for someapp.appspot.com Email: [email protected] Password for [email protected]: [INFO ] Opening database: bulkloader-progress-20110620.103156.sql3 [INFO ] Opening database: bulkloader-results-20110620.103156.sql3 [INFO ] Connecting to someapp.appspot.com/_ah/remote_api [INFO ] Downloading kinds: ['SomeKind'] Traceback (most recent call last): File "E:\google_appengine\appcfg.py", line 76, in <module> run_file(__file__, globals()) File "E:\google_appengine\appcfg.py", line 72, in run_file execfile(script_path, globals_) File "E:\google_appengine\google\appengine\tools\appcfg.py", line 3592, in <module> main(sys.argv) File "E:\google_appengine\google\appengine\tools\appcfg.py", line 3583, in main result = AppCfgApp(argv).Run() File "E:\google_appengine\google\appengine\tools\appcfg.py", line 2229, in Run self.action(self) File "E:\google_appengine\google\appengine\tools\appcfg.py", line 3368, in __call__ return method() File "E:\google_appengine\google\appengine\tools\appcfg.py", line 3174, in PerformDownload run_fn(args) File "E:\google_appengine\google\appengine\tools\appcfg.py", line 3087, in RunBulkloader sys.exit(bulkloader.Run(arg_dict)) File "E:\google_appengine\bulkloader.py", line 4360, in Run File "E:\google_appengine\bulkloader.py", line 4255, in _PerformBulkload File "E:\google_appengine\bulkloader.py", line 3551, in Run File "E:\google_appengine\bulkloader.py", line 2489, in WorkFinished File "C:\google_appengine\google\appengine\ext\bulkload \bulkloader_config.py", line 500, in output_entities File "C:\google_appengine\google\appengine\ext\bulkload \simpletext_connector.py", line 112, in write_dict IOError: [Errno 0] Error -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
