Wikus, I enabled the remote api in app.yaml as you suggested.
Then, after changing the path slightly in the following command: python appcfg.py create_bulkloader_config --filename=config.yaml -e $em...@gmail.com --url=https://$APP.appspot.com/_ah/remote_api your suggestion for creating a config file worked great! I then set the connector_options for all the types, by replacing the following string in the config.yaml file: # TODO: Choose a connector here: csv, simplexml, etc... with csv I then ran the command: python appcfg.py download_data --filename=$TYPE.csv --kind=$TYPE -- config_file=config.yaml -e $em...@gmail.com --url=https:// $APP.appspot.com/_ah/remote_api I was receiving an error with two similar blocks of the config file, so for the moment, I deleted both properties: # Warning: This property is a duplicate, but with a different type. # TODO: Edit this transform so only one property with this name remains. - property: description external_name: description # Type: Text Stats: 16 properties of this type in this kind. import_transform: db.Text # Warning: This property is a duplicate, but with a different type. # TODO: Edit this transform so only one property with this name remains. - property: description external_name: description # Type: Text Stats: 3 properties of this type in this kind. import_transform: db.Text Off the top of your head, do you have any suggestions about what this warning might mean and how to go about resolving this? After deleting these two sections of the config file, I then was able to successfully run the command. I then modified this command for another table in my app, and it ran fine as well. python appcfg.py download_data --filename=$TYPE.csv --kind=$TYPE -- config_file=config.yaml -e $em...@gmail.com --url=https:// $APP.appspot.com/_ah/remote_api So the idea is to run this command, changing the type for each of the models in my app? Thank you for sharing the script to upload this data, I have not tried to upload the data back to google app engine, but I will give this a try and will write back if I have any questions ;) By the way, this command did not appear to work properly with the latest release of GAE: python appcfg.py download_data -e $em...@gmail.com --application=$APP --url=https://$APP.appspot.com/_ah/remote_api --filename=dump.csv It turns out that the file downloaded, not in csv format, but in squlite 3. Also, the tables did not seem to export as one would expect. There were only two tables in the database, a “bulkloader_database_signature” and a “result” table. At this point, one question comes to mind. If I am able to successfully download and upload data back to my app using the bulkloader, does this mean that I do not have to change my data models to include a uuid and and last_modified date. I think a web2py tutorial about this would be useful. I would be willing to contribute all notes from my experience to document this process. I also have one slightly related comment about the web2py book regarding backing up data from a web2py app hosted on google app engine. Only in the exporting data as csv section does it mention the idea of including a uuid or last_modified date. I think this should be mentioned prominently in the section on google app engine. The idea being that we do not want to say that web2py deploys easily on GAE in one breath, but then not let people know that if their models are not set up correctly, that they run the risk of not being able to easily backup and restore their data. It also may be that the bulkloader is a better alternative to the csv export within web2py’s appadmin, and so adding a uuid to ones models may not necessarily apply... or maybe it should just be good practice to do this. Either way, if there is a way where we could give others a heads up about this issue ahead of time, that would be great. On Jan 12, 12:21 am, Wikus van de Merwe <dupakrop...@googlemail.com> wrote: > To get your data as CSV from a GAE app you need to: > 1) enable remote api in app.yaml > 2) create bulkloader config file:> $GAE_DIR/appcfg.py > create_bulkloader_config --filename=config.yaml -e > > $EMAIL --url=https://$APP.appspot.com/remote_api> rm bulkloader* > > 3) edit config.yaml as needed (mainly export_transform functions for your > fields) > 4) dump specific entity kinds to CSV: > KINDS="abc efgh ijk" > for KIND in $KINDS: > do > rm $KIND.csv > $GAE_DIR/appcfg.py download_data --filename=$KIND.csv --kind=$KIND \ > --config_file=config.yaml -e $EMAIL > --url=https://$APP.appspot.com/remote_api > done > rm bulkloader* > > If you have large entities and the get error messages from the csv module: > 1) put the following into a file e.g. csv_limits.py: > import csv > csv.field_size_limit(1048576) > 2) import the file in config.yaml using: > - import: csv_limit > > You can also try the following to dump everything (not sure if this still > works with new releases): > $GAE_DIR/appcfg.py download_data -e $EMAIL --application=$APP > --url=https://$APP.appspot.com/remote_api --filename=dump.csv > > To upload the modified CSV data (after previous download) to GAE do: > for KIND in $KINDS > do > echo "x" | $GAE_DIR/appcfg.py upload_data --filename=$KIND.csv > --kind=$KIND \ > --config_file=config.yaml -e $EMAIL --passin > --url=https://$APP.appspot.com/remote_api \ > --application $APP --batch_size=1 > done > rm bulkloader* > > Hope this helps :)
