Background:
Working with an app on GAE currently I've run across a slight issue
with uploading data into my app. I initially tried to download my
tables to CSV from my development machine then use simply re-upload
them into GAE. It's been a while since I tried that but it was timing
out.
Problem:
Because of that I decided to upload with bulkloader.py. I did my
first table (~2K entries) with out a hitch. My second Table is where
I'm having an issue. This table is about 24K entities. After
uploading all of the data trying to view it on the appadmin returns
the following error (web2py error ticket):
The table is "sqldb.item" if it matters.
-------------
Traceback (most recent call last):
File "/base/data/home/apps/aion-info/2.336452968225099838/gluon/
main.py", line 396, in wsgibase
session._try_store_in_db(request, response)
File "/base/data/home/apps/aion-info/2.336452968225099838/gluon/
globals.py", line 342, in _try_store_in_db
table._db(table.id == record_id).update(**dd)
File "/base/data/home/apps/aion-info/2.336452968225099838/gluon/
contrib/gql.py", line 698, in update
(item, tablename, fields) = self._getitem_exception()
File "/base/data/home/apps/aion-info/2.336452968225099838/gluon/
contrib/gql.py", line 632, in _getitem_exception
item = self._db[tablename]._tableobj.get_by_id(long(id))
File "/base/python_lib/versions/1/google/appengine/ext/db/
__init__.py", line 943, in get_by_id
return get(*keys)
File "/base/python_lib/versions/1/google/appengine/ext/db/
__init__.py", line 1123, in get
entities = datastore.Get(keys)
File "/base/python_lib/versions/1/google/appengine/api/
datastore.py", line 224, in Get
apiproxy_stub_map.MakeSyncCall('datastore_v3', 'Get', req, resp)
File "/base/python_lib/versions/1/google/appengine/api/
apiproxy_stub_map.py", line 72, in MakeSyncCall
apiproxy.MakeSyncCall(service, call, request, response)
File "/base/python_lib/versions/1/google/appengine/api/
apiproxy_stub_map.py", line 255, in MakeSyncCall
rpc.CheckSuccess()
File "/base/python_lib/versions/1/google/appengine/api/
apiproxy_rpc.py", line 111, in CheckSuccess
raise self.exception
CancelledError: The API call datastore_v3.Get() was explicitly
cancelled.
-----------------------------
Looking over two tables here are the things I've identified as
possible problems:
1 - This table has a 'double' in it. Not just int/string like the
working one. For GAE uploading I defined the double field as a GAE
"float" property. Is it possible web2py defines a 'double' as
something else on GAE? If so is there a mapping that would tell me
how to map web2py fields into GAE fields for uploading data?
2 - Searching the Data Viewer on GAE shows that due to a previously
failed data upload I now have multiple copies of some of the
entities. These copies the exact same data in the except for the ID/
Name fields. I would like to remove the duplicates sometime but that
wasn't my first concern. Would duplicate copies cause this problem?
I figured since the ID is unique it would just show up as another
entery in the table. I haven't set any of the entries to 'unique'.
3 - Both tables are set with migrate=False, specifically b/c I've
seperated the data upload and process from the data reading process.
I'm not even sure what web2py would do if I removed the migrate=false
flag. What is the recommendation here?
Inserting/deleting items from the GAE Datastore is very expensive in
CPU time, I messed with this yesterday and in a matter of a few
minutes (delete all, re-upload) was over my quota. Because of that
I'd really like to avoid having to change data in the Datastore as
much as I can. With 20K+ items you don't get a lot of 'do-overs'.
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~----------~----~----~----~------~----~------~--~---
