Hi, We have a large application (TapFishWebService) which is used by more than 3 millions users. This application contains a table Device whose size is about 3 million entities and about 2.6GB. I want to perform some complex quires over it but due to limitation of gql and slow indexing it is not possible practically (I ran one query involving some datetime and I took more than three days to index it).
Now I am in process of downloading this table in CSV format so that I can translate it into some relational database for better querying. I am following the instructions given at :http://code.google.com/ appengine/docs/python/tools/uploadingdata.html It was all working fine but unfortunately my system hanged while only about 2GB (out of 2.6) download was completed. And when I tried to resume the process it is giving me ERROR: "BadArgumentError: kind should be a string; received 3 (a int):" The complete log is given below. The exporter config file is also given after the ERROR LOG. Thanks and regards, Atif ERROR LOG: ====================================== D:\TapFish\TApFishWebService>appcfg.py download_data -- config_file=format.py --filename=Device.csv --kind=Device -- batch_size=100 --db_filename=dbprogress --re sult_db_filename=resultprogress . C:\Program Files\Google\google_appengine\appcfg.py:41: DeprecationWarning: the sha module is deprecated; use the hashlib module instead os.path.join(DIR_PATH, 'lib', 'antlr3'), C:\Program Files\Google\google_appengine\google\appengine\tools \dev_appserver_login.py:33: DeprecationWarning: the md5 module is deprecated; use hashlib instead import md5 Application: tapfishwebservice; version: 115. Downloading data records. [INFO ] Logging to bulkloader-log-20100520.142808 [INFO ] Throttling transfers: [INFO ] Bandwidth: 250000 bytes/second [INFO ] HTTP connections: 8/second [INFO ] Entities inserted/fetched/modified: 20/second [INFO ] Batch Size: 100 [INFO ] Opening database: dbprogress C:\Program Files\Google\google_appengine\google\appengine\tools \bulkloader.py:1614: DeprecationWarning: BaseException.message has been deprecated as of Python 2 .6 if 'already exists' not in e.message: C:\Program Files\Google\google_appengine\google\appengine\tools \bulkloader.py:1621: DeprecationWarning: BaseException.message has been deprecated as of Python 2 .6 if 'already exists' not in e.message: C:\Program Files\Google\google_appengine\google\appengine\tools \bulkloader.py:1636: DeprecationWarning: BaseException.message has been deprecated as of Python 2 .6 if 'already exists' not in e.message: [INFO ] Opening database: resultprogress [INFO ] Connecting to tapfishwebservice.appspot.com/remote_api [INFO ] Downloading kinds: ['Device'] [ERROR ] [WorkerThread-0] WorkerThread: Traceback (most recent call last): File "C:\Program Files\Google\google_appengine\google\appengine\tools \adaptive_thread_pool.py", line 150, in WorkOnItems status, instruction = item.PerformWork(self.__thread_pool) File "C:\Program Files\Google\google_appengine\google\appengine\tools \bulkloader.py", line 684, in PerformWork transfer_time = self._TransferItem(thread_pool) File "C:\Program Files\Google\google_appengine\google\appengine\tools \bulkloader.py", line 1071, in _TransferItem self, retry_parallel=self.first) File "C:\Program Files\Google\google_appengine\google\appengine\tools \bulkloader.py", line 1329, in GetEntities kind, keys_only=keys_only) File "C:\Program Files\Google\google_appengine\google\appengine\ext \key_range\__init__.py", line 201, in make_directed_datastore_query query = datastore.Query(kind, keys_only=keys_only) File "C:\Program Files\Google\google_appengine\google\appengine\api \datastore.py", line 933, in __init__ datastore_errors.BadArgumentError) File "C:\Program Files\Google\google_appengine\google\appengine\api \datastore_types.py", line 105, in ValidateString (name, value, typename(value))) BadArgumentError: kind should be a string; received 3 (a int): [INFO ] [WorkerThread-1] Backing off due to errors: 1.0 seconds [INFO ] An error occurred. Shutting down... [ERROR ] Error in WorkerThread-0: kind should be a string; received 3 (a int): CONFIG FILE: ========================================== from google.appengine.ext import db from google.appengine.tools import bulkloader import tapfishservice class DeviceExporter(bulkloader.Exporter): def __init__(self): bulkloader.Exporter.__init__(self, 'Device', [('udid', str, ""), ('creationDate', str, ""), ('lastConnected', str, ""), ('numTimesConnected', str, ""), ('offersCurrencyCoinsAwarded', str, ""), ('offersCurrencyBucksAwarded', str, ""), ('purchasedCoins', str, ""), ('purchasedBucks', str, ""), ('deviceType', str, ""), ('deviceOS', str, ""), ('gameVersion', str, ""), ('originalGameVersion', str, ""), ('offersPageClickedNumTimes', str, ""), ('offersClickedNumTimes', str, ""), ('offersCompletedNumTimes', str, ""), ('superUser', str, ""), ('game', str, ""), ('TFinstalled', str, ""), ('TF1installed', str, ""), ('TF2installed', str, ""), ('TF3installed', str, ""), ('TF4installed', str, ""), ('TF5installed', str, ""), ('TF6installed', str, ""), ('TF8installed', str, ""), ('TF10installed', str, ""), ('TF15installed', str, ""), ('TF20installed', str, ""), ('TF25installed', str, ""), ('TF30installed', str, ""), ('TF40installed', str, ""), ('TF50installed', str, ""), ('TFfree5installed', str, ""), ('TFfree6installed', str, ""), ('TFfree8installed', str, ""), ('TFfree10installed', str, ""), ('TFfree12installed', str, ""), ('needsCounter', str, ""), ('counterDateString', str, "") ]) exporters = [DeviceExporter] -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
