I have seen this error as well and had to change my code. My situation is python 2.7/HRD/db.Query() in a module with manual scaling.
q = db.Query(…) … for ent in q.run(): do stuff The iteration goes well for a large number of entities and then gives up in a similar way. Something seems to timeout for long lived queries. Needless to say it works fine on the dev server but there of course I do not have so much data. Breaking it down with cursors or equivalent works fine and this is what I am doing as a work around. I did not even bother to see if there is an issue for it but I would happily star it if there is one. PK http://www.gae123.com On October 7, 2014 at 8:26:11 AM, Joshua Smith ([email protected]) wrote: I have an app that once a day does a big data processing task. Every now and then it would throw a datastore timeout error. But now it’s throwing them constantly. I thought maybe my data had tripped over some limit on how much you can read, but I just added some instrumentation and it’s only reading less than half of the entities. If I was tripping over an undocumented limit, I’d think it would read all most all of them (since only a few get added each day). Basically, the code is simply: for h in HitModel.all(): (do collect up info about h) and there are about 85K HitModel objects in the database. It’s dying after reading 35,000 of them (which takes about a minute). It’s on HR data store. Still on Python 2.5. App ID is “kaon-log” The error I’m getting is: 2014-10-07 11:16:46.925 The datastore operation timed out, or the data was temporarily unavailable. Traceback (most recent call last): File "/base/data/home/runtimes/python/python_lib/versions/1/google/appengine/ext/webapp/_webapp25.py", line 714, in __call__ handler.get(*groups) File "/base/data/home/apps/s~kaon-log/33.379217403803985923/main.py", line 648, in get for h in HitModel.all(): File "/base/data/home/runtimes/python/python_lib/versions/1/google/appengine/ext/db/__init__.py", line 2326, in next return self.__model_class.from_entity(self.__iterator.next()) File "/base/data/home/runtimes/python/python_lib/versions/1/google/appengine/datastore/datastore_query.py", line 3091, in next next_batch = self.__batcher.next_batch(Batcher.AT_LEAST_OFFSET) File "/base/data/home/runtimes/python/python_lib/versions/1/google/appengine/datastore/datastore_query.py", line 2977, in next_batch batch = self.__next_batch.get_result() File "/base/data/home/runtimes/python/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 612, in get_result return self.__get_result_hook(self) File "/base/data/home/runtimes/python/python_lib/versions/1/google/appengine/datastore/datastore_query.py", line 2710, in __query_result_hook self._batch_shared.conn.check_rpc_success(rpc) File "/base/data/home/runtimes/python/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 1333, in check_rpc_success raise _ToDatastoreError(err) Timeout: The datastore operation timed out, or the data was temporarily unavailable. Any ideas? (Breaking this up into multiple tasks would be really hard.) -Joshua -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
