I have an app that once a day does a big data processing task.
Every now and then it would throw a datastore timeout error. But now it's
throwing them constantly.
I thought maybe my data had tripped over some limit on how much you can read,
but I just added some instrumentation and it's only reading less than half of
the entities. If I was tripping over an undocumented limit, I'd think it would
read all most all of them (since only a few get added each day).
Basically, the code is simply:
for h in HitModel.all():
(do collect up info about h)
and there are about 85K HitModel objects in the database. It's dying after
reading 35,000 of them (which takes about a minute).
It's on HR data store. Still on Python 2.5.
App ID is "kaon-log"
The error I'm getting is:
2014-10-07 11:16:46.925
The datastore operation timed out, or the data was temporarily unavailable.
Traceback (most recent call last):
File
"/base/data/home/runtimes/python/python_lib/versions/1/google/appengine/ext/webapp/_webapp25.py",
line 714, in __call__
handler.get(*groups)
File "/base/data/home/apps/s~kaon-log/33.379217403803985923/main.py", line
648, in get
for h in HitModel.all():
File
"/base/data/home/runtimes/python/python_lib/versions/1/google/appengine/ext/db/__init__.py",
line 2326, in next
return self.__model_class.from_entity(self.__iterator.next())
File
"/base/data/home/runtimes/python/python_lib/versions/1/google/appengine/datastore/datastore_query.py",
line 3091, in next
next_batch = self.__batcher.next_batch(Batcher.AT_LEAST_OFFSET)
File
"/base/data/home/runtimes/python/python_lib/versions/1/google/appengine/datastore/datastore_query.py",
line 2977, in next_batch
batch = self.__next_batch.get_result()
File
"/base/data/home/runtimes/python/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py",
line 612, in get_result
return self.__get_result_hook(self)
File
"/base/data/home/runtimes/python/python_lib/versions/1/google/appengine/datastore/datastore_query.py",
line 2710, in __query_result_hook
self._batch_shared.conn.check_rpc_success(rpc)
File
"/base/data/home/runtimes/python/python_lib/versions/1/google/appengine/datastore/datastore_rpc.py",
line 1333, in check_rpc_success
raise _ToDatastoreError(err)
Timeout: The datastore operation timed out, or the data was temporarily
unavailable.
Any ideas?
(Breaking this up into multiple tasks would be really hard.)
-Joshua
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at http://groups.google.com/group/google-appengine.
For more options, visit https://groups.google.com/d/optout.
