Hello,
The Appengine Datastore has been throwing lots of timeout and
contention errors these last couple of days (9/14, 9/15) due to
latency spikes. One error in particular has resulted in duplicate
items getting pushed to subscribers.
At the end of a push_event, after all the subscribers have been
notified a datastore Delete is performed. When this error occurs the
hub attempts to process the push_event again resulting in duplicate
items getting sent to subscribers. Even though the datastore has given
up performing
The push_event log entries include the normal debug messages
concluding with the following:
EventToDeliver complete: ... delivery_mode = normal
too much contention on these datastore entities. please try again.
Traceback (most recent call last):
File "/base/python_runtime/python_lib/versions/1/google/appengine/
ext/webapp/__init__.py", line 513, in __call__
handler.post(*groups)
File "/base/data/home/apps/bringit-pshb-test-1/1-0-alpha.
343674745300678296/main.py", line 648, in decorated
return func(myself, *args, **kwargs)
File "/base/data/home/apps/bringit-pshb-test-1/1-0-alpha.
343674745300678296/main.py", line 2672, in post
work.update(more_subscribers, failed_callbacks)
File "/base/data/home/apps/bringit-pshb-test-1/1-0-alpha.
343674745300678296/main.py", line 1544, in update
self.delete()
File "/base/python_runtime/python_lib/versions/1/google/appengine/
ext/db/__init__.py", line 940, in delete
datastore.Delete(self.key(), rpc=rpc)
File "/base/python_runtime/python_lib/versions/1/google/appengine/
api/datastore.py", line 397, in Delete
raise _ToDatastoreError(err)
TransactionFailedError: too much contention on these datastore
entities. please try again.
