I'll follow up off the mailing list.
On Fri, Sep 17, 2010 at 10:04 AM, pkarp <[email protected]> wrote: > Hi Brett, > > This is a reasonable suggestion for subscribers particularly given the > error rates that I was seeing before this week. > > However our implementation of the hub is still experiencing high > latency (> 3000 ms) since monday on even the simple publish ping and > failure rates of 15 % on push_event. The errors are all related to > DataStore timeouts, contention, failures. > > /work/pull_feeds > 2164 24% > > /work/push_events > 932 15% > > / > 487 6.7% > > /work/subscription_cleanup > 27 4.8% > > /work/event_cleanup > 26 4.6% > > /work/poll_bootstrap > 3 2.4% > > If you have any way to determine why our instance is performing so > poorly this week on google appengine I would appreciate it. We are > currently live and preparing to find an alternative to this service. > > appID : bringit-pshb-test-1 > > > On Sep 17, 12:52 pm, Brett Slatkin <[email protected]> wrote: >> Yep that's been frustrating. Looks much better now. There's always >> going to be some errors in the hub like this which means we may repeat >> delivery for some feeds. That's why it's essential for subscribers to >> use feed entry IDs (in Atom and RSS2) to de-dupe feed entries. Such >> idempotent behavior means it's a waste of bandwidth but causes no >> other side-effects. >> >> >> >> On Wed, Sep 15, 2010 at 4:45 PM, pkarp <[email protected]> wrote: >> > Hello, >> >> > The Appengine Datastore has been throwing lots of timeout and >> > contention errors these last couple of days (9/14, 9/15) due to >> > latency spikes. One error in particular has resulted in duplicate >> > items getting pushed to subscribers. >> >> > At the end of a push_event, after all the subscribers have been >> > notified a datastore Delete is performed. When this error occurs the >> > hub attempts to process the push_event again resulting in duplicate >> > items getting sent to subscribers. Even though the datastore has given >> > up performing >> >> > The push_event log entries include the normal debug messages >> > concluding with the following: >> >> > EventToDeliver complete: ... delivery_mode = normal >> >> > too much contention on these datastore entities. please try again. >> > Traceback (most recent call last): >> > File "/base/python_runtime/python_lib/versions/1/google/appengine/ >> > ext/webapp/__init__.py", line 513, in __call__ >> > handler.post(*groups) >> > File "/base/data/home/apps/bringit-pshb-test-1/1-0-alpha. >> > 343674745300678296/main.py", line 648, in decorated >> > return func(myself, *args, **kwargs) >> > File "/base/data/home/apps/bringit-pshb-test-1/1-0-alpha. >> > 343674745300678296/main.py", line 2672, in post >> > work.update(more_subscribers, failed_callbacks) >> > File "/base/data/home/apps/bringit-pshb-test-1/1-0-alpha. >> > 343674745300678296/main.py", line 1544, in update >> > self.delete() >> > File "/base/python_runtime/python_lib/versions/1/google/appengine/ >> > ext/db/__init__.py", line 940, in delete >> > datastore.Delete(self.key(), rpc=rpc) >> > File "/base/python_runtime/python_lib/versions/1/google/appengine/ >> > api/datastore.py", line 397, in Delete >> > raise _ToDatastoreError(err) >> > TransactionFailedError: too much contention on these datastore >> > entities. please try again.
