Ok Got the the bottom of it all.
You need to re-use connections. If you make too many authenticated connections then googles higher up infrastructure shuts you down for about 10 min. It does highlight some problems in the remote_api code as the error messages passed back are particularly informative unless you really dig. but at least I understand where the problem is. T On Dec 2, 11:31 am, Tim Hoffman <[email protected]> wrote: > OK > > Started debugging inside the apiproxy_rpc.py(113)CheckSuccess() > and what I believe is happening is correct. > > the rpc call MaekSyncCall is getting a redirect to a request to login > > apiproxy_rpc.py(113)CheckSuccess() > > The exception is a HTTPErrror which is a redirect (302) > to > > 'https://www.google.com/a/polytechnic.wa.edu.au/ServiceLogin? > service=ah&passive=true&continue=http://psc-prod1.appspot.com/_ah/login > %3Fcon.... stuff deleted', > > Obviously requesting me to log in again. > > So somewhere I am running afoul of higher layers in google that watch > what is going on. > I am going to need some help from google engineers to get to the > bottom of this one. > > Rgds > > Tim Hoffman > > On Dec 2, 11:04 am, Tim Hoffman <[email protected]> wrote: > > > Something I have observerd and can now prove repeatably > > > Once I get the error 302 inside the remote_api call. It takes > > somewhere between 2 and 10 mins before I can resubmit the transaction. > > During that that time, if I try the same call from another process (on > > the same box with the same auth details) the call still gets the error > > until such time as something in google lets it go. > > > It really is looking like I am being clobbered by some google watch > > dog. > > > T > > > On Dec 2, 9:58 am, Tim Hoffman <[email protected]> wrote: > > > > I am still getting this problem. > > > > One of the things I have noticed is the errors (302) are being > > > propogated by google infrastructure and not recorded any way > > > in the apps logs. > > > > My gut feel is that some other google infrastructure thinks I am > > > doing something I shouldn't requires me to re-authenticate but > > > the remote_api proxies can't deal with a 302 inside a Namerror in > > > CheckSuccess see stack below. > > > > Obviously what I am doing is different to the bulkloader client. > > > > Anyone got any suggestions on how I might go about getting to the > > > bottom of this. > > > > T > > > > 2009-12-02T01:55:49 INFO root Exception sending Rollback: > > > Traceback (most recent call last): > > > File "/opt/ktstudio/zope/instances/prod/swantafe.buildout/ > > > google_appengine/google/appengine/api/datastore.py", line 1989, in > > > RunInTransactionCustomRetries > > > tx.handle, resp) > > > File "/opt/ktstudio/zope/instances/prod/swantafe.buildout/ > > > google_appengine/google/appengine/api/apiproxy_stub_map.py", line 72, > > > in MakeSyncCall > > > apiproxy.MakeSyncCall(service, call, request, response) > > > File "/opt/ktstudio/zope/instances/prod/swantafe.buildout/ > > > google_appengine/google/appengine/api/apiproxy_stub_map.py", line 266, > > > in MakeSyncCall > > > rpc.CheckSuccess() > > > File "/opt/ktstudio/zope/instances/prod/swantafe.buildout/ > > > google_appengine/google/appengine/api/apiproxy_rpc.py", line 111, in > > > CheckSuccess > > > raise self.exception > > > NameError: global name 'txdata' is not defined > > > > ------ > > > 2009-12-02T01:55:49 ERROR > > > Zope.SiteErrorLoghttp://10.8.0.134:49081/swan/portal_skins/custom/send_uos > > > Traceback (innermost last): > > > Module ZPublisher.Publish, line 119, in publish > > > Module ZPublisher.mapply, line 88, in mapply > > > Module ZPublisher.Publish, line 42, in call_object > > > Module Shared.DC.Scripts.Bindings, line 313, in __call__ > > > Module Shared.DC.Scripts.Bindings, line 350, in _bindAndExec > > > Module Products.PythonScripts.PythonScript, line 328, in _exec > > > Module None, line 25, in send_uos > > > - <PythonScript at /swan/portal_skins/custom/send_uos> > > > - Line 25 > > > Module None, line 11, in push > > > - <PythonScript at /swan/portal_skins/custom/send_uos> > > > - Line 11 > > > Module Products.PlonePSC.tools.psc_manager, line 639, in > > > push_to_appengine > > > Module google.appengine.ext.db, line 1064, in get_or_insert > > > Module google.appengine.api.datastore, line 1884, in > > > RunInTransaction > > > Module google.appengine.api.datastore, line 1982, in > > > RunInTransactionCustomRetries > > > Module google.appengine.ext.db, line 1059, in txn > > > Module google.appengine.ext.db, line 981, in get_by_key_name > > > Module google.appengine.ext.db, line 1180, in get > > > Module google.appengine.api.datastore, line 234, in Get > > > Module google.appengine.api.apiproxy_stub_map, line 72, in > > > MakeSyncCall > > > Module google.appengine.api.apiproxy_stub_map, line 266, in > > > MakeSyncCall > > > Module google.appengine.api.apiproxy_rpc, line 111, in CheckSuccess > > > HTTPError: HTTP Error 302: Found > > > > On Nov 18, 7:14 am, Tim Hoffman <[email protected]> wrote: > > > > > Hi Mathew > > > > > Appid sent in mail. (Need to keep the site relatively unpublic for > > > > about 1 more week) > > > > > Here is some additional information. > > > > > I encountered this problem last week - see this > > > > threadhttp://groups.google.com.au/group/google-appengine/browse_thread/thre... > > > > > I found then that I would hit the barrier at exactly 200 puts > > > > > In this round of tests (will be our prod instance) I was doing the > > > > transactions a little different and pushing via from two seperate > > > > connections from the same ec2 instance > > > > (rather than from my home connection.) > > > > > It appears that again I was hitting the problem and a combined total > > > > (across the 2 connections ) of 200 puts (not 100% certain on this as > > > > I wasn't > > > > counting puts because some of the records had lots of puts - and I was > > > > tracking documents not puts). > > > > > As I mentioned in my email I am not retrying transactions, and there > > > > is no point becuase the minute I get this particular error I can't get > > > > anything else through to appengine over the remote_api > > > > (not even a get) > > > > > Rgds > > > > > Tim > > > > > On Nov 18, 3:57 am, Matthew Blain <[email protected]> wrote: > > > > > > Hi Tim, > > > > > Can you tell me the appid so we can look into this further? > > > > > Note that the transaction error is interesting--are you seeing lots of > > > > > transaction failures and thus retrying the same call repeatedly? > > > > > > --Matthew > > > > > > On Nov 17, 5:54 am, Tim Hoffman <[email protected]> wrote: > > > > > > > Hi > > > > > > > I am pushing about 20,000 entities from Plone to Appengine via the > > > > > > remote_api > > > > > > and every so often (200-300 entities (my guess is it is exactly > > > > > > 200) I > > > > > > start getting reponses from the remote_api > > > > > > that say > > > > > > > Please go tohttps://www.google.com/accounts/DisplayUnlockCaptcha > > > > > > and verify you are a human. Then try again. > > > > > > 2009-11-17 13:47:26 INFO root Exception sending Rollback: > > > > > > Traceback (most recent call last): > > > > > > File "/opt/ktstudio/zope/instances/prod/swantafe.buildout/ > > > > > > google_appengine/google/appengine/api/datastore.py", line 1989, in > > > > > > RunInTransactionCustomRetries > > > > > > tx.handle, resp) > > > > > > File "/opt/ktstudio/zope/instances/prod/swantafe.buildout/ > > > > > > google_appengine/google/appengine/api/apiproxy_stub_map.py", line > > > > > > 72, > > > > > > in MakeSyncCall > > > > > > apiproxy.MakeSyncCall(service, call, request, response) > > > > > > File "/opt/ktstudio/zope/instances/prod/swantafe.buildout/ > > > > > > google_appengine/google/appengine/api/apiproxy_stub_map.py", line > > > > > > 266, > > > > > > in MakeSyncCall > > > > > > rpc.CheckSuccess() > > > > > > File "/opt/ktstudio/zope/instances/prod/swantafe.buildout/ > > > > > > google_appengine/google/appengine/api/apiproxy_rpc.py", line 111, in > > > > > > CheckSuccess > > > > > > raise self.exception > > > > > > NameError: global name 'txdata' is not defined > > > > > > > Surley repeated use of the remote_api shouldn't make our google > > > > > > overlords think that I am some evil spamming robot. (What I am > > > > > > doing > > > > > > is robot like, but it is authenticated and it is my application and > > > > > > I > > > > > > am not spamming anyone ;-.) > > > > > > > Is there some sort of limit to the number of calls that can be made > > > > > > before we have to reconnect or create a new remote_api connection ? > > > > > > As obviously something is triggering this reponse from google. > > > > > > > Rgds > > > > > > > Tim -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
