Hi Ikai
I am using remote_api called from within plone. Because I could potentially get multiple concurrent calls from plone to push content I was creating a new connection (requiring new authentication ) on each request to push content from plone to app engine. Unfortunately when I started pushing large amounts of content each content entity being pushed was creating a new connection (invoking authentication). When I hit 200 of these in the space of say 10mins I would get hit by google infrastructure which would basically stop transactions. CheckSuccess didn't like getting a redirect to the authentication stuff. And I couldn't re-authenticate for about 10 mins anyway. Looks like I was running afoul of systems designed to prevent spamming etc.... Any other connection attempts from the same host using that user id would also fail. And none of this would be seen in the app engine logs, because the remote_api auth/connection was being redirected even before we got to appengine. I found that if I cached the connection and didn't continually re- authenticate I had no problems. Hope that makes sense. Rgds Tim On Dec 3, 3:35 am, "Ikai L (Google)" <[email protected]> wrote: > Tim, great find. What do you mean re-use connections? If you let me know, > I'll update the docs and open an issue to provide a better error message. > > On Tue, Dec 1, 2009 at 8:56 PM, Tim Hoffman <[email protected]> wrote: > > Ok > > > Got the the bottom of it all. > > > You need to re-use connections. If you make too many authenticated > > connections then googles higher up infrastructure > > shuts you down for about 10 min. > > > It does highlight some problems in the remote_api code as the error > > messages passed back are particularly informative unless you really > > dig. but at least I understand where the problem is. > > > T > > > On Dec 2, 11:31 am, Tim Hoffman <[email protected]> wrote: > > > OK > > > > Started debugging inside the apiproxy_rpc.py(113)CheckSuccess() > > > and what I believe is happening is correct. > > > > the rpc call MaekSyncCall is getting a redirect to a request to login > > > > apiproxy_rpc.py(113)CheckSuccess() > > > > The exception is a HTTPErrror which is a redirect (302) > > > to > > > > 'https://www.google.com/a/polytechnic.wa.edu.au/ServiceLogin? > > > service=ah&passive=true&continue=http://psc-prod1.appspot.com/_ah/login > > > %3Fcon.... stuff deleted', > > > > Obviously requesting me to log in again. > > > > So somewhere I am running afoul of higher layers in google that watch > > > what is going on. > > > I am going to need some help from google engineers to get to the > > > bottom of this one. > > > > Rgds > > > > Tim Hoffman > > > > On Dec 2, 11:04 am, Tim Hoffman <[email protected]> wrote: > > > > > Something I have observerd and can now prove repeatably > > > > > Once I get the error 302 inside the remote_api call. It takes > > > > somewhere between 2 and 10 mins before I can resubmit the transaction. > > > > During that that time, if I try the same call from another process (on > > > > the same box with the same auth details) the call still gets the error > > > > until such time as something in google lets it go. > > > > > It really is looking like I am being clobbered by some google watch > > > > dog. > > > > > T > > > > > On Dec 2, 9:58 am, Tim Hoffman <[email protected]> wrote: > > > > > > I am still getting this problem. > > > > > > One of the things I have noticed is the errors (302) are being > > > > > propogated by google infrastructure and not recorded any way > > > > > in the apps logs. > > > > > > My gut feel is that some other google infrastructure thinks I am > > > > > doing something I shouldn't requires me to re-authenticate but > > > > > the remote_api proxies can't deal with a 302 inside a Namerror in > > > > > CheckSuccess see stack below. > > > > > > Obviously what I am doing is different to the bulkloader client. > > > > > > Anyone got any suggestions on how I might go about getting to the > > > > > bottom of this. > > > > > > T > > > > > > 2009-12-02T01:55:49 INFO root Exception sending Rollback: > > > > > Traceback (most recent call last): > > > > > File "/opt/ktstudio/zope/instances/prod/swantafe.buildout/ > > > > > google_appengine/google/appengine/api/datastore.py", line 1989, in > > > > > RunInTransactionCustomRetries > > > > > tx.handle, resp) > > > > > File "/opt/ktstudio/zope/instances/prod/swantafe.buildout/ > > > > > google_appengine/google/appengine/api/apiproxy_stub_map.py", line 72, > > > > > in MakeSyncCall > > > > > apiproxy.MakeSyncCall(service, call, request, response) > > > > > File "/opt/ktstudio/zope/instances/prod/swantafe.buildout/ > > > > > google_appengine/google/appengine/api/apiproxy_stub_map.py", line > > 266, > > > > > in MakeSyncCall > > > > > rpc.CheckSuccess() > > > > > File "/opt/ktstudio/zope/instances/prod/swantafe.buildout/ > > > > > google_appengine/google/appengine/api/apiproxy_rpc.py", line 111, in > > > > > CheckSuccess > > > > > raise self.exception > > > > > NameError: global name 'txdata' is not defined > > > > > > ------ > > > > > 2009-12-02T01:55:49 ERROR Zope.SiteErrorLoghttp:// > > 10.8.0.134:49081/swan/portal_skins/custom/send_uos > > > > > Traceback (innermost last): > > > > > Module ZPublisher.Publish, line 119, in publish > > > > > Module ZPublisher.mapply, line 88, in mapply > > > > > Module ZPublisher.Publish, line 42, in call_object > > > > > Module Shared.DC.Scripts.Bindings, line 313, in __call__ > > > > > Module Shared.DC.Scripts.Bindings, line 350, in _bindAndExec > > > > > Module Products.PythonScripts.PythonScript, line 328, in _exec > > > > > Module None, line 25, in send_uos > > > > > - <PythonScript at /swan/portal_skins/custom/send_uos> > > > > > - Line 25 > > > > > Module None, line 11, in push > > > > > - <PythonScript at /swan/portal_skins/custom/send_uos> > > > > > - Line 11 > > > > > Module Products.PlonePSC.tools.psc_manager, line 639, in > > > > > push_to_appengine > > > > > Module google.appengine.ext.db, line 1064, in get_or_insert > > > > > Module google.appengine.api.datastore, line 1884, in > > > > > RunInTransaction > > > > > Module google.appengine.api.datastore, line 1982, in > > > > > RunInTransactionCustomRetries > > > > > Module google.appengine.ext.db, line 1059, in txn > > > > > Module google.appengine.ext.db, line 981, in get_by_key_name > > > > > Module google.appengine.ext.db, line 1180, in get > > > > > Module google.appengine.api.datastore, line 234, in Get > > > > > Module google.appengine.api.apiproxy_stub_map, line 72, in > > > > > MakeSyncCall > > > > > Module google.appengine.api.apiproxy_stub_map, line 266, in > > > > > MakeSyncCall > > > > > Module google.appengine.api.apiproxy_rpc, line 111, in CheckSuccess > > > > > HTTPError: HTTP Error 302: Found > > > > > > On Nov 18, 7:14 am, Tim Hoffman <[email protected]> wrote: > > > > > > > Hi Mathew > > > > > > > Appid sent in mail. (Need to keep the site relatively unpublic for > > > > > > about 1 more week) > > > > > > > Here is some additional information. > > > > > > > I encountered this problem last week - see this threadhttp:// > > groups.google.com.au/group/google-appengine/browse_thread/thre... > > > > > > > I found then that I would hit the barrier at exactly 200 puts > > > > > > > In this round of tests (will be our prod instance) I was doing the > > > > > > transactions a little different and pushing via from two seperate > > > > > > connections from the same ec2 instance > > > > > > (rather than from my home connection.) > > > > > > > It appears that again I was hitting the problem and a combined > > total > > > > > > (across the 2 connections ) of 200 puts (not 100% certain on this > > as > > > > > > I wasn't > > > > > > counting puts because some of the records had lots of puts - and I > > was > > > > > > tracking documents not puts). > > > > > > > As I mentioned in my email I am not retrying transactions, and > > there > > > > > > is no point becuase the minute I get this particular error I can't > > get > > > > > > anything else through to appengine over the remote_api > > > > > > (not even a get) > > > > > > > Rgds > > > > > > > Tim > > > > > > > On Nov 18, 3:57 am, Matthew Blain <[email protected]> > > wrote: > > > > > > > > Hi Tim, > > > > > > > Can you tell me the appid so we can look into this further? > > > > > > > Note that the transaction error is interesting--are you seeing > > lots of > > > > > > > transaction failures and thus retrying the same call repeatedly? > > > > > > > > --Matthew > > > > > > > > On Nov 17, 5:54 am, Tim Hoffman <[email protected]> wrote: > > > > > > > > > Hi > > > > > > > > > I am pushing about 20,000 entities from Plone to Appengine via > > the > > > > > > > > remote_api > > > > > > > > and every so often (200-300 entities (my guess is it is exactly > > 200) I > > > > > > > > start getting reponses from the remote_api > > > > > > > > that say > > > > > > > > > Please go tohttps:// > >www.google.com/accounts/DisplayUnlockCaptcha > > > > > > > > and verify you are a human. Then try again. > > > > > > > > 2009-11-17 13:47:26 INFO root Exception sending Rollback: > > > > > > > > Traceback (most recent call last): > > > > > > > > File "/opt/ktstudio/zope/instances/prod/swantafe.buildout/ > > > > > > > > google_appengine/google/appengine/api/datastore.py", line 1989, > > in > > > > > > > > RunInTransactionCustomRetries > > > > > > > > tx.handle, resp) > > > > > > > > File "/opt/ktstudio/zope/instances/prod/swantafe.buildout/ > > > > > > > > google_appengine/google/appengine/api/apiproxy_stub_map.py", > > line 72, > > > > > > > > in MakeSyncCall > > > > > > > > apiproxy.MakeSyncCall(service, call, request, response) > > > > > > > > File "/opt/ktstudio/zope/instances/prod/swantafe.buildout/ > > > > > > > > google_appengine/google/appengine/api/apiproxy_stub_map.py", > > line 266, > > > > > > > > in MakeSyncCall > > > > > > > > rpc.CheckSuccess() > > > > > > > > File "/opt/ktstudio/zope/instances/prod/swantafe.buildout/ > > > > > > > > google_appengine/google/appengine/api/apiproxy_rpc.py", line > > 111, in > > > > > > > > CheckSuccess > > > > > > > > raise self.exception > > > > > > > > NameError: global name 'txdata' is not defined > > > > > > > > > Surley repeated use of the remote_api shouldn't make our google > > > > > > > > overlords think that I am some evil spamming robot. (What I am > > doing > > > > > > > > is robot like, but it is authenticated and it is my application > > and I > > > > > > > > am not spamming anyone ;-.) > > > > > > > > > Is there some sort of limit to the number of calls that can be > > made > > > > > > > > before we have to reconnect or create a new remote_api > > connection ? > > > > > > > > As obviously something is triggering this reponse from google. > > > > > > > > > Rgds > > > > > > > > > Tim > > > -- > > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > ... > > read more » -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
