Tim, Ah, great find. If I'm to understand you correctly, you were using a different process off App Engine to log into App Engine, hitting the login page N times in a span of M minutes. This resulted in you not being able to authenticate anymore, but after you cached the auth cookie, it started working fine.
On Thu, Dec 3, 2009 at 3:01 AM, Tim Hoffman <[email protected]> wrote: > Hi Ikai > > > I am using remote_api called from within plone. > > Because I could potentially get multiple concurrent calls from plone > to push content I was creating a new connection > (requiring new authentication ) on each request to push content from > plone to app engine. > > Unfortunately when I started pushing large amounts of content each > content entity being pushed was creating a new > connection (invoking authentication). When I hit 200 of these in the > space of say 10mins I would get hit by google infrastructure > which would basically stop transactions. CheckSuccess didn't like > getting a redirect to the authentication stuff. > And I couldn't re-authenticate for about 10 mins anyway. Looks like I > was running afoul of systems designed to prevent > spamming etc.... Any other connection attempts from the same host > using that user id would also fail. > And none of this would be seen in the app engine logs, because the > remote_api auth/connection was being redirected > even before we got to appengine. > > I found that if I cached the connection and didn't continually re- > authenticate I had no problems. > > Hope that makes sense. > > Rgds > > Tim > On Dec 3, 3:35 am, "Ikai L (Google)" <[email protected]> wrote: > > Tim, great find. What do you mean re-use connections? If you let me know, > > I'll update the docs and open an issue to provide a better error message. > > > > On Tue, Dec 1, 2009 at 8:56 PM, Tim Hoffman <[email protected]> wrote: > > > Ok > > > > > Got the the bottom of it all. > > > > > You need to re-use connections. If you make too many authenticated > > > connections then googles higher up infrastructure > > > shuts you down for about 10 min. > > > > > It does highlight some problems in the remote_api code as the error > > > messages passed back are particularly informative unless you really > > > dig. but at least I understand where the problem is. > > > > > T > > > > > On Dec 2, 11:31 am, Tim Hoffman <[email protected]> wrote: > > > > OK > > > > > > Started debugging inside the apiproxy_rpc.py(113)CheckSuccess() > > > > and what I believe is happening is correct. > > > > > > the rpc call MaekSyncCall is getting a redirect to a request to login > > > > > > apiproxy_rpc.py(113)CheckSuccess() > > > > > > The exception is a HTTPErrror which is a redirect (302) > > > > to > > > > > > 'https://www.google.com/a/polytechnic.wa.edu.au/ServiceLogin? > > > > service=ah&passive=true&continue= > http://psc-prod1.appspot.com/_ah/login > > > > %3Fcon.... stuff deleted', > > > > > > Obviously requesting me to log in again. > > > > > > So somewhere I am running afoul of higher layers in google that watch > > > > what is going on. > > > > I am going to need some help from google engineers to get to the > > > > bottom of this one. > > > > > > Rgds > > > > > > Tim Hoffman > > > > > > On Dec 2, 11:04 am, Tim Hoffman <[email protected]> wrote: > > > > > > > Something I have observerd and can now prove repeatably > > > > > > > Once I get the error 302 inside the remote_api call. It takes > > > > > somewhere between 2 and 10 mins before I can resubmit the > transaction. > > > > > During that that time, if I try the same call from another process > (on > > > > > the same box with the same auth details) the call still gets the > error > > > > > until such time as something in google lets it go. > > > > > > > It really is looking like I am being clobbered by some google watch > > > > > dog. > > > > > > > T > > > > > > > On Dec 2, 9:58 am, Tim Hoffman <[email protected]> wrote: > > > > > > > > I am still getting this problem. > > > > > > > > One of the things I have noticed is the errors (302) are being > > > > > > propogated by google infrastructure and not recorded any way > > > > > > in the apps logs. > > > > > > > > My gut feel is that some other google infrastructure thinks I am > > > > > > doing something I shouldn't requires me to re-authenticate but > > > > > > the remote_api proxies can't deal with a 302 inside a Namerror in > > > > > > CheckSuccess see stack below. > > > > > > > > Obviously what I am doing is different to the bulkloader client. > > > > > > > > Anyone got any suggestions on how I might go about getting to the > > > > > > bottom of this. > > > > > > > > T > > > > > > > > 2009-12-02T01:55:49 INFO root Exception sending Rollback: > > > > > > Traceback (most recent call last): > > > > > > File "/opt/ktstudio/zope/instances/prod/swantafe.buildout/ > > > > > > google_appengine/google/appengine/api/datastore.py", line 1989, > in > > > > > > RunInTransactionCustomRetries > > > > > > tx.handle, resp) > > > > > > File "/opt/ktstudio/zope/instances/prod/swantafe.buildout/ > > > > > > google_appengine/google/appengine/api/apiproxy_stub_map.py", line > 72, > > > > > > in MakeSyncCall > > > > > > apiproxy.MakeSyncCall(service, call, request, response) > > > > > > File "/opt/ktstudio/zope/instances/prod/swantafe.buildout/ > > > > > > google_appengine/google/appengine/api/apiproxy_stub_map.py", line > > > 266, > > > > > > in MakeSyncCall > > > > > > rpc.CheckSuccess() > > > > > > File "/opt/ktstudio/zope/instances/prod/swantafe.buildout/ > > > > > > google_appengine/google/appengine/api/apiproxy_rpc.py", line 111, > in > > > > > > CheckSuccess > > > > > > raise self.exception > > > > > > NameError: global name 'txdata' is not defined > > > > > > > > ------ > > > > > > 2009-12-02T01:55:49 ERROR Zope.SiteErrorLoghttp:// > > > 10.8.0.134:49081/swan/portal_skins/custom/send_uos > > > > > > Traceback (innermost last): > > > > > > Module ZPublisher.Publish, line 119, in publish > > > > > > Module ZPublisher.mapply, line 88, in mapply > > > > > > Module ZPublisher.Publish, line 42, in call_object > > > > > > Module Shared.DC.Scripts.Bindings, line 313, in __call__ > > > > > > Module Shared.DC.Scripts.Bindings, line 350, in _bindAndExec > > > > > > Module Products.PythonScripts.PythonScript, line 328, in _exec > > > > > > Module None, line 25, in send_uos > > > > > > - <PythonScript at /swan/portal_skins/custom/send_uos> > > > > > > - Line 25 > > > > > > Module None, line 11, in push > > > > > > - <PythonScript at /swan/portal_skins/custom/send_uos> > > > > > > - Line 11 > > > > > > Module Products.PlonePSC.tools.psc_manager, line 639, in > > > > > > push_to_appengine > > > > > > Module google.appengine.ext.db, line 1064, in get_or_insert > > > > > > Module google.appengine.api.datastore, line 1884, in > > > > > > RunInTransaction > > > > > > Module google.appengine.api.datastore, line 1982, in > > > > > > RunInTransactionCustomRetries > > > > > > Module google.appengine.ext.db, line 1059, in txn > > > > > > Module google.appengine.ext.db, line 981, in get_by_key_name > > > > > > Module google.appengine.ext.db, line 1180, in get > > > > > > Module google.appengine.api.datastore, line 234, in Get > > > > > > Module google.appengine.api.apiproxy_stub_map, line 72, in > > > > > > MakeSyncCall > > > > > > Module google.appengine.api.apiproxy_stub_map, line 266, in > > > > > > MakeSyncCall > > > > > > Module google.appengine.api.apiproxy_rpc, line 111, in > CheckSuccess > > > > > > HTTPError: HTTP Error 302: Found > > > > > > > > On Nov 18, 7:14 am, Tim Hoffman <[email protected]> wrote: > > > > > > > > > Hi Mathew > > > > > > > > > Appid sent in mail. (Need to keep the site relatively unpublic > for > > > > > > > about 1 more week) > > > > > > > > > Here is some additional information. > > > > > > > > > I encountered this problem last week - see this threadhttp:// > > > groups.google.com.au/group/google-appengine/browse_thread/thre... > > > > > > > > > I found then that I would hit the barrier at exactly 200 puts > > > > > > > > > In this round of tests (will be our prod instance) I was doing > the > > > > > > > transactions a little different and pushing via from two > seperate > > > > > > > connections from the same ec2 instance > > > > > > > (rather than from my home connection.) > > > > > > > > > It appears that again I was hitting the problem and a combined > > > total > > > > > > > (across the 2 connections ) of 200 puts (not 100% certain on > this > > > as > > > > > > > I wasn't > > > > > > > counting puts because some of the records had lots of puts - > and I > > > was > > > > > > > tracking documents not puts). > > > > > > > > > As I mentioned in my email I am not retrying transactions, and > > > there > > > > > > > is no point becuase the minute I get this particular error I > can't > > > get > > > > > > > anything else through to appengine over the remote_api > > > > > > > (not even a get) > > > > > > > > > Rgds > > > > > > > > > Tim > > > > > > > > > On Nov 18, 3:57 am, Matthew Blain <[email protected]> > > > wrote: > > > > > > > > > > Hi Tim, > > > > > > > > Can you tell me the appid so we can look into this further? > > > > > > > > Note that the transaction error is interesting--are you > seeing > > > lots of > > > > > > > > transaction failures and thus retrying the same call > repeatedly? > > > > > > > > > > --Matthew > > > > > > > > > > On Nov 17, 5:54 am, Tim Hoffman <[email protected]> wrote: > > > > > > > > > > > Hi > > > > > > > > > > > I am pushing about 20,000 entities from Plone to Appengine > via > > > the > > > > > > > > > remote_api > > > > > > > > > and every so often (200-300 entities (my guess is it is > exactly > > > 200) I > > > > > > > > > start getting reponses from the remote_api > > > > > > > > > that say > > > > > > > > > > > Please go tohttps:// > > >www.google.com/accounts/DisplayUnlockCaptcha > > > > > > > > > and verify you are a human. Then try again. > > > > > > > > > 2009-11-17 13:47:26 INFO root Exception sending Rollback: > > > > > > > > > Traceback (most recent call last): > > > > > > > > > File > "/opt/ktstudio/zope/instances/prod/swantafe.buildout/ > > > > > > > > > google_appengine/google/appengine/api/datastore.py", line > 1989, > > > in > > > > > > > > > RunInTransactionCustomRetries > > > > > > > > > tx.handle, resp) > > > > > > > > > File > "/opt/ktstudio/zope/instances/prod/swantafe.buildout/ > > > > > > > > > > google_appengine/google/appengine/api/apiproxy_stub_map.py", > > > line 72, > > > > > > > > > in MakeSyncCall > > > > > > > > > apiproxy.MakeSyncCall(service, call, request, response) > > > > > > > > > File > "/opt/ktstudio/zope/instances/prod/swantafe.buildout/ > > > > > > > > > > google_appengine/google/appengine/api/apiproxy_stub_map.py", > > > line 266, > > > > > > > > > in MakeSyncCall > > > > > > > > > rpc.CheckSuccess() > > > > > > > > > File > "/opt/ktstudio/zope/instances/prod/swantafe.buildout/ > > > > > > > > > google_appengine/google/appengine/api/apiproxy_rpc.py", > line > > > 111, in > > > > > > > > > CheckSuccess > > > > > > > > > raise self.exception > > > > > > > > > NameError: global name 'txdata' is not defined > > > > > > > > > > > Surley repeated use of the remote_api shouldn't make our > google > > > > > > > > > overlords think that I am some evil spamming robot. (What > I am > > > doing > > > > > > > > > is robot like, but it is authenticated and it is my > application > > > and I > > > > > > > > > am not spamming anyone ;-.) > > > > > > > > > > > Is there some sort of limit to the number of calls that can > be > > > made > > > > > > > > > before we have to reconnect or create a new remote_api > > > connection ? > > > > > > > > > As obviously something is triggering this reponse from > google. > > > > > > > > > > > Rgds > > > > > > > > > > > Tim > > > > > -- > > > > > You received this message because you are subscribed to the Google > Groups > > > "Google App Engine" group. > > > To post to this group, send email to [email protected] > . > > > To unsubscribe from this group, send email to > > > > ... > > > > read more ยป > > -- > > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]<google-appengine%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > > > -- Ikai Lan Developer Programs Engineer, Google App Engine -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
