Hi Abhijit, On Tue, Oct 9, 2012 at 4:41 AM, Abhijit Rao <[email protected]> wrote:
> Hi there, > > Our (python) app uses a pull queue to send work to dynamic backend > instances. The backends sit in a tight loop polling for new work, 5 tasks > at a time. As long as tasks are coming through, the backends periodically > send 'keepalive' tasks to themselves via a push queue. Once the pull queue > is exhausted, the backends spin down since they go idle and no keepalive > tasks come through. > > Up until yesterday, this system was working fine, with the caveat that > lease_tasks was throwing large numbers of TransientErrors, which we would > catch and ignore. > > As of today, lease_tasks has started to fail silently - there are over 30k > tasks in the pull queue with ETAs in the past, yet lease_tasks > intermittently (but frequently) returns 0 results. It's happening often > enough that there are insufficient keepalive tasks, meaning the backends > die off even though there is tons of work to do. > Can you file a production issue with more detailed information(exact occurrence of the issue instead of just 'today', app-ids, etc)? > > Anyone else seeing this? Did something change in the taskqueue > implementation? > > Thanks, > -Abhijit > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To view this discussion on the web visit > https://groups.google.com/d/msg/google-appengine/-/mxlK_vp0TuEJ. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- Takashi Matsuo | Developers Advocate | [email protected] -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
