Hey Camilo, This definitely sounds like an issue with the purge function. Your report is pretty detailed, and is good as an explanation of the issue, although it's hard to proceed from here without being able to reproduce the issue for fixing. Is it possible that you have a minimal reproducing example which can show this behaviour? You could file an issue at the github issue tracker for GAE Mapreduce <https://github.com/GoogleCloudPlatform/appengine-mapreduce/issues/> and hopefully get this addressed. There seems to be a related issue already <https://github.com/GoogleCloudPlatform/appengine-mapreduce/issues/69>, although you'll want to create a clear issue of your own, linking to it.
Best wishes, Nick On Tuesday, July 28, 2015 at 11:23:35 AM UTC-4, Camilo Silva wrote: > > Dear GAE Team & Enthusiasts: > > At work, I run daily automated map reduce jobs using Python as a pipeline > from the Datastore to GCS. > The issue is that in Python, the "mrcontroller" is always lingering > (I already posted this issue - and someone got back to me) > ( > https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/google-appengine/TUf4OvITSG4/wmftKhQIQBQJ > ) > > Anyways, in order to circumvent for now that challenge, I opted to use the > purge queue functionality in order to clear my task queue handling the map > reduce stuff after it has been completed (see Python code below): > q = taskqueue.Queue(configvars.TASK_QUEUE) > q.purge() > > While the call seems to execute without exceptions, unfortunately, the > tasks present ("POST /mapreduce/controller_callback/1574180594759CAA69707") > in my tasks queue above are never purged. > > I always need to go back to the admin panel, pause the queue, and then > delete each spawning non-ending "mrcontroller" callback task. > > This is a *serious* issue because these tasks accumulate and keep making > call backs which are translated to frontend calls which later on translate > to $$$ (hundreds) -- that's right -- we are literally paying for a stupid > bug of the mapreduce API. > > Anyways, any help or insight on this would be super helpful. > Maybe there's another way to clear the tasks of the taskqueue? > Thanks for your time and help. > > Regards, > Camilo > > > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/google-appengine. To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/ec003d0d-3d29-4dc4-a095-ab8ffe49000f%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
