After some thoughts and feedback I'm thinking of using PULL queue after all
in the following way:
- have two PULL queues (medium-tasks and large-tasks)
- have a dispatcher (PUSH) queue with a concurrency of 1 (so that only
one dispatch task runs at any time). Dispatch tasks are created in multiple
ways:
- by a once-a-minute cron job
- after adding a medium/large task to the push queues
- after a worker task finishes
- have a worker (PUSH) queue with a concurrency equal to the number of
workers
And the workflow:
- small tasks are added directly to the worker queue
- the dispatcher task, whenever it is triggered, does the following:
- estimates the number of free workers (by looking at the number of
running tasks in the worker queue)
- for any "free" slots it takes a task from the medium/large tasks
PULL queue and enqueues it on a worker (or more precisely: adds it to the
worker PUSH queue which will result in it being executed - eventually -
on
a worker).
I'll report back once this is implemented and at least moderately tested.
--
You received this message because you are subscribed to the Google Groups
"Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/google-appengine.
To view this discussion on the web visit
https://groups.google.com/d/msgid/google-appengine/dc0efade-306c-4988-82a1-66c9ad030ff7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
