Hi,
We have a production pipeline running a series of jobs, with each job
creating a custom mesos framework to execute all tasks related to that job.
Both scheduler and executor are written using the Python mesos API.
Here's a snippet (modified for brevity) of the scheduler code:
class MyMesosScheduler(mesos.Scheduler):
<...>
def run_jobs(self, jobs):
for job in jobs:
framework = mesos_pb2.FrameworkInfo()
<...>
driver = MesosSchedulerDriver(self, framework, Flags.mesos_master)
driver.start()
for task in job.generate_tasks():
<...>
# wait for all tasks to complete
driver.stop()
<...>
This usually works just fine, but sometimes the pipeline gets "stuck" on
the second framework, and I can see on the mesos dashboard that the first
framework is still "active":
[image: Inline image 1]
I know driver.stop() for the first framework was called and has returned
(from my logs, and from the fact that the following job started). I also
see this in the console where the scheduler is running:
I0514 07:05:16.819201 29486 sched.cpp:1286] Asked to stop the driver
*If I add time.sleep(0.1) after driver.stop() the problem disappears!*
I tried adding driver.join() after driver.stop(), but the behavior was the
same (the join() returned immediately).
I tried adding "del driver" after driver.stop(), but the behavior was the
same.
*So my questions are:*
- What is promised to me by the mesos API on return from "driver.stop()" ?
(I thought the promise is that the framework successfully stopped,
including stopping all executors)
- Is it safe to "recycle" the same instance of MyMesosScheduler for
multiple (consecutive, not overlapping) frameworks? (note that the driver
object is brand new for every framework)
- Any thoughts on the problem I'm describing, and potential solutions that
are not based on a voodoo sleep?
Thanks!
- Itamar.
