> On Nov. 21, 2013, 10:03 p.m., Ben Mahler wrote: > > Slaves continue to retry status updates until an acknowledgement is > > received (the scheduler driver sends an acknowledgement once > > Scheduler::statusUpdate was invoked). So in Case 1 you've described, if a > > task transitions while a framework if failing over, the slave will continue > > to retry sending the status update. When the framework reconnects to the > > master, the retried status update would be delivered and so reconciliation > > of this would be unnecessary and possibly error prone: > > > > Normally: > > > > Framework fails over with T1 in STARTING. > > T1 transitions from STARTING->RUNNING->FINISHED in the slave and master. > > Framework reconnects thinking T1 is STARTING. > > First the slave retries RUNNING and this now gets processed by the > > Framework. Acknowledgement is sent to the slave. > > Then the slave retries FINISHED and this now gets processed by the > > Framework. Acknowledgement is sent to the slave. > > > > Framework sees the following order: STARTING -> RUNNING -> FINISHED > > > > Assuming reconciliation Case 1 here: > > Framework fails over with T1 in STARTING. > > T1 transitions from STARTING->RUNNING->FINISHED in the slave and master. > > Framework reconnects thinking T1 is STARTING. > > Framework reconciles and master sends FINISHED. > > The slave retries RUNNING and this now gets processed by the Framework. > > Acknowledgement is sent to the slave. > > Then the slave retries FINISHED and this now gets processed by the > > Framework. Acknowledgement is sent to the slave. > > > > Framework sees the following order: STARTING -> FINISHED -> RUNNING -> > > FINISHED > > > > Case 1 here can unfortunately lead to out-of-order update delivery to > > frameworks.
I think the latter option is still the lesser of two evils. When the framework makes a call to reconcileTasks(), the master should always inform the framework of any differences between the framework and the master's current accounting for task states. In the case you describe, it seems to be the expected behaviour. I wouldn't expect the master to replay the queue of task statuses. Alternatively, you could use the task status timestamp to replay every status update since the last status update which the framework knows about. Thoughts? - Brenden ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15745/#review29249 ----------------------------------------------------------- On Nov. 21, 2013, 12:30 a.m., Brenden Matthews wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/15745/ > ----------------------------------------------------------- > > (Updated Nov. 21, 2013, 12:30 a.m.) > > > Review request for mesos and Niklas Nielsen. > > > Repository: mesos-git > > > Description > ------- > > Fixed some task reconciliation cases. > > Case 1: > > If a slave is known but the task cannot be found, we should assume that > the task has been lost. It's possible that the following events > occurred: > > 1) Framework disconnected from master > 2) Master terminated framework's tasks > 3) Framework reconnects to master, and (incorrectly) assumes tasks are > still running > > Case 2: > > If a framework loses track of running tasks, the master should inform > the framework of which tasks it knows to be running, in addition to any > which have had a state change. > > Review: https://reviews.apache.org/r/15745 > > > Diffs > ----- > > src/master/master.cpp a08d01208ff7bbb878b2d50d8406efee4de86171 > > Diff: https://reviews.apache.org/r/15745/diff/ > > > Testing > ------- > > `make check` & tested in staging cluster. > > > Thanks, > > Brenden Matthews > >
