[
https://issues.apache.org/jira/browse/MESOS-6026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15419779#comment-15419779
]
Benjamin Mahler commented on MESOS-6026:
----------------------------------------
Took a look with [~vinodkone] and [~kaysoky], it appears the race is that the
second terminal update (TASK_FAILED) in this case, races past the first one
because it {{_statusUpdate}} it does not have to wait for the
{{containerizer->update()}} to release the resources (which is a bug! resources
should be released before we notify the world that the task is terminal):
https://github.com/apache/mesos/blob/a536cef67d2f250f60eb2f991e62402dae0590a1/src/slave/slave.cpp#L3457-L3496
{code}
void Slave::_statusUpdate(
StatusUpdate update,
const Option<process::UPID>& pid,
const ExecutorID& executorId,
const Future<ContainerStatus>& future)
{
...
executor->updateTaskState(status);
// Handle the task appropriately if it is terminated.
// TODO(vinod): Revisit these semantics when we disallow duplicate
// terminal updates (e.g., when slave recovery is always enabled).
if (protobuf::isTerminalState(status.state()) &&
(executor->queuedTasks.contains(status.task_id()) ||
executor->launchedTasks.contains(status.task_id()))) {
executor->terminateTask(status.task_id(), status);
// XXX: TASK_FINISHED has to wait for the containerizer to update the
resources.
containerizer->update(executor->containerId, executor->resources)
.onAny(defer(self(),
&Slave::__statusUpdate,
lambda::_1,
update,
pid,
executor->id,
executor->containerId,
executor->checkpoint));
} else {
// XXX: TASK_FAILED races through directly to __statusUpdate!
__statusUpdate(None(),
update,
pid,
executor->id,
executor->containerId,
executor->checkpoint);
}
{code}
We should fix this race. Probably the simplest thing to do here is to ignore
double-terminal updates (notice the TODO) and drop them. We should probably log
a warning in this case and still ACK them in case some executors retry these
indefinitely.
> Tasks mistakenly marked as FAILED due to race b/w
> sendExecutorTerminatedStatusUpdate() and _statusUpdate()
> ------------------------------------------------------------------------------------------------------------------
>
> Key: MESOS-6026
> URL: https://issues.apache.org/jira/browse/MESOS-6026
> Project: Mesos
> Issue Type: Bug
> Components: slave
> Reporter: Kapil Arya
> Labels: mesosphere
>
> Due to a race between sendExecutorTerminatedStatusUpdate() and
> _statusUpdate() that happens when the task has just finished and the
> executor is exiting.
> Here is an example of slave log messages:
> {code}
> Aug 10 21:32:53 ip-10-10-0-205 mesos-slave[20413]: I0810 21:32:53.959374
> 20418 slave.cpp:3211] Handling status update TASK_FINISHED (UUID:
> fd79d0bd-4ece-41dc-bced-b93491f6bb2e) for task 291 of framework
> 340dfe26-a09f-4857-85b8-faba5f8d95df-0008 from executor(1)@10.10.0.205:53504
> Aug 10 21:32:53 ip-10-10-0-205 mesos-slave[20413]: I0810 21:32:53.959604
> 20418 slave.cpp:3732] executor(1)@10.10.0.205:53504 exited
> Aug 10 21:32:53 ip-10-10-0-205 mesos-slave[20413]: I0810 21:32:53.959643
> 20418 slave.cpp:4089] Executor '291' of framework
> 340dfe26-a09f-4857-85b8-faba5f8d95df-0008 exited with status 0
> Aug 10 21:32:53 ip-10-10-0-205 mesos-slave[20413]: I0810 21:32:53.959744
> 20418 slave.cpp:3211] Handling status update TASK_FAILED (UUID:
> b94722fb-1658-4936-b604-6d642ffe20a0) for task 291 of framework
> 340dfe26-a09f-4857-85b8-faba5f8d95df-0008 from @0.0.0.0:0
> {code}
> As can be noticed, the task is marked as TASK_FAILED after the executor has
> exited.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
