[ 
https://issues.apache.org/jira/browse/MESOS-8624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16386580#comment-16386580
 ] 

Meng Zhu commented on MESOS-8624:
---------------------------------

Ticket description updated to better clarify the agent behavior before/after 
MESOS-1720.

> Valid tasks may be explicitly dropped by agent due to race conditions
> ---------------------------------------------------------------------
>
>                 Key: MESOS-8624
>                 URL: https://issues.apache.org/jira/browse/MESOS-8624
>             Project: Mesos
>          Issue Type: Bug
>    Affects Versions: 1.5.0
>            Reporter: Meng Zhu
>            Assignee: Meng Zhu
>            Priority: Critical
>
> Tasks may be explicitly dropped by the agent if all the following conditions 
> are met:
> (1) Several `LAUNCH_TASK` or `LAUNCH_GROUP` calls use the same executor.
> (2) The executor currently does not exist on the agent.
> (3) Due to some race conditions, these tasks are trying to launch on the 
> agent in a different order from their original launch order. (See below how 
> this could happen)
> In this case, tasks that are trying to launch on the agent before the first 
> task in the original order will be explicitly dropped by the agent 
> (TASK_DROPPED` or `TASK_LOST` will be sent)). 
> Up until now, Mesos does not guarantee in-order task launch on the agent. 
> Let's say Mesos master sends two `launchTask` messages (launch Task1 and 
> Task2) to an agent. In most cases (except MESOS-3870), these messages are 
> delivered to the agent in order. However, currently, there are two 
> asynchronous steps (unschedule GC and task authorization) in the agent task 
> launch path. Depending on the CPU scheduling order, task2 launch may finish 
> these two steps earlier than task1 and get to the launch executor stage 
> before task1.
> In this case, prior to MESOS-1720, these two tasks will still get launched. 
> If task1 and task2 use the same executor, whoever reaches the launch executor 
> stage first, will launch the executor.
> However, after resolving MESOS-1720, agents start to enforce some order for 
> tasks using the same executor. Specifically, when master crafts the launch 
> task message, it will specify the `launch_executor` flag. Thus Task1 in the 
> above case will have `launch_executor` flag set to true. And task2 (and any 
> subsequent tasks that use the same executor) will have the flag set to false.
> If task2 reaches the launch executor stage before task1 (due to the race 
> condition described above), the agent will see that its `launch_executor ` is 
> false but the executor specified in the `launchTask` message is not running. 
> As a result, it will explicitly drop task2 as in:
> https://github.com/apache/mesos/blob/32f6d4eec2724414e217875f4f7d3b2538db5381/src/slave/slave.cpp#L2888
> Based on discussion with [~chhsia0] and [~bmahler],  we should take an 
> explicit approach of using process:: Sequence to ensure ordered task delivery 
> (on both the master and agent).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to