A. Dukhovniy created MESOS-7752: ----------------------------------- Summary: Command executor still active after terminal task state update. Key: MESOS-7752 URL: https://issues.apache.org/jira/browse/MESOS-7752 Project: Mesos Issue Type: Bug Affects Versions: 1.3.0 Reporter: A. Dukhovniy
Here is a rather simple scenario to reproduce this error: * Frameworks starts a task with taskId = _task1_ * Framework kills _task1_ *successfully* and *acknowledges* TASK_KILLED * Framework starts another task with the same _task1_ but receives "_TASK_FAILED (Attempted to run multiple tasks using a "command" executor)_" *Note*: this test is racy so this scenario fails occasionally. *Here is a full log* from that show a life-cycle of a task id _app-restart-resident-app-with-five-instances.8882bd16-5fdd-11e7-a00e-0242aceef95c_: {code:java} # Starting... WARN [10:51:14 ResidentTaskIntegrationTest-MesosMaster-32782] I0703 10:51:14.476085 14666 master.cpp:3352] Authorizing framework principal 'principal' to launch task app-restart-resident-app-with-five-instances.8882bd16-5fdd-11e7-a00e-0242aceef95c WARN [10:51:14 ResidentTaskIntegrationTest-MesosMaster-32782] I0703 10:51:14.510136 14666 master.cpp:4426] Launching task app-restart-resident-app-with-five-instances.8882bd16-5fdd-11e7-a00e-0242aceef95c of framework 76d8f3e7-8f3a-4764-bb7d-2bcf8e85e2be-0000 (marathon) at scheduler-6dbbac16-7355-4a33-aee6-b9697c83e51c@127.0.1.1:61567 with resources... WARN [10:51:14 ResidentTaskIntegrationTest-MesosAgent-32788] I0703 10:51:14.513908 14697 slave.cpp:2118] Queued task 'app-restart-resident-app-with-five-instances.8882bd16-5fdd-11e7-a00e-0242aceef95c' for executor 'app-restart-resident-app-with-five-instances.8882bd16-5fdd-11e7-a00e-0242aceef95c' of framework 76d8f3e7-8f3a-4764-bb7d-2bcf8e85e2be-0000 WARN [10:51:15 ResidentTaskIntegrationTest-MesosMaster-32782] I0703 10:51:15.011696 14671 master.cpp:6222] Forwarding status update TASK_RUNNING (UUID: ed2d0475-9d83-4e09-9f54-5b4d323e4558) for task app-restart-resident-app-with-five-instances.8882bd16-5fdd-11e7-a00e-0242aceef95c of framework 76d8f3e7-8f3a-4764-bb7d-2bcf8e85e2be-0000 WARN [10:51:15 ResidentTaskIntegrationTest-MesosMaster-32782] I0703 10:51:15.036391 14671 master.cpp:5092] Processing ACKNOWLEDGE call ed2d0475-9d83-4e09-9f54-5b4d323e4558 for task app-restart-resident-app-with-five-instances.8882bd16-5fdd-11e7-a00e-0242aceef95c of framework 76d8f3e7-8f3a-4764-bb7d-2bcf8e85e2be-0000 (marathon) at scheduler-6dbbac16-7355-4a33-aee6-b9697c83e51c@127.0.1.1:61567 on agent 76d8f3e7-8f3a-4764-bb7d-2bcf8e85e2be-S0 {code} {code:java} # Killing... DEBUG[10:51:15 ResidentTaskIntegrationTest-LocalMarathon-32800] WARN [10:51:15 KillAction$] Killing known task [app-restart-resident-app-with-five-instances.8882bd16-5fdd-11e7-a00e-0242aceef95c] of instance instance [app-restart-resident-app-with-five-instances.marathon-8882bd16-5fdd-11e7-a00e-0242aceef95c] WARN [10:51:15 ResidentTaskIntegrationTest-MesosAgent-32788] I0703 10:51:15.196702 14697 slave.cpp:3816] Handling status update TASK_KILLED (UUID: f7e9d0bc-726c-43aa-9ddc-3b082a68642e) for task app-restart-resident-app-with-five-instances.8882bd16-5fdd-11e7-a00e-0242aceef95c of framework 76d8f3e7-8f3a-4764-bb7d-2bcf8e85e2be-0000 from executor(1)@172.16.10.121:35184 WARN [10:51:15 ResidentTaskIntegrationTest-MesosAgent-32788] I0703 10:51:15.197676 14697 slave.cpp:4166] Sending acknowledgement for status update TASK_KILLED (UUID: f7e9d0bc-726c-43aa-9ddc-3b082a68642e) for task app-restart-resident-app-with-five-instances.8882bd16-5fdd-11e7-a00e-0242aceef95c of framework 76d8f3e7-8f3a-4764-bb7d-2bcf8e85e2be-0000 to executor(1)@172.16.10.121:35184 WARN [10:51:15 ResidentTaskIntegrationTest-MesosMaster-32782] I0703 10:51:15.198299 14671 master.cpp:6154] Status update TASK_KILLED (UUID: f7e9d0bc-726c-43aa-9ddc-3b082a68642e) for task app-restart-resident-app-with-five-instances.8882bd16-5fdd-11e7-a00e-0242aceef95c of framework 76d8f3e7-8f3a-4764-bb7d-2bcf8e85e2be-0000 from agent 76d8f3e7-8f3a-4764-bb7d-2bcf8e85e2be-S0 at slave(1)@172.16.10.121:32788 (172.16.10.121) DEBUG[10:51:15 ResidentTaskIntegrationTest-LocalMarathon-32800] INFO [10:51:15 MarathonScheduler] Received status update for task app-restart-resident-app-with-five-instances.8882bd16-5fdd-11e7-a00e-0242aceef95c: TASK_KILLED (Command terminated with signal Terminated) WARN [10:51:15 ResidentTaskIntegrationTest-MesosMaster-32782] I0703 10:51:15.216081 14671 master.cpp:5092] Processing ACKNOWLEDGE call f7e9d0bc-726c-43aa-9ddc-3b082a68642e for task app-restart-resident-app-with-five-instances.8882bd16-5fdd-11e7-a00e-0242aceef95c of framework 76d8f3e7-8f3a-4764-bb7d-2bcf8e85e2be-0000 (marathon) at scheduler-6dbbac16-7355-4a33-aee6-b9697c83e51c@127.0.1.1:61567 on agent 76d8f3e7-8f3a-4764-bb7d-2bcf8e85e2be-S0 WARN [10:51:15 ResidentTaskIntegrationTest-MesosMaster-32782] I0703 10:51:15.216107 14671 master.cpp:8396] Removing task app-restart-resident-app-with-five-instances.8882bd16-5fdd-11e7-a00e-0242aceef95c with resources... WARN [10:51:15 ResidentTaskIntegrationTest-MesosAgent-32788] I0703 10:51:15.216667 14697 status_update_manager.cpp:395] Received status update acknowledgement (UUID: f7e9d0bc-726c-43aa-9ddc-3b082a68642e) for task app-restart-resident-app-with-five-instances.8882bd16-5fdd-11e7-a00e-0242aceef95c of framework 76d8f3e7-8f3a-4764-bb7d-2bcf8e85e2be-0000 WARN [10:51:15 ResidentTaskIntegrationTest-MesosAgent-32788] I0703 10:51:15.216722 14697 status_update_manager.cpp:832] Checkpointing ACK for status update TASK_KILLED (UUID: f7e9d0bc-726c-43aa-9ddc-3b082a68642e) for task app-restart-resident-app-with-five-instances.8882bd16-5fdd-11e7-a00e-0242aceef95c of framework 76d8f3e7-8f3a-4764-bb7d-2bcf8e85e2be-0000 {code} {code:java} # and starting again: WARN [10:51:15 ResidentTaskIntegrationTest-MesosMaster-32782] I0703 10:51:15.247561 14671 master.cpp:3352] Authorizing framework principal 'principal' to launch task app-restart-resident-app-with-five-instances.8882bd16-5fdd-11e7-a00e-0242aceef95c WARN [10:51:15 ResidentTaskIntegrationTest-MesosAgent-32788] I0703 10:51:15.252348 14697 slave.cpp:1625] Got assigned task 'app-restart-resident-app-with-five-instances.8882bd16-5fdd-11e7-a00e-0242aceef95c' for framework 76d8f3e7-8f3a-4764-bb7d-2bcf8e85e2be-0000 WARN [10:51:15 ResidentTaskIntegrationTest-MesosAgent-32788] I0703 10:51:15.252707 14697 slave.cpp:1785] Launching task 'app-restart-resident-app-with-five-instances.8882bd16-5fdd-11e7-a00e-0242aceef95c' for framework 76d8f3e7-8f3a-4764-bb7d-2bcf8e85e2be-0000 WARN [10:51:15 ResidentTaskIntegrationTest-MesosAgent-32788] I0703 10:51:15.253159 14697 slave.cpp:2140] Queued task 'app-restart-resident-app-with-five-instances.8882bd16-5fdd-11e7-a00e-0242aceef95c' for executor 'app-restart-resident-app-with-five-instances.8882bd16-5fdd-11e7-a00e-0242aceef95c' of framework 76d8f3e7-8f3a-4764-bb7d-2bcf8e85e2be-0000 at executor(1)@172.16.10.121:35184 DEBUG[10:51:15 ResidentTaskIntegrationTest-LocalMarathon-32800] INFO [10:51:15 MarathonScheduler] Received status update for task app-restart-resident-app-with-five-instances.8882bd16-5fdd-11e7-a00e-0242aceef95c: TASK_FAILED (Attempted to run multiple tasks using a "command" executor) {code} *TL;DR*: framework receives and acknowledges _TASK_KILLED_ status but fails to re-start the task because _"Attempted to run multiple tasks using a "command" executor"_ Though reusing task Ids is discouraged {code:java} /** * A framework-generated ID to distinguish a task. The ID must remain * unique while the task is active. A framework can reuse an ID _only_ * if the previous task with the same ID has reached a terminal state * (e.g., TASK_FINISHED, TASK_LOST, TASK_KILLED, etc.). However, * reusing task IDs is strongly discouraged (MESOS-2198). */{code} it is acceptable after receiving a terminal tasks status which happened above. *Possible cause*: I assume that occasionally the executor is not yet cleaned and is reused during task restart. This however fails here: https://github.com/apache/mesos/blob/35dd2b600b8af0204d03c4ee5348a1a6672b136c/src/launcher/executor.cpp#L512 /cc [~tillt] -- This message was sent by Atlassian JIRA (v6.4.14#64029)