I ran across a situation with the same symptoms last year (with Mesos & Marathon) when we had network problems. The mesos task did exit normally (eventually found same in the logs), therefore the UUID had aged out.
<======> "Who do you think made the first stone spear? The Asperger guy. If you get rid of the autism genetics, there would be no Silicon Valley" Temple Grandin *Daemeon C.M. ReiydelleSan Francisco 1.415.501.0198London 44 020 8144 9872* On Tue, Mar 20, 2018 at 1:34 AM, Adam Cecile <[email protected]> wrote: > Hi Greg, > > Yes I can confirm i cannot find this UUID in Mesos interface. > > Regards, Adam. > > On 03/15/2018 05:47 PM, Greg Mann wrote: > > Hi Adam, > The KILL calls are being sent to Mesos by Marathon. Since the KILL call is > being forwarded to the agent, it seems that the Mesos master is aware of > the task. Could you verify that the tasks show up as running in the Mesos > UI? You say that the tasks don't exist anymore - how did you verify this? > If the tasks show up as running in the Mesos state, but the actual task > processes are not running on the agent, then it could indicate an issue > with the Mesos agent or executor. > > Cheers, > Greg > > > On Wed, Mar 14, 2018 at 1:59 AM, Adam Cecile <[email protected]> wrote: > >> Hello, >> >> I see two old tasks being stuck in Mesos. These tasks don't exist anymore >> since ages but Mesos still tries to kill them: >> >> >> Mar 14 09:56:49 mario mesos-master[23570]: I0314 09:56:49.441572 23602 >> master.cpp:5297] Processing KILL call for task >> 'pub_api_oecd-rest-api-on-port-20015.196f414a-f61f-11e7-856c-f6e84742f1ef' >> of framework 346d7333-a980-43a8-93ab-343ea12d77d7-0000 (marathon) at >> [email protected]:40487 >> >> Mar 14 09:56:49 mario mesos-master[23570]: I0314 09:56:49.441658 23602 >> master.cpp:5371] Telling agent 2215ab84-177b-478b-ab62-4453803fde6c-S6 >> at slave(1)@10.99.50.3:5051 (zelda.service.domain.com) to kill task >> pub_api_oecd-rest-api-on-port-20015.196f414a-f61f-11e7-856c-f6e84742f1ef >> of framework 346d7333-a980-43a8-93ab-343ea12d77d7-0000 (marathon) at >> [email protected]:40487 >> >> Mar 14 09:57:09 mario mesos-master[23570]: I0314 09:57:09.441529 23607 >> master.cpp:5297] Processing KILL call for task >> 'pub_api_oecd-rest-api-on-port-20015.196f414a-f61f-11e7-856c-f6e84742f1ef' >> of framework 346d7333-a980-43a8-93ab-343ea12d77d7-0000 (marathon) at >> [email protected]:40487 >> >> Mar 14 09:57:09 mario mesos-master[23570]: I0314 09:57:09.441617 23607 >> master.cpp:5371] Telling agent 2215ab84-177b-478b-ab62-4453803fde6c-S6 >> at slave(1)@10.99.50.3:5051 (zelda.service.domain.com) to kill task >> pub_api_oecd-rest-api-on-port-20015.196f414a-f61f-11e7-856c-f6e84742f1ef >> of framework 346d7333-a980-43a8-93ab-343ea12d77d7-0000 (marathon) at >> [email protected]:40487 >> >> >> Could you please tell me how to "purge" them from Mesos master ? >> >> Thanks in advance, >> >> Adam. >> > > >
