Hi Adam,
that's exactly what happened. Thanks a lot for the explanation and
suggestion. Now mesos is clean again :)
On 03/31/16 03:51, Adam Bordelon wrote:
I suspect that after your maintenance operation, Marathon may have
registered with a new frameworkId and launched is own copies of your
tasks (why you see double). However, the old Marathon frameworkId
probably has a failover_timeout of a week, so it will continue to be
considered "registered", but "disconnected".
Check the /frameworks endpoint to see if Mesos thinks you have two
Marathons registered.
If so, you can use the /teardown endpoint to unregister the old one,
which will cause all of its tasks to be killed.
On Wed, Mar 30, 2016 at 4:56 AM, Alberto del Barrio
<[email protected]
<mailto:[email protected]>> wrote:
Hi haosdent,
thanks for your reply. It is actually very weird, first time I see
this situation in around one year using mesos.
I am pasting here the truncate output you asked for. It is showing
one of the tasks with "Failed" state under "Active tasks":
{
"executor_id": "",
"framework_id":
"c857c625-25dc-4650-89b8-de4b597026ed-0000",
"id":
"pixie.33f85e8f-f03b-11e5-af6c-fa6389efeef1",
"labels": [
......................
],
"name": "myTask",
"resources": {
"cpus": 4.0,
"disk": 0,
"mem": 2560.0,
"ports": "[31679-31679]"
},
"slave_id":
"c857c625-25dc-4650-89b8-de4b597026ed-S878",
"state": "TASK_FAILED",
"statuses": [
{
"container_status": {
"network_infos": [
{
"ip_address": "10.XX.XX.XX"
}
]
},
"state": "TASK_RUNNING",
"timestamp": 1458657321.16671
},
{
"container_status": {
"network_infos": [
{
"ip_address": "10.XX.XX.XX"
}
]
},
"state": "TASK_FAILED",
"timestamp": 1459329310.13663
}
]
},
t
On 03/30/16 13:30, haosdent wrote:
>"Active tasks" with status "Failed"
A bit wired here. According to my test, it should exists in
"Completed Tasks". If possible, could you show you /master/state
endpoint result. I think the frameworks node in state response
would be helpful to analyze the problem.
On Wed, Mar 30, 2016 at 6:26 PM, Alberto del Barrio
<[email protected]
<mailto:[email protected]>> wrote:
Hi all,
after a maintenance carried on in a mesos cluster (0.25)
using marathon (0.10) as a only scheduler , I've finished
with the double of tasks for each application. But marathon
was recognizing only half of them.
For getting rid of this orphaned tasks, I've did a "kill PID"
of them, so they free up their resources.
My problem now is that these tasks I've killed, are still
appearing in the mesos UI under "Active tasks" with status
"Failed". This is not affecting my system, but I would like
to clean them up.
Googling I can't find anything.
Can someone point me to a solution for cleaning those tasks?
Cheers,
Alberto.
--
Best Regards,
Haosdent Huang
