Vinod Kone created MESOS-439:
--------------------------------
Summary: Slave crashes on the duplicate ACK when waiting for the
next update
Key: MESOS-439
URL: https://issues.apache.org/jira/browse/MESOS-439
Project: Mesos
Issue Type: Bug
Reporter: Vinod Kone
Assignee: Vinod Kone
I0418 15:17:04.299052 43021 slave.cpp:719] Got assigned task
1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93 for
framework 201103282247-0000000019-0000
I0418 15:17:04.305749 43021 slave.cpp:792] Launching task
1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93 for
framework 201103282247-0000000019-0000
I0418 15:17:04.307135 43021 paths.hpp:302] Created executor directory
'/var/lib/mesos/slaves/201303281614-1937777162-5050-34776-36/frameworks/201103282247-0000000019-0000/executors/thermos-1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93/runs/e02f6720-c049-4a54-9865-e537a9d47ec6'
I0418 15:17:04.322979 43021 slave.cpp:940] Queuing task
'1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93' for
executor
thermos-1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93
of framework '201103282247-0000000019-0000
I0418 15:17:04.323269 43028 cgroups_isolator.cpp:520] Launching
thermos-1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93
(./thermos_executor) in
/var/lib/mesos/slaves/201303281614-1937777162-5050-34776-36/frameworks/201103282247-0000000019-0000/executors/thermos-1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93/runs/e02f6720-c049-4a54-9865-e537a9d47ec6
with resources cpus=0.25; mem=128 for framework 201103282247-0000000019-0000
in cgroup
mesos/framework_201103282247-0000000019-0000_executor_thermos-1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93_tag_e02f6720-c049-4a54-9865-e537a9d47ec6
I0418 15:17:04.325932 43028 cgroups_isolator.cpp:655] Changing cgroup controls
for executor
thermos-1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93
of framework 201103282247-0000000019-0000 with resources cpus=0.25; mem=128
I0418 15:17:04.326355 43028 cgroups_isolator.cpp:839] Updated 'cpu.shares' to
256 for executor
thermos-1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93
of framework 201103282247-0000000019-0000
I0418 15:17:04.326457 43032 slave.cpp:512] Successfully attached file
'/var/lib/mesos/slaves/201303281614-1937777162-5050-34776-36/frameworks/201103282247-0000000019-0000/executors/thermos-1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93/runs/e02f6720-c049-4a54-9865-e537a9d47ec6'
I0418 15:17:04.326828 43028 cgroups_isolator.cpp:977] Updated
'memory.limit_in_bytes' to 134217728 for executor
thermos-1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93
of framework 201103282247-0000000019-0000
I0418 15:17:04.329628 43028 cgroups_isolator.cpp:1003] Started listening for
OOM events for executor
thermos-1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93
of framework 201103282247-0000000019-0000
Fetching resources into
'/var/lib/mesos/slaves/201303281614-1937777162-5050-34776-36/frameworks/201103282247-0000000019-0000/executors/thermos-1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93/runs/e02f6720-c049-4a54-9865-e537a9d47ec6'
I0418 15:17:05.550911 43022 slave.cpp:1391] Got registration for executor
'thermos-1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93'
of framework 201103282247-0000000019-0000
I0418 15:17:05.551237 43023 cgroups_isolator.cpp:655] Changing cgroup controls
for executor
thermos-1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93
of framework 201103282247-0000000019-0000 with resources cpus=0.35; mem=384;
disk=1024
I0418 15:17:05.551877 43023 cgroups_isolator.cpp:839] Updated 'cpu.shares' to
358 for executor
thermos-1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93
of framework 201103282247-0000000019-0000
I0418 15:17:05.552738 43023 cgroups_isolator.cpp:977] Updated
'memory.limit_in_bytes' to 402653184 for executor
thermos-1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93
of framework 201103282247-0000000019-0000
I0418 15:17:05.600024 43030 slave.cpp:1733] Handling status update
TASK_STARTING from task
1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93 of
framework 201103282247-0000000019-0000
I0418 15:17:05.600225 43028 status_update_manager.cpp:289] Received status
update TASK_STARTING from task
1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93 of
framework 201103282247-0000000019-0000 with checkpoint=false
I0418 15:17:05.600306 43028 status_update_manager.cpp:451] Creating
StatusUpdate stream for task
1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93 of
framework 201103282247-0000000019-0000
I0418 15:17:05.600374 43028 status_update_manager.hpp:336] Handling UPDATE for
status update TASK_STARTING from task
1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93 of
framework 201103282247-0000000019-0000
I0418 15:17:05.600409 43028 status_update_manager.cpp:335] Forwarding status
update TASK_STARTING from task
1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93 of
framework 201103282247-0000000019-0000 to the master at
[email protected]:5050
I0418 15:17:05.600632 43021 slave.cpp:1793] Sending ACK for status update
TASK_STARTING from task
1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93 of
framework 201103282247-0000000019-0000 to executor
executor(1)@10.34.135.114:42980
I0418 15:17:07.127419 43036 slave.cpp:1733] Handling status update TASK_RUNNING
from task
1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93 of
framework 201103282247-0000000019-0000
I0418 15:17:07.127655 43025 status_update_manager.cpp:289] Received status
update TASK_RUNNING from task
1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93 of
framework 201103282247-0000000019-0000 with checkpoint=false
I0418 15:17:07.127707 43025 status_update_manager.hpp:336] Handling UPDATE for
status update TASK_RUNNING from task
1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93 of
framework 201103282247-0000000019-0000
I0418 15:17:07.127779 43025 slave.cpp:1793] Sending ACK for status update
TASK_RUNNING from task
1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93 of
framework 201103282247-0000000019-0000 to executor
executor(1)@10.34.135.114:42980
W0418 15:17:15.601752 43024 status_update_manager.cpp:434] Resending status
update TASK_STARTING from task
1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93 of
framework 201103282247-0000000019-0000
I0418 15:17:15.601836 43024 status_update_manager.cpp:335] Forwarding status
update TASK_STARTING from task
1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93 of
framework 201103282247-0000000019-0000 to the master at
[email protected]:5050
I0418 15:17:18.861799 43021 slave.cpp:1307] Got acknowledgement of status
update for task
1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93 of
framework 201103282247-0000000019-0000
I0418 15:17:18.869899 43025 status_update_manager.cpp:360] Received status
update acknowledgement for task
1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93 of
framework 201103282247-0000000019-0000
I0418 15:17:18.870002 43025 status_update_manager.hpp:336] Handling ACK for
status update TASK_STARTING from task
1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93 of
framework 201103282247-0000000019-0000
I0418 15:17:18.870111 43025 status_update_manager.cpp:335] Forwarding status
update TASK_RUNNING from task
1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93 of
framework 201103282247-0000000019-0000 to the master at
[email protected]:5050
I0418 15:17:18.870247 43025 slave.cpp:1344] Status update manager successfully
handled status update acknowledgement for task
1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93 of
framework 201103282247-0000000019-0000
I0418 15:17:19.301548 43024 slave.cpp:1307] Got acknowledgement of status
update for task
1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93 of
framework 201103282247-0000000019-0000
I0418 15:17:19.301774 43024 status_update_manager.cpp:360] Received status
update acknowledgement for task
1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93 of
framework 201103282247-0000000019-0000
F0418 15:17:19.375743 43024 status_update_manager.hpp:236] Check failed: uuid
== UUID::fromBytes(update.uuid()) Unexpected UUID mismatch! (received
72fae945-1afb-4f86-a80e-c0b67df0aa04, expecting
e1ea786a-7a7a-4f02-ae74-ae09b525ce11) for update TASK_RUNNING from task
1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93 of
framework 201103282247-0000000019-0000
I0418 15:17:22.727701 24131 cgroups_isolator.cpp:784] Removing orphaned cgroup
'mesos/framework_201103282247-0000000019-0000_executor_thermos-1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93_tag_e02f6720-c049-4a54-9865-e537a9d47ec6'
I0418 15:17:22.729791 24126 cgroups.cpp:1175] Trying to freeze cgroup
/cgroup/mesos/framework_201103282247-0000000019-0000_executor_thermos-1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93_tag_e02f6720-c049-4a54-9865-e537a9d47ec6
I0418 15:17:23.363044 24126 cgroups.cpp:1214] Successfully froze cgroup
/cgroup/mesos/framework_201103282247-0000000019-0000_executor_thermos-1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93_tag_e02f6720-c049-4a54-9865-e537a9d47ec6
after 7 attempts
I0418 15:17:23.365375 24123 cgroups.cpp:1190] Trying to thaw cgroup
/cgroup/mesos/framework_201103282247-0000000019-0000_executor_thermos-1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93_tag_e02f6720-c049-4a54-9865-e537a9d47ec6
I0418 15:17:23.365535 24123 cgroups.cpp:1298] Successfully thawed
/cgroup/mesos/framework_201103282247-0000000019-0000_executor_thermos-1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93_tag_e02f6720-c049-4a54-9865-e537a9d47ec6
I0418 15:17:23.475486 24132 cgroups_isolator.cpp:1125] Successfully destroyed
cgroup
mesos/framework_201103282247-0000000019-0000_executor_thermos-1366298223270-mesos-meta_slave_9-13-6b22df3f-d77f-44ac-a0b2-6047cebf0a93_tag_e02f6720-c049-4a54-9865-e537a9d47ec6
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
