Alexander Rojas created MESOS-7587:
--------------------------------------
Summary: Launching tasks with the Mesos Containerizer after a long
time without launching new tasks fails
Key: MESOS-7587
URL: https://issues.apache.org/jira/browse/MESOS-7587
Project: Mesos
Issue Type: Bug
Components: containerization
Affects Versions: 1.2.0
Reporter: Alexander Rojas
After having a cluster running without launching new tasks for an extended
period of time, ~1week. When launching a new task using the Mesos
Containerizer, the task fails to launch with the error:
[{{Failed to execute command: No such file or
directory}}|https://github.com/apache/mesos/blob/8245981b889ec3725cc0be4150b15d1fe9d64b86/src/slave/containerizer/mesos/launch.cpp#L778]
The task is launched from Marathon with the app definition:
{code}
{
"container": {
"type": "MESOS",
"docker": {
"forcePullImage": true,
"image": "private.repository.local/updated:fixed",
"privileged": false
}
},
"cpus": 0.1,
"id": "/20150530/mesos9",
"instances": 1,
"minimumHealthCapacity": 1,
"acceptedResourceRoles": ["*"],
"constraints": [["hostname", "UNIQUE"]],
"mem": 128
}
{code}
and {{Docerfile}}
{code}
FROM private.repository.local/centos:stable
MAINTAINER Sebastian Gerlach "s...@boreus.de"
CMD python -m SimpleHTTPServer 80
{code}
The obtained stdout is:
{noformat}
Executing pre-exec command
'{"arguments":["mesos-containerizer","mount","--help=false","--operation=make-rslave","--path=\/"],"shell":false,"value":"\/usr\/libexec\/mesos\/mesos-containerizer"}'
Executing pre-exec command
'{"arguments":["mount","-n","--rbind","\/var\/lib\/mesos\/slaves\/562f1892-41c6-4c55-955e-662c63ab3ddf-S8\/frameworks\/562f1892-41c6-4c55-955e-662c63ab3ddf-0000\/executors\/20150530_mesos13.5f8c3db1-4517-11e7-8d95-024276be29e0\/runs\/da1cb066-1354-42a6-82bc-eee688d61b94","\/var\/lib\/mesos\/provisioner\/containers\/da1cb066-1354-42a6-82bc-eee688d61b94\/backends\/overlay\/rootfses\/f74816b3-c115-4a5e-a51c-fbd502efd1fe\/mnt\/mesos\/sandbox"],"shell":false,"value":"mount"}'
Received SUBSCRIBED event
Subscribed executor on bp-mesos8.private.local
Received LAUNCH event
Starting task 20150530_mesos13.5f8c3db1-4517-11e7-8d95-024276be29e0
/usr/libexec/mesos/mesos-containerizer launch --help="false"
--launch_info="{"command":{"arguments":["\/bin\/sh","-c","python -m
SimpleHTTPServer
80"],"environment":{"variables":[{"name":"MARATHON_APP_VERSION","value":"2017-05-30T09:06:59.666Z"},{"name":"HOST","value":"bp-mesos8.private.local"},{"name":"MARATHON_APP_RESOURCE_CPUS","value":"0.1"},{"name":"MARATHON_APP_RESOURCE_GPUS","value":"0"},{"name":"MESOS_TASK_ID","value":"20150530_mesos13.5f8c3db1-4517-11e7-8d95-024276be29e0"},{"name":"MARATHON_APP_RESOURCE_MEM","value":"128.0"},{"name":"MARATHON_APP_RESOURCE_DISK","value":"0.0"},{"name":"MARATHON_APP_LABELS","value":""},{"name":"MARATHON_APP_ID","value":"\/20150530\/mesos13"}]},"shell":false,"value":"\/bin\/sh"},"environment":{"variables":[{"name":"HOST","value":"bp-mesos8.private.local"},{"name":"LIBPROCESS_IP","value":"0.0.0.0"},{"name":"LIBPROCESS_PORT","value":"0"},{"name":"MARATHON_APP_ID","value":"\/20150530\/mesos13"},{"name":"MARATHON_APP_LABELS","value":""},{"name":"MARATHON_APP_RESOURCE_CPUS","value":"0.1"},{"name":"MARATHON_APP_RESOURCE_DISK","value":"0.0"},{"name":"MARATHON_APP_RESOURCE_GPUS","value":"0"},{"name":"MARATHON_APP_RESOURCE_MEM","value":"128.0"},{"name":"MARATHON_APP_VERSION","value":"2017-05-30T09:06:59.666Z"},{"name":"MESOS_AGENT_ENDPOINT","value":"10.107.19.26:5051"},{"name":"MESOS_CHECKPOINT","value":"1"},{"name":"MESOS_DIRECTORY","value":"\/var\/lib\/mesos\/slaves\/562f1892-41c6-4c55-955e-662c63ab3ddf-S8\/frameworks\/562f1892-41c6-4c55-955e-662c63ab3ddf-0000\/executors\/20150530_mesos13.5f8c3db1-4517-11e7-8d95-024276be29e0\/runs\/da1cb066-1354-42a6-82bc-eee688d61b94"},{"name":"MESOS_EXECUTOR_ID","value":"20150530_mesos13.5f8c3db1-4517-11e7-8d95-024276be29e0"},{"name":"MESOS_EXECUTOR_SHUTDOWN_GRACE_PERIOD","value":"5secs"},{"name":"MESOS_FRAMEWORK_ID","value":"562f1892-41c6-4c55-955e-662c63ab3ddf-0000"},{"name":"MESOS_HTTP_COMMAND_EXECUTOR","value":"0"},{"name":"MESOS_NATIVE_JAVA_LIBRARY","value":"\/usr\/lib\/libmesos-1.2.0.so"},{"name":"MESOS_NATIVE_LIBRARY","value":"\/usr\/lib\/libmesos-1.2.0.so"},{"name":"MESOS_RECOVERY_TIMEOUT","value":"15mins"},{"name":"MESOS_SANDBOX","value":"\/mnt\/mesos\/sandbox"},{"name":"MESOS_SLAVE_ID","value":"562f1892-41c6-4c55-955e-662c63ab3ddf-S8"},{"name":"MESOS_SLAVE_PID","value":"slave(1)@10.107.19.26:5051"},{"name":"MESOS_SUBSCRIPTION_BACKOFF_MAX","value":"2secs"},{"name":"MESOS_TASK_ID","value":"20150530_mesos13.5f8c3db1-4517-11e7-8d95-024276be29e0"},{"name":"PATH","value":"\/usr\/local\/sbin:\/usr\/local\/bin:\/usr\/sbin:\/usr\/bin:\/sbin:\/bin"},{"name":"PATH","value":"\/usr\/local\/sbin:\/usr\/local\/bin:\/usr\/sbin:\/usr\/bin:\/sbin:\/bin"},{"name":"ftp_proxy","value":"http:\/\/bp-proxy.private.local:3128"},{"name":"http_proxy","value":"http:\/\/bp-proxy.private.local:3128"},{"name":"https_proxy","value":"http:\/\/bp-proxy.private.local:3128"},{"name":"no_proxy","value":"localhost,127.0.0.1,svc.boreus.de,private.local"},{"name":"MARATHON_APP_VERSION","value":"2017-05-30T09:06:59.666Z"},{"name":"HOST","value":"bp-mesos8.private.local"},{"name":"MARATHON_APP_RESOURCE_CPUS","value":"0.1"},{"name":"MARATHON_APP_RESOURCE_GPUS","value":"0"},{"name":"MESOS_TASK_ID","value":"20150530_mesos13.5f8c3db1-4517-11e7-8d95-024276be29e0"},{"name":"MARATHON_APP_RESOURCE_MEM","value":"128.0"},{"name":"MARATHON_APP_RESOURCE_DISK","value":"0.0"},{"name":"MARATHON_APP_LABELS","value":""},{"name":"MARATHON_APP_ID","value":"\/20150530\/mesos13"}]},"rootfs":"\/var\/lib\/mesos\/provisioner\/containers\/da1cb066-1354-42a6-82bc-eee688d61b94\/backends\/overlay\/rootfses\/f74816b3-c115-4a5e-a51c-fbd502efd1fe","user":"root","working_directory":"\/mnt\/mesos\/sandbox"}"
--unshare_namespace_mnt="true"
Forked command at 10438
Changing root to
/var/lib/mesos/provisioner/containers/da1cb066-1354-42a6-82bc-eee688d61b94/backends/overlay/rootfses/f74816b3-c115-4a5e-a51c-fbd502efd1fe
Overwriting environment variable 'PATH', original:
'/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', new:
'/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
Overwriting environment variable 'MARATHON_APP_VERSION', original:
'2017-05-30T09:06:59.666Z', new: '2017-05-30T09:06:59.666Z'
Overwriting environment variable 'HOST', original: 'bp-mesos8.private.local',
new: 'bp-mesos8.private.local'
Overwriting environment variable 'MARATHON_APP_RESOURCE_CPUS', original: '0.1',
new: '0.1'
Overwriting environment variable 'MARATHON_APP_RESOURCE_GPUS', original: '0',
new: '0'
Overwriting environment variable 'MESOS_TASK_ID', original:
'20150530_mesos13.5f8c3db1-4517-11e7-8d95-024276be29e0', new:
'20150530_mesos13.5f8c3db1-4517-11e7-8d95-024276be29e0'
Overwriting environment variable 'MARATHON_APP_RESOURCE_MEM', original:
'128.0', new: '128.0'
Overwriting environment variable 'MARATHON_APP_RESOURCE_DISK', original: '0.0',
new: '0.0'
Overwriting environment variable 'MARATHON_APP_LABELS', original: '', new: ''
Overwriting environment variable 'MARATHON_APP_ID', original:
'/20150530/mesos13', new: '/20150530/mesos13'
Command exited with status 1 (pid: 10438)
{noformat}
And a portion of the logs:
{noformat}
tail -f mesos-slave.ERROR mesos-slave.INFO mesos-slave.WARNING
==> mesos-slave.ERROR <==
Log file created at: 2017/05/30 10:48:17
Running on machine: bp-mesos7.private.local
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0530 10:48:17.653518 8876 shell.hpp:107] Command 'hadoop version 2>&1'
failed; this is the output:
sh: hadoop: command not found
==> mesos-slave.INFO <==
I0530 10:48:19.953788 8906 slave.cpp:5558] Finished recovery
I0530 10:48:19.954376 8906 slave.cpp:931] New master detected at
master@10.107.19.21:5050
I0530 10:48:19.954401 8906 slave.cpp:955] No credentials provided. Attempting
to register without authentication
I0530 10:48:19.954416 8906 slave.cpp:966] Detecting new master
I0530 10:48:19.954444 8906 status_update_manager.cpp:177] Pausing sending
status updates
I0530 10:48:20.208361 8907 slave.cpp:1242] Re-registered with master
master@10.107.19.21:5050
I0530 10:48:20.208427 8907 slave.cpp:1279] Forwarding total oversubscribed
resources {}
I0530 10:48:20.208452 8906 status_update_manager.cpp:184] Resuming sending
status updates
I0530 10:48:20.208501 8907 slave.cpp:2827] Updating framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000 pid to
scheduler-afcaa9d7-6040-476d-8a86-d97138cc23c8@10.107.19.22:39431
I0530 10:48:20.208871 8907 status_update_manager.cpp:184] Resuming sending
status updates
==> mesos-slave.WARNING <==
Log file created at: 2017/05/30 10:48:17
Running on machine: bp-mesos7.private.local
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0530 10:48:17.653518 8876 shell.hpp:107] Command 'hadoop version 2>&1'
failed; this is the output:
sh: hadoop: command not found
W0530 10:48:17.694730 8903 state.cpp:477] Failed to find executor forked pid
file
'/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150530_mesos1.7c34f793-44fd-11e7-8f05-02420d613ec3/runs/0ac924bb-4ba5-41e9-b563-6768329f6f7d/pids/forked.pid'
W0530 10:48:17.695217 8903 state.cpp:477] Failed to find executor forked pid
file
'/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150530_mesos1.7041e28f-44fd-11e7-8f05-02420d613ec3/runs/d8afe43c-292c-4069-88d7-7edf4a730575/pids/forked.pid'
W0530 10:48:17.713217 8903 state.cpp:477] Failed to find executor forked pid
file
'/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150530_mesos1.0cad114c-44fe-11e7-8f05-02420d613ec3/runs/3adaddbb-5fd9-4ca6-8225-660daf91a7d9/pids/forked.pid'
W0530 10:48:17.741708 8904 linux_launcher.cpp:357] Couldn't find pid '14251'
in 'mesos_executors.slice'. This can lead to lack of proper resource isolation
W0530 10:48:17.741747 8904 linux_launcher.cpp:357] Couldn't find pid '19408'
in 'mesos_executors.slice'. This can lead to lack of proper resource isolation
==> mesos-slave.INFO <==
I0530 10:49:17.692493 8903 slave.cpp:5267] Current disk usage 0.88%. Max
allowed age: 6.238522372392419days
I0530 10:49:32.356741 8905 slave.cpp:1625] Got assigned task
'20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0' for framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:32.357070 8905 slave.cpp:1785] Launching task
'20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0' for framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:32.357969 8905 paths.cpp:547] Trying to chown
'/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0/runs/424fd077-caf5-4b67-90b0-760a23775ec1'
to user 'root'
I0530 10:49:32.358539 8905 slave.cpp:6480] Launching executor
'20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0' of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000 with resources cpus(*)(allocated:
*):0.1; mem(*)(allocated: *):32 in work directory
'/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0/runs/424fd077-caf5-4b67-90b0-760a23775ec1'
I0530 10:49:32.359091 8903 docker.cpp:1104] Skipping non-docker container
I0530 10:49:32.359094 8905 slave.cpp:2118] Queued task
'20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0' for executor
'20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0' of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:32.359357 8903 containerizer.cpp:992] Starting container
424fd077-caf5-4b67-90b0-760a23775ec1 for executor
'20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0' of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:32.542762 8902 provisioner.cpp:453] Provisioning image rootfs
'/var/lib/mesos/provisioner/containers/424fd077-caf5-4b67-90b0-760a23775ec1/backends/overlay/rootfses/ea0f760c-5079-4893-a5ec-469af9715abd'
for container 424fd077-caf5-4b67-90b0-760a23775ec1 using overlay backend
I0530 10:49:32.555275 8901 memory.cpp:451] Started listening for OOM events
for container 424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:32.555717 8901 memory.cpp:562] Started listening on 'low' memory
pressure events for container 424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:32.556246 8901 memory.cpp:562] Started listening on 'medium'
memory pressure events for container 424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:32.556879 8901 memory.cpp:562] Started listening on 'critical'
memory pressure events for container 424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:32.559020 8901 memory.cpp:199] Updated
'memory.soft_limit_in_bytes' to 160MB for container
424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:32.559835 8901 memory.cpp:251] Updated 'memory.limit_in_bytes' to
160MB for container 424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:32.560269 8901 cpu.cpp:101] Updated 'cpu.shares' to 204 (cpus 0.2)
for container 424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:32.566457 8902 linux_launcher.cpp:429] Launching container
424fd077-caf5-4b67-90b0-760a23775ec1 and cloning with namespaces CLONE_NEWNS |
CLONE_NEWUTS | CLONE_NEWNET
I0530 10:49:32.568379 8902 systemd.cpp:96] Assigned child process '9019' to
'mesos_executors.slice'
I0530 10:49:32.570240 8901 containerizer.cpp:1637] Checkpointing container's
forked pid 9019 to
'/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0/runs/424fd077-caf5-4b67-90b0-760a23775ec1/pids/forked.pid'
I0530 10:49:32.570843 8901 cni.cpp:872] Bind mounted '/proc/9019/ns/net' to
'/run/mesos/isolators/network/cni/424fd077-caf5-4b67-90b0-760a23775ec1/ns' for
container 424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:35.541935 8908 cni.cpp:1290] Got assigned IPv4 address
'192.168.235.73/32' from CNI network 'calico-net-1' for container
424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:35.542150 8908 cni.cpp:999] Unable to find DNS nameservers for
container 424fd077-caf5-4b67-90b0-760a23775ec1. Using host '/etc/resolv.conf'
I0530 10:49:35.703557 8906 slave.cpp:3385] Got registration for executor
'20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0' of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000 from executor(1)@192.168.235.73:38212
I0530 10:49:35.704555 8908 disk.cpp:207] Updating the disk resources for
container 424fd077-caf5-4b67-90b0-760a23775ec1 to cpus(*)(allocated: *):0.2;
mem(*)(allocated: *):160
I0530 10:49:35.705009 8906 memory.cpp:199] Updated
'memory.soft_limit_in_bytes' to 160MB for container
424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:35.705909 8906 cpu.cpp:101] Updated 'cpu.shares' to 204 (cpus 0.2)
for container 424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:35.706162 8906 slave.cpp:2331] Sending queued task
'20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0' to executor
'20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0' of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000 at executor(1)@192.168.235.73:38212
I0530 10:49:35.713090 8905 slave.cpp:3816] Handling status update TASK_RUNNING
(UUID: d99c77b3-7a18-4940-9e84-9bc4b7eea948) for task
20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000 from executor(1)@192.168.235.73:38212
I0530 10:49:35.713526 8905 status_update_manager.cpp:323] Received status
update TASK_RUNNING (UUID: d99c77b3-7a18-4940-9e84-9bc4b7eea948) for task
20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:35.713686 8905 status_update_manager.cpp:832] Checkpointing UPDATE
for status update TASK_RUNNING (UUID: d99c77b3-7a18-4940-9e84-9bc4b7eea948) for
task 20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:35.713820 8905 slave.cpp:4256] Forwarding the update TASK_RUNNING
(UUID: d99c77b3-7a18-4940-9e84-9bc4b7eea948) for task
20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to master@10.107.19.21:5050
I0530 10:49:35.713904 8905 slave.cpp:4166] Sending acknowledgement for status
update TASK_RUNNING (UUID: d99c77b3-7a18-4940-9e84-9bc4b7eea948) for task
20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to executor(1)@192.168.235.73:38212
I0530 10:49:35.757575 8905 status_update_manager.cpp:395] Received status
update acknowledgement (UUID: d99c77b3-7a18-4940-9e84-9bc4b7eea948) for task
20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:35.757642 8905 status_update_manager.cpp:832] Checkpointing ACK
for status update TASK_RUNNING (UUID: d99c77b3-7a18-4940-9e84-9bc4b7eea948) for
task 20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:35.801589 8905 slave.cpp:3816] Handling status update TASK_FAILED
(UUID: 75ca5b0c-7582-4524-9cde-a64039cecc97) for task
20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000 from executor(1)@192.168.235.73:38212
I0530 10:49:35.802206 8906 disk.cpp:207] Updating the disk resources for
container 424fd077-caf5-4b67-90b0-760a23775ec1 to cpus(*)(allocated: *):0.1;
mem(*)(allocated: *):32
I0530 10:49:35.802685 8908 memory.cpp:199] Updated
'memory.soft_limit_in_bytes' to 32MB for container
424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:35.803598 8908 cpu.cpp:101] Updated 'cpu.shares' to 102 (cpus 0.1)
for container 424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:35.803773 8908 status_update_manager.cpp:323] Received status
update TASK_FAILED (UUID: 75ca5b0c-7582-4524-9cde-a64039cecc97) for task
20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:35.803802 8908 status_update_manager.cpp:832] Checkpointing UPDATE
for status update TASK_FAILED (UUID: 75ca5b0c-7582-4524-9cde-a64039cecc97) for
task 20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:35.803984 8903 slave.cpp:4256] Forwarding the update TASK_FAILED
(UUID: 75ca5b0c-7582-4524-9cde-a64039cecc97) for task
20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to master@10.107.19.21:5050
I0530 10:49:35.804077 8903 slave.cpp:4166] Sending acknowledgement for status
update TASK_FAILED (UUID: 75ca5b0c-7582-4524-9cde-a64039cecc97) for task
20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to executor(1)@192.168.235.73:38212
I0530 10:49:35.817703 8906 status_update_manager.cpp:395] Received status
update acknowledgement (UUID: 75ca5b0c-7582-4524-9cde-a64039cecc97) for task
20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:35.817754 8906 status_update_manager.cpp:832] Checkpointing ACK
for status update TASK_FAILED (UUID: 75ca5b0c-7582-4524-9cde-a64039cecc97) for
task 20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:36.804152 8906 slave.cpp:4388] Got exited event for
executor(1)@192.168.235.73:38212
I0530 10:49:36.847506 8906 containerizer.cpp:2479] Container
424fd077-caf5-4b67-90b0-760a23775ec1 has exited
I0530 10:49:36.847568 8906 containerizer.cpp:2116] Destroying container
424fd077-caf5-4b67-90b0-760a23775ec1 in RUNNING state
I0530 10:49:36.847664 8906 linux_launcher.cpp:505] Asked to destroy container
424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:36.848474 8906 linux_launcher.cpp:548] Using freezer to destroy
cgroup mesos/424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:36.849936 8906 cgroups.cpp:2692] Freezing cgroup
/sys/fs/cgroup/freezer/mesos/424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:36.851379 8907 cgroups.cpp:1405] Successfully froze cgroup
/sys/fs/cgroup/freezer/mesos/424fd077-caf5-4b67-90b0-760a23775ec1 after
1.411072ms
I0530 10:49:36.853444 8906 cgroups.cpp:2710] Thawing cgroup
/sys/fs/cgroup/freezer/mesos/424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:36.855401 8906 cgroups.cpp:1434] Successfully thawed cgroup
/sys/fs/cgroup/freezer/mesos/424fd077-caf5-4b67-90b0-760a23775ec1 after
1.910016ms
I0530 10:49:37.375128 8902 slave.cpp:1625] Got assigned task
'20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0' for framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:37.375424 8902 slave.cpp:1785] Launching task
'20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0' for framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:37.375635 8902 paths.cpp:547] Trying to chown
'/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0/runs/1248ce41-4143-4adb-98d0-04c6cf8ff000'
to user 'root'
I0530 10:49:37.376144 8902 slave.cpp:6480] Launching executor
'20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0' of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000 with resources cpus(*)(allocated:
*):0.1; mem(*)(allocated: *):32 in work directory
'/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0/runs/1248ce41-4143-4adb-98d0-04c6cf8ff000'
I0530 10:49:37.376639 8902 slave.cpp:2118] Queued task
'20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0' for executor
'20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0' of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:37.376938 8902 docker.cpp:1104] Skipping non-docker container
I0530 10:49:37.377231 8902 containerizer.cpp:992] Starting container
1248ce41-4143-4adb-98d0-04c6cf8ff000 for executor
'20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0' of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:37.565088 8901 provisioner.cpp:453] Provisioning image rootfs
'/var/lib/mesos/provisioner/containers/1248ce41-4143-4adb-98d0-04c6cf8ff000/backends/overlay/rootfses/585e6e95-cd8b-4ed9-8341-49360770675b'
for container 1248ce41-4143-4adb-98d0-04c6cf8ff000 using overlay backend
I0530 10:49:37.578210 8901 memory.cpp:451] Started listening for OOM events
for container 1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:37.578721 8901 memory.cpp:562] Started listening on 'low' memory
pressure events for container 1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:37.579267 8901 memory.cpp:562] Started listening on 'medium'
memory pressure events for container 1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:37.579759 8901 memory.cpp:562] Started listening on 'critical'
memory pressure events for container 1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:37.581714 8901 memory.cpp:199] Updated
'memory.soft_limit_in_bytes' to 160MB for container
1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:37.582581 8901 memory.cpp:251] Updated 'memory.limit_in_bytes' to
160MB for container 1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:37.583048 8901 cpu.cpp:101] Updated 'cpu.shares' to 204 (cpus 0.2)
for container 1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:37.585544 8905 linux_launcher.cpp:429] Launching container
1248ce41-4143-4adb-98d0-04c6cf8ff000 and cloning with namespaces CLONE_NEWNS |
CLONE_NEWUTS | CLONE_NEWNET
I0530 10:49:37.587668 8905 systemd.cpp:96] Assigned child process '9101' to
'mesos_executors.slice'
I0530 10:49:37.589622 8903 containerizer.cpp:1637] Checkpointing container's
forked pid 9101 to
'/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0/runs/1248ce41-4143-4adb-98d0-04c6cf8ff000/pids/forked.pid'
I0530 10:49:37.590232 8904 cni.cpp:872] Bind mounted '/proc/9101/ns/net' to
'/run/mesos/isolators/network/cni/1248ce41-4143-4adb-98d0-04c6cf8ff000/ns' for
container 1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:38.826182 8907 http.cpp:307] HTTP GET for /slave(1)/state from
10.114.99.73:54369 with User-Agent='Mozilla/5.0 (Macintosh; Intel Mac OS X
10_12_5) AppleWebKit/603.2.4 (KHTML, like Gecko) Version/10.1.1 Safari/603.2.4'
I0530 10:49:39.358474 8905 cni.cpp:1468] Unmounted the network namespace
handle
'/run/mesos/isolators/network/cni/424fd077-caf5-4b67-90b0-760a23775ec1/ns' for
container 424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:39.358549 8905 cni.cpp:1479] Removed the container directory
'/run/mesos/isolators/network/cni/424fd077-caf5-4b67-90b0-760a23775ec1'
I0530 10:49:39.378610 8901 provisioner.cpp:570] Destroying container rootfs at
'/var/lib/mesos/provisioner/containers/424fd077-caf5-4b67-90b0-760a23775ec1/backends/overlay/rootfses/ea0f760c-5079-4893-a5ec-469af9715abd'
for container 424fd077-caf5-4b67-90b0-760a23775ec1
I0530 10:49:39.380601 8907 slave.cpp:4769] Executor
'20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0' of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000 exited with status 0
I0530 10:49:39.380643 8907 slave.cpp:4869] Cleaning up executor
'20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0' of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000 at executor(1)@192.168.235.73:38212
I0530 10:49:39.380894 8906 gc.cpp:55] Scheduling
'/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0/runs/424fd077-caf5-4b67-90b0-760a23775ec1'
for gc 6.99999559317037days in the future
I0530 10:49:39.380941 8906 gc.cpp:55] Scheduling
'/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0'
for gc 6.99999559270815days in the future
I0530 10:49:39.380957 8906 gc.cpp:55] Scheduling
'/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0/runs/424fd077-caf5-4b67-90b0-760a23775ec1'
for gc 6.99999559248889days in the future
I0530 10:49:39.380970 8906 gc.cpp:55] Scheduling
'/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0'
for gc 6.99999559230222days in the future
I0530 10:49:39.961740 8904 cni.cpp:1290] Got assigned IPv4 address
'192.168.235.74/32' from CNI network 'calico-net-1' for container
1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:39.961976 8904 cni.cpp:999] Unable to find DNS nameservers for
container 1248ce41-4143-4adb-98d0-04c6cf8ff000. Using host '/etc/resolv.conf'
I0530 10:49:40.125231 8902 slave.cpp:3385] Got registration for executor
'20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0' of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000 from executor(1)@192.168.235.74:43999
I0530 10:49:40.126565 8902 memory.cpp:199] Updated
'memory.soft_limit_in_bytes' to 160MB for container
1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:40.127490 8902 cpu.cpp:101] Updated 'cpu.shares' to 204 (cpus 0.2)
for container 1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:40.127542 8902 disk.cpp:207] Updating the disk resources for
container 1248ce41-4143-4adb-98d0-04c6cf8ff000 to cpus(*)(allocated: *):0.2;
mem(*)(allocated: *):160
I0530 10:49:40.127712 8902 slave.cpp:2331] Sending queued task
'20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0' to executor
'20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0' of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000 at executor(1)@192.168.235.74:43999
I0530 10:49:40.134186 8902 slave.cpp:3816] Handling status update TASK_RUNNING
(UUID: 3f7b6737-e154-47cb-93e7-4e0c55b233d3) for task
20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000 from executor(1)@192.168.235.74:43999
I0530 10:49:40.134565 8902 status_update_manager.cpp:323] Received status
update TASK_RUNNING (UUID: 3f7b6737-e154-47cb-93e7-4e0c55b233d3) for task
20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:40.134727 8902 status_update_manager.cpp:832] Checkpointing UPDATE
for status update TASK_RUNNING (UUID: 3f7b6737-e154-47cb-93e7-4e0c55b233d3) for
task 20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:40.134851 8902 slave.cpp:4256] Forwarding the update TASK_RUNNING
(UUID: 3f7b6737-e154-47cb-93e7-4e0c55b233d3) for task
20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to master@10.107.19.21:5050
I0530 10:49:40.134941 8902 slave.cpp:4166] Sending acknowledgement for status
update TASK_RUNNING (UUID: 3f7b6737-e154-47cb-93e7-4e0c55b233d3) for task
20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to executor(1)@192.168.235.74:43999
I0530 10:49:40.143187 8908 status_update_manager.cpp:395] Received status
update acknowledgement (UUID: 3f7b6737-e154-47cb-93e7-4e0c55b233d3) for task
20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:40.143262 8908 status_update_manager.cpp:832] Checkpointing ACK
for status update TASK_RUNNING (UUID: 3f7b6737-e154-47cb-93e7-4e0c55b233d3) for
task 20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:40.223562 8908 slave.cpp:3816] Handling status update TASK_FAILED
(UUID: 4dec875b-3c98-4c13-943b-9aa9f015fa64) for task
20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000 from executor(1)@192.168.235.74:43999
I0530 10:49:40.224624 8908 memory.cpp:199] Updated
'memory.soft_limit_in_bytes' to 32MB for container
1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:40.225495 8903 disk.cpp:207] Updating the disk resources for
container 1248ce41-4143-4adb-98d0-04c6cf8ff000 to cpus(*)(allocated: *):0.1;
mem(*)(allocated: *):32
I0530 10:49:40.225520 8908 cpu.cpp:101] Updated 'cpu.shares' to 102 (cpus 0.1)
for container 1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:40.225795 8908 status_update_manager.cpp:323] Received status
update TASK_FAILED (UUID: 4dec875b-3c98-4c13-943b-9aa9f015fa64) for task
20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:40.225888 8908 status_update_manager.cpp:832] Checkpointing UPDATE
for status update TASK_FAILED (UUID: 4dec875b-3c98-4c13-943b-9aa9f015fa64) for
task 20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:40.226018 8902 slave.cpp:4256] Forwarding the update TASK_FAILED
(UUID: 4dec875b-3c98-4c13-943b-9aa9f015fa64) for task
20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to master@10.107.19.21:5050
I0530 10:49:40.226104 8902 slave.cpp:4166] Sending acknowledgement for status
update TASK_FAILED (UUID: 4dec875b-3c98-4c13-943b-9aa9f015fa64) for task
20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to executor(1)@192.168.235.74:43999
I0530 10:49:40.233405 8902 status_update_manager.cpp:395] Received status
update acknowledgement (UUID: 4dec875b-3c98-4c13-943b-9aa9f015fa64) for task
20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:40.233520 8902 status_update_manager.cpp:832] Checkpointing ACK
for status update TASK_FAILED (UUID: 4dec875b-3c98-4c13-943b-9aa9f015fa64) for
task 20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:41.225358 8903 slave.cpp:4388] Got exited event for
executor(1)@192.168.235.74:43999
I0530 10:49:41.266679 8903 containerizer.cpp:2479] Container
1248ce41-4143-4adb-98d0-04c6cf8ff000 has exited
I0530 10:49:41.266719 8903 containerizer.cpp:2116] Destroying container
1248ce41-4143-4adb-98d0-04c6cf8ff000 in RUNNING state
I0530 10:49:41.266774 8903 linux_launcher.cpp:505] Asked to destroy container
1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:41.267336 8903 linux_launcher.cpp:548] Using freezer to destroy
cgroup mesos/1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:41.268402 8903 cgroups.cpp:2692] Freezing cgroup
/sys/fs/cgroup/freezer/mesos/1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:41.269942 8903 cgroups.cpp:1405] Successfully froze cgroup
/sys/fs/cgroup/freezer/mesos/1248ce41-4143-4adb-98d0-04c6cf8ff000 after
1.48992ms
I0530 10:49:41.271653 8902 cgroups.cpp:2710] Thawing cgroup
/sys/fs/cgroup/freezer/mesos/1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:41.273180 8903 cgroups.cpp:1434] Successfully thawed cgroup
/sys/fs/cgroup/freezer/mesos/1248ce41-4143-4adb-98d0-04c6cf8ff000 after
1.490944ms
I0530 10:49:42.395020 8908 slave.cpp:1625] Got assigned task
'20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0' for framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:42.395308 8908 slave.cpp:1785] Launching task
'20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0' for framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:42.396659 8908 paths.cpp:547] Trying to chown
'/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0/runs/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8'
to user 'root'
I0530 10:49:42.406666 8908 slave.cpp:6480] Launching executor
'20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0' of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000 with resources cpus(*)(allocated:
*):0.1; mem(*)(allocated: *):32 in work directory
'/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0/runs/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8'
I0530 10:49:42.407295 8908 slave.cpp:2118] Queued task
'20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0' for executor
'20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0' of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:42.407347 8908 docker.cpp:1104] Skipping non-docker container
I0530 10:49:42.407714 8908 containerizer.cpp:992] Starting container
fd4bb1b7-0fb1-46ad-860d-3080d1670ba8 for executor
'20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0' of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:42.576038 8903 provisioner.cpp:453] Provisioning image rootfs
'/var/lib/mesos/provisioner/containers/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8/backends/overlay/rootfses/843753d1-28f7-4bf1-bbe5-84463f30fa2b'
for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8 using overlay backend
I0530 10:49:42.588337 8907 memory.cpp:451] Started listening for OOM events
for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:42.588832 8907 memory.cpp:562] Started listening on 'low' memory
pressure events for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:42.589339 8907 memory.cpp:562] Started listening on 'medium'
memory pressure events for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:42.589799 8907 memory.cpp:562] Started listening on 'critical'
memory pressure events for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:42.591732 8907 memory.cpp:199] Updated
'memory.soft_limit_in_bytes' to 160MB for container
fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:42.593526 8907 memory.cpp:251] Updated 'memory.limit_in_bytes' to
160MB for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:42.593991 8907 cpu.cpp:101] Updated 'cpu.shares' to 204 (cpus 0.2)
for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:42.596405 8907 linux_launcher.cpp:429] Launching container
fd4bb1b7-0fb1-46ad-860d-3080d1670ba8 and cloning with namespaces CLONE_NEWNS |
CLONE_NEWUTS | CLONE_NEWNET
I0530 10:49:42.598201 8907 systemd.cpp:96] Assigned child process '9206' to
'mesos_executors.slice'
I0530 10:49:42.600131 8903 containerizer.cpp:1637] Checkpointing container's
forked pid 9206 to
'/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0/runs/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8/pids/forked.pid'
I0530 10:49:42.600713 8901 cni.cpp:872] Bind mounted '/proc/9206/ns/net' to
'/run/mesos/isolators/network/cni/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8/ns' for
container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:42.974720 8905 cni.cpp:1468] Unmounted the network namespace
handle
'/run/mesos/isolators/network/cni/1248ce41-4143-4adb-98d0-04c6cf8ff000/ns' for
container 1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:42.974813 8905 cni.cpp:1479] Removed the container directory
'/run/mesos/isolators/network/cni/1248ce41-4143-4adb-98d0-04c6cf8ff000'
I0530 10:49:42.987810 8902 provisioner.cpp:570] Destroying container rootfs at
'/var/lib/mesos/provisioner/containers/1248ce41-4143-4adb-98d0-04c6cf8ff000/backends/overlay/rootfses/585e6e95-cd8b-4ed9-8341-49360770675b'
for container 1248ce41-4143-4adb-98d0-04c6cf8ff000
I0530 10:49:42.989212 8908 slave.cpp:4769] Executor
'20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0' of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000 exited with status 0
I0530 10:49:42.989243 8908 slave.cpp:4869] Cleaning up executor
'20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0' of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000 at executor(1)@192.168.235.74:43999
I0530 10:49:42.989409 8908 gc.cpp:55] Scheduling
'/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0/runs/1248ce41-4143-4adb-98d0-04c6cf8ff000'
for gc 6.99998854937482days in the future
I0530 10:49:42.989439 8908 gc.cpp:55] Scheduling
'/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0'
for gc 6.9999885490163days in the future
I0530 10:49:42.989454 8908 gc.cpp:55] Scheduling
'/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0/runs/1248ce41-4143-4adb-98d0-04c6cf8ff000'
for gc 6.99998854880889days in the future
I0530 10:49:42.989467 8908 gc.cpp:55] Scheduling
'/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0'
for gc 6.99998854864593days in the future
I0530 10:49:45.888653 8906 cni.cpp:1290] Got assigned IPv4 address
'192.168.235.75/32' from CNI network 'calico-net-1' for container
fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:45.888829 8906 cni.cpp:999] Unable to find DNS nameservers for
container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8. Using host '/etc/resolv.conf'
I0530 10:49:46.050271 8902 slave.cpp:3385] Got registration for executor
'20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0' of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000 from executor(1)@192.168.235.75:40717
I0530 10:49:46.051580 8902 memory.cpp:199] Updated
'memory.soft_limit_in_bytes' to 160MB for container
fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:46.052515 8902 cpu.cpp:101] Updated 'cpu.shares' to 204 (cpus 0.2)
for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:46.052567 8902 disk.cpp:207] Updating the disk resources for
container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8 to cpus(*)(allocated: *):0.2;
mem(*)(allocated: *):160
I0530 10:49:46.052742 8902 slave.cpp:2331] Sending queued task
'20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0' to executor
'20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0' of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000 at executor(1)@192.168.235.75:40717
I0530 10:49:46.056938 8902 slave.cpp:3816] Handling status update TASK_RUNNING
(UUID: fb7d9371-cd0e-49c6-943b-e897f7afbd62) for task
20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000 from executor(1)@192.168.235.75:40717
I0530 10:49:46.057296 8902 status_update_manager.cpp:323] Received status
update TASK_RUNNING (UUID: fb7d9371-cd0e-49c6-943b-e897f7afbd62) for task
20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:46.057456 8902 status_update_manager.cpp:832] Checkpointing UPDATE
for status update TASK_RUNNING (UUID: fb7d9371-cd0e-49c6-943b-e897f7afbd62) for
task 20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:46.057593 8902 slave.cpp:4256] Forwarding the update TASK_RUNNING
(UUID: fb7d9371-cd0e-49c6-943b-e897f7afbd62) for task
20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to master@10.107.19.21:5050
I0530 10:49:46.057663 8902 slave.cpp:4166] Sending acknowledgement for status
update TASK_RUNNING (UUID: fb7d9371-cd0e-49c6-943b-e897f7afbd62) for task
20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to executor(1)@192.168.235.75:40717
I0530 10:49:46.064831 8902 status_update_manager.cpp:395] Received status
update acknowledgement (UUID: fb7d9371-cd0e-49c6-943b-e897f7afbd62) for task
20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:46.064899 8902 status_update_manager.cpp:832] Checkpointing ACK
for status update TASK_RUNNING (UUID: fb7d9371-cd0e-49c6-943b-e897f7afbd62) for
task 20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:46.148407 8902 slave.cpp:3816] Handling status update TASK_FAILED
(UUID: 68132cf3-98fd-4d35-a8cc-c72fe3924d70) for task
20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000 from executor(1)@192.168.235.75:40717
I0530 10:49:46.149554 8908 memory.cpp:199] Updated
'memory.soft_limit_in_bytes' to 32MB for container
fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:46.150315 8901 disk.cpp:207] Updating the disk resources for
container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8 to cpus(*)(allocated: *):0.1;
mem(*)(allocated: *):32
I0530 10:49:46.150490 8908 cpu.cpp:101] Updated 'cpu.shares' to 102 (cpus 0.1)
for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:46.150804 8908 status_update_manager.cpp:323] Received status
update TASK_FAILED (UUID: 68132cf3-98fd-4d35-a8cc-c72fe3924d70) for task
20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:46.150835 8908 status_update_manager.cpp:832] Checkpointing UPDATE
for status update TASK_FAILED (UUID: 68132cf3-98fd-4d35-a8cc-c72fe3924d70) for
task 20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:46.151010 8903 slave.cpp:4256] Forwarding the update TASK_FAILED
(UUID: 68132cf3-98fd-4d35-a8cc-c72fe3924d70) for task
20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to master@10.107.19.21:5050
I0530 10:49:46.151093 8903 slave.cpp:4166] Sending acknowledgement for status
update TASK_FAILED (UUID: 68132cf3-98fd-4d35-a8cc-c72fe3924d70) for task
20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to executor(1)@192.168.235.75:40717
I0530 10:49:46.158910 8904 status_update_manager.cpp:395] Received status
update acknowledgement (UUID: 68132cf3-98fd-4d35-a8cc-c72fe3924d70) for task
20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:46.158972 8904 status_update_manager.cpp:832] Checkpointing ACK
for status update TASK_FAILED (UUID: 68132cf3-98fd-4d35-a8cc-c72fe3924d70) for
task 20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000
I0530 10:49:47.151319 8903 slave.cpp:4388] Got exited event for
executor(1)@192.168.235.75:40717
I0530 10:49:47.193552 8903 containerizer.cpp:2479] Container
fd4bb1b7-0fb1-46ad-860d-3080d1670ba8 has exited
I0530 10:49:47.193584 8903 containerizer.cpp:2116] Destroying container
fd4bb1b7-0fb1-46ad-860d-3080d1670ba8 in RUNNING state
I0530 10:49:47.193639 8903 linux_launcher.cpp:505] Asked to destroy container
fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:47.194249 8903 linux_launcher.cpp:548] Using freezer to destroy
cgroup mesos/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:47.195314 8903 cgroups.cpp:2692] Freezing cgroup
/sys/fs/cgroup/freezer/mesos/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:47.196699 8904 cgroups.cpp:1405] Successfully froze cgroup
/sys/fs/cgroup/freezer/mesos/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8 after
1.355008ms
I0530 10:49:47.198119 8901 cgroups.cpp:2710] Thawing cgroup
/sys/fs/cgroup/freezer/mesos/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:47.199431 8904 cgroups.cpp:1434] Successfully thawed cgroup
/sys/fs/cgroup/freezer/mesos/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8 after
1.277952ms
I0530 10:49:48.698365 8906 cni.cpp:1468] Unmounted the network namespace
handle
'/run/mesos/isolators/network/cni/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8/ns' for
container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:48.698437 8906 cni.cpp:1479] Removed the container directory
'/run/mesos/isolators/network/cni/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8'
I0530 10:49:48.714375 8906 provisioner.cpp:570] Destroying container rootfs at
'/var/lib/mesos/provisioner/containers/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8/backends/overlay/rootfses/843753d1-28f7-4bf1-bbe5-84463f30fa2b'
for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
I0530 10:49:48.715656 8906 slave.cpp:4769] Executor
'20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0' of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000 exited with status 0
I0530 10:49:48.715685 8906 slave.cpp:4869] Cleaning up executor
'20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0' of framework
562f1892-41c6-4c55-955e-662c63ab3ddf-0000 at executor(1)@192.168.235.75:40717
I0530 10:49:48.715879 8906 gc.cpp:55] Scheduling
'/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0/runs/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8'
for gc 6.99999171553185days in the future
I0530 10:49:48.715922 8906 gc.cpp:55] Scheduling
'/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0'
for gc 6.99999171524444days in the future
I0530 10:49:48.715948 8906 gc.cpp:55] Scheduling
'/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0/runs/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8'
for gc 6.99999171495407days in the future
I0530 10:49:48.715963 8906 gc.cpp:55] Scheduling
'/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0'
for gc 6.99999171468741days in the future
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
