[
https://issues.apache.org/jira/browse/MESOS-7587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16162996#comment-16162996
]
Sebastian Gerlach commented on MESOS-7587:
------------------------------------------
We have done some more research / debugging on this issue. We could isolate the
problem a little bit further. If we switch from overlay backend to the copy
backend we are able to start processes with the Mesos Containerizer without any
problem.
With strace we try to analyze why we get the message “No such file or
directory”. It looks like the path for “python” wasn’t correct. Even if we use
the full path it didn’t show up and we couldn’t start the container. We didn’t
follow this path to deeply. So treat the strace results with care.
So we think there is a problem with overlayfs. If we remove the fetcher tmp
folder we could start the container again. But if we restore the fetcher folder
we could reproduce the behavior.
We are not sure when and why the cache or the overlay breaks.
> Launching tasks with the Mesos Containerizer after a long time without
> launching new tasks fails
> ------------------------------------------------------------------------------------------------
>
> Key: MESOS-7587
> URL: https://issues.apache.org/jira/browse/MESOS-7587
> Project: Mesos
> Issue Type: Bug
> Components: containerization
> Affects Versions: 1.2.0
> Reporter: Alexander Rojas
> Priority: Critical
> Labels: mesos-containerizer, mesosphere
>
> After having a cluster running without launching new tasks for an extended
> period of time, ~1week. When launching a new task using the Mesos
> Containerizer, the task fails to launch with the error:
> [{{Failed to execute command: No such file or
> directory}}|https://github.com/apache/mesos/blob/8245981b889ec3725cc0be4150b15d1fe9d64b86/src/slave/containerizer/mesos/launch.cpp#L778]
> The task is launched from Marathon with the app definition:
> {code}
> {
> "container": {
> "type": "MESOS",
> "docker": {
> "forcePullImage": true,
> "image": "private.repository.local/updated:fixed",
> "privileged": false
> }
> },
> "cpus": 0.1,
> "id": "/20150530/mesos9",
> "instances": 1,
> "minimumHealthCapacity": 1,
> "acceptedResourceRoles": ["*"],
> "constraints": [["hostname", "UNIQUE"]],
> "mem": 128
> }
> {code}
> and {{Dockerfile}}
> {code}
> FROM private.repository.local/centos:stable
> MAINTAINER Sebastian Gerlach "[email protected]"
> CMD python -m SimpleHTTPServer 80
> {code}
> The obtained stdout is:
> {noformat}
> Executing pre-exec command
> '{"arguments":["mesos-containerizer","mount","--help=false","--operation=make-rslave","--path=\/"],"shell":false,"value":"\/usr\/libexec\/mesos\/mesos-containerizer"}'
> Executing pre-exec command
> '{"arguments":["mount","-n","--rbind","\/var\/lib\/mesos\/slaves\/562f1892-41c6-4c55-955e-662c63ab3ddf-S8\/frameworks\/562f1892-41c6-4c55-955e-662c63ab3ddf-0000\/executors\/20150530_mesos13.5f8c3db1-4517-11e7-8d95-024276be29e0\/runs\/da1cb066-1354-42a6-82bc-eee688d61b94","\/var\/lib\/mesos\/provisioner\/containers\/da1cb066-1354-42a6-82bc-eee688d61b94\/backends\/overlay\/rootfses\/f74816b3-c115-4a5e-a51c-fbd502efd1fe\/mnt\/mesos\/sandbox"],"shell":false,"value":"mount"}'
> Received SUBSCRIBED event
> Subscribed executor on bp-mesos8.private.local
> Received LAUNCH event
> Starting task 20150530_mesos13.5f8c3db1-4517-11e7-8d95-024276be29e0
> /usr/libexec/mesos/mesos-containerizer launch --help="false"
> --launch_info="{"command":{"arguments":["\/bin\/sh","-c","python -m
> SimpleHTTPServer
> 80"],"environment":{"variables":[{"name":"MARATHON_APP_VERSION","value":"2017-05-30T09:06:59.666Z"},{"name":"HOST","value":"bp-mesos8.private.local"},{"name":"MARATHON_APP_RESOURCE_CPUS","value":"0.1"},{"name":"MARATHON_APP_RESOURCE_GPUS","value":"0"},{"name":"MESOS_TASK_ID","value":"20150530_mesos13.5f8c3db1-4517-11e7-8d95-024276be29e0"},{"name":"MARATHON_APP_RESOURCE_MEM","value":"128.0"},{"name":"MARATHON_APP_RESOURCE_DISK","value":"0.0"},{"name":"MARATHON_APP_LABELS","value":""},{"name":"MARATHON_APP_ID","value":"\/20150530\/mesos13"}]},"shell":false,"value":"\/bin\/sh"},"environment":{"variables":[{"name":"HOST","value":"bp-mesos8.private.local"},{"name":"LIBPROCESS_IP","value":"0.0.0.0"},{"name":"LIBPROCESS_PORT","value":"0"},{"name":"MARATHON_APP_ID","value":"\/20150530\/mesos13"},{"name":"MARATHON_APP_LABELS","value":""},{"name":"MARATHON_APP_RESOURCE_CPUS","value":"0.1"},{"name":"MARATHON_APP_RESOURCE_DISK","value":"0.0"},{"name":"MARATHON_APP_RESOURCE_GPUS","value":"0"},{"name":"MARATHON_APP_RESOURCE_MEM","value":"128.0"},{"name":"MARATHON_APP_VERSION","value":"2017-05-30T09:06:59.666Z"},{"name":"MESOS_AGENT_ENDPOINT","value":"10.107.19.26:5051"},{"name":"MESOS_CHECKPOINT","value":"1"},{"name":"MESOS_DIRECTORY","value":"\/var\/lib\/mesos\/slaves\/562f1892-41c6-4c55-955e-662c63ab3ddf-S8\/frameworks\/562f1892-41c6-4c55-955e-662c63ab3ddf-0000\/executors\/20150530_mesos13.5f8c3db1-4517-11e7-8d95-024276be29e0\/runs\/da1cb066-1354-42a6-82bc-eee688d61b94"},{"name":"MESOS_EXECUTOR_ID","value":"20150530_mesos13.5f8c3db1-4517-11e7-8d95-024276be29e0"},{"name":"MESOS_EXECUTOR_SHUTDOWN_GRACE_PERIOD","value":"5secs"},{"name":"MESOS_FRAMEWORK_ID","value":"562f1892-41c6-4c55-955e-662c63ab3ddf-0000"},{"name":"MESOS_HTTP_COMMAND_EXECUTOR","value":"0"},{"name":"MESOS_NATIVE_JAVA_LIBRARY","value":"\/usr\/lib\/libmesos-1.2.0.so"},{"name":"MESOS_NATIVE_LIBRARY","value":"\/usr\/lib\/libmesos-1.2.0.so"},{"name":"MESOS_RECOVERY_TIMEOUT","value":"15mins"},{"name":"MESOS_SANDBOX","value":"\/mnt\/mesos\/sandbox"},{"name":"MESOS_SLAVE_ID","value":"562f1892-41c6-4c55-955e-662c63ab3ddf-S8"},{"name":"MESOS_SLAVE_PID","value":"slave(1)@10.107.19.26:5051"},{"name":"MESOS_SUBSCRIPTION_BACKOFF_MAX","value":"2secs"},{"name":"MESOS_TASK_ID","value":"20150530_mesos13.5f8c3db1-4517-11e7-8d95-024276be29e0"},{"name":"PATH","value":"\/usr\/local\/sbin:\/usr\/local\/bin:\/usr\/sbin:\/usr\/bin:\/sbin:\/bin"},{"name":"PATH","value":"\/usr\/local\/sbin:\/usr\/local\/bin:\/usr\/sbin:\/usr\/bin:\/sbin:\/bin"},{"name":"ftp_proxy","value":"http:\/\/bp-proxy.private.local:3128"},{"name":"http_proxy","value":"http:\/\/bp-proxy.private.local:3128"},{"name":"https_proxy","value":"http:\/\/bp-proxy.private.local:3128"},{"name":"no_proxy","value":"localhost,127.0.0.1,svc.boreus.de,private.local"},{"name":"MARATHON_APP_VERSION","value":"2017-05-30T09:06:59.666Z"},{"name":"HOST","value":"bp-mesos8.private.local"},{"name":"MARATHON_APP_RESOURCE_CPUS","value":"0.1"},{"name":"MARATHON_APP_RESOURCE_GPUS","value":"0"},{"name":"MESOS_TASK_ID","value":"20150530_mesos13.5f8c3db1-4517-11e7-8d95-024276be29e0"},{"name":"MARATHON_APP_RESOURCE_MEM","value":"128.0"},{"name":"MARATHON_APP_RESOURCE_DISK","value":"0.0"},{"name":"MARATHON_APP_LABELS","value":""},{"name":"MARATHON_APP_ID","value":"\/20150530\/mesos13"}]},"rootfs":"\/var\/lib\/mesos\/provisioner\/containers\/da1cb066-1354-42a6-82bc-eee688d61b94\/backends\/overlay\/rootfses\/f74816b3-c115-4a5e-a51c-fbd502efd1fe","user":"root","working_directory":"\/mnt\/mesos\/sandbox"}"
> --unshare_namespace_mnt="true"
> Forked command at 10438
> Changing root to
> /var/lib/mesos/provisioner/containers/da1cb066-1354-42a6-82bc-eee688d61b94/backends/overlay/rootfses/f74816b3-c115-4a5e-a51c-fbd502efd1fe
> Overwriting environment variable 'PATH', original:
> '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', new:
> '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
> Overwriting environment variable 'MARATHON_APP_VERSION', original:
> '2017-05-30T09:06:59.666Z', new: '2017-05-30T09:06:59.666Z'
> Overwriting environment variable 'HOST', original: 'bp-mesos8.private.local',
> new: 'bp-mesos8.private.local'
> Overwriting environment variable 'MARATHON_APP_RESOURCE_CPUS', original:
> '0.1', new: '0.1'
> Overwriting environment variable 'MARATHON_APP_RESOURCE_GPUS', original: '0',
> new: '0'
> Overwriting environment variable 'MESOS_TASK_ID', original:
> '20150530_mesos13.5f8c3db1-4517-11e7-8d95-024276be29e0', new:
> '20150530_mesos13.5f8c3db1-4517-11e7-8d95-024276be29e0'
> Overwriting environment variable 'MARATHON_APP_RESOURCE_MEM', original:
> '128.0', new: '128.0'
> Overwriting environment variable 'MARATHON_APP_RESOURCE_DISK', original:
> '0.0', new: '0.0'
> Overwriting environment variable 'MARATHON_APP_LABELS', original: '', new: ''
> Overwriting environment variable 'MARATHON_APP_ID', original:
> '/20150530/mesos13', new: '/20150530/mesos13'
> Command exited with status 1 (pid: 10438)
> {noformat}
> And a portion of the logs:
> {noformat}
> tail -f mesos-slave.ERROR mesos-slave.INFO mesos-slave.WARNING
> ==> mesos-slave.ERROR <==
> Log file created at: 2017/05/30 10:48:17
> Running on machine: bp-mesos7.private.local
> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
> E0530 10:48:17.653518 8876 shell.hpp:107] Command 'hadoop version 2>&1'
> failed; this is the output:
> sh: hadoop: command not found
> ==> mesos-slave.INFO <==
> I0530 10:48:19.953788 8906 slave.cpp:5558] Finished recovery
> I0530 10:48:19.954376 8906 slave.cpp:931] New master detected at
> [email protected]:5050
> I0530 10:48:19.954401 8906 slave.cpp:955] No credentials provided.
> Attempting to register without authentication
> I0530 10:48:19.954416 8906 slave.cpp:966] Detecting new master
> I0530 10:48:19.954444 8906 status_update_manager.cpp:177] Pausing sending
> status updates
> I0530 10:48:20.208361 8907 slave.cpp:1242] Re-registered with master
> [email protected]:5050
> I0530 10:48:20.208427 8907 slave.cpp:1279] Forwarding total oversubscribed
> resources {}
> I0530 10:48:20.208452 8906 status_update_manager.cpp:184] Resuming sending
> status updates
> I0530 10:48:20.208501 8907 slave.cpp:2827] Updating framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 pid to
> [email protected]:39431
> I0530 10:48:20.208871 8907 status_update_manager.cpp:184] Resuming sending
> status updates
> ==> mesos-slave.WARNING <==
> Log file created at: 2017/05/30 10:48:17
> Running on machine: bp-mesos7.private.local
> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
> E0530 10:48:17.653518 8876 shell.hpp:107] Command 'hadoop version 2>&1'
> failed; this is the output:
> sh: hadoop: command not found
> W0530 10:48:17.694730 8903 state.cpp:477] Failed to find executor forked pid
> file
> '/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150530_mesos1.7c34f793-44fd-11e7-8f05-02420d613ec3/runs/0ac924bb-4ba5-41e9-b563-6768329f6f7d/pids/forked.pid'
> W0530 10:48:17.695217 8903 state.cpp:477] Failed to find executor forked pid
> file
> '/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150530_mesos1.7041e28f-44fd-11e7-8f05-02420d613ec3/runs/d8afe43c-292c-4069-88d7-7edf4a730575/pids/forked.pid'
> W0530 10:48:17.713217 8903 state.cpp:477] Failed to find executor forked pid
> file
> '/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150530_mesos1.0cad114c-44fe-11e7-8f05-02420d613ec3/runs/3adaddbb-5fd9-4ca6-8225-660daf91a7d9/pids/forked.pid'
> W0530 10:48:17.741708 8904 linux_launcher.cpp:357] Couldn't find pid '14251'
> in 'mesos_executors.slice'. This can lead to lack of proper resource isolation
> W0530 10:48:17.741747 8904 linux_launcher.cpp:357] Couldn't find pid '19408'
> in 'mesos_executors.slice'. This can lead to lack of proper resource isolation
> ==> mesos-slave.INFO <==
> I0530 10:49:17.692493 8903 slave.cpp:5267] Current disk usage 0.88%. Max
> allowed age: 6.238522372392419days
> I0530 10:49:32.356741 8905 slave.cpp:1625] Got assigned task
> '20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0' for framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:32.357070 8905 slave.cpp:1785] Launching task
> '20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0' for framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:32.357969 8905 paths.cpp:547] Trying to chown
> '/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0/runs/424fd077-caf5-4b67-90b0-760a23775ec1'
> to user 'root'
> I0530 10:49:32.358539 8905 slave.cpp:6480] Launching executor
> '20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0' of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 with resources cpus(*)(allocated:
> *):0.1; mem(*)(allocated: *):32 in work directory
> '/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0/runs/424fd077-caf5-4b67-90b0-760a23775ec1'
> I0530 10:49:32.359091 8903 docker.cpp:1104] Skipping non-docker container
> I0530 10:49:32.359094 8905 slave.cpp:2118] Queued task
> '20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0' for executor
> '20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0' of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:32.359357 8903 containerizer.cpp:992] Starting container
> 424fd077-caf5-4b67-90b0-760a23775ec1 for executor
> '20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0' of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:32.542762 8902 provisioner.cpp:453] Provisioning image rootfs
> '/var/lib/mesos/provisioner/containers/424fd077-caf5-4b67-90b0-760a23775ec1/backends/overlay/rootfses/ea0f760c-5079-4893-a5ec-469af9715abd'
> for container 424fd077-caf5-4b67-90b0-760a23775ec1 using overlay backend
> I0530 10:49:32.555275 8901 memory.cpp:451] Started listening for OOM events
> for container 424fd077-caf5-4b67-90b0-760a23775ec1
> I0530 10:49:32.555717 8901 memory.cpp:562] Started listening on 'low' memory
> pressure events for container 424fd077-caf5-4b67-90b0-760a23775ec1
> I0530 10:49:32.556246 8901 memory.cpp:562] Started listening on 'medium'
> memory pressure events for container 424fd077-caf5-4b67-90b0-760a23775ec1
> I0530 10:49:32.556879 8901 memory.cpp:562] Started listening on 'critical'
> memory pressure events for container 424fd077-caf5-4b67-90b0-760a23775ec1
> I0530 10:49:32.559020 8901 memory.cpp:199] Updated
> 'memory.soft_limit_in_bytes' to 160MB for container
> 424fd077-caf5-4b67-90b0-760a23775ec1
> I0530 10:49:32.559835 8901 memory.cpp:251] Updated 'memory.limit_in_bytes'
> to 160MB for container 424fd077-caf5-4b67-90b0-760a23775ec1
> I0530 10:49:32.560269 8901 cpu.cpp:101] Updated 'cpu.shares' to 204 (cpus
> 0.2) for container 424fd077-caf5-4b67-90b0-760a23775ec1
> I0530 10:49:32.566457 8902 linux_launcher.cpp:429] Launching container
> 424fd077-caf5-4b67-90b0-760a23775ec1 and cloning with namespaces CLONE_NEWNS
> | CLONE_NEWUTS | CLONE_NEWNET
> I0530 10:49:32.568379 8902 systemd.cpp:96] Assigned child process '9019' to
> 'mesos_executors.slice'
> I0530 10:49:32.570240 8901 containerizer.cpp:1637] Checkpointing container's
> forked pid 9019 to
> '/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0/runs/424fd077-caf5-4b67-90b0-760a23775ec1/pids/forked.pid'
> I0530 10:49:32.570843 8901 cni.cpp:872] Bind mounted '/proc/9019/ns/net' to
> '/run/mesos/isolators/network/cni/424fd077-caf5-4b67-90b0-760a23775ec1/ns'
> for container 424fd077-caf5-4b67-90b0-760a23775ec1
> I0530 10:49:35.541935 8908 cni.cpp:1290] Got assigned IPv4 address
> '192.168.235.73/32' from CNI network 'calico-net-1' for container
> 424fd077-caf5-4b67-90b0-760a23775ec1
> I0530 10:49:35.542150 8908 cni.cpp:999] Unable to find DNS nameservers for
> container 424fd077-caf5-4b67-90b0-760a23775ec1. Using host '/etc/resolv.conf'
> I0530 10:49:35.703557 8906 slave.cpp:3385] Got registration for executor
> '20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0' of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 from
> executor(1)@192.168.235.73:38212
> I0530 10:49:35.704555 8908 disk.cpp:207] Updating the disk resources for
> container 424fd077-caf5-4b67-90b0-760a23775ec1 to cpus(*)(allocated: *):0.2;
> mem(*)(allocated: *):160
> I0530 10:49:35.705009 8906 memory.cpp:199] Updated
> 'memory.soft_limit_in_bytes' to 160MB for container
> 424fd077-caf5-4b67-90b0-760a23775ec1
> I0530 10:49:35.705909 8906 cpu.cpp:101] Updated 'cpu.shares' to 204 (cpus
> 0.2) for container 424fd077-caf5-4b67-90b0-760a23775ec1
> I0530 10:49:35.706162 8906 slave.cpp:2331] Sending queued task
> '20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0' to executor
> '20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0' of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 at executor(1)@192.168.235.73:38212
> I0530 10:49:35.713090 8905 slave.cpp:3816] Handling status update
> TASK_RUNNING (UUID: d99c77b3-7a18-4940-9e84-9bc4b7eea948) for task
> 20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 from
> executor(1)@192.168.235.73:38212
> I0530 10:49:35.713526 8905 status_update_manager.cpp:323] Received status
> update TASK_RUNNING (UUID: d99c77b3-7a18-4940-9e84-9bc4b7eea948) for task
> 20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:35.713686 8905 status_update_manager.cpp:832] Checkpointing
> UPDATE for status update TASK_RUNNING (UUID:
> d99c77b3-7a18-4940-9e84-9bc4b7eea948) for task
> 20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:35.713820 8905 slave.cpp:4256] Forwarding the update
> TASK_RUNNING (UUID: d99c77b3-7a18-4940-9e84-9bc4b7eea948) for task
> 20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to [email protected]:5050
> I0530 10:49:35.713904 8905 slave.cpp:4166] Sending acknowledgement for
> status update TASK_RUNNING (UUID: d99c77b3-7a18-4940-9e84-9bc4b7eea948) for
> task 20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to executor(1)@192.168.235.73:38212
> I0530 10:49:35.757575 8905 status_update_manager.cpp:395] Received status
> update acknowledgement (UUID: d99c77b3-7a18-4940-9e84-9bc4b7eea948) for task
> 20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:35.757642 8905 status_update_manager.cpp:832] Checkpointing ACK
> for status update TASK_RUNNING (UUID: d99c77b3-7a18-4940-9e84-9bc4b7eea948)
> for task 20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:35.801589 8905 slave.cpp:3816] Handling status update
> TASK_FAILED (UUID: 75ca5b0c-7582-4524-9cde-a64039cecc97) for task
> 20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 from
> executor(1)@192.168.235.73:38212
> I0530 10:49:35.802206 8906 disk.cpp:207] Updating the disk resources for
> container 424fd077-caf5-4b67-90b0-760a23775ec1 to cpus(*)(allocated: *):0.1;
> mem(*)(allocated: *):32
> I0530 10:49:35.802685 8908 memory.cpp:199] Updated
> 'memory.soft_limit_in_bytes' to 32MB for container
> 424fd077-caf5-4b67-90b0-760a23775ec1
> I0530 10:49:35.803598 8908 cpu.cpp:101] Updated 'cpu.shares' to 102 (cpus
> 0.1) for container 424fd077-caf5-4b67-90b0-760a23775ec1
> I0530 10:49:35.803773 8908 status_update_manager.cpp:323] Received status
> update TASK_FAILED (UUID: 75ca5b0c-7582-4524-9cde-a64039cecc97) for task
> 20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:35.803802 8908 status_update_manager.cpp:832] Checkpointing
> UPDATE for status update TASK_FAILED (UUID:
> 75ca5b0c-7582-4524-9cde-a64039cecc97) for task
> 20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:35.803984 8903 slave.cpp:4256] Forwarding the update TASK_FAILED
> (UUID: 75ca5b0c-7582-4524-9cde-a64039cecc97) for task
> 20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to [email protected]:5050
> I0530 10:49:35.804077 8903 slave.cpp:4166] Sending acknowledgement for
> status update TASK_FAILED (UUID: 75ca5b0c-7582-4524-9cde-a64039cecc97) for
> task 20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to executor(1)@192.168.235.73:38212
> I0530 10:49:35.817703 8906 status_update_manager.cpp:395] Received status
> update acknowledgement (UUID: 75ca5b0c-7582-4524-9cde-a64039cecc97) for task
> 20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:35.817754 8906 status_update_manager.cpp:832] Checkpointing ACK
> for status update TASK_FAILED (UUID: 75ca5b0c-7582-4524-9cde-a64039cecc97)
> for task 20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:36.804152 8906 slave.cpp:4388] Got exited event for
> executor(1)@192.168.235.73:38212
> I0530 10:49:36.847506 8906 containerizer.cpp:2479] Container
> 424fd077-caf5-4b67-90b0-760a23775ec1 has exited
> I0530 10:49:36.847568 8906 containerizer.cpp:2116] Destroying container
> 424fd077-caf5-4b67-90b0-760a23775ec1 in RUNNING state
> I0530 10:49:36.847664 8906 linux_launcher.cpp:505] Asked to destroy
> container 424fd077-caf5-4b67-90b0-760a23775ec1
> I0530 10:49:36.848474 8906 linux_launcher.cpp:548] Using freezer to destroy
> cgroup mesos/424fd077-caf5-4b67-90b0-760a23775ec1
> I0530 10:49:36.849936 8906 cgroups.cpp:2692] Freezing cgroup
> /sys/fs/cgroup/freezer/mesos/424fd077-caf5-4b67-90b0-760a23775ec1
> I0530 10:49:36.851379 8907 cgroups.cpp:1405] Successfully froze cgroup
> /sys/fs/cgroup/freezer/mesos/424fd077-caf5-4b67-90b0-760a23775ec1 after
> 1.411072ms
> I0530 10:49:36.853444 8906 cgroups.cpp:2710] Thawing cgroup
> /sys/fs/cgroup/freezer/mesos/424fd077-caf5-4b67-90b0-760a23775ec1
> I0530 10:49:36.855401 8906 cgroups.cpp:1434] Successfully thawed cgroup
> /sys/fs/cgroup/freezer/mesos/424fd077-caf5-4b67-90b0-760a23775ec1 after
> 1.910016ms
> I0530 10:49:37.375128 8902 slave.cpp:1625] Got assigned task
> '20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0' for framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:37.375424 8902 slave.cpp:1785] Launching task
> '20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0' for framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:37.375635 8902 paths.cpp:547] Trying to chown
> '/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0/runs/1248ce41-4143-4adb-98d0-04c6cf8ff000'
> to user 'root'
> I0530 10:49:37.376144 8902 slave.cpp:6480] Launching executor
> '20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0' of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 with resources cpus(*)(allocated:
> *):0.1; mem(*)(allocated: *):32 in work directory
> '/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0/runs/1248ce41-4143-4adb-98d0-04c6cf8ff000'
> I0530 10:49:37.376639 8902 slave.cpp:2118] Queued task
> '20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0' for executor
> '20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0' of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:37.376938 8902 docker.cpp:1104] Skipping non-docker container
> I0530 10:49:37.377231 8902 containerizer.cpp:992] Starting container
> 1248ce41-4143-4adb-98d0-04c6cf8ff000 for executor
> '20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0' of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:37.565088 8901 provisioner.cpp:453] Provisioning image rootfs
> '/var/lib/mesos/provisioner/containers/1248ce41-4143-4adb-98d0-04c6cf8ff000/backends/overlay/rootfses/585e6e95-cd8b-4ed9-8341-49360770675b'
> for container 1248ce41-4143-4adb-98d0-04c6cf8ff000 using overlay backend
> I0530 10:49:37.578210 8901 memory.cpp:451] Started listening for OOM events
> for container 1248ce41-4143-4adb-98d0-04c6cf8ff000
> I0530 10:49:37.578721 8901 memory.cpp:562] Started listening on 'low' memory
> pressure events for container 1248ce41-4143-4adb-98d0-04c6cf8ff000
> I0530 10:49:37.579267 8901 memory.cpp:562] Started listening on 'medium'
> memory pressure events for container 1248ce41-4143-4adb-98d0-04c6cf8ff000
> I0530 10:49:37.579759 8901 memory.cpp:562] Started listening on 'critical'
> memory pressure events for container 1248ce41-4143-4adb-98d0-04c6cf8ff000
> I0530 10:49:37.581714 8901 memory.cpp:199] Updated
> 'memory.soft_limit_in_bytes' to 160MB for container
> 1248ce41-4143-4adb-98d0-04c6cf8ff000
> I0530 10:49:37.582581 8901 memory.cpp:251] Updated 'memory.limit_in_bytes'
> to 160MB for container 1248ce41-4143-4adb-98d0-04c6cf8ff000
> I0530 10:49:37.583048 8901 cpu.cpp:101] Updated 'cpu.shares' to 204 (cpus
> 0.2) for container 1248ce41-4143-4adb-98d0-04c6cf8ff000
> I0530 10:49:37.585544 8905 linux_launcher.cpp:429] Launching container
> 1248ce41-4143-4adb-98d0-04c6cf8ff000 and cloning with namespaces CLONE_NEWNS
> | CLONE_NEWUTS | CLONE_NEWNET
> I0530 10:49:37.587668 8905 systemd.cpp:96] Assigned child process '9101' to
> 'mesos_executors.slice'
> I0530 10:49:37.589622 8903 containerizer.cpp:1637] Checkpointing container's
> forked pid 9101 to
> '/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0/runs/1248ce41-4143-4adb-98d0-04c6cf8ff000/pids/forked.pid'
> I0530 10:49:37.590232 8904 cni.cpp:872] Bind mounted '/proc/9101/ns/net' to
> '/run/mesos/isolators/network/cni/1248ce41-4143-4adb-98d0-04c6cf8ff000/ns'
> for container 1248ce41-4143-4adb-98d0-04c6cf8ff000
> I0530 10:49:38.826182 8907 http.cpp:307] HTTP GET for /slave(1)/state from
> 10.114.99.73:54369 with User-Agent='Mozilla/5.0 (Macintosh; Intel Mac OS X
> 10_12_5) AppleWebKit/603.2.4 (KHTML, like Gecko) Version/10.1.1
> Safari/603.2.4'
> I0530 10:49:39.358474 8905 cni.cpp:1468] Unmounted the network namespace
> handle
> '/run/mesos/isolators/network/cni/424fd077-caf5-4b67-90b0-760a23775ec1/ns'
> for container 424fd077-caf5-4b67-90b0-760a23775ec1
> I0530 10:49:39.358549 8905 cni.cpp:1479] Removed the container directory
> '/run/mesos/isolators/network/cni/424fd077-caf5-4b67-90b0-760a23775ec1'
> I0530 10:49:39.378610 8901 provisioner.cpp:570] Destroying container rootfs
> at
> '/var/lib/mesos/provisioner/containers/424fd077-caf5-4b67-90b0-760a23775ec1/backends/overlay/rootfses/ea0f760c-5079-4893-a5ec-469af9715abd'
> for container 424fd077-caf5-4b67-90b0-760a23775ec1
> I0530 10:49:39.380601 8907 slave.cpp:4769] Executor
> '20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0' of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 exited with status 0
> I0530 10:49:39.380643 8907 slave.cpp:4869] Cleaning up executor
> '20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0' of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 at executor(1)@192.168.235.73:38212
> I0530 10:49:39.380894 8906 gc.cpp:55] Scheduling
> '/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0/runs/424fd077-caf5-4b67-90b0-760a23775ec1'
> for gc 6.99999559317037days in the future
> I0530 10:49:39.380941 8906 gc.cpp:55] Scheduling
> '/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0'
> for gc 6.99999559270815days in the future
> I0530 10:49:39.380957 8906 gc.cpp:55] Scheduling
> '/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0/runs/424fd077-caf5-4b67-90b0-760a23775ec1'
> for gc 6.99999559248889days in the future
> I0530 10:49:39.380970 8906 gc.cpp:55] Scheduling
> '/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e64ba952-4514-11e7-8d95-024276be29e0'
> for gc 6.99999559230222days in the future
> I0530 10:49:39.961740 8904 cni.cpp:1290] Got assigned IPv4 address
> '192.168.235.74/32' from CNI network 'calico-net-1' for container
> 1248ce41-4143-4adb-98d0-04c6cf8ff000
> I0530 10:49:39.961976 8904 cni.cpp:999] Unable to find DNS nameservers for
> container 1248ce41-4143-4adb-98d0-04c6cf8ff000. Using host '/etc/resolv.conf'
> I0530 10:49:40.125231 8902 slave.cpp:3385] Got registration for executor
> '20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0' of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 from
> executor(1)@192.168.235.74:43999
> I0530 10:49:40.126565 8902 memory.cpp:199] Updated
> 'memory.soft_limit_in_bytes' to 160MB for container
> 1248ce41-4143-4adb-98d0-04c6cf8ff000
> I0530 10:49:40.127490 8902 cpu.cpp:101] Updated 'cpu.shares' to 204 (cpus
> 0.2) for container 1248ce41-4143-4adb-98d0-04c6cf8ff000
> I0530 10:49:40.127542 8902 disk.cpp:207] Updating the disk resources for
> container 1248ce41-4143-4adb-98d0-04c6cf8ff000 to cpus(*)(allocated: *):0.2;
> mem(*)(allocated: *):160
> I0530 10:49:40.127712 8902 slave.cpp:2331] Sending queued task
> '20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0' to executor
> '20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0' of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 at executor(1)@192.168.235.74:43999
> I0530 10:49:40.134186 8902 slave.cpp:3816] Handling status update
> TASK_RUNNING (UUID: 3f7b6737-e154-47cb-93e7-4e0c55b233d3) for task
> 20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 from
> executor(1)@192.168.235.74:43999
> I0530 10:49:40.134565 8902 status_update_manager.cpp:323] Received status
> update TASK_RUNNING (UUID: 3f7b6737-e154-47cb-93e7-4e0c55b233d3) for task
> 20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:40.134727 8902 status_update_manager.cpp:832] Checkpointing
> UPDATE for status update TASK_RUNNING (UUID:
> 3f7b6737-e154-47cb-93e7-4e0c55b233d3) for task
> 20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:40.134851 8902 slave.cpp:4256] Forwarding the update
> TASK_RUNNING (UUID: 3f7b6737-e154-47cb-93e7-4e0c55b233d3) for task
> 20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to [email protected]:5050
> I0530 10:49:40.134941 8902 slave.cpp:4166] Sending acknowledgement for
> status update TASK_RUNNING (UUID: 3f7b6737-e154-47cb-93e7-4e0c55b233d3) for
> task 20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to executor(1)@192.168.235.74:43999
> I0530 10:49:40.143187 8908 status_update_manager.cpp:395] Received status
> update acknowledgement (UUID: 3f7b6737-e154-47cb-93e7-4e0c55b233d3) for task
> 20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:40.143262 8908 status_update_manager.cpp:832] Checkpointing ACK
> for status update TASK_RUNNING (UUID: 3f7b6737-e154-47cb-93e7-4e0c55b233d3)
> for task 20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:40.223562 8908 slave.cpp:3816] Handling status update
> TASK_FAILED (UUID: 4dec875b-3c98-4c13-943b-9aa9f015fa64) for task
> 20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 from
> executor(1)@192.168.235.74:43999
> I0530 10:49:40.224624 8908 memory.cpp:199] Updated
> 'memory.soft_limit_in_bytes' to 32MB for container
> 1248ce41-4143-4adb-98d0-04c6cf8ff000
> I0530 10:49:40.225495 8903 disk.cpp:207] Updating the disk resources for
> container 1248ce41-4143-4adb-98d0-04c6cf8ff000 to cpus(*)(allocated: *):0.1;
> mem(*)(allocated: *):32
> I0530 10:49:40.225520 8908 cpu.cpp:101] Updated 'cpu.shares' to 102 (cpus
> 0.1) for container 1248ce41-4143-4adb-98d0-04c6cf8ff000
> I0530 10:49:40.225795 8908 status_update_manager.cpp:323] Received status
> update TASK_FAILED (UUID: 4dec875b-3c98-4c13-943b-9aa9f015fa64) for task
> 20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:40.225888 8908 status_update_manager.cpp:832] Checkpointing
> UPDATE for status update TASK_FAILED (UUID:
> 4dec875b-3c98-4c13-943b-9aa9f015fa64) for task
> 20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:40.226018 8902 slave.cpp:4256] Forwarding the update TASK_FAILED
> (UUID: 4dec875b-3c98-4c13-943b-9aa9f015fa64) for task
> 20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to [email protected]:5050
> I0530 10:49:40.226104 8902 slave.cpp:4166] Sending acknowledgement for
> status update TASK_FAILED (UUID: 4dec875b-3c98-4c13-943b-9aa9f015fa64) for
> task 20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to executor(1)@192.168.235.74:43999
> I0530 10:49:40.233405 8902 status_update_manager.cpp:395] Received status
> update acknowledgement (UUID: 4dec875b-3c98-4c13-943b-9aa9f015fa64) for task
> 20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:40.233520 8902 status_update_manager.cpp:832] Checkpointing ACK
> for status update TASK_FAILED (UUID: 4dec875b-3c98-4c13-943b-9aa9f015fa64)
> for task 20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:41.225358 8903 slave.cpp:4388] Got exited event for
> executor(1)@192.168.235.74:43999
> I0530 10:49:41.266679 8903 containerizer.cpp:2479] Container
> 1248ce41-4143-4adb-98d0-04c6cf8ff000 has exited
> I0530 10:49:41.266719 8903 containerizer.cpp:2116] Destroying container
> 1248ce41-4143-4adb-98d0-04c6cf8ff000 in RUNNING state
> I0530 10:49:41.266774 8903 linux_launcher.cpp:505] Asked to destroy
> container 1248ce41-4143-4adb-98d0-04c6cf8ff000
> I0530 10:49:41.267336 8903 linux_launcher.cpp:548] Using freezer to destroy
> cgroup mesos/1248ce41-4143-4adb-98d0-04c6cf8ff000
> I0530 10:49:41.268402 8903 cgroups.cpp:2692] Freezing cgroup
> /sys/fs/cgroup/freezer/mesos/1248ce41-4143-4adb-98d0-04c6cf8ff000
> I0530 10:49:41.269942 8903 cgroups.cpp:1405] Successfully froze cgroup
> /sys/fs/cgroup/freezer/mesos/1248ce41-4143-4adb-98d0-04c6cf8ff000 after
> 1.48992ms
> I0530 10:49:41.271653 8902 cgroups.cpp:2710] Thawing cgroup
> /sys/fs/cgroup/freezer/mesos/1248ce41-4143-4adb-98d0-04c6cf8ff000
> I0530 10:49:41.273180 8903 cgroups.cpp:1434] Successfully thawed cgroup
> /sys/fs/cgroup/freezer/mesos/1248ce41-4143-4adb-98d0-04c6cf8ff000 after
> 1.490944ms
> I0530 10:49:42.395020 8908 slave.cpp:1625] Got assigned task
> '20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0' for framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:42.395308 8908 slave.cpp:1785] Launching task
> '20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0' for framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:42.396659 8908 paths.cpp:547] Trying to chown
> '/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0/runs/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8'
> to user 'root'
> I0530 10:49:42.406666 8908 slave.cpp:6480] Launching executor
> '20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0' of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 with resources cpus(*)(allocated:
> *):0.1; mem(*)(allocated: *):32 in work directory
> '/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0/runs/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8'
> I0530 10:49:42.407295 8908 slave.cpp:2118] Queued task
> '20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0' for executor
> '20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0' of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:42.407347 8908 docker.cpp:1104] Skipping non-docker container
> I0530 10:49:42.407714 8908 containerizer.cpp:992] Starting container
> fd4bb1b7-0fb1-46ad-860d-3080d1670ba8 for executor
> '20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0' of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:42.576038 8903 provisioner.cpp:453] Provisioning image rootfs
> '/var/lib/mesos/provisioner/containers/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8/backends/overlay/rootfses/843753d1-28f7-4bf1-bbe5-84463f30fa2b'
> for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8 using overlay backend
> I0530 10:49:42.588337 8907 memory.cpp:451] Started listening for OOM events
> for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
> I0530 10:49:42.588832 8907 memory.cpp:562] Started listening on 'low' memory
> pressure events for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
> I0530 10:49:42.589339 8907 memory.cpp:562] Started listening on 'medium'
> memory pressure events for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
> I0530 10:49:42.589799 8907 memory.cpp:562] Started listening on 'critical'
> memory pressure events for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
> I0530 10:49:42.591732 8907 memory.cpp:199] Updated
> 'memory.soft_limit_in_bytes' to 160MB for container
> fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
> I0530 10:49:42.593526 8907 memory.cpp:251] Updated 'memory.limit_in_bytes'
> to 160MB for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
> I0530 10:49:42.593991 8907 cpu.cpp:101] Updated 'cpu.shares' to 204 (cpus
> 0.2) for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
> I0530 10:49:42.596405 8907 linux_launcher.cpp:429] Launching container
> fd4bb1b7-0fb1-46ad-860d-3080d1670ba8 and cloning with namespaces CLONE_NEWNS
> | CLONE_NEWUTS | CLONE_NEWNET
> I0530 10:49:42.598201 8907 systemd.cpp:96] Assigned child process '9206' to
> 'mesos_executors.slice'
> I0530 10:49:42.600131 8903 containerizer.cpp:1637] Checkpointing container's
> forked pid 9206 to
> '/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0/runs/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8/pids/forked.pid'
> I0530 10:49:42.600713 8901 cni.cpp:872] Bind mounted '/proc/9206/ns/net' to
> '/run/mesos/isolators/network/cni/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8/ns'
> for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
> I0530 10:49:42.974720 8905 cni.cpp:1468] Unmounted the network namespace
> handle
> '/run/mesos/isolators/network/cni/1248ce41-4143-4adb-98d0-04c6cf8ff000/ns'
> for container 1248ce41-4143-4adb-98d0-04c6cf8ff000
> I0530 10:49:42.974813 8905 cni.cpp:1479] Removed the container directory
> '/run/mesos/isolators/network/cni/1248ce41-4143-4adb-98d0-04c6cf8ff000'
> I0530 10:49:42.987810 8902 provisioner.cpp:570] Destroying container rootfs
> at
> '/var/lib/mesos/provisioner/containers/1248ce41-4143-4adb-98d0-04c6cf8ff000/backends/overlay/rootfses/585e6e95-cd8b-4ed9-8341-49360770675b'
> for container 1248ce41-4143-4adb-98d0-04c6cf8ff000
> I0530 10:49:42.989212 8908 slave.cpp:4769] Executor
> '20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0' of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 exited with status 0
> I0530 10:49:42.989243 8908 slave.cpp:4869] Cleaning up executor
> '20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0' of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 at executor(1)@192.168.235.74:43999
> I0530 10:49:42.989409 8908 gc.cpp:55] Scheduling
> '/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0/runs/1248ce41-4143-4adb-98d0-04c6cf8ff000'
> for gc 6.99998854937482days in the future
> I0530 10:49:42.989439 8908 gc.cpp:55] Scheduling
> '/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0'
> for gc 6.9999885490163days in the future
> I0530 10:49:42.989454 8908 gc.cpp:55] Scheduling
> '/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0/runs/1248ce41-4143-4adb-98d0-04c6cf8ff000'
> for gc 6.99998854880889days in the future
> I0530 10:49:42.989467 8908 gc.cpp:55] Scheduling
> '/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.e94958f3-4514-11e7-8d95-024276be29e0'
> for gc 6.99998854864593days in the future
> I0530 10:49:45.888653 8906 cni.cpp:1290] Got assigned IPv4 address
> '192.168.235.75/32' from CNI network 'calico-net-1' for container
> fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
> I0530 10:49:45.888829 8906 cni.cpp:999] Unable to find DNS nameservers for
> container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8. Using host '/etc/resolv.conf'
> I0530 10:49:46.050271 8902 slave.cpp:3385] Got registration for executor
> '20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0' of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 from
> executor(1)@192.168.235.75:40717
> I0530 10:49:46.051580 8902 memory.cpp:199] Updated
> 'memory.soft_limit_in_bytes' to 160MB for container
> fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
> I0530 10:49:46.052515 8902 cpu.cpp:101] Updated 'cpu.shares' to 204 (cpus
> 0.2) for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
> I0530 10:49:46.052567 8902 disk.cpp:207] Updating the disk resources for
> container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8 to cpus(*)(allocated: *):0.2;
> mem(*)(allocated: *):160
> I0530 10:49:46.052742 8902 slave.cpp:2331] Sending queued task
> '20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0' to executor
> '20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0' of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 at executor(1)@192.168.235.75:40717
> I0530 10:49:46.056938 8902 slave.cpp:3816] Handling status update
> TASK_RUNNING (UUID: fb7d9371-cd0e-49c6-943b-e897f7afbd62) for task
> 20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 from
> executor(1)@192.168.235.75:40717
> I0530 10:49:46.057296 8902 status_update_manager.cpp:323] Received status
> update TASK_RUNNING (UUID: fb7d9371-cd0e-49c6-943b-e897f7afbd62) for task
> 20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:46.057456 8902 status_update_manager.cpp:832] Checkpointing
> UPDATE for status update TASK_RUNNING (UUID:
> fb7d9371-cd0e-49c6-943b-e897f7afbd62) for task
> 20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:46.057593 8902 slave.cpp:4256] Forwarding the update
> TASK_RUNNING (UUID: fb7d9371-cd0e-49c6-943b-e897f7afbd62) for task
> 20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to [email protected]:5050
> I0530 10:49:46.057663 8902 slave.cpp:4166] Sending acknowledgement for
> status update TASK_RUNNING (UUID: fb7d9371-cd0e-49c6-943b-e897f7afbd62) for
> task 20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to executor(1)@192.168.235.75:40717
> I0530 10:49:46.064831 8902 status_update_manager.cpp:395] Received status
> update acknowledgement (UUID: fb7d9371-cd0e-49c6-943b-e897f7afbd62) for task
> 20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:46.064899 8902 status_update_manager.cpp:832] Checkpointing ACK
> for status update TASK_RUNNING (UUID: fb7d9371-cd0e-49c6-943b-e897f7afbd62)
> for task 20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:46.148407 8902 slave.cpp:3816] Handling status update
> TASK_FAILED (UUID: 68132cf3-98fd-4d35-a8cc-c72fe3924d70) for task
> 20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 from
> executor(1)@192.168.235.75:40717
> I0530 10:49:46.149554 8908 memory.cpp:199] Updated
> 'memory.soft_limit_in_bytes' to 32MB for container
> fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
> I0530 10:49:46.150315 8901 disk.cpp:207] Updating the disk resources for
> container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8 to cpus(*)(allocated: *):0.1;
> mem(*)(allocated: *):32
> I0530 10:49:46.150490 8908 cpu.cpp:101] Updated 'cpu.shares' to 102 (cpus
> 0.1) for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
> I0530 10:49:46.150804 8908 status_update_manager.cpp:323] Received status
> update TASK_FAILED (UUID: 68132cf3-98fd-4d35-a8cc-c72fe3924d70) for task
> 20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:46.150835 8908 status_update_manager.cpp:832] Checkpointing
> UPDATE for status update TASK_FAILED (UUID:
> 68132cf3-98fd-4d35-a8cc-c72fe3924d70) for task
> 20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:46.151010 8903 slave.cpp:4256] Forwarding the update TASK_FAILED
> (UUID: 68132cf3-98fd-4d35-a8cc-c72fe3924d70) for task
> 20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to [email protected]:5050
> I0530 10:49:46.151093 8903 slave.cpp:4166] Sending acknowledgement for
> status update TASK_FAILED (UUID: 68132cf3-98fd-4d35-a8cc-c72fe3924d70) for
> task 20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 to executor(1)@192.168.235.75:40717
> I0530 10:49:46.158910 8904 status_update_manager.cpp:395] Received status
> update acknowledgement (UUID: 68132cf3-98fd-4d35-a8cc-c72fe3924d70) for task
> 20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:46.158972 8904 status_update_manager.cpp:832] Checkpointing ACK
> for status update TASK_FAILED (UUID: 68132cf3-98fd-4d35-a8cc-c72fe3924d70)
> for task 20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0 of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000
> I0530 10:49:47.151319 8903 slave.cpp:4388] Got exited event for
> executor(1)@192.168.235.75:40717
> I0530 10:49:47.193552 8903 containerizer.cpp:2479] Container
> fd4bb1b7-0fb1-46ad-860d-3080d1670ba8 has exited
> I0530 10:49:47.193584 8903 containerizer.cpp:2116] Destroying container
> fd4bb1b7-0fb1-46ad-860d-3080d1670ba8 in RUNNING state
> I0530 10:49:47.193639 8903 linux_launcher.cpp:505] Asked to destroy
> container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
> I0530 10:49:47.194249 8903 linux_launcher.cpp:548] Using freezer to destroy
> cgroup mesos/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
> I0530 10:49:47.195314 8903 cgroups.cpp:2692] Freezing cgroup
> /sys/fs/cgroup/freezer/mesos/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
> I0530 10:49:47.196699 8904 cgroups.cpp:1405] Successfully froze cgroup
> /sys/fs/cgroup/freezer/mesos/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8 after
> 1.355008ms
> I0530 10:49:47.198119 8901 cgroups.cpp:2710] Thawing cgroup
> /sys/fs/cgroup/freezer/mesos/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
> I0530 10:49:47.199431 8904 cgroups.cpp:1434] Successfully thawed cgroup
> /sys/fs/cgroup/freezer/mesos/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8 after
> 1.277952ms
> I0530 10:49:48.698365 8906 cni.cpp:1468] Unmounted the network namespace
> handle
> '/run/mesos/isolators/network/cni/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8/ns'
> for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
> I0530 10:49:48.698437 8906 cni.cpp:1479] Removed the container directory
> '/run/mesos/isolators/network/cni/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8'
> I0530 10:49:48.714375 8906 provisioner.cpp:570] Destroying container rootfs
> at
> '/var/lib/mesos/provisioner/containers/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8/backends/overlay/rootfses/843753d1-28f7-4bf1-bbe5-84463f30fa2b'
> for container fd4bb1b7-0fb1-46ad-860d-3080d1670ba8
> I0530 10:49:48.715656 8906 slave.cpp:4769] Executor
> '20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0' of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 exited with status 0
> I0530 10:49:48.715685 8906 slave.cpp:4869] Cleaning up executor
> '20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0' of framework
> 562f1892-41c6-4c55-955e-662c63ab3ddf-0000 at executor(1)@192.168.235.75:40717
> I0530 10:49:48.715879 8906 gc.cpp:55] Scheduling
> '/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0/runs/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8'
> for gc 6.99999171553185days in the future
> I0530 10:49:48.715922 8906 gc.cpp:55] Scheduling
> '/var/lib/mesos/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0'
> for gc 6.99999171524444days in the future
> I0530 10:49:48.715948 8906 gc.cpp:55] Scheduling
> '/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0/runs/fd4bb1b7-0fb1-46ad-860d-3080d1670ba8'
> for gc 6.99999171495407days in the future
> I0530 10:49:48.715963 8906 gc.cpp:55] Scheduling
> '/var/lib/mesos/meta/slaves/562f1892-41c6-4c55-955e-662c63ab3ddf-S7/frameworks/562f1892-41c6-4c55-955e-662c63ab3ddf-0000/executors/20150518_mesos10.ec477dc4-4514-11e7-8d95-024276be29e0'
> for gc 6.99999171468741days in the future
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
