[
https://issues.apache.org/jira/browse/MESOS-6004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Thomas updated MESOS-6004:
----------------------------------
Description:
When scheduling more than one task on the same agent, all tasks fail a as
containers seem to be destroyed during provisioning.
Specifically, the errors on the agent logs are:
{noformat}
E0808 15:53:09.691315 30996 slave.cpp:3976] Container
'eb20f642-bb90-4293-8eec-6f1576ccaeb1' for executor '3' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0001 failed to start: Container is being
destroyed during provisioning
{noformat}
and
{noformat}
I0808 15:52:32.510210 30999 slave.cpp:4539] Terminating executor ''2' of
framework c9852a23-bc07-422d-8d69-23c167a1924d-0001' because it did not
register within 3mins
{noformat}
As the default provisioning method {{copy}} is being used, I assume this is due
to the provisioning of multiple containers taking too long and the agent will
not wait. For large images, this method is simply not performant.
The issue did not occur, when only one tasks was scheduled.
Increasing the {{executor_registration_timeout}} parameter, seemed to help a
bit as it allowed to schedule at least 2 tasks at the same time. But still
fails with more (5 in this case)
h4. Complete logs
(with GLOG_v=1)
{noformat}
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.800375 3738
slave.cpp:198] Agent started on 1)@172.31.23.17:5051
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.800403 3738
slave.cpp:199] Flags at startup: --appc_simple_discovery_uri_prefix="http://";
--appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="false"
--authenticate_http_readwrite="false" --authenticatee="crammd5"
--authentication_backoff_factor="1secs" --authorizer="local"
--cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false"
--cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false"
--cgroups_root="mesos" --container_disk_watch_interval="15secs"
--containerizers="mesos,docker" --default_role="*"
--disk_watch_interval="1mins" --docker="docker" --docker_config="XXX"
--docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io";
--docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock"
--docker_stop_timeout="0ns" --docker_store_dir="/mnt/t" --docker_volume_checkp
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]:
oint_dir="/var/run/mesos/isolators/docker/volume"
--enforce_container_disk_quota="false" --executor_registration_timeout="1mins"
--executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch"
--fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks"
--gc_disk_headroom="0.1" --hadoop_home="" --help="false"
--hostname="ec2-52-59-113-0.eu-central-1.compute.amazonaws.com"
--hostname_lookup="true" --http_authenticators="basic"
--http_command_executor="false" --image_providers="docker"
--image_provisioner_backend="copy" --initialize_driver_logging="true"
--isolation="filesystem/linux,docker/runtime,cgroups/devices,gpu/nvidia"
--launcher_dir="/usr/libexec/mesos" --log_dir="/var/log/mesos" --logbufsecs="0"
--logging_level="INFO" --master="zk://172.31.19.240:2181/mesos"
--oversubscribed_resources_interval="15secs" --perf_duration="10secs"
--perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns"
--quiet="false" --recover="reconnect" --re
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: covery_timeout="15mins"
--registration_backoff_factor="1secs" --revocable_cpu_low_priority="true"
--sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="false"
--systemd_enable_support="true"
--systemd_runtime_directory="/run/systemd/system" --version="false"
--work_dir="/mnt/mesos"
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: 2016-08-09
10:11:41,800:3738(0x7f71172a1700):ZOO_INFO@check_events@1728: initiated
connection to server [172.31.19.240:2181]
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.801007 3738
slave.cpp:519] Agent resources: gpus(*):1; cpus(*):8; mem(*):14014;
disk(*):60257; ports(*):[31000-32000]
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.801059 3738
slave.cpp:527] Agent attributes: [ ]
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.801077 3738
slave.cpp:532] Agent hostname:
ec2-52-59-113-0.eu-central-1.compute.amazonaws.com
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.802232 3772
state.cpp:57] Recovering state from '/mnt/mesos/meta'
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.803539 3774
status_update_manager.cpp:200] Recovering status update manager
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.803745 3772
containerizer.cpp:522] Recovering containerizer
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.803788 3770
docker.cpp:775] Recovering Docker containers
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.808550 3777
metadata_manager.cpp:251] Successfully loaded 2 Docker images
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.808688 3771
provisioner.cpp:253] Provisioner recovery complete
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: 2016-08-09
10:11:41,816:3738(0x7f71172a1700):ZOO_INFO@check_events@1775: session
establishment complete on server [172.31.19.240:2181],
sessionId=0x1566a66ab9b0010, negotiated timeout=10000
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.817003 3774
group.cpp:349] Group process (group(1)@172.31.23.17:5051) connected to ZooKeeper
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.817116 3774
group.cpp:837] Syncing group operations: queue size (joins, cancels, datas) =
(0, 0, 0)
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.817147 3774
group.cpp:427] Trying to create path '/mesos' in ZooKeeper
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.818857 3774
detector.cpp:152] Detected a new leader: (id='64')
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.819005 3776
group.cpp:706] Trying to get '/mesos/json.info_0000000064' in ZooKeeper
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.819855 3775
zookeeper.cpp:259] A new leading master ([email protected]:5050) is
detected
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.899876 3777
slave.cpp:4782] Finished recovery
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900573 3777
slave.cpp:4815] Garbage collecting old agent
524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S2
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900687 3773
gc.cpp:55] Scheduling
'/mnt/mesos/slaves/524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S2' for gc
6.99998957623704days in the future
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900729 3777
slave.cpp:4815] Garbage collecting old agent
c9852a23-bc07-422d-8d69-23c167a1924d-S0
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900779 3773
gc.cpp:55] Scheduling
'/mnt/mesos/meta/slaves/524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S2' for gc
6.99998957520889days in the future
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900817 3777
slave.cpp:4815] Garbage collecting old agent
524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S1
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900825 3773
gc.cpp:55] Scheduling
'/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S0' for gc
6.99998957451259days in the future
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900899 3773
gc.cpp:55] Scheduling
'/mnt/mesos/meta/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S0' for gc
6.99998957412148days in the future
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900933 3777
slave.cpp:4815] Garbage collecting old agent
524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S3
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900944 3773
gc.cpp:55] Scheduling
'/mnt/mesos/slaves/524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S1' for gc
6.99998957332148days in the future
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901010 3773
gc.cpp:55] Scheduling
'/mnt/mesos/meta/slaves/524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S1' for gc
6.99998957291556days in the future
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901051 3773
gc.cpp:55] Scheduling
'/mnt/mesos/slaves/524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S3' for gc
6.99998957196741days in the future
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901101 3773
gc.cpp:55] Scheduling
'/mnt/mesos/meta/slaves/524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S3' for gc
6.99998957156148days in the future
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901201 3775
status_update_manager.cpp:174] Pausing sending status updates
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901203 3777
slave.cpp:895] New master detected at [email protected]:5050
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901278 3777
slave.cpp:916] No credentials provided. Attempting to register without
authentication
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901317 3777
slave.cpp:927] Detecting new master
Aug 9 10:11:42 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:42.267201 3774
slave.cpp:1197] Re-registered with master [email protected]:5050
Aug 9 10:11:42 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:42.267302 3771
status_update_manager.cpp:181] Resuming sending status updates
Aug 9 10:11:42 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:42.267302 3774
slave.cpp:1233] Forwarding total oversubscribed resources
Aug 9 10:11:42 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:42.267577 3774
slave.cpp:2526] Updated checkpointed resources from to
Aug 9 10:11:48 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:48.269104 3777
slave.cpp:2218] Asked to shut down framework
c9852a23-bc07-422d-8d69-23c167a1924d-0002 by [email protected]:5050
Aug 9 10:11:48 ip-172-31-23-17 mesos-slave[3738]: W0809 10:11:48.269186 3777
slave.cpp:2233] Cannot shut down unknown framework
c9852a23-bc07-422d-8d69-23c167a1924d-0002
Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.969521 3772
slave.cpp:1495] Got assigned task 0 for framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.970026 3772
slave.cpp:1614] Launching task 0 for framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.970826 3772
slave.cpp:5674] Launching executor 0 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 with resources cpus(*):0.1; mem(*):32
in work directory
'/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/0/runs/09a77a80-f904-4581-966a-734d24ae96f3'
Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.971051 3772
slave.cpp:1840] Queuing task '0' for executor '0' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.971212 3773
containerizer.cpp:781] Starting container
'09a77a80-f904-4581-966a-734d24ae96f3' for executor '0' of framework
'c9852a23-bc07-422d-8d69-23c167a1924d-0003'
Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.973953 3774
provisioner.cpp:294] Provisioning image rootfs
'/mnt/mesos/provisioner/containers/09a77a80-f904-4581-966a-734d24ae96f3/backends/copy/rootfses/ec9bb10e-6726-4872-be2b-4a72dfc0fafc'
for container 09a77a80-f904-4581-966a-734d24ae96f3
Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.978344 3774
slave.cpp:1495] Got assigned task 1 for framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.978687 3774
slave.cpp:1614] Launching task 1 for framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.978976 3774
slave.cpp:5674] Launching executor 1 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 with resources cpus(*):0.1; mem(*):32
in work directory
'/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/1/runs/0cea1dc7-b523-4b68-851a-531ed101de5a'
Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.979173 3774
slave.cpp:1840] Queuing task '1' for executor '1' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.979387 3777
containerizer.cpp:781] Starting container
'0cea1dc7-b523-4b68-851a-531ed101de5a' for executor '1' of framework
'c9852a23-bc07-422d-8d69-23c167a1924d-0003'
Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.981374 3776
provisioner.cpp:294] Provisioning image rootfs
'/mnt/mesos/provisioner/containers/0cea1dc7-b523-4b68-851a-531ed101de5a/backends/copy/rootfses/e3af89fe-c889-45fc-929b-f008b18e5d4e'
for container 0cea1dc7-b523-4b68-851a-531ed101de5a
Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.983610 3776
slave.cpp:1495] Got assigned task 2 for framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.983947 3776
slave.cpp:1614] Launching task 2 for framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.984236 3776
slave.cpp:5674] Launching executor 2 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 with resources cpus(*):0.1; mem(*):32
in work directory
'/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/2/runs/a4c972da-b12d-4b4b-8481-79ca320ddab7'
Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.984450 3776
slave.cpp:1840] Queuing task '2' for executor '2' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.984778 3777
containerizer.cpp:781] Starting container
'a4c972da-b12d-4b4b-8481-79ca320ddab7' for executor '2' of framework
'c9852a23-bc07-422d-8d69-23c167a1924d-0003'
Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.986330 3776
provisioner.cpp:294] Provisioning image rootfs
'/mnt/mesos/provisioner/containers/a4c972da-b12d-4b4b-8481-79ca320ddab7/backends/copy/rootfses/feaf70a4-df43-45c9-b6be-a654c9210a54'
for container a4c972da-b12d-4b4b-8481-79ca320ddab7
Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.996104 3771
slave.cpp:1495] Got assigned task 3 for framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.996462 3771
slave.cpp:1614] Launching task 3 for framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.997241 3771
slave.cpp:5674] Launching executor 3 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 with resources cpus(*):0.1; mem(*):32
in work directory
'/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/3/runs/d9363f0d-645b-468f-947a-8bf97d15a630'
Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.997406 3771
slave.cpp:1840] Queuing task '3' for executor '3' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.997640 3777
containerizer.cpp:781] Starting container
'd9363f0d-645b-468f-947a-8bf97d15a630' for executor '3' of framework
'c9852a23-bc07-422d-8d69-23c167a1924d-0003'
Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.999593 3777
provisioner.cpp:294] Provisioning image rootfs
'/mnt/mesos/provisioner/containers/d9363f0d-645b-468f-947a-8bf97d15a630/backends/copy/rootfses/eeb1b066-94b8-4880-a696-090f830cf2d6'
for container d9363f0d-645b-468f-947a-8bf97d15a630
Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.003924 3775
slave.cpp:1495] Got assigned task 4 for framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.004278 3775
slave.cpp:1614] Launching task 4 for framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.005383 3775
slave.cpp:5674] Launching executor 4 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 with resources cpus(*):0.1; mem(*):32
in work directory
'/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/4/runs/58c398e2-38f9-4389-87ba-8696c9a523eb'
Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.005681 3775
slave.cpp:1840] Queuing task '4' for executor '4' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.005971 3772
containerizer.cpp:781] Starting container
'58c398e2-38f9-4389-87ba-8696c9a523eb' for executor '4' of framework
'c9852a23-bc07-422d-8d69-23c167a1924d-0003'
Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.007786 3777
provisioner.cpp:294] Provisioning image rootfs
'/mnt/mesos/provisioner/containers/58c398e2-38f9-4389-87ba-8696c9a523eb/backends/copy/rootfses/78b7bf3a-6f3b-4f8a-b4fa-98a6ea4bf8f5'
for container 58c398e2-38f9-4389-87ba-8696c9a523eb
Aug 9 10:12:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:41.801560 3777
slave.cpp:4591] Current disk usage 43.28%. Max allowed age:
3.270091966898553days
Aug 9 10:12:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:54.972057 3772
slave.cpp:4539] Terminating executor ''0' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003' because it did not register within
1mins
Aug 9 10:12:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:54.972322 3775
containerizer.cpp:1622] Destroying container
'09a77a80-f904-4581-966a-734d24ae96f3'
Aug 9 10:13:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:00.979907 3777
slave.cpp:4539] Terminating executor ''1' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003' because it did not register within
1mins
Aug 9 10:13:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:00.980058 3775
containerizer.cpp:1622] Destroying container
'0cea1dc7-b523-4b68-851a-531ed101de5a'
Aug 9 10:13:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:06.985003 3777
slave.cpp:4539] Terminating executor ''2' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003' because it did not register within
1mins
Aug 9 10:13:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:06.985255 3774
containerizer.cpp:1622] Destroying container
'a4c972da-b12d-4b4b-8481-79ca320ddab7'
Aug 9 10:13:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:12.998438 3777
slave.cpp:4539] Terminating executor ''3' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003' because it did not register within
1mins
Aug 9 10:13:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:12.998641 3776
containerizer.cpp:1622] Destroying container
'd9363f0d-645b-468f-947a-8bf97d15a630'
Aug 9 10:13:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:19.006058 3773
slave.cpp:4539] Terminating executor ''4' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003' because it did not register within
1mins
Aug 9 10:13:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:19.006228 3773
containerizer.cpp:1622] Destroying container
'58c398e2-38f9-4389-87ba-8696c9a523eb'
Aug 9 10:13:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:41.802209 3775
slave.cpp:4591] Current disk usage 57.59%. Max allowed age:
2.269006805055764days
Aug 9 10:14:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:14:41.803022 3776
slave.cpp:4591] Current disk usage 68.70%. Max allowed age:
1.491204118572176days
Aug 9 10:15:01 ip-172-31-23-17 mesos-slave[3738]: E0809 10:15:01.516207 3775
slave.cpp:3976] Container '58c398e2-38f9-4389-87ba-8696c9a523eb' for executor
'4' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 failed to start:
Container is being destroyed during provisioning
Aug 9 10:15:01 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:01.518277 3772
composing.cpp:541] Container '58c398e2-38f9-4389-87ba-8696c9a523eb' is already
destroyed
Aug 9 10:15:02 ip-172-31-23-17 mesos-slave[3738]: E0809 10:15:02.291764 3775
slave.cpp:3976] Container '09a77a80-f904-4581-966a-734d24ae96f3' for executor
'0' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 failed to start:
Container is being destroyed during provisioning
Aug 9 10:15:02 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:02.291877 3775
composing.cpp:541] Container '09a77a80-f904-4581-966a-734d24ae96f3' is already
destroyed
Aug 9 10:15:03 ip-172-31-23-17 mesos-slave[3738]: E0809 10:15:03.022406 3774
slave.cpp:3976] Container 'd9363f0d-645b-468f-947a-8bf97d15a630' for executor
'3' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 failed to start:
Container is being destroyed during provisioning
Aug 9 10:15:03 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:03.022595 3771
composing.cpp:541] Container 'd9363f0d-645b-468f-947a-8bf97d15a630' is already
destroyed
Aug 9 10:15:03 ip-172-31-23-17 mesos-slave[3738]: E0809 10:15:03.461519 3775
slave.cpp:3976] Container '0cea1dc7-b523-4b68-851a-531ed101de5a' for executor
'1' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 failed to start:
Container is being destroyed during provisioning
Aug 9 10:15:03 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:03.461643 3774
composing.cpp:541] Container '0cea1dc7-b523-4b68-851a-531ed101de5a' is already
destroyed
Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:04.190901 3776
provisioner.cpp:434] Destroying container rootfs at
'/mnt/mesos/provisioner/containers/58c398e2-38f9-4389-87ba-8696c9a523eb/backends/copy/rootfses/78b7bf3a-6f3b-4f8a-b4fa-98a6ea4bf8f5'
for container 58c398e2-38f9-4389-87ba-8696c9a523eb
Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: E0809 10:15:04.190904 3777
slave.cpp:3976] Container 'a4c972da-b12d-4b4b-8481-79ca320ddab7' for executor
'2' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 failed to start:
Container is being destroyed during provisioning
Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:04.191128 3770
composing.cpp:541] Container 'a4c972da-b12d-4b4b-8481-79ca320ddab7' is already
destroyed
Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:04.197221 3776
provisioner.cpp:434] Destroying container rootfs at
'/mnt/mesos/provisioner/containers/09a77a80-f904-4581-966a-734d24ae96f3/backends/copy/rootfses/ec9bb10e-6726-4872-be2b-4a72dfc0fafc'
for container 09a77a80-f904-4581-966a-734d24ae96f3
Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:04.197410 3776
provisioner.cpp:434] Destroying container rootfs at
'/mnt/mesos/provisioner/containers/d9363f0d-645b-468f-947a-8bf97d15a630/backends/copy/rootfses/eeb1b066-94b8-4880-a696-090f830cf2d6'
for container d9363f0d-645b-468f-947a-8bf97d15a630
Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:04.197510 3776
provisioner.cpp:434] Destroying container rootfs at
'/mnt/mesos/provisioner/containers/0cea1dc7-b523-4b68-851a-531ed101de5a/backends/copy/rootfses/e3af89fe-c889-45fc-929b-f008b18e5d4e'
for container 0cea1dc7-b523-4b68-851a-531ed101de5a
Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:04.197619 3776
provisioner.cpp:434] Destroying container rootfs at
'/mnt/mesos/provisioner/containers/a4c972da-b12d-4b4b-8481-79ca320ddab7/backends/copy/rootfses/feaf70a4-df43-45c9-b6be-a654c9210a54'
for container a4c972da-b12d-4b4b-8481-79ca320ddab7
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.390928 3776
slave.cpp:4082] Executor '3' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 has terminated with unknown status
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.395370 3776
slave.cpp:3211] Handling status update TASK_FAILED (UUID:
543e1e1e-ae47-48ab-ad64-ee82fdd27418) for task 3 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 from @0.0.0.0:0
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.402472 3776
slave.cpp:4082] Executor '2' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 has terminated with unknown status
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.402556 3776
slave.cpp:3211] Handling status update TASK_FAILED (UUID:
1a54664d-4157-42a3-a97f-7189c137a769) for task 2 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 from @0.0.0.0:0
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:07.406177 3772
containerizer.cpp:1451] Ignoring update for unknown container:
d9363f0d-645b-468f-947a-8bf97d15a630
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.407325 3777
status_update_manager.cpp:320] Received status update TASK_FAILED (UUID:
543e1e1e-ae47-48ab-ad64-ee82fdd27418) for task 3 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:07.407598 3773
containerizer.cpp:1451] Ignoring update for unknown container:
a4c972da-b12d-4b4b-8481-79ca320ddab7
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.409890 3774
slave.cpp:3604] Forwarding the update TASK_FAILED (UUID:
543e1e1e-ae47-48ab-ad64-ee82fdd27418) for task 3 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 to [email protected]:5050
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.409893 3777
status_update_manager.cpp:320] Received status update TASK_FAILED (UUID:
1a54664d-4157-42a3-a97f-7189c137a769) for task 2 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.414727 3774
slave.cpp:3604] Forwarding the update TASK_FAILED (UUID:
1a54664d-4157-42a3-a97f-7189c137a769) for task 2 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 to [email protected]:5050
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.418180 3777
status_update_manager.cpp:392] Received status update acknowledgement (UUID:
543e1e1e-ae47-48ab-ad64-ee82fdd27418) for task 3 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.419615 3777
status_update_manager.cpp:392] Received status update acknowledgement (UUID:
1a54664d-4157-42a3-a97f-7189c137a769) for task 2 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.419636 3774
slave.cpp:4193] Cleaning up executor '3' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.419924 3772
gc.cpp:55] Scheduling
'/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/3/runs/d9363f0d-645b-468f-947a-8bf97d15a630'
for gc 6.9999951410637days in the future
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.420703 3773
gc.cpp:55] Scheduling
'/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/3'
for gc 6.99999513154074days in the future
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.420722 3774
slave.cpp:4193] Cleaning up executor '2' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.420897 3774
gc.cpp:55] Scheduling
'/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/2/runs/a4c972da-b12d-4b4b-8481-79ca320ddab7'
for gc 6.99999512931852days in the future
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.420940 3774
gc.cpp:55] Scheduling
'/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/2'
for gc 6.99999512869333days in the future
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.490155 3772
slave.cpp:4082] Executor '4' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 has terminated with unknown status
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.490289 3772
slave.cpp:3211] Handling status update TASK_FAILED (UUID:
9c5528ec-0725-42f2-a2e2-465d3c73b8d5) for task 4 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 from @0.0.0.0:0
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.490414 3772
slave.cpp:4082] Executor '0' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 has terminated with unknown status
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.490489 3772
slave.cpp:3211] Handling status update TASK_FAILED (UUID:
3f801210-b029-427f-af3f-e51b1dfa5cc8) for task 0 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 from @0.0.0.0:0
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:07.490687 3773
containerizer.cpp:1451] Ignoring update for unknown container:
58c398e2-38f9-4389-87ba-8696c9a523eb
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:07.490774 3773
containerizer.cpp:1451] Ignoring update for unknown container:
09a77a80-f904-4581-966a-734d24ae96f3
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.490867 3773
status_update_manager.cpp:320] Received status update TASK_FAILED (UUID:
9c5528ec-0725-42f2-a2e2-465d3c73b8d5) for task 4 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.491082 3773
status_update_manager.cpp:320] Received status update TASK_FAILED (UUID:
3f801210-b029-427f-af3f-e51b1dfa5cc8) for task 0 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.491082 3772
slave.cpp:3604] Forwarding the update TASK_FAILED (UUID:
9c5528ec-0725-42f2-a2e2-465d3c73b8d5) for task 4 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 to [email protected]:5050
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.491303 3774
slave.cpp:3604] Forwarding the update TASK_FAILED (UUID:
3f801210-b029-427f-af3f-e51b1dfa5cc8) for task 0 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 to [email protected]:5050
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494438 3775
status_update_manager.cpp:392] Received status update acknowledgement (UUID:
9c5528ec-0725-42f2-a2e2-465d3c73b8d5) for task 4 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494572 3775
slave.cpp:4193] Cleaning up executor '4' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494757 3773
gc.cpp:55] Scheduling
'/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/4/runs/58c398e2-38f9-4389-87ba-8696c9a523eb'
for gc 6.99999427436444days in the future
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494817 3775
status_update_manager.cpp:392] Received status update acknowledgement (UUID:
3f801210-b029-427f-af3f-e51b1dfa5cc8) for task 0 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494937 3775
slave.cpp:4193] Cleaning up executor '0' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494823 3773
gc.cpp:55] Scheduling
'/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/4'
for gc 6.99999427361185days in the future
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.495107 3773
gc.cpp:55] Scheduling
'/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/0/runs/09a77a80-f904-4581-966a-734d24ae96f3'
for gc 6.99999427055407days in the future
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.495149 3773
gc.cpp:55] Scheduling
'/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/0'
for gc 6.99999427024296days in the future
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.591338 3775
slave.cpp:4082] Executor '1' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 has terminated with unknown status
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.591469 3775
slave.cpp:3211] Handling status update TASK_FAILED (UUID:
85b7c18a-c59c-4449-827c-a3bfaa2a7de8) for task 1 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 from @0.0.0.0:0
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:07.591841 3776
containerizer.cpp:1451] Ignoring update for unknown container:
0cea1dc7-b523-4b68-851a-531ed101de5a
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.591974 3770
status_update_manager.cpp:320] Received status update TASK_FAILED (UUID:
85b7c18a-c59c-4449-827c-a3bfaa2a7de8) for task 1 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.592157 3772
slave.cpp:3604] Forwarding the update TASK_FAILED (UUID:
85b7c18a-c59c-4449-827c-a3bfaa2a7de8) for task 1 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 to [email protected]:5050
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594518 3771
status_update_manager.cpp:392] Received status update acknowledgement (UUID:
85b7c18a-c59c-4449-827c-a3bfaa2a7de8) for task 1 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594636 3771
slave.cpp:4193] Cleaning up executor '1' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594792 3770
gc.cpp:55] Scheduling
'/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/1/runs/0cea1dc7-b523-4b68-851a-531ed101de5a'
for gc 6.99999311635556days in the future
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594820 3771
slave.cpp:4281] Cleaning up framework c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594861 3770
gc.cpp:55] Scheduling
'/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/1'
for gc 6.99999311568296days in the future
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594897 3770
status_update_manager.cpp:282] Closing status update streams for framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
{noformat}
was:
When scheduling more than one task on the same agent, all tasks fail a as
containers seem to be destroyed during provisioning.
Specifically, the errors on the agent logs are:
{noformat}
E0808 15:53:09.691315 30996 slave.cpp:3976] Container
'eb20f642-bb90-4293-8eec-6f1576ccaeb1' for executor '3' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0001 failed to start: Container is being
destroyed during provisioning
{noformat}
and
{noformat}
I0808 15:52:32.510210 30999 slave.cpp:4539] Terminating executor ''2' of
framework c9852a23-bc07-422d-8d69-23c167a1924d-0001' because it did not
register within 3mins
{noformat}
As the default provisioning method {{copy}} is being used, I assume this is due
to the provisioning of multiple containers taking too long and the agent will
not wait. For large images, this method is simply not performant.
The issue did not occur, when only one tasks was scheduled.
Increasing the {{executor_registration_timeout}} parameter, seemed to help a
bit as it allowed to schedule at least 2 tasks at the same time. But still
fails with more (5 in this case)
h4. Complete logs
(with GLOG_v=1)
{noformat}
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.800375 3738
slave.cpp:198] Agent started on 1)@172.31.23.17:5051
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.800403 3738
slave.cpp:199] Flags at startup: --appc_simple_discovery_uri_prefix="http://";
--appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="false"
--authenticate_http_readwrite="false" --authenticatee="crammd5"
--authentication_backoff_factor="1secs" --authorizer="local"
--cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false"
--cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false"
--cgroups_root="mesos" --container_disk_watch_interval="15secs"
--containerizers="mesos,docker" --default_role="*"
--disk_watch_interval="1mins" --docker="docker"
--docker_config="{"auths":{"https:\/\/index.docker.io\/v1\/":{"auth":"dGVycmFsb3VwZTpUYWxFWUFOSXR5","email":"[email protected]"}}}"
--docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io";
--docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock"
--docker_stop_timeout="0ns" --docker_store_dir="/mnt/t" --docker_volume_checkp
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]:
oint_dir="/var/run/mesos/isolators/docker/volume"
--enforce_container_disk_quota="false" --executor_registration_timeout="1mins"
--executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch"
--fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks"
--gc_disk_headroom="0.1" --hadoop_home="" --help="false"
--hostname="ec2-52-59-113-0.eu-central-1.compute.amazonaws.com"
--hostname_lookup="true" --http_authenticators="basic"
--http_command_executor="false" --image_providers="docker"
--image_provisioner_backend="copy" --initialize_driver_logging="true"
--isolation="filesystem/linux,docker/runtime,cgroups/devices,gpu/nvidia"
--launcher_dir="/usr/libexec/mesos" --log_dir="/var/log/mesos" --logbufsecs="0"
--logging_level="INFO" --master="zk://172.31.19.240:2181/mesos"
--oversubscribed_resources_interval="15secs" --perf_duration="10secs"
--perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns"
--quiet="false" --recover="reconnect" --re
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: covery_timeout="15mins"
--registration_backoff_factor="1secs" --revocable_cpu_low_priority="true"
--sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="false"
--systemd_enable_support="true"
--systemd_runtime_directory="/run/systemd/system" --version="false"
--work_dir="/mnt/mesos"
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: 2016-08-09
10:11:41,800:3738(0x7f71172a1700):ZOO_INFO@check_events@1728: initiated
connection to server [172.31.19.240:2181]
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.801007 3738
slave.cpp:519] Agent resources: gpus(*):1; cpus(*):8; mem(*):14014;
disk(*):60257; ports(*):[31000-32000]
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.801059 3738
slave.cpp:527] Agent attributes: [ ]
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.801077 3738
slave.cpp:532] Agent hostname:
ec2-52-59-113-0.eu-central-1.compute.amazonaws.com
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.802232 3772
state.cpp:57] Recovering state from '/mnt/mesos/meta'
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.803539 3774
status_update_manager.cpp:200] Recovering status update manager
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.803745 3772
containerizer.cpp:522] Recovering containerizer
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.803788 3770
docker.cpp:775] Recovering Docker containers
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.808550 3777
metadata_manager.cpp:251] Successfully loaded 2 Docker images
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.808688 3771
provisioner.cpp:253] Provisioner recovery complete
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: 2016-08-09
10:11:41,816:3738(0x7f71172a1700):ZOO_INFO@check_events@1775: session
establishment complete on server [172.31.19.240:2181],
sessionId=0x1566a66ab9b0010, negotiated timeout=10000
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.817003 3774
group.cpp:349] Group process (group(1)@172.31.23.17:5051) connected to ZooKeeper
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.817116 3774
group.cpp:837] Syncing group operations: queue size (joins, cancels, datas) =
(0, 0, 0)
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.817147 3774
group.cpp:427] Trying to create path '/mesos' in ZooKeeper
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.818857 3774
detector.cpp:152] Detected a new leader: (id='64')
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.819005 3776
group.cpp:706] Trying to get '/mesos/json.info_0000000064' in ZooKeeper
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.819855 3775
zookeeper.cpp:259] A new leading master ([email protected]:5050) is
detected
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.899876 3777
slave.cpp:4782] Finished recovery
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900573 3777
slave.cpp:4815] Garbage collecting old agent
524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S2
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900687 3773
gc.cpp:55] Scheduling
'/mnt/mesos/slaves/524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S2' for gc
6.99998957623704days in the future
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900729 3777
slave.cpp:4815] Garbage collecting old agent
c9852a23-bc07-422d-8d69-23c167a1924d-S0
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900779 3773
gc.cpp:55] Scheduling
'/mnt/mesos/meta/slaves/524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S2' for gc
6.99998957520889days in the future
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900817 3777
slave.cpp:4815] Garbage collecting old agent
524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S1
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900825 3773
gc.cpp:55] Scheduling
'/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S0' for gc
6.99998957451259days in the future
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900899 3773
gc.cpp:55] Scheduling
'/mnt/mesos/meta/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S0' for gc
6.99998957412148days in the future
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900933 3777
slave.cpp:4815] Garbage collecting old agent
524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S3
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900944 3773
gc.cpp:55] Scheduling
'/mnt/mesos/slaves/524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S1' for gc
6.99998957332148days in the future
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901010 3773
gc.cpp:55] Scheduling
'/mnt/mesos/meta/slaves/524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S1' for gc
6.99998957291556days in the future
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901051 3773
gc.cpp:55] Scheduling
'/mnt/mesos/slaves/524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S3' for gc
6.99998957196741days in the future
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901101 3773
gc.cpp:55] Scheduling
'/mnt/mesos/meta/slaves/524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S3' for gc
6.99998957156148days in the future
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901201 3775
status_update_manager.cpp:174] Pausing sending status updates
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901203 3777
slave.cpp:895] New master detected at [email protected]:5050
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901278 3777
slave.cpp:916] No credentials provided. Attempting to register without
authentication
Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901317 3777
slave.cpp:927] Detecting new master
Aug 9 10:11:42 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:42.267201 3774
slave.cpp:1197] Re-registered with master [email protected]:5050
Aug 9 10:11:42 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:42.267302 3771
status_update_manager.cpp:181] Resuming sending status updates
Aug 9 10:11:42 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:42.267302 3774
slave.cpp:1233] Forwarding total oversubscribed resources
Aug 9 10:11:42 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:42.267577 3774
slave.cpp:2526] Updated checkpointed resources from to
Aug 9 10:11:48 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:48.269104 3777
slave.cpp:2218] Asked to shut down framework
c9852a23-bc07-422d-8d69-23c167a1924d-0002 by [email protected]:5050
Aug 9 10:11:48 ip-172-31-23-17 mesos-slave[3738]: W0809 10:11:48.269186 3777
slave.cpp:2233] Cannot shut down unknown framework
c9852a23-bc07-422d-8d69-23c167a1924d-0002
Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.969521 3772
slave.cpp:1495] Got assigned task 0 for framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.970026 3772
slave.cpp:1614] Launching task 0 for framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.970826 3772
slave.cpp:5674] Launching executor 0 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 with resources cpus(*):0.1; mem(*):32
in work directory
'/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/0/runs/09a77a80-f904-4581-966a-734d24ae96f3'
Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.971051 3772
slave.cpp:1840] Queuing task '0' for executor '0' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.971212 3773
containerizer.cpp:781] Starting container
'09a77a80-f904-4581-966a-734d24ae96f3' for executor '0' of framework
'c9852a23-bc07-422d-8d69-23c167a1924d-0003'
Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.973953 3774
provisioner.cpp:294] Provisioning image rootfs
'/mnt/mesos/provisioner/containers/09a77a80-f904-4581-966a-734d24ae96f3/backends/copy/rootfses/ec9bb10e-6726-4872-be2b-4a72dfc0fafc'
for container 09a77a80-f904-4581-966a-734d24ae96f3
Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.978344 3774
slave.cpp:1495] Got assigned task 1 for framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.978687 3774
slave.cpp:1614] Launching task 1 for framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.978976 3774
slave.cpp:5674] Launching executor 1 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 with resources cpus(*):0.1; mem(*):32
in work directory
'/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/1/runs/0cea1dc7-b523-4b68-851a-531ed101de5a'
Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.979173 3774
slave.cpp:1840] Queuing task '1' for executor '1' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.979387 3777
containerizer.cpp:781] Starting container
'0cea1dc7-b523-4b68-851a-531ed101de5a' for executor '1' of framework
'c9852a23-bc07-422d-8d69-23c167a1924d-0003'
Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.981374 3776
provisioner.cpp:294] Provisioning image rootfs
'/mnt/mesos/provisioner/containers/0cea1dc7-b523-4b68-851a-531ed101de5a/backends/copy/rootfses/e3af89fe-c889-45fc-929b-f008b18e5d4e'
for container 0cea1dc7-b523-4b68-851a-531ed101de5a
Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.983610 3776
slave.cpp:1495] Got assigned task 2 for framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.983947 3776
slave.cpp:1614] Launching task 2 for framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.984236 3776
slave.cpp:5674] Launching executor 2 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 with resources cpus(*):0.1; mem(*):32
in work directory
'/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/2/runs/a4c972da-b12d-4b4b-8481-79ca320ddab7'
Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.984450 3776
slave.cpp:1840] Queuing task '2' for executor '2' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.984778 3777
containerizer.cpp:781] Starting container
'a4c972da-b12d-4b4b-8481-79ca320ddab7' for executor '2' of framework
'c9852a23-bc07-422d-8d69-23c167a1924d-0003'
Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.986330 3776
provisioner.cpp:294] Provisioning image rootfs
'/mnt/mesos/provisioner/containers/a4c972da-b12d-4b4b-8481-79ca320ddab7/backends/copy/rootfses/feaf70a4-df43-45c9-b6be-a654c9210a54'
for container a4c972da-b12d-4b4b-8481-79ca320ddab7
Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.996104 3771
slave.cpp:1495] Got assigned task 3 for framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.996462 3771
slave.cpp:1614] Launching task 3 for framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.997241 3771
slave.cpp:5674] Launching executor 3 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 with resources cpus(*):0.1; mem(*):32
in work directory
'/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/3/runs/d9363f0d-645b-468f-947a-8bf97d15a630'
Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.997406 3771
slave.cpp:1840] Queuing task '3' for executor '3' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.997640 3777
containerizer.cpp:781] Starting container
'd9363f0d-645b-468f-947a-8bf97d15a630' for executor '3' of framework
'c9852a23-bc07-422d-8d69-23c167a1924d-0003'
Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.999593 3777
provisioner.cpp:294] Provisioning image rootfs
'/mnt/mesos/provisioner/containers/d9363f0d-645b-468f-947a-8bf97d15a630/backends/copy/rootfses/eeb1b066-94b8-4880-a696-090f830cf2d6'
for container d9363f0d-645b-468f-947a-8bf97d15a630
Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.003924 3775
slave.cpp:1495] Got assigned task 4 for framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.004278 3775
slave.cpp:1614] Launching task 4 for framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.005383 3775
slave.cpp:5674] Launching executor 4 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 with resources cpus(*):0.1; mem(*):32
in work directory
'/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/4/runs/58c398e2-38f9-4389-87ba-8696c9a523eb'
Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.005681 3775
slave.cpp:1840] Queuing task '4' for executor '4' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.005971 3772
containerizer.cpp:781] Starting container
'58c398e2-38f9-4389-87ba-8696c9a523eb' for executor '4' of framework
'c9852a23-bc07-422d-8d69-23c167a1924d-0003'
Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.007786 3777
provisioner.cpp:294] Provisioning image rootfs
'/mnt/mesos/provisioner/containers/58c398e2-38f9-4389-87ba-8696c9a523eb/backends/copy/rootfses/78b7bf3a-6f3b-4f8a-b4fa-98a6ea4bf8f5'
for container 58c398e2-38f9-4389-87ba-8696c9a523eb
Aug 9 10:12:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:41.801560 3777
slave.cpp:4591] Current disk usage 43.28%. Max allowed age:
3.270091966898553days
Aug 9 10:12:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:54.972057 3772
slave.cpp:4539] Terminating executor ''0' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003' because it did not register within
1mins
Aug 9 10:12:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:54.972322 3775
containerizer.cpp:1622] Destroying container
'09a77a80-f904-4581-966a-734d24ae96f3'
Aug 9 10:13:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:00.979907 3777
slave.cpp:4539] Terminating executor ''1' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003' because it did not register within
1mins
Aug 9 10:13:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:00.980058 3775
containerizer.cpp:1622] Destroying container
'0cea1dc7-b523-4b68-851a-531ed101de5a'
Aug 9 10:13:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:06.985003 3777
slave.cpp:4539] Terminating executor ''2' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003' because it did not register within
1mins
Aug 9 10:13:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:06.985255 3774
containerizer.cpp:1622] Destroying container
'a4c972da-b12d-4b4b-8481-79ca320ddab7'
Aug 9 10:13:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:12.998438 3777
slave.cpp:4539] Terminating executor ''3' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003' because it did not register within
1mins
Aug 9 10:13:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:12.998641 3776
containerizer.cpp:1622] Destroying container
'd9363f0d-645b-468f-947a-8bf97d15a630'
Aug 9 10:13:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:19.006058 3773
slave.cpp:4539] Terminating executor ''4' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003' because it did not register within
1mins
Aug 9 10:13:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:19.006228 3773
containerizer.cpp:1622] Destroying container
'58c398e2-38f9-4389-87ba-8696c9a523eb'
Aug 9 10:13:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:41.802209 3775
slave.cpp:4591] Current disk usage 57.59%. Max allowed age:
2.269006805055764days
Aug 9 10:14:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:14:41.803022 3776
slave.cpp:4591] Current disk usage 68.70%. Max allowed age:
1.491204118572176days
Aug 9 10:15:01 ip-172-31-23-17 mesos-slave[3738]: E0809 10:15:01.516207 3775
slave.cpp:3976] Container '58c398e2-38f9-4389-87ba-8696c9a523eb' for executor
'4' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 failed to start:
Container is being destroyed during provisioning
Aug 9 10:15:01 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:01.518277 3772
composing.cpp:541] Container '58c398e2-38f9-4389-87ba-8696c9a523eb' is already
destroyed
Aug 9 10:15:02 ip-172-31-23-17 mesos-slave[3738]: E0809 10:15:02.291764 3775
slave.cpp:3976] Container '09a77a80-f904-4581-966a-734d24ae96f3' for executor
'0' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 failed to start:
Container is being destroyed during provisioning
Aug 9 10:15:02 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:02.291877 3775
composing.cpp:541] Container '09a77a80-f904-4581-966a-734d24ae96f3' is already
destroyed
Aug 9 10:15:03 ip-172-31-23-17 mesos-slave[3738]: E0809 10:15:03.022406 3774
slave.cpp:3976] Container 'd9363f0d-645b-468f-947a-8bf97d15a630' for executor
'3' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 failed to start:
Container is being destroyed during provisioning
Aug 9 10:15:03 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:03.022595 3771
composing.cpp:541] Container 'd9363f0d-645b-468f-947a-8bf97d15a630' is already
destroyed
Aug 9 10:15:03 ip-172-31-23-17 mesos-slave[3738]: E0809 10:15:03.461519 3775
slave.cpp:3976] Container '0cea1dc7-b523-4b68-851a-531ed101de5a' for executor
'1' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 failed to start:
Container is being destroyed during provisioning
Aug 9 10:15:03 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:03.461643 3774
composing.cpp:541] Container '0cea1dc7-b523-4b68-851a-531ed101de5a' is already
destroyed
Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:04.190901 3776
provisioner.cpp:434] Destroying container rootfs at
'/mnt/mesos/provisioner/containers/58c398e2-38f9-4389-87ba-8696c9a523eb/backends/copy/rootfses/78b7bf3a-6f3b-4f8a-b4fa-98a6ea4bf8f5'
for container 58c398e2-38f9-4389-87ba-8696c9a523eb
Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: E0809 10:15:04.190904 3777
slave.cpp:3976] Container 'a4c972da-b12d-4b4b-8481-79ca320ddab7' for executor
'2' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 failed to start:
Container is being destroyed during provisioning
Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:04.191128 3770
composing.cpp:541] Container 'a4c972da-b12d-4b4b-8481-79ca320ddab7' is already
destroyed
Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:04.197221 3776
provisioner.cpp:434] Destroying container rootfs at
'/mnt/mesos/provisioner/containers/09a77a80-f904-4581-966a-734d24ae96f3/backends/copy/rootfses/ec9bb10e-6726-4872-be2b-4a72dfc0fafc'
for container 09a77a80-f904-4581-966a-734d24ae96f3
Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:04.197410 3776
provisioner.cpp:434] Destroying container rootfs at
'/mnt/mesos/provisioner/containers/d9363f0d-645b-468f-947a-8bf97d15a630/backends/copy/rootfses/eeb1b066-94b8-4880-a696-090f830cf2d6'
for container d9363f0d-645b-468f-947a-8bf97d15a630
Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:04.197510 3776
provisioner.cpp:434] Destroying container rootfs at
'/mnt/mesos/provisioner/containers/0cea1dc7-b523-4b68-851a-531ed101de5a/backends/copy/rootfses/e3af89fe-c889-45fc-929b-f008b18e5d4e'
for container 0cea1dc7-b523-4b68-851a-531ed101de5a
Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:04.197619 3776
provisioner.cpp:434] Destroying container rootfs at
'/mnt/mesos/provisioner/containers/a4c972da-b12d-4b4b-8481-79ca320ddab7/backends/copy/rootfses/feaf70a4-df43-45c9-b6be-a654c9210a54'
for container a4c972da-b12d-4b4b-8481-79ca320ddab7
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.390928 3776
slave.cpp:4082] Executor '3' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 has terminated with unknown status
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.395370 3776
slave.cpp:3211] Handling status update TASK_FAILED (UUID:
543e1e1e-ae47-48ab-ad64-ee82fdd27418) for task 3 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 from @0.0.0.0:0
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.402472 3776
slave.cpp:4082] Executor '2' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 has terminated with unknown status
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.402556 3776
slave.cpp:3211] Handling status update TASK_FAILED (UUID:
1a54664d-4157-42a3-a97f-7189c137a769) for task 2 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 from @0.0.0.0:0
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:07.406177 3772
containerizer.cpp:1451] Ignoring update for unknown container:
d9363f0d-645b-468f-947a-8bf97d15a630
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.407325 3777
status_update_manager.cpp:320] Received status update TASK_FAILED (UUID:
543e1e1e-ae47-48ab-ad64-ee82fdd27418) for task 3 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:07.407598 3773
containerizer.cpp:1451] Ignoring update for unknown container:
a4c972da-b12d-4b4b-8481-79ca320ddab7
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.409890 3774
slave.cpp:3604] Forwarding the update TASK_FAILED (UUID:
543e1e1e-ae47-48ab-ad64-ee82fdd27418) for task 3 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 to [email protected]:5050
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.409893 3777
status_update_manager.cpp:320] Received status update TASK_FAILED (UUID:
1a54664d-4157-42a3-a97f-7189c137a769) for task 2 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.414727 3774
slave.cpp:3604] Forwarding the update TASK_FAILED (UUID:
1a54664d-4157-42a3-a97f-7189c137a769) for task 2 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 to [email protected]:5050
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.418180 3777
status_update_manager.cpp:392] Received status update acknowledgement (UUID:
543e1e1e-ae47-48ab-ad64-ee82fdd27418) for task 3 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.419615 3777
status_update_manager.cpp:392] Received status update acknowledgement (UUID:
1a54664d-4157-42a3-a97f-7189c137a769) for task 2 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.419636 3774
slave.cpp:4193] Cleaning up executor '3' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.419924 3772
gc.cpp:55] Scheduling
'/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/3/runs/d9363f0d-645b-468f-947a-8bf97d15a630'
for gc 6.9999951410637days in the future
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.420703 3773
gc.cpp:55] Scheduling
'/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/3'
for gc 6.99999513154074days in the future
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.420722 3774
slave.cpp:4193] Cleaning up executor '2' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.420897 3774
gc.cpp:55] Scheduling
'/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/2/runs/a4c972da-b12d-4b4b-8481-79ca320ddab7'
for gc 6.99999512931852days in the future
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.420940 3774
gc.cpp:55] Scheduling
'/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/2'
for gc 6.99999512869333days in the future
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.490155 3772
slave.cpp:4082] Executor '4' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 has terminated with unknown status
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.490289 3772
slave.cpp:3211] Handling status update TASK_FAILED (UUID:
9c5528ec-0725-42f2-a2e2-465d3c73b8d5) for task 4 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 from @0.0.0.0:0
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.490414 3772
slave.cpp:4082] Executor '0' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 has terminated with unknown status
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.490489 3772
slave.cpp:3211] Handling status update TASK_FAILED (UUID:
3f801210-b029-427f-af3f-e51b1dfa5cc8) for task 0 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 from @0.0.0.0:0
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:07.490687 3773
containerizer.cpp:1451] Ignoring update for unknown container:
58c398e2-38f9-4389-87ba-8696c9a523eb
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:07.490774 3773
containerizer.cpp:1451] Ignoring update for unknown container:
09a77a80-f904-4581-966a-734d24ae96f3
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.490867 3773
status_update_manager.cpp:320] Received status update TASK_FAILED (UUID:
9c5528ec-0725-42f2-a2e2-465d3c73b8d5) for task 4 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.491082 3773
status_update_manager.cpp:320] Received status update TASK_FAILED (UUID:
3f801210-b029-427f-af3f-e51b1dfa5cc8) for task 0 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.491082 3772
slave.cpp:3604] Forwarding the update TASK_FAILED (UUID:
9c5528ec-0725-42f2-a2e2-465d3c73b8d5) for task 4 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 to [email protected]:5050
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.491303 3774
slave.cpp:3604] Forwarding the update TASK_FAILED (UUID:
3f801210-b029-427f-af3f-e51b1dfa5cc8) for task 0 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 to [email protected]:5050
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494438 3775
status_update_manager.cpp:392] Received status update acknowledgement (UUID:
9c5528ec-0725-42f2-a2e2-465d3c73b8d5) for task 4 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494572 3775
slave.cpp:4193] Cleaning up executor '4' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494757 3773
gc.cpp:55] Scheduling
'/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/4/runs/58c398e2-38f9-4389-87ba-8696c9a523eb'
for gc 6.99999427436444days in the future
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494817 3775
status_update_manager.cpp:392] Received status update acknowledgement (UUID:
3f801210-b029-427f-af3f-e51b1dfa5cc8) for task 0 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494937 3775
slave.cpp:4193] Cleaning up executor '0' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494823 3773
gc.cpp:55] Scheduling
'/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/4'
for gc 6.99999427361185days in the future
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.495107 3773
gc.cpp:55] Scheduling
'/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/0/runs/09a77a80-f904-4581-966a-734d24ae96f3'
for gc 6.99999427055407days in the future
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.495149 3773
gc.cpp:55] Scheduling
'/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/0'
for gc 6.99999427024296days in the future
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.591338 3775
slave.cpp:4082] Executor '1' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 has terminated with unknown status
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.591469 3775
slave.cpp:3211] Handling status update TASK_FAILED (UUID:
85b7c18a-c59c-4449-827c-a3bfaa2a7de8) for task 1 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 from @0.0.0.0:0
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:07.591841 3776
containerizer.cpp:1451] Ignoring update for unknown container:
0cea1dc7-b523-4b68-851a-531ed101de5a
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.591974 3770
status_update_manager.cpp:320] Received status update TASK_FAILED (UUID:
85b7c18a-c59c-4449-827c-a3bfaa2a7de8) for task 1 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.592157 3772
slave.cpp:3604] Forwarding the update TASK_FAILED (UUID:
85b7c18a-c59c-4449-827c-a3bfaa2a7de8) for task 1 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003 to [email protected]:5050
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594518 3771
status_update_manager.cpp:392] Received status update acknowledgement (UUID:
85b7c18a-c59c-4449-827c-a3bfaa2a7de8) for task 1 of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594636 3771
slave.cpp:4193] Cleaning up executor '1' of framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594792 3770
gc.cpp:55] Scheduling
'/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/1/runs/0cea1dc7-b523-4b68-851a-531ed101de5a'
for gc 6.99999311635556days in the future
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594820 3771
slave.cpp:4281] Cleaning up framework c9852a23-bc07-422d-8d69-23c167a1924d-0003
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594861 3770
gc.cpp:55] Scheduling
'/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/1'
for gc 6.99999311568296days in the future
Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594897 3770
status_update_manager.cpp:282] Closing status update streams for framework
c9852a23-bc07-422d-8d69-23c167a1924d-0003
{noformat}
> Tasks fail when provisioning multiple containers with large docker images
> using copy backend
> ---------------------------------------------------------------------------------------------
>
> Key: MESOS-6004
> URL: https://issues.apache.org/jira/browse/MESOS-6004
> Project: Mesos
> Issue Type: Bug
> Components: containerization, docker
> Affects Versions: 0.28.2, 1.0.0
> Environment: h4. Agent Platform
> - Ubuntu 16.04
> - AWS g2.x2large instance
> - Nvidia support enabled
> h4. Agent Configuration
> -{noformat}
> --containerizers=mesos,docker
> --docker_config=<docker auth json>
> --docker_store_dir=/mnt/mesos/store/docker
> --executor_registration_timeout=3mins
> --hostname=<aws public dns>
> --image_providers=docker
> --image_provisioner_backend=copy
> --isolation=filesystem/linux,docker/runtime,cgroups/devices,gpu/nvidia
> --switch_user=false
> --work_dir=/mnt/mesos
> {noformat}
> h4. Framework
> - custom framework written in python
> - using unified containerizer with docker images
> h4. Test Setup
> * 1 master
> * 1 agent
> * 5 tasks scheduled at the same time:
> ** resources: cpus: 0.1, mem: 128
> ** command: `echo test`
> ** docker image: custom docker image, based on nvidia/cuda ~5gb
> ** the same docker image was for all tasks, already pulled.
> Reporter: Michael Thomas
> Labels: containerizer, docker, performance
>
> When scheduling more than one task on the same agent, all tasks fail a as
> containers seem to be destroyed during provisioning.
> Specifically, the errors on the agent logs are:
> {noformat}
> E0808 15:53:09.691315 30996 slave.cpp:3976] Container
> 'eb20f642-bb90-4293-8eec-6f1576ccaeb1' for executor '3' of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0001 failed to start: Container is being
> destroyed during provisioning
> {noformat}
> and
> {noformat}
> I0808 15:52:32.510210 30999 slave.cpp:4539] Terminating executor ''2' of
> framework c9852a23-bc07-422d-8d69-23c167a1924d-0001' because it did not
> register within 3mins
> {noformat}
> As the default provisioning method {{copy}} is being used, I assume this is
> due to the provisioning of multiple containers taking too long and the agent
> will not wait. For large images, this method is simply not performant.
> The issue did not occur, when only one tasks was scheduled.
> Increasing the {{executor_registration_timeout}} parameter, seemed to help a
> bit as it allowed to schedule at least 2 tasks at the same time. But still
> fails with more (5 in this case)
> h4. Complete logs
> (with GLOG_v=1)
> {noformat}
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.800375
> 3738 slave.cpp:198] Agent started on 1)@172.31.23.17:5051
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.800403
> 3738 slave.cpp:199] Flags at startup:
> --appc_simple_discovery_uri_prefix="http://";
> --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="false"
> --authenticate_http_readwrite="false" --authenticatee="crammd5"
> --authentication_backoff_factor="1secs" --authorizer="local"
> --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false"
> --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false"
> --cgroups_root="mesos" --container_disk_watch_interval="15secs"
> --containerizers="mesos,docker" --default_role="*"
> --disk_watch_interval="1mins" --docker="docker" --docker_config="XXX"
> --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io";
> --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock"
> --docker_stop_timeout="0ns" --docker_store_dir="/mnt/t" --docker_volume_checkp
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]:
> oint_dir="/var/run/mesos/isolators/docker/volume"
> --enforce_container_disk_quota="false"
> --executor_registration_timeout="1mins"
> --executor_shutdown_grace_period="5secs"
> --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB"
> --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1"
> --hadoop_home="" --help="false"
> --hostname="ec2-52-59-113-0.eu-central-1.compute.amazonaws.com"
> --hostname_lookup="true" --http_authenticators="basic"
> --http_command_executor="false" --image_providers="docker"
> --image_provisioner_backend="copy" --initialize_driver_logging="true"
> --isolation="filesystem/linux,docker/runtime,cgroups/devices,gpu/nvidia"
> --launcher_dir="/usr/libexec/mesos" --log_dir="/var/log/mesos"
> --logbufsecs="0" --logging_level="INFO"
> --master="zk://172.31.19.240:2181/mesos"
> --oversubscribed_resources_interval="15secs" --perf_duration="10secs"
> --perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns"
> --quiet="false" --recover="reconnect" --re
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: covery_timeout="15mins"
> --registration_backoff_factor="1secs" --revocable_cpu_low_priority="true"
> --sandbox_directory="/mnt/mesos/sandbox" --strict="true"
> --switch_user="false" --systemd_enable_support="true"
> --systemd_runtime_directory="/run/systemd/system" --version="false"
> --work_dir="/mnt/mesos"
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: 2016-08-09
> 10:11:41,800:3738(0x7f71172a1700):ZOO_INFO@check_events@1728: initiated
> connection to server [172.31.19.240:2181]
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.801007
> 3738 slave.cpp:519] Agent resources: gpus(*):1; cpus(*):8; mem(*):14014;
> disk(*):60257; ports(*):[31000-32000]
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.801059
> 3738 slave.cpp:527] Agent attributes: [ ]
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.801077
> 3738 slave.cpp:532] Agent hostname:
> ec2-52-59-113-0.eu-central-1.compute.amazonaws.com
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.802232
> 3772 state.cpp:57] Recovering state from '/mnt/mesos/meta'
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.803539
> 3774 status_update_manager.cpp:200] Recovering status update manager
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.803745
> 3772 containerizer.cpp:522] Recovering containerizer
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.803788
> 3770 docker.cpp:775] Recovering Docker containers
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.808550
> 3777 metadata_manager.cpp:251] Successfully loaded 2 Docker images
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.808688
> 3771 provisioner.cpp:253] Provisioner recovery complete
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: 2016-08-09
> 10:11:41,816:3738(0x7f71172a1700):ZOO_INFO@check_events@1775: session
> establishment complete on server [172.31.19.240:2181],
> sessionId=0x1566a66ab9b0010, negotiated timeout=10000
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.817003
> 3774 group.cpp:349] Group process (group(1)@172.31.23.17:5051) connected to
> ZooKeeper
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.817116
> 3774 group.cpp:837] Syncing group operations: queue size (joins, cancels,
> datas) = (0, 0, 0)
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.817147
> 3774 group.cpp:427] Trying to create path '/mesos' in ZooKeeper
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.818857
> 3774 detector.cpp:152] Detected a new leader: (id='64')
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.819005
> 3776 group.cpp:706] Trying to get '/mesos/json.info_0000000064' in ZooKeeper
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.819855
> 3775 zookeeper.cpp:259] A new leading master ([email protected]:5050)
> is detected
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.899876
> 3777 slave.cpp:4782] Finished recovery
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900573
> 3777 slave.cpp:4815] Garbage collecting old agent
> 524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S2
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900687
> 3773 gc.cpp:55] Scheduling
> '/mnt/mesos/slaves/524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S2' for gc
> 6.99998957623704days in the future
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900729
> 3777 slave.cpp:4815] Garbage collecting old agent
> c9852a23-bc07-422d-8d69-23c167a1924d-S0
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900779
> 3773 gc.cpp:55] Scheduling
> '/mnt/mesos/meta/slaves/524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S2' for gc
> 6.99998957520889days in the future
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900817
> 3777 slave.cpp:4815] Garbage collecting old agent
> 524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S1
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900825
> 3773 gc.cpp:55] Scheduling
> '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S0' for gc
> 6.99998957451259days in the future
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900899
> 3773 gc.cpp:55] Scheduling
> '/mnt/mesos/meta/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S0' for gc
> 6.99998957412148days in the future
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900933
> 3777 slave.cpp:4815] Garbage collecting old agent
> 524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S3
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.900944
> 3773 gc.cpp:55] Scheduling
> '/mnt/mesos/slaves/524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S1' for gc
> 6.99998957332148days in the future
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901010
> 3773 gc.cpp:55] Scheduling
> '/mnt/mesos/meta/slaves/524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S1' for gc
> 6.99998957291556days in the future
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901051
> 3773 gc.cpp:55] Scheduling
> '/mnt/mesos/slaves/524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S3' for gc
> 6.99998957196741days in the future
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901101
> 3773 gc.cpp:55] Scheduling
> '/mnt/mesos/meta/slaves/524105e7-de7a-43f0-8b28-d3ff3e0c4a44-S3' for gc
> 6.99998957156148days in the future
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901201
> 3775 status_update_manager.cpp:174] Pausing sending status updates
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901203
> 3777 slave.cpp:895] New master detected at [email protected]:5050
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901278
> 3777 slave.cpp:916] No credentials provided. Attempting to register without
> authentication
> Aug 9 10:11:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:41.901317
> 3777 slave.cpp:927] Detecting new master
> Aug 9 10:11:42 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:42.267201
> 3774 slave.cpp:1197] Re-registered with master [email protected]:5050
> Aug 9 10:11:42 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:42.267302
> 3771 status_update_manager.cpp:181] Resuming sending status updates
> Aug 9 10:11:42 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:42.267302
> 3774 slave.cpp:1233] Forwarding total oversubscribed resources
> Aug 9 10:11:42 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:42.267577
> 3774 slave.cpp:2526] Updated checkpointed resources from to
> Aug 9 10:11:48 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:48.269104
> 3777 slave.cpp:2218] Asked to shut down framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0002 by [email protected]:5050
> Aug 9 10:11:48 ip-172-31-23-17 mesos-slave[3738]: W0809 10:11:48.269186
> 3777 slave.cpp:2233] Cannot shut down unknown framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0002
> Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.969521
> 3772 slave.cpp:1495] Got assigned task 0 for framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.970026
> 3772 slave.cpp:1614] Launching task 0 for framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.970826
> 3772 slave.cpp:5674] Launching executor 0 of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003 with resources cpus(*):0.1;
> mem(*):32 in work directory
> '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/0/runs/09a77a80-f904-4581-966a-734d24ae96f3'
> Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.971051
> 3772 slave.cpp:1840] Queuing task '0' for executor '0' of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.971212
> 3773 containerizer.cpp:781] Starting container
> '09a77a80-f904-4581-966a-734d24ae96f3' for executor '0' of framework
> 'c9852a23-bc07-422d-8d69-23c167a1924d-0003'
> Aug 9 10:11:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:11:54.973953
> 3774 provisioner.cpp:294] Provisioning image rootfs
> '/mnt/mesos/provisioner/containers/09a77a80-f904-4581-966a-734d24ae96f3/backends/copy/rootfses/ec9bb10e-6726-4872-be2b-4a72dfc0fafc'
> for container 09a77a80-f904-4581-966a-734d24ae96f3
> Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.978344
> 3774 slave.cpp:1495] Got assigned task 1 for framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.978687
> 3774 slave.cpp:1614] Launching task 1 for framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.978976
> 3774 slave.cpp:5674] Launching executor 1 of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003 with resources cpus(*):0.1;
> mem(*):32 in work directory
> '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/1/runs/0cea1dc7-b523-4b68-851a-531ed101de5a'
> Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.979173
> 3774 slave.cpp:1840] Queuing task '1' for executor '1' of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.979387
> 3777 containerizer.cpp:781] Starting container
> '0cea1dc7-b523-4b68-851a-531ed101de5a' for executor '1' of framework
> 'c9852a23-bc07-422d-8d69-23c167a1924d-0003'
> Aug 9 10:12:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:00.981374
> 3776 provisioner.cpp:294] Provisioning image rootfs
> '/mnt/mesos/provisioner/containers/0cea1dc7-b523-4b68-851a-531ed101de5a/backends/copy/rootfses/e3af89fe-c889-45fc-929b-f008b18e5d4e'
> for container 0cea1dc7-b523-4b68-851a-531ed101de5a
> Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.983610
> 3776 slave.cpp:1495] Got assigned task 2 for framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.983947
> 3776 slave.cpp:1614] Launching task 2 for framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.984236
> 3776 slave.cpp:5674] Launching executor 2 of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003 with resources cpus(*):0.1;
> mem(*):32 in work directory
> '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/2/runs/a4c972da-b12d-4b4b-8481-79ca320ddab7'
> Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.984450
> 3776 slave.cpp:1840] Queuing task '2' for executor '2' of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.984778
> 3777 containerizer.cpp:781] Starting container
> 'a4c972da-b12d-4b4b-8481-79ca320ddab7' for executor '2' of framework
> 'c9852a23-bc07-422d-8d69-23c167a1924d-0003'
> Aug 9 10:12:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:06.986330
> 3776 provisioner.cpp:294] Provisioning image rootfs
> '/mnt/mesos/provisioner/containers/a4c972da-b12d-4b4b-8481-79ca320ddab7/backends/copy/rootfses/feaf70a4-df43-45c9-b6be-a654c9210a54'
> for container a4c972da-b12d-4b4b-8481-79ca320ddab7
> Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.996104
> 3771 slave.cpp:1495] Got assigned task 3 for framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.996462
> 3771 slave.cpp:1614] Launching task 3 for framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.997241
> 3771 slave.cpp:5674] Launching executor 3 of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003 with resources cpus(*):0.1;
> mem(*):32 in work directory
> '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/3/runs/d9363f0d-645b-468f-947a-8bf97d15a630'
> Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.997406
> 3771 slave.cpp:1840] Queuing task '3' for executor '3' of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.997640
> 3777 containerizer.cpp:781] Starting container
> 'd9363f0d-645b-468f-947a-8bf97d15a630' for executor '3' of framework
> 'c9852a23-bc07-422d-8d69-23c167a1924d-0003'
> Aug 9 10:12:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:12.999593
> 3777 provisioner.cpp:294] Provisioning image rootfs
> '/mnt/mesos/provisioner/containers/d9363f0d-645b-468f-947a-8bf97d15a630/backends/copy/rootfses/eeb1b066-94b8-4880-a696-090f830cf2d6'
> for container d9363f0d-645b-468f-947a-8bf97d15a630
> Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.003924
> 3775 slave.cpp:1495] Got assigned task 4 for framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.004278
> 3775 slave.cpp:1614] Launching task 4 for framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.005383
> 3775 slave.cpp:5674] Launching executor 4 of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003 with resources cpus(*):0.1;
> mem(*):32 in work directory
> '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/4/runs/58c398e2-38f9-4389-87ba-8696c9a523eb'
> Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.005681
> 3775 slave.cpp:1840] Queuing task '4' for executor '4' of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.005971
> 3772 containerizer.cpp:781] Starting container
> '58c398e2-38f9-4389-87ba-8696c9a523eb' for executor '4' of framework
> 'c9852a23-bc07-422d-8d69-23c167a1924d-0003'
> Aug 9 10:12:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:19.007786
> 3777 provisioner.cpp:294] Provisioning image rootfs
> '/mnt/mesos/provisioner/containers/58c398e2-38f9-4389-87ba-8696c9a523eb/backends/copy/rootfses/78b7bf3a-6f3b-4f8a-b4fa-98a6ea4bf8f5'
> for container 58c398e2-38f9-4389-87ba-8696c9a523eb
> Aug 9 10:12:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:41.801560
> 3777 slave.cpp:4591] Current disk usage 43.28%. Max allowed age:
> 3.270091966898553days
> Aug 9 10:12:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:54.972057
> 3772 slave.cpp:4539] Terminating executor ''0' of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003' because it did not register within
> 1mins
> Aug 9 10:12:54 ip-172-31-23-17 mesos-slave[3738]: I0809 10:12:54.972322
> 3775 containerizer.cpp:1622] Destroying container
> '09a77a80-f904-4581-966a-734d24ae96f3'
> Aug 9 10:13:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:00.979907
> 3777 slave.cpp:4539] Terminating executor ''1' of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003' because it did not register within
> 1mins
> Aug 9 10:13:00 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:00.980058
> 3775 containerizer.cpp:1622] Destroying container
> '0cea1dc7-b523-4b68-851a-531ed101de5a'
> Aug 9 10:13:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:06.985003
> 3777 slave.cpp:4539] Terminating executor ''2' of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003' because it did not register within
> 1mins
> Aug 9 10:13:06 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:06.985255
> 3774 containerizer.cpp:1622] Destroying container
> 'a4c972da-b12d-4b4b-8481-79ca320ddab7'
> Aug 9 10:13:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:12.998438
> 3777 slave.cpp:4539] Terminating executor ''3' of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003' because it did not register within
> 1mins
> Aug 9 10:13:12 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:12.998641
> 3776 containerizer.cpp:1622] Destroying container
> 'd9363f0d-645b-468f-947a-8bf97d15a630'
> Aug 9 10:13:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:19.006058
> 3773 slave.cpp:4539] Terminating executor ''4' of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003' because it did not register within
> 1mins
> Aug 9 10:13:19 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:19.006228
> 3773 containerizer.cpp:1622] Destroying container
> '58c398e2-38f9-4389-87ba-8696c9a523eb'
> Aug 9 10:13:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:13:41.802209
> 3775 slave.cpp:4591] Current disk usage 57.59%. Max allowed age:
> 2.269006805055764days
> Aug 9 10:14:41 ip-172-31-23-17 mesos-slave[3738]: I0809 10:14:41.803022
> 3776 slave.cpp:4591] Current disk usage 68.70%. Max allowed age:
> 1.491204118572176days
> Aug 9 10:15:01 ip-172-31-23-17 mesos-slave[3738]: E0809 10:15:01.516207
> 3775 slave.cpp:3976] Container '58c398e2-38f9-4389-87ba-8696c9a523eb' for
> executor '4' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 failed to
> start: Container is being destroyed during provisioning
> Aug 9 10:15:01 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:01.518277
> 3772 composing.cpp:541] Container '58c398e2-38f9-4389-87ba-8696c9a523eb' is
> already destroyed
> Aug 9 10:15:02 ip-172-31-23-17 mesos-slave[3738]: E0809 10:15:02.291764
> 3775 slave.cpp:3976] Container '09a77a80-f904-4581-966a-734d24ae96f3' for
> executor '0' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 failed to
> start: Container is being destroyed during provisioning
> Aug 9 10:15:02 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:02.291877
> 3775 composing.cpp:541] Container '09a77a80-f904-4581-966a-734d24ae96f3' is
> already destroyed
> Aug 9 10:15:03 ip-172-31-23-17 mesos-slave[3738]: E0809 10:15:03.022406
> 3774 slave.cpp:3976] Container 'd9363f0d-645b-468f-947a-8bf97d15a630' for
> executor '3' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 failed to
> start: Container is being destroyed during provisioning
> Aug 9 10:15:03 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:03.022595
> 3771 composing.cpp:541] Container 'd9363f0d-645b-468f-947a-8bf97d15a630' is
> already destroyed
> Aug 9 10:15:03 ip-172-31-23-17 mesos-slave[3738]: E0809 10:15:03.461519
> 3775 slave.cpp:3976] Container '0cea1dc7-b523-4b68-851a-531ed101de5a' for
> executor '1' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 failed to
> start: Container is being destroyed during provisioning
> Aug 9 10:15:03 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:03.461643
> 3774 composing.cpp:541] Container '0cea1dc7-b523-4b68-851a-531ed101de5a' is
> already destroyed
> Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:04.190901
> 3776 provisioner.cpp:434] Destroying container rootfs at
> '/mnt/mesos/provisioner/containers/58c398e2-38f9-4389-87ba-8696c9a523eb/backends/copy/rootfses/78b7bf3a-6f3b-4f8a-b4fa-98a6ea4bf8f5'
> for container 58c398e2-38f9-4389-87ba-8696c9a523eb
> Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: E0809 10:15:04.190904
> 3777 slave.cpp:3976] Container 'a4c972da-b12d-4b4b-8481-79ca320ddab7' for
> executor '2' of framework c9852a23-bc07-422d-8d69-23c167a1924d-0003 failed to
> start: Container is being destroyed during provisioning
> Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:04.191128
> 3770 composing.cpp:541] Container 'a4c972da-b12d-4b4b-8481-79ca320ddab7' is
> already destroyed
> Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:04.197221
> 3776 provisioner.cpp:434] Destroying container rootfs at
> '/mnt/mesos/provisioner/containers/09a77a80-f904-4581-966a-734d24ae96f3/backends/copy/rootfses/ec9bb10e-6726-4872-be2b-4a72dfc0fafc'
> for container 09a77a80-f904-4581-966a-734d24ae96f3
> Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:04.197410
> 3776 provisioner.cpp:434] Destroying container rootfs at
> '/mnt/mesos/provisioner/containers/d9363f0d-645b-468f-947a-8bf97d15a630/backends/copy/rootfses/eeb1b066-94b8-4880-a696-090f830cf2d6'
> for container d9363f0d-645b-468f-947a-8bf97d15a630
> Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:04.197510
> 3776 provisioner.cpp:434] Destroying container rootfs at
> '/mnt/mesos/provisioner/containers/0cea1dc7-b523-4b68-851a-531ed101de5a/backends/copy/rootfses/e3af89fe-c889-45fc-929b-f008b18e5d4e'
> for container 0cea1dc7-b523-4b68-851a-531ed101de5a
> Aug 9 10:15:04 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:04.197619
> 3776 provisioner.cpp:434] Destroying container rootfs at
> '/mnt/mesos/provisioner/containers/a4c972da-b12d-4b4b-8481-79ca320ddab7/backends/copy/rootfses/feaf70a4-df43-45c9-b6be-a654c9210a54'
> for container a4c972da-b12d-4b4b-8481-79ca320ddab7
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.390928
> 3776 slave.cpp:4082] Executor '3' of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003 has terminated with unknown status
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.395370
> 3776 slave.cpp:3211] Handling status update TASK_FAILED (UUID:
> 543e1e1e-ae47-48ab-ad64-ee82fdd27418) for task 3 of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003 from @0.0.0.0:0
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.402472
> 3776 slave.cpp:4082] Executor '2' of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003 has terminated with unknown status
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.402556
> 3776 slave.cpp:3211] Handling status update TASK_FAILED (UUID:
> 1a54664d-4157-42a3-a97f-7189c137a769) for task 2 of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003 from @0.0.0.0:0
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:07.406177
> 3772 containerizer.cpp:1451] Ignoring update for unknown container:
> d9363f0d-645b-468f-947a-8bf97d15a630
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.407325
> 3777 status_update_manager.cpp:320] Received status update TASK_FAILED (UUID:
> 543e1e1e-ae47-48ab-ad64-ee82fdd27418) for task 3 of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:07.407598
> 3773 containerizer.cpp:1451] Ignoring update for unknown container:
> a4c972da-b12d-4b4b-8481-79ca320ddab7
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.409890
> 3774 slave.cpp:3604] Forwarding the update TASK_FAILED (UUID:
> 543e1e1e-ae47-48ab-ad64-ee82fdd27418) for task 3 of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003 to [email protected]:5050
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.409893
> 3777 status_update_manager.cpp:320] Received status update TASK_FAILED (UUID:
> 1a54664d-4157-42a3-a97f-7189c137a769) for task 2 of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.414727
> 3774 slave.cpp:3604] Forwarding the update TASK_FAILED (UUID:
> 1a54664d-4157-42a3-a97f-7189c137a769) for task 2 of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003 to [email protected]:5050
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.418180
> 3777 status_update_manager.cpp:392] Received status update acknowledgement
> (UUID: 543e1e1e-ae47-48ab-ad64-ee82fdd27418) for task 3 of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.419615
> 3777 status_update_manager.cpp:392] Received status update acknowledgement
> (UUID: 1a54664d-4157-42a3-a97f-7189c137a769) for task 2 of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.419636
> 3774 slave.cpp:4193] Cleaning up executor '3' of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.419924
> 3772 gc.cpp:55] Scheduling
> '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/3/runs/d9363f0d-645b-468f-947a-8bf97d15a630'
> for gc 6.9999951410637days in the future
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.420703
> 3773 gc.cpp:55] Scheduling
> '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/3'
> for gc 6.99999513154074days in the future
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.420722
> 3774 slave.cpp:4193] Cleaning up executor '2' of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.420897
> 3774 gc.cpp:55] Scheduling
> '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/2/runs/a4c972da-b12d-4b4b-8481-79ca320ddab7'
> for gc 6.99999512931852days in the future
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.420940
> 3774 gc.cpp:55] Scheduling
> '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/2'
> for gc 6.99999512869333days in the future
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.490155
> 3772 slave.cpp:4082] Executor '4' of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003 has terminated with unknown status
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.490289
> 3772 slave.cpp:3211] Handling status update TASK_FAILED (UUID:
> 9c5528ec-0725-42f2-a2e2-465d3c73b8d5) for task 4 of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003 from @0.0.0.0:0
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.490414
> 3772 slave.cpp:4082] Executor '0' of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003 has terminated with unknown status
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.490489
> 3772 slave.cpp:3211] Handling status update TASK_FAILED (UUID:
> 3f801210-b029-427f-af3f-e51b1dfa5cc8) for task 0 of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003 from @0.0.0.0:0
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:07.490687
> 3773 containerizer.cpp:1451] Ignoring update for unknown container:
> 58c398e2-38f9-4389-87ba-8696c9a523eb
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:07.490774
> 3773 containerizer.cpp:1451] Ignoring update for unknown container:
> 09a77a80-f904-4581-966a-734d24ae96f3
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.490867
> 3773 status_update_manager.cpp:320] Received status update TASK_FAILED (UUID:
> 9c5528ec-0725-42f2-a2e2-465d3c73b8d5) for task 4 of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.491082
> 3773 status_update_manager.cpp:320] Received status update TASK_FAILED (UUID:
> 3f801210-b029-427f-af3f-e51b1dfa5cc8) for task 0 of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.491082
> 3772 slave.cpp:3604] Forwarding the update TASK_FAILED (UUID:
> 9c5528ec-0725-42f2-a2e2-465d3c73b8d5) for task 4 of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003 to [email protected]:5050
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.491303
> 3774 slave.cpp:3604] Forwarding the update TASK_FAILED (UUID:
> 3f801210-b029-427f-af3f-e51b1dfa5cc8) for task 0 of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003 to [email protected]:5050
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494438
> 3775 status_update_manager.cpp:392] Received status update acknowledgement
> (UUID: 9c5528ec-0725-42f2-a2e2-465d3c73b8d5) for task 4 of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494572
> 3775 slave.cpp:4193] Cleaning up executor '4' of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494757
> 3773 gc.cpp:55] Scheduling
> '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/4/runs/58c398e2-38f9-4389-87ba-8696c9a523eb'
> for gc 6.99999427436444days in the future
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494817
> 3775 status_update_manager.cpp:392] Received status update acknowledgement
> (UUID: 3f801210-b029-427f-af3f-e51b1dfa5cc8) for task 0 of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494937
> 3775 slave.cpp:4193] Cleaning up executor '0' of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.494823
> 3773 gc.cpp:55] Scheduling
> '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/4'
> for gc 6.99999427361185days in the future
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.495107
> 3773 gc.cpp:55] Scheduling
> '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/0/runs/09a77a80-f904-4581-966a-734d24ae96f3'
> for gc 6.99999427055407days in the future
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.495149
> 3773 gc.cpp:55] Scheduling
> '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/0'
> for gc 6.99999427024296days in the future
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.591338
> 3775 slave.cpp:4082] Executor '1' of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003 has terminated with unknown status
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.591469
> 3775 slave.cpp:3211] Handling status update TASK_FAILED (UUID:
> 85b7c18a-c59c-4449-827c-a3bfaa2a7de8) for task 1 of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003 from @0.0.0.0:0
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: W0809 10:15:07.591841
> 3776 containerizer.cpp:1451] Ignoring update for unknown container:
> 0cea1dc7-b523-4b68-851a-531ed101de5a
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.591974
> 3770 status_update_manager.cpp:320] Received status update TASK_FAILED (UUID:
> 85b7c18a-c59c-4449-827c-a3bfaa2a7de8) for task 1 of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.592157
> 3772 slave.cpp:3604] Forwarding the update TASK_FAILED (UUID:
> 85b7c18a-c59c-4449-827c-a3bfaa2a7de8) for task 1 of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003 to [email protected]:5050
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594518
> 3771 status_update_manager.cpp:392] Received status update acknowledgement
> (UUID: 85b7c18a-c59c-4449-827c-a3bfaa2a7de8) for task 1 of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594636
> 3771 slave.cpp:4193] Cleaning up executor '1' of framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594792
> 3770 gc.cpp:55] Scheduling
> '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/1/runs/0cea1dc7-b523-4b68-851a-531ed101de5a'
> for gc 6.99999311635556days in the future
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594820
> 3771 slave.cpp:4281] Cleaning up framework
> c9852a23-bc07-422d-8d69-23c167a1924d-0003
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594861
> 3770 gc.cpp:55] Scheduling
> '/mnt/mesos/slaves/c9852a23-bc07-422d-8d69-23c167a1924d-S1/frameworks/c9852a23-bc07-422d-8d69-23c167a1924d-0003/executors/1'
> for gc 6.99999311568296days in the future
> Aug 9 10:15:07 ip-172-31-23-17 mesos-slave[3738]: I0809 10:15:07.594897
> 3770 status_update_manager.cpp:282] Closing status update streams for
> framework c9852a23-bc07-422d-8d69-23c167a1924d-0003
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
