[
https://issues.apache.org/jira/browse/MESOS-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15157704#comment-15157704
]
Alexander Rojas commented on MESOS-4047:
----------------------------------------
Reproduced again with following message (CentOS 6.7):
{noformat}
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from MemoryPressureMesosTest
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.000394345 s, 2.7 GB/s
[ RUN ] MemoryPressureMesosTest.CGROUPS_ROOT_SlaveRecovery
I0222 09:32:20.622694 20868 leveldb.cpp:174] Opened db in 5.153509ms
I0222 09:32:20.624688 20868 leveldb.cpp:181] Compacted db in 1.914323ms
I0222 09:32:20.624778 20868 leveldb.cpp:196] Created db iterator in 24549ns
I0222 09:32:20.624795 20868 leveldb.cpp:202] Seeked to beginning of db in 2610ns
I0222 09:32:20.624804 20868 leveldb.cpp:271] Iterated through 0 keys in the db
in 323ns
I0222 09:32:20.624874 20868 replica.cpp:779] Replica recovered with log
positions 0 -> 0 with 1 holes and 0 unlearned
I0222 09:32:20.625977 20888 recover.cpp:447] Starting replica recovery
I0222 09:32:20.626901 20888 recover.cpp:473] Replica is in EMPTY status
I0222 09:32:20.634701 20889 replica.cpp:673] Replica in EMPTY status received a
broadcasted recover request from (11193)@127.0.0.1:54769
I0222 09:32:20.634953 20888 master.cpp:376] Master
17b7da64-0c4d-4e46-ae1f-2b356dc5f266 (localhost) started on 127.0.0.1:54769
I0222 09:32:20.634986 20888 master.cpp:378] Flags at startup: --acls=""
--allocation_interval="1secs" --allocator="HierarchicalDRF"
--authenticate="true" --authenticate_http="true" --authenticate_slaves="true"
--authenticators="crammd5" --authorizers="local"
--credentials="/tmp/0rXncF/credentials" --framework_sorter="drf" --help="false"
--hostname_lookup="true" --http_authenticators="basic"
--initialize_driver_logging="true" --log_auto_initialize="true"
--logbufsecs="0" --logging_level="INFO" --max_completed_frameworks="50"
--max_completed_tasks_per_framework="1000" --max_slave_ping_timeouts="5"
--quiet="false" --recovery_slave_removal_limit="100%"
--registry="replicated_log" --registry_fetch_timeout="1mins"
--registry_store_timeout="100secs" --registry_strict="true"
--root_submissions="true" --slave_ping_timeout="15secs"
--slave_reregister_timeout="10mins" --user_sorter="drf" --version="false"
--webui_dir="/usr/local/share/mesos/webui" --work_dir="/tmp/0rXncF/master"
--zk_session_timeout="10secs"
W0222 09:32:20.635417 20888 master.cpp:381]
**************************************************
Master bound to loopback interface! Cannot communicate with remote schedulers
or slaves. You might want to set '--ip' flag to a routable IP address.
**************************************************
I0222 09:32:20.635587 20888 master.cpp:423] Master only allowing authenticated
frameworks to register
I0222 09:32:20.635601 20888 master.cpp:428] Master only allowing authenticated
slaves to register
I0222 09:32:20.635622 20888 credentials.hpp:35] Loading credentials for
authentication from '/tmp/0rXncF/credentials'
I0222 09:32:20.636018 20888 master.cpp:468] Using default 'crammd5'
authenticator
I0222 09:32:20.636190 20888 master.cpp:537] Using default 'basic' HTTP
authenticator
I0222 09:32:20.636174 20887 recover.cpp:193] Received a recover response from a
replica in EMPTY status
I0222 09:32:20.636425 20888 master.cpp:571] Authorization enabled
I0222 09:32:20.637810 20885 recover.cpp:564] Updating replica status to STARTING
I0222 09:32:20.640805 20887 leveldb.cpp:304] Persisting metadata (8 bytes) to
leveldb took 2.741248ms
I0222 09:32:20.640964 20887 replica.cpp:320] Persisted replica status to
STARTING
I0222 09:32:20.641525 20885 recover.cpp:473] Replica is in STARTING status
I0222 09:32:20.642133 20888 master.cpp:1712] The newly elected leader is
[email protected]:54769 with id 17b7da64-0c4d-4e46-ae1f-2b356dc5f266
I0222 09:32:20.642236 20888 master.cpp:1725] Elected as the leading master!
I0222 09:32:20.642253 20888 master.cpp:1470] Recovering from registrar
I0222 09:32:20.642496 20885 registrar.cpp:307] Recovering registrar
I0222 09:32:20.643162 20889 replica.cpp:673] Replica in STARTING status
received a broadcasted recover request from (11195)@127.0.0.1:54769
I0222 09:32:20.643590 20885 recover.cpp:193] Received a recover response from a
replica in STARTING status
I0222 09:32:20.644120 20887 recover.cpp:564] Updating replica status to VOTING
I0222 09:32:20.646817 20889 leveldb.cpp:304] Persisting metadata (8 bytes) to
leveldb took 1.190281ms
I0222 09:32:20.646870 20889 replica.cpp:320] Persisted replica status to VOTING
I0222 09:32:20.647094 20885 recover.cpp:578] Successfully joined the Paxos group
I0222 09:32:20.647337 20885 recover.cpp:462] Recover process terminated
I0222 09:32:20.647781 20887 log.cpp:659] Attempting to start the writer
I0222 09:32:20.648854 20890 replica.cpp:493] Replica received implicit promise
request from (11196)@127.0.0.1:54769 with proposal 1
I0222 09:32:20.650074 20890 leveldb.cpp:304] Persisting metadata (8 bytes) to
leveldb took 1.17538ms
I0222 09:32:20.650133 20890 replica.cpp:342] Persisted promised to 1
I0222 09:32:20.650879 20883 coordinator.cpp:238] Coordinator attempting to fill
missing positions
I0222 09:32:20.652029 20889 replica.cpp:388] Replica received explicit promise
request from (11197)@127.0.0.1:54769 for position 0 with proposal 2
I0222 09:32:20.653084 20889 leveldb.cpp:341] Persisting action (8 bytes) to
leveldb took 1.006874ms
I0222 09:32:20.653152 20889 replica.cpp:712] Persisted action at 0
I0222 09:32:20.654109 20888 replica.cpp:537] Replica received write request for
position 0 from (11198)@127.0.0.1:54769
I0222 09:32:20.654157 20888 leveldb.cpp:436] Reading position from leveldb took
19941ns
I0222 09:32:20.655339 20888 leveldb.cpp:341] Persisting action (14 bytes) to
leveldb took 914288ns
I0222 09:32:20.655375 20888 replica.cpp:712] Persisted action at 0
I0222 09:32:20.655944 20890 replica.cpp:691] Replica received learned notice
for position 0 from @0.0.0.0:0
I0222 09:32:20.657099 20890 leveldb.cpp:341] Persisting action (16 bytes) to
leveldb took 1.124148ms
I0222 09:32:20.657166 20890 replica.cpp:712] Persisted action at 0
I0222 09:32:20.657232 20890 replica.cpp:697] Replica learned NOP action at
position 0
I0222 09:32:20.657861 20890 log.cpp:675] Writer started with ending position 0
I0222 09:32:20.658861 20886 leveldb.cpp:436] Reading position from leveldb took
30090ns
I0222 09:32:20.659822 20888 registrar.cpp:340] Successfully fetched the
registry (0B) in 17.27488ms
I0222 09:32:20.660028 20888 registrar.cpp:439] Applied 1 operations in
123880ns; attempting to update the 'registry'
I0222 09:32:20.660781 20886 log.cpp:683] Attempting to append 160 bytes to the
log
I0222 09:32:20.661144 20885 coordinator.cpp:348] Coordinator attempting to
write APPEND action at position 1
I0222 09:32:20.663028 20890 replica.cpp:537] Replica received write request for
position 1 from (11199)@127.0.0.1:54769
I0222 09:32:20.666872 20890 leveldb.cpp:341] Persisting action (179 bytes) to
leveldb took 3.746467ms
I0222 09:32:20.666934 20890 replica.cpp:712] Persisted action at 1
I0222 09:32:20.667937 20890 replica.cpp:691] Replica received learned notice
for position 1 from @0.0.0.0:0
I0222 09:32:20.670030 20890 leveldb.cpp:341] Persisting action (181 bytes) to
leveldb took 2.029339ms
I0222 09:32:20.670084 20890 replica.cpp:712] Persisted action at 1
I0222 09:32:20.670105 20890 replica.cpp:697] Replica learned APPEND action at
position 1
I0222 09:32:20.671183 20889 registrar.cpp:484] Successfully updated the
'registry' in 11.046144ms
I0222 09:32:20.671344 20889 registrar.cpp:370] Successfully recovered registrar
I0222 09:32:20.671454 20890 log.cpp:702] Attempting to truncate the log to 1
I0222 09:32:20.672252 20889 coordinator.cpp:348] Coordinator attempting to
write TRUNCATE action at position 2
I0222 09:32:20.672366 20885 master.cpp:1522] Recovered 0 slaves from the
Registry (122B) ; allowing 10mins for slaves to re-register
I0222 09:32:20.673305 20884 replica.cpp:537] Replica received write request for
position 2 from (11200)@127.0.0.1:54769
I0222 09:32:20.675009 20884 leveldb.cpp:341] Persisting action (16 bytes) to
leveldb took 1.632734ms
I0222 09:32:20.675067 20884 replica.cpp:712] Persisted action at 2
I0222 09:32:20.675851 20883 replica.cpp:691] Replica received learned notice
for position 2 from @0.0.0.0:0
I0222 09:32:20.677466 20883 leveldb.cpp:341] Persisting action (18 bytes) to
leveldb took 1.548407ms
I0222 09:32:20.677618 20883 leveldb.cpp:399] Deleting ~1 keys from leveldb took
69924ns
I0222 09:32:20.677649 20883 replica.cpp:712] Persisted action at 2
I0222 09:32:20.677686 20883 replica.cpp:697] Replica learned TRUNCATE action at
position 2
I0222 09:32:20.685191 20868 containerizer.cpp:149] Using isolation:
cgroups/mem,filesystem/posix
I0222 09:32:20.692539 20868 linux_launcher.cpp:101] Using /cgroup/freezer as
the freezer hierarchy for the Linux launcher
I0222 09:32:20.717495 20889 slave.cpp:193] Slave started on 247)@127.0.0.1:54769
I0222 09:32:20.717723 20889 slave.cpp:194] Flags at startup:
--appc_simple_discovery_uri_prefix="http://";
--appc_store_dir="/tmp/mesos/store/appc" --authenticatee="crammd5"
--cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false"
--cgroups_hierarchy="/cgroup" --cgroups_limit_swap="false"
--cgroups_root="mesos_test_7adedc17-08f0-4c8b-9974-66c009b1da18"
--container_disk_watch_interval="15secs" --containerizers="mesos"
--credential="/tmp/MemoryPressureMesosTest_CGROUPS_ROOT_SlaveRecovery_YcfvOI/credential"
--default_role="*" --disk_watch_interval="1mins" --docker="docker"
--docker_auth_server="https://auth.docker.io"; --docker_kill_orphans="true"
--docker_puller_timeout="60" --docker_registry="https://registry-1.docker.io";
--docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock"
--docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker"
--enforce_container_disk_quota="false" --executor_registration_timeout="1mins"
--executor_shutdown_grace_period="5secs"
--fetcher_cache_dir="/tmp/MemoryPressureMesosTest_CGROUPS_ROOT_SlaveRecovery_YcfvOI/fetch"
--fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks"
--gc_disk_headroom="0.1" --hadoop_home="" --help="false"
--hostname_lookup="true" --image_provisioner_backend="copy"
--initialize_driver_logging="true" --isolation="cgroups/mem"
--launcher_dir="/home/alexander/workspace/mesos/build/src" --logbufsecs="0"
--logging_level="INFO" --oversubscribed_resources_interval="15secs"
--perf_duration="10secs" --perf_interval="1mins"
--qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect"
--recovery_timeout="15mins" --registration_backoff_factor="10ms"
--resources="cpus:2;mem:1024;disk:1024;ports:[31000-32000]"
--revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox"
--strict="true" --switch_user="true" --systemd_enable_support="true"
--systemd_runtime_directory="/run/systemd/system" --version="false"
--work_dir="/tmp/MemoryPressureMesosTest_CGROUPS_ROOT_SlaveRecovery_YcfvOI"
W0222 09:32:20.718128 20889 slave.cpp:197]
**************************************************
Slave bound to loopback interface! Cannot communicate with remote master(s).
You might want to set '--ip' flag to a routable IP address.
**************************************************
I0222 09:32:20.718827 20889 credentials.hpp:83] Loading credential for
authentication from
'/tmp/MemoryPressureMesosTest_CGROUPS_ROOT_SlaveRecovery_YcfvOI/credential'
I0222 09:32:20.719223 20889 slave.cpp:324] Slave using credential for:
test-principal
W0222 09:32:20.719915 20868 sched.cpp:1642]
**************************************************
Scheduler driver bound to loopback interface! Cannot communicate with remote
master(s). You might want to set 'LIBPROCESS_IP' environment variable to use a
routable IP address.
**************************************************
I0222 09:32:20.720688 20889 slave.cpp:464] Slave resources: cpus(*):2;
mem(*):1024; disk(*):1024; ports(*):[31000-32000]
I0222 09:32:20.720878 20889 slave.cpp:472] Slave attributes: [ ]
I0222 09:32:20.720979 20889 slave.cpp:477] Slave hostname: localhost
I0222 09:32:20.721812 20868 sched.cpp:222] Version: 0.28.0
I0222 09:32:20.722302 20889 state.cpp:58] Recovering state from
'/tmp/MemoryPressureMesosTest_CGROUPS_ROOT_SlaveRecovery_YcfvOI/meta'
I0222 09:32:20.722332 20886 sched.cpp:326] New master detected at
[email protected]:54769
I0222 09:32:20.725953 20885 status_update_manager.cpp:200] Recovering status
update manager
I0222 09:32:20.726177 20886 sched.cpp:382] Authenticating with master
[email protected]:54769
I0222 09:32:20.726552 20886 sched.cpp:389] Using default CRAM-MD5 authenticatee
I0222 09:32:20.726557 20889 containerizer.cpp:407] Recovering containerizer
I0222 09:32:20.727334 20884 authenticatee.cpp:121] Creating new client SASL
connection
I0222 09:32:20.727854 20886 master.cpp:5526] Authenticating
[email protected]:54769
I0222 09:32:20.728514 20889 authenticator.cpp:98] Creating new server SASL
connection
I0222 09:32:20.728842 20889 authenticatee.cpp:212] Received SASL authentication
mechanisms: CRAM-MD5
I0222 09:32:20.728987 20889 authenticatee.cpp:238] Attempting to authenticate
with mechanism 'CRAM-MD5'
I0222 09:32:20.729140 20889 authenticator.cpp:203] Received SASL authentication
start
I0222 09:32:20.729305 20889 authenticator.cpp:325] Authentication requires more
steps
I0222 09:32:20.729557 20889 authenticatee.cpp:258] Received SASL authentication
step
I0222 09:32:20.729763 20889 authenticator.cpp:231] Received SASL authentication
step
I0222 09:32:20.730895 20889 authenticator.cpp:317] Authentication success
I0222 09:32:20.731767 20889 master.cpp:5556] Successfully authenticated
principal 'test-principal' at
[email protected]:54769
I0222 09:32:20.731767 20885 authenticatee.cpp:298] Authentication success
I0222 09:32:20.735591 20885 sched.cpp:471] Successfully authenticated with
master [email protected]:54769
I0222 09:32:20.736048 20890 provisioner.cpp:245] Provisioner recovery complete
I0222 09:32:20.739279 20889 slave.cpp:4565] Finished recovery
I0222 09:32:20.739347 20887 master.cpp:2280] Received SUBSCRIBE call for
framework 'default' at
[email protected]:54769
I0222 09:32:20.739572 20887 master.cpp:1751] Authorizing framework principal
'test-principal' to receive offers for role '*'
I0222 09:32:20.740470 20889 master.cpp:2351] Subscribing framework default with
checkpointing enabled and capabilities [ ]
I0222 09:32:20.740790 20887 slave.cpp:796] New master detected at
[email protected]:54769
I0222 09:32:20.741118 20885 hierarchical.cpp:265] Added framework
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000
I0222 09:32:20.741659 20883 sched.cpp:703] Framework registered with
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000
I0222 09:32:20.740885 20886 status_update_manager.cpp:174] Pausing sending
status updates
I0222 09:32:20.741971 20887 slave.cpp:859] Authenticating with master
[email protected]:54769
I0222 09:32:20.742007 20887 slave.cpp:864] Using default CRAM-MD5 authenticatee
I0222 09:32:20.742254 20887 slave.cpp:832] Detecting new master
I0222 09:32:20.742301 20889 authenticatee.cpp:121] Creating new client SASL
connection
I0222 09:32:20.742700 20885 master.cpp:5526] Authenticating
slave(247)@127.0.0.1:54769
I0222 09:32:20.743103 20884 authenticator.cpp:98] Creating new server SASL
connection
I0222 09:32:20.743357 20884 authenticatee.cpp:212] Received SASL authentication
mechanisms: CRAM-MD5
I0222 09:32:20.743404 20884 authenticatee.cpp:238] Attempting to authenticate
with mechanism 'CRAM-MD5'
I0222 09:32:20.743563 20884 authenticator.cpp:203] Received SASL authentication
start
I0222 09:32:20.743628 20884 authenticator.cpp:325] Authentication requires more
steps
I0222 09:32:20.743881 20886 authenticatee.cpp:258] Received SASL authentication
step
I0222 09:32:20.743976 20886 authenticator.cpp:231] Received SASL authentication
step
I0222 09:32:20.744055 20886 authenticator.cpp:317] Authentication success
I0222 09:32:20.744320 20886 authenticatee.cpp:298] Authentication success
I0222 09:32:20.744364 20883 master.cpp:5556] Successfully authenticated
principal 'test-principal' at slave(247)@127.0.0.1:54769
I0222 09:32:20.744813 20883 slave.cpp:927] Successfully authenticated with
master [email protected]:54769
I0222 09:32:20.745275 20889 master.cpp:4240] Registering slave at
slave(247)@127.0.0.1:54769 (localhost) with id
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0
I0222 09:32:20.745687 20883 registrar.cpp:439] Applied 1 operations in 46410ns;
attempting to update the 'registry'
I0222 09:32:20.746387 20883 log.cpp:683] Attempting to append 327 bytes to the
log
I0222 09:32:20.746670 20889 coordinator.cpp:348] Coordinator attempting to
write APPEND action at position 3
I0222 09:32:20.747566 20886 replica.cpp:537] Replica received write request for
position 3 from (11220)@127.0.0.1:54769
I0222 09:32:20.749735 20886 leveldb.cpp:341] Persisting action (346 bytes) to
leveldb took 2.033429ms
I0222 09:32:20.749877 20886 replica.cpp:712] Persisted action at 3
I0222 09:32:20.751174 20890 replica.cpp:691] Replica received learned notice
for position 3 from @0.0.0.0:0
I0222 09:32:20.752976 20883 master.cpp:4228] Ignoring register slave message
from slave(247)@127.0.0.1:54769 (localhost) as admission is already in progress
I0222 09:32:20.753378 20890 leveldb.cpp:341] Persisting action (348 bytes) to
leveldb took 2.085441ms
I0222 09:32:20.753435 20890 replica.cpp:712] Persisted action at 3
I0222 09:32:20.753473 20890 replica.cpp:697] Replica learned APPEND action at
position 3
I0222 09:32:20.755097 20888 registrar.cpp:484] Successfully updated the
'registry' in 9.328128ms
I0222 09:32:20.755506 20884 log.cpp:702] Attempting to truncate the log to 3
I0222 09:32:20.755728 20888 coordinator.cpp:348] Coordinator attempting to
write TRUNCATE action at position 4
I0222 09:32:20.756162 20886 master.cpp:4308] Registered slave
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0 at slave(247)@127.0.0.1:54769
(localhost) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000]
I0222 09:32:20.756397 20883 hierarchical.cpp:473] Added slave
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0 (localhost) with cpus(*):2;
mem(*):1024; disk(*):1024; ports(*):[31000-32000] (allocated: )
I0222 09:32:20.756494 20889 slave.cpp:971] Registered with master
[email protected]:54769; given slave ID 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0
I0222 09:32:20.756803 20888 status_update_manager.cpp:181] Resuming sending
status updates
I0222 09:32:20.757164 20889 slave.cpp:1030] Forwarding total oversubscribed
resources
I0222 09:32:20.757809 20883 replica.cpp:537] Replica received write request for
position 4 from (11221)@127.0.0.1:54769
I0222 09:32:20.757522 20888 master.cpp:5355] Sending 1 offers to framework
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 (default) at
[email protected]:54769
I0222 09:32:20.758471 20888 master.cpp:4649] Received update of slave
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0 at slave(247)@127.0.0.1:54769
(localhost) with total oversubscribed resources
I0222 09:32:20.759210 20883 leveldb.cpp:341] Persisting action (16 bytes) to
leveldb took 1.362326ms
I0222 09:32:20.759312 20883 replica.cpp:712] Persisted action at 4
I0222 09:32:20.760362 20888 hierarchical.cpp:531] Slave
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0 (localhost) updated with oversubscribed
resources (total: cpus(*):2; mem(*):1024; disk(*):1024;
ports(*):[31000-32000], allocated: cpus(*):2; mem(*):1024; disk(*):1024;
ports(*):[31000-32000])
I0222 09:32:20.760388 20889 replica.cpp:691] Replica received learned notice
for position 4 from @0.0.0.0:0
I0222 09:32:20.762329 20888 master.cpp:3138] Processing ACCEPT call for offers:
[ 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-O0 ] on slave
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0 at slave(247)@127.0.0.1:54769
(localhost) for framework 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 (default)
at [email protected]:54769
I0222 09:32:20.762415 20888 master.cpp:2825] Authorizing framework principal
'test-principal' to launch task 24458ae6-dde3-40e0-a4f7-84a901bd9f7d as user
'root'
I0222 09:32:20.762817 20889 leveldb.cpp:341] Persisting action (18 bytes) to
leveldb took 2.201003ms
I0222 09:32:20.764372 20889 leveldb.cpp:399] Deleting ~2 keys from leveldb took
1.136088ms
I0222 09:32:20.764535 20889 replica.cpp:712] Persisted action at 4
I0222 09:32:20.764650 20889 replica.cpp:697] Replica learned TRUNCATE action at
position 4
I0222 09:32:20.765740 20884 master.hpp:176] Adding task
24458ae6-dde3-40e0-a4f7-84a901bd9f7d with resources cpus(*):1; mem(*):256;
disk(*):1024 on slave 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0 (localhost)
I0222 09:32:20.766749 20884 master.cpp:3623] Launching task
24458ae6-dde3-40e0-a4f7-84a901bd9f7d of framework
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 (default) at
[email protected]:54769 with resources
cpus(*):1; mem(*):256; disk(*):1024 on slave
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0 at slave(247)@127.0.0.1:54769
(localhost)
I0222 09:32:20.768129 20888 slave.cpp:1361] Got assigned task
24458ae6-dde3-40e0-a4f7-84a901bd9f7d for framework
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000
I0222 09:32:20.769053 20888 slave.cpp:1480] Launching task
24458ae6-dde3-40e0-a4f7-84a901bd9f7d for framework
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000
I0222 09:32:20.769868 20888 paths.cpp:474] Trying to chown
'/tmp/MemoryPressureMesosTest_CGROUPS_ROOT_SlaveRecovery_YcfvOI/slaves/17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0/frameworks/17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000/executors/24458ae6-dde3-40e0-a4f7-84a901bd9f7d/runs/0e8d2278-bb2b-4480-bfb1-39b47abcb707'
to user 'root'
I0222 09:32:20.774318 20888 slave.cpp:5367] Launching executor
24458ae6-dde3-40e0-a4f7-84a901bd9f7d of framework
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 with resources cpus(*):0.1; mem(*):32
in work directory
'/tmp/MemoryPressureMesosTest_CGROUPS_ROOT_SlaveRecovery_YcfvOI/slaves/17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0/frameworks/17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000/executors/24458ae6-dde3-40e0-a4f7-84a901bd9f7d/runs/0e8d2278-bb2b-4480-bfb1-39b47abcb707'
I0222 09:32:20.775251 20890 containerizer.cpp:666] Starting container
'0e8d2278-bb2b-4480-bfb1-39b47abcb707' for executor
'24458ae6-dde3-40e0-a4f7-84a901bd9f7d' of framework
'17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000'
I0222 09:32:20.775647 20888 slave.cpp:1698] Queuing task
'24458ae6-dde3-40e0-a4f7-84a901bd9f7d' for executor
'24458ae6-dde3-40e0-a4f7-84a901bd9f7d' of framework
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000
I0222 09:32:20.779909 20883 mem.cpp:602] Started listening for OOM events for
container 0e8d2278-bb2b-4480-bfb1-39b47abcb707
I0222 09:32:20.780675 20883 mem.cpp:722] Started listening on low memory
pressure events for container 0e8d2278-bb2b-4480-bfb1-39b47abcb707
I0222 09:32:20.781409 20883 mem.cpp:722] Started listening on medium memory
pressure events for container 0e8d2278-bb2b-4480-bfb1-39b47abcb707
I0222 09:32:20.781992 20883 mem.cpp:722] Started listening on critical memory
pressure events for container 0e8d2278-bb2b-4480-bfb1-39b47abcb707
I0222 09:32:20.782661 20883 mem.cpp:353] Updated 'memory.soft_limit_in_bytes'
to 288MB for container 0e8d2278-bb2b-4480-bfb1-39b47abcb707
I0222 09:32:20.783485 20883 mem.cpp:388] Updated 'memory.limit_in_bytes' to
288MB for container 0e8d2278-bb2b-4480-bfb1-39b47abcb707
I0222 09:32:20.785579 20890 linux_launcher.cpp:304] Cloning child process with
flags =
I0222 09:32:20.788130 20890 containerizer.cpp:1104] Checkpointing executor's
forked pid 26589 to
'/tmp/MemoryPressureMesosTest_CGROUPS_ROOT_SlaveRecovery_YcfvOI/meta/slaves/17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0/frameworks/17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000/executors/24458ae6-dde3-40e0-a4f7-84a901bd9f7d/runs/0e8d2278-bb2b-4480-bfb1-39b47abcb707/pids/forked.pid'
I0222 09:32:20.911543 26589 exec.cpp:143] Version: 0.28.0
I0222 09:32:20.920542 20884 slave.cpp:2643] Got registration for executor
'24458ae6-dde3-40e0-a4f7-84a901bd9f7d' of framework
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 from executor(1)@127.0.0.1:49224
I0222 09:32:20.923596 20884 mem.cpp:353] Updated 'memory.soft_limit_in_bytes'
to 288MB for container 0e8d2278-bb2b-4480-bfb1-39b47abcb707
I0222 09:32:20.923749 26619 exec.cpp:217] Executor registered on slave
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0
I0222 09:32:20.924268 20886 slave.cpp:1863] Sending queued task
'24458ae6-dde3-40e0-a4f7-84a901bd9f7d' to executor
'24458ae6-dde3-40e0-a4f7-84a901bd9f7d' of framework
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 at executor(1)@127.0.0.1:49224
Registered executor on localhost
Starting task 24458ae6-dde3-40e0-a4f7-84a901bd9f7d
sh -c 'while true; do dd count=512 bs=1M if=/dev/zero of=./temp; done'
Forked command at 26627
I0222 09:32:20.929365 20884 slave.cpp:3002] Handling status update TASK_RUNNING
(UUID: e57e1d63-60c7-4132-812e-4d980eeb3039) for task
24458ae6-dde3-40e0-a4f7-84a901bd9f7d of framework
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 from executor(1)@127.0.0.1:49224
I0222 09:32:20.930857 20886 status_update_manager.cpp:320] Received status
update TASK_RUNNING (UUID: e57e1d63-60c7-4132-812e-4d980eeb3039) for task
24458ae6-dde3-40e0-a4f7-84a901bd9f7d of framework
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000
I0222 09:32:20.931435 20886 status_update_manager.cpp:824] Checkpointing UPDATE
for status update TASK_RUNNING (UUID: e57e1d63-60c7-4132-812e-4d980eeb3039) for
task 24458ae6-dde3-40e0-a4f7-84a901bd9f7d of framework
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000
I0222 09:32:20.935792 20887 slave.cpp:3400] Forwarding the update TASK_RUNNING
(UUID: e57e1d63-60c7-4132-812e-4d980eeb3039) for task
24458ae6-dde3-40e0-a4f7-84a901bd9f7d of framework
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 to [email protected]:54769
I0222 09:32:20.936246 20887 slave.cpp:3310] Sending acknowledgement for status
update TASK_RUNNING (UUID: e57e1d63-60c7-4132-812e-4d980eeb3039) for task
24458ae6-dde3-40e0-a4f7-84a901bd9f7d of framework
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 to executor(1)@127.0.0.1:49224
I0222 09:32:20.936532 20886 master.cpp:4794] Status update TASK_RUNNING (UUID:
e57e1d63-60c7-4132-812e-4d980eeb3039) for task
24458ae6-dde3-40e0-a4f7-84a901bd9f7d of framework
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 from slave
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0 at slave(247)@127.0.0.1:54769
(localhost)
I0222 09:32:20.936666 20886 master.cpp:4842] Forwarding status update
TASK_RUNNING (UUID: e57e1d63-60c7-4132-812e-4d980eeb3039) for task
24458ae6-dde3-40e0-a4f7-84a901bd9f7d of framework
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000
I0222 09:32:20.936980 20886 master.cpp:6450] Updating the state of task
24458ae6-dde3-40e0-a4f7-84a901bd9f7d of framework
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 (latest state: TASK_RUNNING, status
update state: TASK_RUNNING)
I0222 09:32:20.938391 20885 master.cpp:3952] Processing ACKNOWLEDGE call
e57e1d63-60c7-4132-812e-4d980eeb3039 for task
24458ae6-dde3-40e0-a4f7-84a901bd9f7d of framework
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 (default) at
[email protected]:54769 on slave
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0
I0222 09:32:20.939096 20883 status_update_manager.cpp:392] Received status
update acknowledgement (UUID: e57e1d63-60c7-4132-812e-4d980eeb3039) for task
24458ae6-dde3-40e0-a4f7-84a901bd9f7d of framework
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000
I0222 09:32:20.939386 20868 slave.cpp:668] Slave terminating
I0222 09:32:20.939414 20883 status_update_manager.cpp:824] Checkpointing ACK
for status update TASK_RUNNING (UUID: e57e1d63-60c7-4132-812e-4d980eeb3039) for
task 24458ae6-dde3-40e0-a4f7-84a901bd9f7d of framework
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000
I0222 09:32:20.939702 20889 master.cpp:1174] Slave
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0 at slave(247)@127.0.0.1:54769
(localhost) disconnected
I0222 09:32:20.939720 20889 master.cpp:2635] Disconnecting slave
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0 at slave(247)@127.0.0.1:54769
(localhost)
I0222 09:32:20.939901 20889 master.cpp:2654] Deactivating slave
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0 at slave(247)@127.0.0.1:54769
(localhost)
I0222 09:32:20.940335 20887 hierarchical.cpp:560] Slave
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0 deactivated
I0222 09:32:20.944319 20868 containerizer.cpp:149] Using isolation:
cgroups/mem,filesystem/posix
I0222 09:32:20.949115 20868 linux_launcher.cpp:101] Using /cgroup/freezer as
the freezer hierarchy for the Linux launcher
I0222 09:32:21.021484 20890 slave.cpp:193] Slave started on 248)@127.0.0.1:54769
I0222 09:32:21.021664 20890 slave.cpp:194] Flags at startup:
--appc_simple_discovery_uri_prefix="http://";
--appc_store_dir="/tmp/mesos/store/appc" --authenticatee="crammd5"
--cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false"
--cgroups_hierarchy="/cgroup" --cgroups_limit_swap="false"
--cgroups_root="mesos_test_7adedc17-08f0-4c8b-9974-66c009b1da18"
--container_disk_watch_interval="15secs" --containerizers="mesos"
--credential="/tmp/MemoryPressureMesosTest_CGROUPS_ROOT_SlaveRecovery_YcfvOI/credential"
--default_role="*" --disk_watch_interval="1mins" --docker="docker"
--docker_auth_server="https://auth.docker.io"; --docker_kill_orphans="true"
--docker_puller_timeout="60" --docker_registry="https://registry-1.docker.io";
--docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock"
--docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker"
--enforce_container_disk_quota="false" --executor_registration_timeout="1mins"
--executor_shutdown_grace_period="5secs"
--fetcher_cache_dir="/tmp/MemoryPressureMesosTest_CGROUPS_ROOT_SlaveRecovery_YcfvOI/fetch"
--fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks"
--gc_disk_headroom="0.1" --hadoop_home="" --help="false"
--hostname_lookup="true" --image_provisioner_backend="copy"
--initialize_driver_logging="true" --isolation="cgroups/mem"
--launcher_dir="/home/alexander/workspace/mesos/build/src" --logbufsecs="0"
--logging_level="INFO" --oversubscribed_resources_interval="15secs"
--perf_duration="10secs" --perf_interval="1mins"
--qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect"
--recovery_timeout="15mins" --registration_backoff_factor="10ms"
--resources="cpus:2;mem:1024;disk:1024;ports:[31000-32000]"
--revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox"
--strict="true" --switch_user="true" --systemd_enable_support="true"
--systemd_runtime_directory="/run/systemd/system" --version="false"
--work_dir="/tmp/MemoryPressureMesosTest_CGROUPS_ROOT_SlaveRecovery_YcfvOI"
W0222 09:32:21.021950 20890 slave.cpp:197]
**************************************************
Slave bound to loopback interface! Cannot communicate with remote master(s).
You might want to set '--ip' flag to a routable IP address.
**************************************************
I0222 09:32:21.021999 20890 credentials.hpp:83] Loading credential for
authentication from
'/tmp/MemoryPressureMesosTest_CGROUPS_ROOT_SlaveRecovery_YcfvOI/credential'
I0222 09:32:21.022282 20890 slave.cpp:324] Slave using credential for:
test-principal
I0222 09:32:21.022814 20890 slave.cpp:464] Slave resources: cpus(*):2;
mem(*):1024; disk(*):1024; ports(*):[31000-32000]
I0222 09:32:21.022861 20890 slave.cpp:472] Slave attributes: [ ]
I0222 09:32:21.022871 20890 slave.cpp:477] Slave hostname: localhost
I0222 09:32:21.024055 20890 state.cpp:58] Recovering state from
'/tmp/MemoryPressureMesosTest_CGROUPS_ROOT_SlaveRecovery_YcfvOI/meta'
I0222 09:32:21.024163 20890 state.cpp:698] No checkpointed resources found at
'/tmp/MemoryPressureMesosTest_CGROUPS_ROOT_SlaveRecovery_YcfvOI/meta/resources/resources.info'
I0222 09:32:21.026868 20889 slave.cpp:4653] Recovering framework
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000
I0222 09:32:21.026999 20889 slave.cpp:5476] Recovering executor
'24458ae6-dde3-40e0-a4f7-84a901bd9f7d' of framework
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000
I0222 09:32:21.028094 20884 status_update_manager.cpp:200] Recovering status
update manager
I0222 09:32:21.028137 20884 status_update_manager.cpp:208] Recovering executor
'24458ae6-dde3-40e0-a4f7-84a901bd9f7d' of framework
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000
I0222 09:32:21.031335 20887 containerizer.cpp:407] Recovering containerizer
I0222 09:32:21.031548 20887 containerizer.cpp:462] Recovering container
'0e8d2278-bb2b-4480-bfb1-39b47abcb707' for executor
'24458ae6-dde3-40e0-a4f7-84a901bd9f7d' of framework
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000
I0222 09:32:21.035538 20885 mem.cpp:602] Started listening for OOM events for
container 0e8d2278-bb2b-4480-bfb1-39b47abcb707
I0222 09:32:21.036461 20885 mem.cpp:722] Started listening on low memory
pressure events for container 0e8d2278-bb2b-4480-bfb1-39b47abcb707
I0222 09:32:21.037284 20885 mem.cpp:722] Started listening on medium memory
pressure events for container 0e8d2278-bb2b-4480-bfb1-39b47abcb707
I0222 09:32:21.038064 20885 mem.cpp:722] Started listening on critical memory
pressure events for container 0e8d2278-bb2b-4480-bfb1-39b47abcb707
I0222 09:32:21.040393 20886 provisioner.cpp:245] Provisioner recovery complete
I0222 09:32:21.041451 20883 slave.cpp:4505] Sending reconnect request to
executor '24458ae6-dde3-40e0-a4f7-84a901bd9f7d' of framework
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 at executor(1)@127.0.0.1:49224
I0222 09:32:21.044780 26624 exec.cpp:263] Received reconnect request from slave
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0
I0222 09:32:21.048050 20886 slave.cpp:2792] Re-registering executor
'24458ae6-dde3-40e0-a4f7-84a901bd9f7d' of framework
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000
I0222 09:32:21.051213 26620 exec.cpp:240] Executor re-registered on slave
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0
I0222 09:32:21.052510 20888 mem.cpp:353] Updated 'memory.soft_limit_in_bytes'
to 288MB for container 0e8d2278-bb2b-4480-bfb1-39b47abcb707
Re-registered executor on localhost
I0222 09:32:21.053912 20888 mem.cpp:388] Updated 'memory.limit_in_bytes' to
288MB for container 0e8d2278-bb2b-4480-bfb1-39b47abcb707
W0222 09:32:21.210090 20886 master.cpp:3869] Cannot kill task
24458ae6-dde3-40e0-a4f7-84a901bd9f7d of framework
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-0000 (default) at
[email protected]:54769 because the
slave 17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0 at slave(247)@127.0.0.1:54769
(localhost) is disconnected. Kill will be retried if the slave re-registers
../../src/tests/containerizer/memory_pressure_tests.cpp:300: Failure
Expected: (usage.get().mem_low_pressure_counter()) >=
(usage.get().mem_medium_pressure_counter()), actual: 6 vs 8
*** Aborted at 1456162341 (unix time) try "date -d @1456162341" if you are
using GNU date ***
PC: @ 0x16b24b0 testing::UnitTest::AddTestPartResult()
*** SIGSEGV (@0x0) received by PID 20868 (TID 0x7f0ee95ae840) from PID 0; stack
trace: ***
@ 0x7f0ee14ee790 (unknown)
@ 0x16b24b0 testing::UnitTest::AddTestPartResult()
@ 0x16a6ee9 testing::internal::AssertHelper::operator=()
@ 0x1688fc0
mesos::internal::tests::MemoryPressureMesosTest_CGROUPS_ROOT_SlaveRecovery_Test::TestBody()
@ 0x16cff10
testing::internal::HandleSehExceptionsInMethodIfSupported<>()
@ 0x16caf3e
testing::internal::HandleExceptionsInMethodIfSupported<>()
@ 0x16ac289 testing::Test::Run()
@ 0x16aca17 testing::TestInfo::Run()
@ 0x16ad052 testing::TestCase::Run()
@ 0x16b39a1 testing::internal::UnitTestImpl::RunAllTests()
@ 0x16d0b9f
testing::internal::HandleSehExceptionsInMethodIfSupported<>()
@ 0x16cbaca
testing::internal::HandleExceptionsInMethodIfSupported<>()
@ 0x16b26d1 testing::UnitTest::Run()
@ 0xe3bcd6 RUN_ALL_TESTS()
@ 0xe3b8ec main
@ 0x7f0ee0356d5d __libc_start_main
@ 0x9c4d89 (unknown)
I0222 09:32:21.558915 26624 exec.cpp:463] Slave exited, but framework has
checkpointing enabled. Waiting 15mins to reconnect with slave
17b7da64-0c4d-4e46-ae1f-2b356dc5f266-S0
I0222 09:32:21.559326 26620 exec.cpp:472] Slave exited ... shutting down
Shutting down
Sending SIGTERM to process tree at pid 26627
/var/tmp/sclcApPT3: line 8: 20868 Segmentation fault
'./bin/mesos-tests.sh'
'--gtest_filter=MemoryPressureMesosTest.CGROUPS_ROOT_SlaveRecovery'
'--gtest_repeat=1000' '--gtest_break_on_failure'
Sent SIGTERM to the following process trees:
[
-+- 26627 sh -c while true; do dd count=512 bs=1M if=/dev/zero of=./temp; done
\--- 26628 dd count=512 bs=1M if=/dev/zero of=./temp
]
Command terminated with signal Terminated (pid: 26627)
E0222 09:32:21.704880 26626 process.cpp:1963] Failed to shutdown socket with fd
9: Transport endpoint is not connected
{noformat}
after running:
{code}
MESOS_VERBOSE=1 sudo .libs/mesos-tests
--gtest_filter="MemoryPressureMesosTest.CGROUPS_ROOT_SlaveRecovery"
--gtest_repeat=1000 --gtest_break_on_failure
{code}
> MemoryPressureMesosTest.CGROUPS_ROOT_SlaveRecovery is flaky
> -----------------------------------------------------------
>
> Key: MESOS-4047
> URL: https://issues.apache.org/jira/browse/MESOS-4047
> Project: Mesos
> Issue Type: Bug
> Components: test
> Affects Versions: 0.26.0
> Environment: Ubuntu 14, gcc 4.8.4
> Reporter: Joseph Wu
> Assignee: Joseph Wu
> Labels: flaky, flaky-test
> Fix For: 0.27.0
>
>
> {code:title=Output from passed test}
> [----------] 1 test from MemoryPressureMesosTest
> 1+0 records in
> 1+0 records out
> 1048576 bytes (1.0 MB) copied, 0.000430889 s, 2.4 GB/s
> [ RUN ] MemoryPressureMesosTest.CGROUPS_ROOT_SlaveRecovery
> I1202 11:09:14.319327 5062 exec.cpp:134] Version: 0.27.0
> I1202 11:09:14.333317 5079 exec.cpp:208] Executor registered on slave
> bea15b35-9aa1-4b57-96fb-29b5f70638ac-S0
> Registered executor on ubuntu
> Starting task 4e62294c-cfcf-4a13-b699-c6a4b7ac5162
> sh -c 'while true; do dd count=512 bs=1M if=/dev/zero of=./temp; done'
> Forked command at 5085
> I1202 11:09:14.391739 5077 exec.cpp:254] Received reconnect request from
> slave bea15b35-9aa1-4b57-96fb-29b5f70638ac-S0
> I1202 11:09:14.398598 5082 exec.cpp:231] Executor re-registered on slave
> bea15b35-9aa1-4b57-96fb-29b5f70638ac-S0
> Re-registered executor on ubuntu
> Shutting down
> Sending SIGTERM to process tree at pid 5085
> Killing the following process trees:
> [
> -+- 5085 sh -c while true; do dd count=512 bs=1M if=/dev/zero of=./temp; done
> \--- 5086 dd count=512 bs=1M if=/dev/zero of=./temp
> ]
> [ OK ] MemoryPressureMesosTest.CGROUPS_ROOT_SlaveRecovery (1096 ms)
> {code}
> {code:title=Output from failed test}
> [----------] 1 test from MemoryPressureMesosTest
> 1+0 records in
> 1+0 records out
> 1048576 bytes (1.0 MB) copied, 0.000404489 s, 2.6 GB/s
> [ RUN ] MemoryPressureMesosTest.CGROUPS_ROOT_SlaveRecovery
> I1202 11:09:15.509950 5109 exec.cpp:134] Version: 0.27.0
> I1202 11:09:15.568183 5123 exec.cpp:208] Executor registered on slave
> 88734acc-718e-45b0-95b9-d8f07cea8a9e-S0
> Registered executor on ubuntu
> Starting task 14b6bab9-9f60-4130-bdc4-44efba262bc6
> Forked command at 5132
> sh -c 'while true; do dd count=512 bs=1M if=/dev/zero of=./temp; done'
> I1202 11:09:15.665498 5129 exec.cpp:254] Received reconnect request from
> slave 88734acc-718e-45b0-95b9-d8f07cea8a9e-S0
> I1202 11:09:15.670995 5123 exec.cpp:381] Executor asked to shutdown
> Shutting down
> Sending SIGTERM to process tree at pid 5132
> ../../src/tests/containerizer/memory_pressure_tests.cpp:283: Failure
> (usage).failure(): Unknown container: ebe90e15-72fa-4519-837b-62f43052c913
> *** Aborted at 1449083355 (unix time) try "date -d @1449083355" if you are
> using GNU date ***
> {code}
> Notice that in the failed test, the executor is asked to shutdown when it
> tries to reconnect to the agent.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
