I saw this on ASF CI <https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/33/BUILDTOOL=autotools,COMPILER=gcc,CONFIGURATION=--verbose%20--enable-libevent%20--enable-ssl,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=centos:7,label_exp=(docker%7C%7CHadoop)&&(!ubuntu-us1)&&(!ubuntu-eu2)/console>. Expected flaky test?
[ RUN ] HTTPCommandExecutorTest.TerminateWithACK I0504 15:43:05.341382 32064 cluster.cpp:158] Creating default 'local' authorizer I0504 15:43:05.345090 32064 leveldb.cpp:174] Opened db in 3.444533ms I0504 15:43:05.345728 32064 leveldb.cpp:181] Compacted db in 603462ns I0504 15:43:05.345772 32064 leveldb.cpp:196] Created db iterator in 16838ns I0504 15:43:05.345788 32064 leveldb.cpp:202] Seeked to beginning of db in 1987ns I0504 15:43:05.345799 32064 leveldb.cpp:271] Iterated through 0 keys in the db in 269ns I0504 15:43:05.345834 32064 replica.cpp:776] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I0504 15:43:05.346590 32091 recover.cpp:451] Starting replica recovery I0504 15:43:05.346793 32091 recover.cpp:477] Replica is in EMPTY status I0504 15:43:05.347823 32098 replica.cpp:673] Replica in EMPTY status received a broadcasted recover request from __req_res__(168)@172.17.0.3:41866 I0504 15:43:05.348352 32090 recover.cpp:197] Received a recover response from a replica in EMPTY status I0504 15:43:05.348784 32098 recover.cpp:568] Updating replica status to STARTING I0504 15:43:05.349874 32095 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 840720ns I0504 15:43:05.349900 32095 replica.cpp:320] Persisted replica status to STARTING I0504 15:43:05.350070 32088 recover.cpp:477] Replica is in STARTING status I0504 15:43:05.350971 32102 master.cpp:380] Master 2075640b-b7dc-44f0-89b5-b0f9af99be7e (41c61dc99119) started on 172.17.0.3:41866 I0504 15:43:05.351112 32088 replica.cpp:673] Replica in STARTING status received a broadcasted recover request from __req_res__(169)@172.17.0.3:41866 I0504 15:43:05.350991 32102 master.cpp:382] Flags at startup: --acls="" --agent_ping_timeout="15secs" --agent_reregister_timeout="10mins" --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate_agents="true" --authenticate_frameworks="true" --authenticate_http_frameworks="true" --authenticate_http_readonly="true" --authenticate_http_readwrite="true" --authenticators="crammd5" --authorizers="local" --credentials="/tmp/t7Ea9P/credentials" --framework_sorter="drf" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_framework_authenticators="basic" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_agent_ping_timeouts="5" --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000" --quiet="false" --recovery_agent_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_gc_interval="15mins" --registry_max_agent_age="2weeks" --registry_max_agent_count="102400" --registry_store_timeout="100secs" --registry_strict="false" --root_submissions="true" --user_sorter="drf" --version="false" --webui_dir="/mesos/mesos-1.1.2/_inst/share/mesos/webui" --work_dir="/tmp/t7Ea9P/master" --zk_session_timeout="10secs" I0504 15:43:05.351322 32102 master.cpp:432] Master only allowing authenticated frameworks to register I0504 15:43:05.351335 32102 master.cpp:446] Master only allowing authenticated agents to register I0504 15:43:05.351341 32102 master.cpp:459] Master only allowing authenticated HTTP frameworks to register I0504 15:43:05.351348 32102 credentials.hpp:37] Loading credentials for authentication from '/tmp/t7Ea9P/credentials' I0504 15:43:05.351394 32094 recover.cpp:197] Received a recover response from a replica in STARTING status I0504 15:43:05.351594 32102 master.cpp:504] Using default 'crammd5' authenticator I0504 15:43:05.351850 32102 http.cpp:887] Using default 'basic' HTTP authenticator for realm 'mesos-master-readonly' I0504 15:43:05.352252 32102 http.cpp:887] Using default 'basic' HTTP authenticator for realm 'mesos-master-readwrite' I0504 15:43:05.352270 32090 recover.cpp:568] Updating replica status to VOTING I0504 15:43:05.352500 32102 http.cpp:887] Using default 'basic' HTTP authenticator for realm 'mesos-master-scheduler' I0504 15:43:05.352635 32092 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 225076ns I0504 15:43:05.352660 32092 replica.cpp:320] Persisted replica status to VOTING I0504 15:43:05.352707 32102 master.cpp:584] Authorization enabled I0504 15:43:05.352778 32087 recover.cpp:582] Successfully joined the Paxos group I0504 15:43:05.352880 32091 hierarchical.cpp:149] Initialized hierarchical allocator process I0504 15:43:05.352883 32089 whitelist_watcher.cpp:77] No whitelist given I0504 15:43:05.353144 32087 recover.cpp:466] Recover process terminated I0504 15:43:05.355403 32085 master.cpp:2017] Elected as the leading master! I0504 15:43:05.355437 32085 master.cpp:1560] Recovering from registrar I0504 15:43:05.355631 32097 registrar.cpp:329] Recovering registrar I0504 15:43:05.356300 32094 log.cpp:553] Attempting to start the writer I0504 15:43:05.357868 32094 replica.cpp:493] Replica received implicit promise request from __req_res__(170)@172.17.0.3:41866 with proposal 1 I0504 15:43:05.358140 32094 leveldb.cpp:304] Persisting metadata (8 bytes) to leveldb took 238736ns I0504 15:43:05.358163 32094 replica.cpp:342] Persisted promised to 1 I0504 15:43:05.358783 32106 coordinator.cpp:238] Coordinator attempting to fill missing positions I0504 15:43:05.360201 32094 replica.cpp:388] Replica received explicit promise request from __req_res__(171)@172.17.0.3:41866 for position 0 with proposal 2 I0504 15:43:05.360452 32094 leveldb.cpp:341] Persisting action (8 bytes) to leveldb took 209201ns I0504 15:43:05.360476 32094 replica.cpp:708] Persisted action NOP at position 0 I0504 15:43:05.361809 32091 replica.cpp:537] Replica received write request for position 0 from __req_res__(172)@172.17.0.3:41866 I0504 15:43:05.361876 32091 leveldb.cpp:436] Reading position from leveldb took 29524ns I0504 15:43:05.362116 32091 leveldb.cpp:341] Persisting action (14 bytes) to leveldb took 194035ns I0504 15:43:05.362143 32091 replica.cpp:708] Persisted action NOP at position 0 I0504 15:43:05.362979 32103 replica.cpp:691] Replica received learned notice for position 0 from @0.0.0.0:0 I0504 15:43:05.363215 32103 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 207499ns I0504 15:43:05.363240 32103 replica.cpp:708] Persisted action NOP at position 0 I0504 15:43:05.363785 32105 log.cpp:569] Writer started with ending position 0 I0504 15:43:05.365002 32089 leveldb.cpp:436] Reading position from leveldb took 26769ns I0504 15:43:05.365936 32094 registrar.cpp:362] Successfully fetched the registry (0B) in 10.246912ms I0504 15:43:05.366044 32094 registrar.cpp:461] Applied 1 operations in 19811ns; attempting to update the registry I0504 15:43:05.366716 32092 log.cpp:577] Attempting to append 168 bytes to the log I0504 15:43:05.366830 32087 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 1 I0504 15:43:05.367691 32095 replica.cpp:537] Replica received write request for position 1 from __req_res__(173)@172.17.0.3:41866 I0504 15:43:05.375273 32095 leveldb.cpp:341] Persisting action (187 bytes) to leveldb took 7.546417ms I0504 15:43:05.375305 32095 replica.cpp:708] Persisted action APPEND at position 1 I0504 15:43:05.376070 32089 replica.cpp:691] Replica received learned notice for position 1 from @0.0.0.0:0 I0504 15:43:05.376334 32089 leveldb.cpp:341] Persisting action (189 bytes) to leveldb took 223609ns I0504 15:43:05.376359 32089 replica.cpp:708] Persisted action APPEND at position 1 I0504 15:43:05.377385 32093 registrar.cpp:506] Successfully updated the registry in 11.236864ms I0504 15:43:05.377521 32093 registrar.cpp:392] Successfully recovered registrar I0504 15:43:05.377568 32098 log.cpp:596] Attempting to truncate the log to 1 I0504 15:43:05.377693 32085 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 2 I0504 15:43:05.378213 32105 master.cpp:1676] Recovered 0 agents from the registry (129B); allowing 10mins for agents to re-register I0504 15:43:05.378229 32100 hierarchical.cpp:176] Skipping recovery of hierarchical allocator: nothing to recover I0504 15:43:05.378677 32086 replica.cpp:537] Replica received write request for position 2 from __req_res__(174)@172.17.0.3:41866 I0504 15:43:05.378937 32086 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 223698ns I0504 15:43:05.378960 32086 replica.cpp:708] Persisted action TRUNCATE at position 2 I0504 15:43:05.379722 32102 replica.cpp:691] Replica received learned notice for position 2 from @0.0.0.0:0 I0504 15:43:05.383610 32102 leveldb.cpp:341] Persisting action (18 bytes) to leveldb took 3.858422ms I0504 15:43:05.383677 32102 leveldb.cpp:399] Deleting ~1 keys from leveldb took 35851ns I0504 15:43:05.383697 32102 replica.cpp:708] Persisted action TRUNCATE at position 2 I0504 15:43:05.386698 32064 containerizer.cpp:200] Using isolation: posix/cpu,posix/mem,filesystem/posix,network/cni W0504 15:43:05.387332 32064 backend.cpp:76] Failed to create 'aufs' backend: AufsBackend requires root privileges, but is running as user mesos W0504 15:43:05.387490 32064 backend.cpp:76] Failed to create 'bind' backend: BindBackend requires root privileges I0504 15:43:05.391284 32096 slave.cpp:208] Mesos agent started on 72.17.0.3:41866 I0504 15:43:05.391306 32096 slave.cpp:209] Flags at startup: --acls="" --appc_simple_discovery_uri_prefix="http://"; --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="true" --authenticate_http_readwrite="true" --authenticatee="crammd5" --authentication_backoff_factor="1secs" --authorizer="local" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --credential="/tmp/HTTPCommandExecutorTest_TerminateWithACK_3LgMk8/credential" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io"; --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker" --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/HTTPCommandExecutorTest_TerminateWithACK_3LgMk8/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --hostname_lookup="true" --http_authenticators="basic" --http_command_executor="true" --http_credentials="/tmp/HTTPCommandExecutorTest_TerminateWithACK_3LgMk8/http_credentials" --image_provisioner_backend="copy" --initialize_driver_logging="true" --isolation="posix/cpu,posix/mem" --launcher="posix" --launcher_dir="/mesos/mesos-1.1.2/_build/src" --logbufsecs="0" --logging_level="INFO" --max_completed_executors_per_framework="150" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="10ms" --resources="cpus:2;gpus:0;mem:1024;disk:1024;ports:[31000-32000]" --revocable_cpu_low_priority="true" --runtime_dir="/tmp/HTTPCommandExecutorTest_TerminateWithACK_3LgMk8" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --systemd_enable_support="true" --systemd_runtime_directory="/run/systemd/system" --version="false" --work_dir="/tmp/HTTPCommandExecutorTest_TerminateWithACK_BRTXeu" I0504 15:43:05.391834 32096 credentials.hpp:86] Loading credential for authentication from '/tmp/HTTPCommandExecutorTest_TerminateWithACK_3LgMk8/credential' I0504 15:43:05.391991 32096 slave.cpp:346] Agent using credential for: test-principal I0504 15:43:05.392012 32096 credentials.hpp:37] Loading credentials for authentication from '/tmp/HTTPCommandExecutorTest_TerminateWithACK_3LgMk8/http_credentials' I0504 15:43:05.392226 32096 http.cpp:887] Using default 'basic' HTTP authenticator for realm 'mesos-agent-readonly' I0504 15:43:05.392369 32096 http.cpp:887] Using default 'basic' HTTP authenticator for realm 'mesos-agent-readwrite' I0504 15:43:05.392388 32064 sched.cpp:226] Version: 1.1.2 I0504 15:43:05.392886 32098 sched.cpp:330] New master detected at master@172.17.0.3:41866 I0504 15:43:05.392966 32098 sched.cpp:396] Authenticating with master master@172.17.0.3:41866 I0504 15:43:05.392990 32098 sched.cpp:403] Using default CRAM-MD5 authenticatee I0504 15:43:05.393358 32095 authenticatee.cpp:121] Creating new client SASL connection I0504 15:43:05.393451 32096 slave.cpp:533] Agent resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0504 15:43:05.393534 32096 slave.cpp:541] Agent attributes: [ ] I0504 15:43:05.393548 32096 slave.cpp:546] Agent hostname: 41c61dc99119 I0504 15:43:05.393612 32086 master.cpp:6680] Authenticating scheduler-16d6a594-fabf-46b8-8d65-3d174b8e2daa@172.17.0.3:41866 I0504 15:43:05.393725 32102 authenticator.cpp:414] Starting authentication session for crammd5-authenticatee(32)@172.17.0.3:41866 I0504 15:43:05.394050 32089 authenticator.cpp:98] Creating new server SASL connection I0504 15:43:05.394294 32101 authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5 I0504 15:43:05.394325 32101 authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5' I0504 15:43:05.394428 32088 authenticator.cpp:204] Received SASL authentication start I0504 15:43:05.394482 32088 authenticator.cpp:326] Authentication requires more steps I0504 15:43:05.394671 32083 authenticatee.cpp:259] Received SASL authentication step I0504 15:43:05.394803 32101 authenticator.cpp:232] Received SASL authentication step I0504 15:43:05.394848 32101 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: '41c61dc99119' server FQDN: '41c61dc99119' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0504 15:43:05.394871 32101 auxprop.cpp:181] Looking up auxiliary property '*userPassword' I0504 15:43:05.394943 32101 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0504 15:43:05.394985 32101 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: '41c61dc99119' server FQDN: '41c61dc99119' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0504 15:43:05.395009 32101 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0504 15:43:05.395025 32101 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0504 15:43:05.395052 32101 authenticator.cpp:318] Authentication success I0504 15:43:05.395088 32084 state.cpp:57] Recovering state from '/tmp/HTTPCommandExecutorTest_TerminateWithACK_BRTXeu/meta' I0504 15:43:05.395241 32102 master.cpp:6710] Successfully authenticated principal 'test-principal' at scheduler-16d6a594-fabf-46b8-8d65-3d174b8e2daa@172.17.0.3:41866 I0504 15:43:05.395313 32097 authenticator.cpp:432] Authentication session cleanup for crammd5-authenticatee(32)@172.17.0.3:41866 I0504 15:43:05.395408 32098 authenticatee.cpp:299] Authentication success I0504 15:43:05.395480 32103 status_update_manager.cpp:203] Recovering status update manager I0504 15:43:05.395692 32087 containerizer.cpp:555] Recovering containerizer I0504 15:43:05.395896 32090 sched.cpp:502] Successfully authenticated with master master@172.17.0.3:41866 I0504 15:43:05.395917 32090 sched.cpp:820] Sending SUBSCRIBE call to master@172.17.0.3:41866 I0504 15:43:05.396005 32090 sched.cpp:853] Will retry registration in 1.712040262secs if necessary I0504 15:43:05.396121 32102 master.cpp:2596] Received SUBSCRIBE call for framework 'default' at scheduler-16d6a594-fabf-46b8-8d65-3d174b8e2daa@172.17.0.3:41866 I0504 15:43:05.396178 32102 master.cpp:2053] Authorizing framework principal 'test-principal' to receive offers for role '*' I0504 15:43:05.396824 32088 master.cpp:2672] Subscribing framework default with checkpointing disabled and capabilities [ ] I0504 15:43:05.397313 32098 provisioner.cpp:253] Provisioner recovery complete I0504 15:43:05.397626 32096 sched.cpp:743] Framework registered with 2075640b-b7dc-44f0-89b5-b0f9af99be7e-0000 I0504 15:43:05.397678 32096 sched.cpp:757] Scheduler::registered took 26397ns I0504 15:43:05.397708 32104 hierarchical.cpp:275] Added framework 2075640b-b7dc-44f0-89b5-b0f9af99be7e-0000 I0504 15:43:05.397784 32104 hierarchical.cpp:1694] No allocations performed I0504 15:43:05.397799 32100 slave.cpp:5281] Finished recovery I0504 15:43:05.397812 32104 hierarchical.cpp:1789] No inverse offers to send out! I0504 15:43:05.397878 32104 hierarchical.cpp:1286] Performed allocation for 0 agents in 140315ns I0504 15:43:05.398274 32100 slave.cpp:5455] Querying resource estimator for oversubscribable resources I0504 15:43:05.398495 32086 status_update_manager.cpp:177] Pausing sending status updates I0504 15:43:05.398500 32100 slave.cpp:915] New master detected at master@172.17.0.3:41866 I0504 15:43:05.398524 32100 slave.cpp:974] Authenticating with master master@172.17.0.3:41866 I0504 15:43:05.398576 32100 slave.cpp:985] Using default CRAM-MD5 authenticatee I0504 15:43:05.398720 32100 slave.cpp:947] Detecting new master I0504 15:43:05.398767 32083 authenticatee.cpp:121] Creating new client SASL connection I0504 15:43:05.399022 32091 master.cpp:6680] Authenticating (1)@172.17.0.3:41866 I0504 15:43:05.399214 32097 authenticator.cpp:414] Starting authentication session for crammd5-authenticatee(33)@172.17.0.3:41866 I0504 15:43:05.399416 32087 authenticator.cpp:98] Creating new server SASL connection I0504 15:43:05.399605 32095 authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5 I0504 15:43:05.399631 32095 authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5' I0504 15:43:05.399798 32087 authenticator.cpp:204] Received SASL authentication start I0504 15:43:05.399850 32087 authenticator.cpp:326] Authentication requires more steps I0504 15:43:05.399952 32090 authenticatee.cpp:259] Received SASL authentication step I0504 15:43:05.400058 32095 authenticator.cpp:232] Received SASL authentication step I0504 15:43:05.400089 32095 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: '41c61dc99119' server FQDN: '41c61dc99119' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: false I0504 15:43:05.400108 32095 auxprop.cpp:181] Looking up auxiliary property '*userPassword' I0504 15:43:05.400138 32095 auxprop.cpp:181] Looking up auxiliary property '*cmusaslsecretCRAM-MD5' I0504 15:43:05.400166 32095 auxprop.cpp:109] Request to lookup properties for user: 'test-principal' realm: '41c61dc99119' server FQDN: '41c61dc99119' SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false SASL_AUXPROP_AUTHZID: true I0504 15:43:05.400176 32095 auxprop.cpp:131] Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true I0504 15:43:05.400184 32095 auxprop.cpp:131] Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since SASL_AUXPROP_AUTHZID == true I0504 15:43:05.400198 32095 authenticator.cpp:318] Authentication success I0504 15:43:05.400274 32090 authenticatee.cpp:299] Authentication success I0504 15:43:05.400344 32102 master.cpp:6710] Successfully authenticated principal 'test-principal' at (1)@172.17.0.3:41866 I0504 15:43:05.400382 32104 authenticator.cpp:432] Authentication session cleanup for crammd5-authenticatee(33)@172.17.0.3:41866 I0504 15:43:05.400687 32089 slave.cpp:1069] Successfully authenticated with master master@172.17.0.3:41866 I0504 15:43:05.400862 32089 slave.cpp:1475] Will retry registration in 4.615564ms if necessary I0504 15:43:05.401051 32088 master.cpp:5103] Registering agent at (1)@172.17.0.3:41866 (41c61dc99119) with id 2075640b-b7dc-44f0-89b5-b0f9af99be7e-S0 I0504 15:43:05.401487 32083 registrar.cpp:461] Applied 1 operations in 49606ns; attempting to update the registry I0504 15:43:05.403597 32097 log.cpp:577] Attempting to append 337 bytes to the log I0504 15:43:05.403753 32085 coordinator.cpp:348] Coordinator attempting to write APPEND action at position 3 I0504 15:43:05.404583 32086 replica.cpp:537] Replica received write request for position 3 from __req_res__(175)@172.17.0.3:41866 I0504 15:43:05.404911 32086 leveldb.cpp:341] Persisting action (356 bytes) to leveldb took 269746ns I0504 15:43:05.404937 32086 replica.cpp:708] Persisted action APPEND at position 3 I0504 15:43:05.405844 32106 replica.cpp:691] Replica received learned notice for position 3 from @0.0.0.0:0 I0504 15:43:05.406158 32106 leveldb.cpp:341] Persisting action (358 bytes) to leveldb took 282857ns I0504 15:43:05.406189 32106 replica.cpp:708] Persisted action APPEND at position 3 I0504 15:43:05.406263 32089 slave.cpp:1475] Will retry registration in 13.993213ms if necessary I0504 15:43:05.406363 32093 master.cpp:5091] Ignoring register agent message from (1)@172.17.0.3:41866 (41c61dc99119) as admission is already in progress I0504 15:43:05.408088 32099 registrar.cpp:506] Successfully updated the registry in 6.545152ms I0504 15:43:05.408306 32093 log.cpp:596] Attempting to truncate the log to 3 I0504 15:43:05.408526 32099 coordinator.cpp:348] Coordinator attempting to write TRUNCATE action at position 4 I0504 15:43:05.408797 32088 slave.cpp:4133] Received ping from slave-observer(8)@172.17.0.3:41866 I0504 15:43:05.408907 32098 master.cpp:5174] Registered agent 2075640b-b7dc-44f0-89b5-b0f9af99be7e-S0 at (1)@172.17.0.3:41866 (41c61dc99119) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] I0504 15:43:05.409015 32088 slave.cpp:1115] Registered with master master@172.17.0.3:41866; given agent ID 2075640b-b7dc-44f0-89b5-b0f9af99be7e-S0 I0504 15:43:05.409046 32088 fetcher.cpp:90] Clearing fetcher cache I0504 15:43:05.409303 32084 hierarchical.cpp:485] Added agent 2075640b-b7dc-44f0-89b5-b0f9af99be7e-S0 (41c61dc99119) with cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (allocated: {}) I0504 15:43:05.409330 32098 status_update_manager.cpp:184] Resuming sending status updates I0504 15:43:05.409369 32099 replica.cpp:537] Replica received write request for position 4 from __req_res__(176)@172.17.0.3:41866 I0504 15:43:05.409466 32088 slave.cpp:1138] Checkpointing SlaveInfo to '/tmp/HTTPCommandExecutorTest_TerminateWithACK_BRTXeu/meta/slaves/2075640b-b7dc-44f0-89b5-b0f9af99be7e-S0/slave.info' I0504 15:43:05.409611 32099 leveldb.cpp:341] Persisting action (16 bytes) to leveldb took 207020ns I0504 15:43:05.409636 32099 replica.cpp:708] Persisted action TRUNCATE at position 4 I0504 15:43:05.410361 32100 replica.cpp:691] Replica received learned notice for position 4 from @0.0.0.0:0 I0504 15:43:05.410599 32100 leveldb.cpp:341] Persisting action (18 bytes) to leveldb took 211227ns I0504 15:43:05.410648 32084 hierarchical.cpp:1789] No inverse offers to send out! I0504 15:43:05.410667 32100 leveldb.cpp:399] Deleting ~2 keys from leveldb took 40031ns I0504 15:43:05.410691 32100 replica.cpp:708] Persisted action TRUNCATE at position 4 I0504 15:43:05.410740 32084 hierarchical.cpp:1309] Performed allocation for agent 2075640b-b7dc-44f0-89b5-b0f9af99be7e-S0 in 1.395222ms I0504 15:43:05.411295 32104 master.cpp:6509] Sending 1 offers to framework 2075640b-b7dc-44f0-89b5-b0f9af99be7e-0000 (default) at scheduler-16d6a594-fabf-46b8-8d65-3d174b8e2daa@172.17.0.3:41866 I0504 15:43:05.411969 32099 sched.cpp:917] Scheduler::resourceOffers took 122658ns I0504 15:43:05.414469 32102 master.cpp:3554] Processing ACCEPT call for offers: [ 2075640b-b7dc-44f0-89b5-b0f9af99be7e-O0 ] on agent 2075640b-b7dc-44f0-89b5-b0f9af99be7e-S0 at (1)@172.17.0.3:41866 (41c61dc99119) for framework 2075640b-b7dc-44f0-89b5-b0f9af99be7e-0000 (default) at scheduler-16d6a594-fabf-46b8-8d65-3d174b8e2daa@172.17.0.3:41866 I0504 15:43:05.414572 32102 master.cpp:3159] Authorizing framework principal 'test-principal' to launch task 600b509d-caf9-4c9e-a7ec-d0cb49ec2f7a I0504 15:43:05.417814 32091 master.cpp:8262] Adding task 600b509d-caf9-4c9e-a7ec-d0cb49ec2f7a with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on agent 2075640b-b7dc-44f0-89b5-b0f9af99be7e-S0 (41c61dc99119) I0504 15:43:05.418177 32091 master.cpp:4192] Launching task 600b509d-caf9-4c9e-a7ec-d0cb49ec2f7a of framework 2075640b-b7dc-44f0-89b5-b0f9af99be7e-0000 (default) at scheduler-16d6a594-fabf-46b8-8d65-3d174b8e2daa@172.17.0.3:41866 with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on agent 2075640b-b7dc-44f0-89b5-b0f9af99be7e-S0 at (1)@172.17.0.3:41866 (41c61dc99119) I0504 15:43:05.418993 32099 slave.cpp:1539] Got assigned task '600b509d-caf9-4c9e-a7ec-d0cb49ec2f7a' for framework 2075640b-b7dc-44f0-89b5-b0f9af99be7e-0000 I0504 15:43:05.420351 32099 slave.cpp:1701] Launching task '600b509d-caf9-4c9e-a7ec-d0cb49ec2f7a' for framework 2075640b-b7dc-44f0-89b5-b0f9af99be7e-0000 I0504 15:43:05.421084 32099 paths.cpp:536] Trying to chown '/tmp/HTTPCommandExecutorTest_TerminateWithACK_BRTXeu/slaves/2075640b-b7dc-44f0-89b5-b0f9af99be7e-S0/frameworks/2075640b-b7dc-44f0-89b5-b0f9af99be7e-0000/executors/600b509d-caf9-4c9e-a7ec-d0cb49ec2f7a/runs/0817f9bd-8e43-4b6f-b6b3-88805365c80c' to user 'mesos' I0504 15:43:05.438580 32099 slave.cpp:6179] Launching executor '600b509d-caf9-4c9e-a7ec-d0cb49ec2f7a' of framework 2075640b-b7dc-44f0-89b5-b0f9af99be7e-0000 with resources cpus(*):0.1; mem(*):32 in work directory '/tmp/HTTPCommandExecutorTest_TerminateWithACK_BRTXeu/slaves/2075640b-b7dc-44f0-89b5-b0f9af99be7e-S0/frameworks/2075640b-b7dc-44f0-89b5-b0f9af99be7e-0000/executors/600b509d-caf9-4c9e-a7ec-d0cb49ec2f7a/runs/0817f9bd-8e43-4b6f-b6b3-88805365c80c' I0504 15:43:05.439748 32097 containerizer.cpp:938] Starting container 0817f9bd-8e43-4b6f-b6b3-88805365c80c for executor '600b509d-caf9-4c9e-a7ec-d0cb49ec2f7a' of framework 2075640b-b7dc-44f0-89b5-b0f9af99be7e-0000 I0504 15:43:05.440160 32099 slave.cpp:1987] Queued task '600b509d-caf9-4c9e-a7ec-d0cb49ec2f7a' for executor '600b509d-caf9-4c9e-a7ec-d0cb49ec2f7a' of framework 2075640b-b7dc-44f0-89b5-b0f9af99be7e-0000 I0504 15:43:05.440285 32099 slave.cpp:868] Successfully attached file '/tmp/HTTPCommandExecutorTest_TerminateWithACK_BRTXeu/slaves/2075640b-b7dc-44f0-89b5-b0f9af99be7e-S0/frameworks/2075640b-b7dc-44f0-89b5-b0f9af99be7e-0000/executors/600b509d-caf9-4c9e-a7ec-d0cb49ec2f7a/runs/0817f9bd-8e43-4b6f-b6b3-88805365c80c' I0504 15:43:05.447082 32097 containerizer.cpp:1463] Launching 'mesos-containerizer' with flags '--command="{"arguments":["mesos-executor","--launcher_dir=\/mesos\/mesos-1.1.2\/_build\/src"],"shell":false,"value":"\/mesos\/mesos-1.1.2\/_build\/src\/mesos-executor"}" --help="false" --pipe_read="9" --pipe_write="12" --pre_exec_commands="[]" --runtime_directory="/tmp/HTTPCommandExecutorTest_TerminateWithACK_3LgMk8/containers/0817f9bd-8e43-4b6f-b6b3-88805365c80c" --unshare_namespace_mnt="false" --user="mesos" --working_directory="/tmp/HTTPCommandExecutorTest_TerminateWithACK_BRTXeu/slaves/2075640b-b7dc-44f0-89b5-b0f9af99be7e-S0/frameworks/2075640b-b7dc-44f0-89b5-b0f9af99be7e-0000/executors/600b509d-caf9-4c9e-a7ec-d0cb49ec2f7a/runs/0817f9bd-8e43-4b6f-b6b3-88805365c80c"' I0504 15:43:05.449122 32097 launcher.cpp:127] Forked child with pid '32121' for container '0817f9bd-8e43-4b6f-b6b3-88805365c80c' I0504 15:43:06.354243 32087 hierarchical.cpp:1694] No allocations performed I0504 15:43:06.354343 32087 hierarchical.cpp:1789] No inverse offers to send out! I0504 15:43:06.354439 32087 hierarchical.cpp:1286] Performed allocation for 1 agents in 488945ns I0504 15:43:07.355734 32101 hierarchical.cpp:1694] No allocations performed I0504 15:43:07.355855 32101 hierarchical.cpp:1789] No inverse offers to send out! I0504 15:43:07.355985 32101 hierarchical.cpp:1286] Performed allocation for 1 agents in 567052ns I0504 15:43:08.356890 32100 hierarchical.cpp:1694] No allocations performed I0504 15:43:08.356999 32100 hierarchical.cpp:1789] No inverse offers to send out! I0504 15:43:08.357092 32100 hierarchical.cpp:1286] Performed allocation for 1 agents in 462937ns I0504 15:43:09.358674 32088 hierarchical.cpp:1694] No allocations performed I0504 15:43:09.358762 32088 hierarchical.cpp:1789] No inverse offers to send out! I0504 15:43:09.358856 32088 hierarchical.cpp:1286] Performed allocation for 1 agents in 460517ns I0504 15:43:10.360016 32105 hierarchical.cpp:1694] No allocations performed I0504 15:43:10.360115 32105 hierarchical.cpp:1789] No inverse offers to send out! I0504 15:43:10.360208 32105 hierarchical.cpp:1286] Performed allocation for 1 agents in 468579ns I0504 15:43:11.361495 32092 hierarchical.cpp:1694] No allocations performed I0504 15:43:11.361596 32092 hierarchical.cpp:1789] No inverse offers to send out! I0504 15:43:11.361706 32092 hierarchical.cpp:1286] Performed allocation for 1 agents in 456795ns I0504 15:43:12.362313 32089 hierarchical.cpp:1694] No allocations performed I0504 15:43:12.362409 32089 hierarchical.cpp:1789] No inverse offers to send out! I0504 15:43:12.362490 32089 hierarchical.cpp:1286] Performed allocation for 1 agents in 421777ns I0504 15:43:13.363725 32090 hierarchical.cpp:1694] No allocations performed I0504 15:43:13.363808 32090 hierarchical.cpp:1789] No inverse offers to send out! I0504 15:43:13.363903 32090 hierarchical.cpp:1286] Performed allocation for 1 agents in 446183ns I0504 15:43:14.365000 32104 hierarchical.cpp:1694] No allocations performed I0504 15:43:14.365072 32104 hierarchical.cpp:1789] No inverse offers to send out! I0504 15:43:14.365156 32104 hierarchical.cpp:1286] Performed allocation for 1 agents in 349292ns I0504 15:43:15.366421 32103 hierarchical.cpp:1694] No allocations performed I0504 15:43:15.366508 32103 hierarchical.cpp:1789] No inverse offers to send out! I0504 15:43:15.366601 32103 hierarchical.cpp:1286] Performed allocation for 1 agents in 396177ns I0504 15:43:16.367655 32085 hierarchical.cpp:1694] No allocations performed I0504 15:43:16.367746 32085 hierarchical.cpp:1789] No inverse offers to send out! I0504 15:43:16.367839 32085 hierarchical.cpp:1286] Performed allocation for 1 agents in 442656ns I0504 15:43:17.369190 32103 hierarchical.cpp:1694] No allocations performed I0504 15:43:17.369278 32103 hierarchical.cpp:1789] No inverse offers to send out! I0504 15:43:17.369369 32103 hierarchical.cpp:1286] Performed allocation for 1 agents in 409472ns I0504 15:43:18.370553 32096 hierarchical.cpp:1694] No allocations performed I0504 15:43:18.370645 32096 hierarchical.cpp:1789] No inverse offers to send out! I0504 15:43:18.370726 32096 hierarchical.cpp:1286] Performed allocation for 1 agents in 369732ns I0504 15:43:19.371739 32087 hierarchical.cpp:1694] No allocations performed I0504 15:43:19.371831 32087 hierarchical.cpp:1789] No inverse offers to send out! I0504 15:43:19.371904 32087 hierarchical.cpp:1286] Performed allocation for 1 agents in 373744ns I0504 15:43:20.373103 32096 hierarchical.cpp:1694] No allocations performed I0504 15:43:20.373210 32096 hierarchical.cpp:1789] No inverse offers to send out! I0504 15:43:20.373299 32096 hierarchical.cpp:1286] Performed allocation for 1 agents in 410026ns I0504 15:43:20.409860 32088 slave.cpp:4133] Received ping from slave-observer(8)@172.17.0.3:41866 ../../src/tests/command_executor_tests.cpp:280: Failure Failed to wait 15secs for statusRunning I0504 15:43:20.414811 32086 master.cpp:1297] Framework 2075640b-b7dc-44f0-89b5-b0f9af99be7e-0000 (default) at scheduler-16d6a594-fabf-46b8-8d65-3d174b8e2daa@172.17.0.3:41866 disconnected I0504 15:43:20.414896 32086 master.cpp:2904] Disconnecting framework 2075640b-b7dc-44f0-89b5-b0f9af99be7e-0000 (default) at scheduler-16d6a594-fabf-46b8-8d65-3d174b8e2daa@172.17.0.3:41866 I0504 15:43:20.414968 32086 master.cpp:2928] Deactivating framework 2075640b-b7dc-44f0-89b5-b0f9af99be7e-0000 (default) at scheduler-16d6a594-fabf-46b8-8d65-3d174b8e2daa@172.17.0.3:41866 I0504 15:43:20.415062 32086 master.cpp:1310] Giving framework 2075640b-b7dc-44f0-89b5-b0f9af99be7e-0000 (default) at scheduler-16d6a594-fabf-46b8-8d65-3d174b8e2daa@172.17.0.3:41866 0ns to failover I0504 15:43:20.415182 32087 hierarchical.cpp:386] Deactivated framework 2075640b-b7dc-44f0-89b5-b0f9af99be7e-0000 ../../src/tests/command_executor_tests.cpp:269: Failure Actual function call count doesn't match EXPECT_CALL(sched, statusUpdate(_, _))... Expected: to be called twice Actual: never called - unsatisfied and active I0504 15:43:20.415565 32085 master.cpp:6361] Framework failover timeout, removing framework 2075640b-b7dc-44f0-89b5-b0f9af99be7e-0000 (default) at scheduler-16d6a594-fabf-46b8-8d65-3d174b8e2daa@172.17.0.3:41866 I0504 15:43:20.415604 32085 master.cpp:7095] Removing framework 2075640b-b7dc-44f0-89b5-b0f9af99be7e-0000 (default) at scheduler-16d6a594-fabf-46b8-8d65-3d174b8e2daa@172.17.0.3:41866 I0504 15:43:20.415940 32104 slave.cpp:2511] Asked to shut down framework 2075640b-b7dc-44f0-89b5-b0f9af99be7e-0000 by master@172.17.0.3:41866 I0504 15:43:20.415990 32104 slave.cpp:2536] Shutting down framework 2075640b-b7dc-44f0-89b5-b0f9af99be7e-0000 I0504 15:43:20.416119 32085 master.cpp:7640] Updating the state of task 600b509d-caf9-4c9e-a7ec-d0cb49ec2f7a of framework 2075640b-b7dc-44f0-89b5-b0f9af99be7e-0000 (latest state: TASK_KILLED, status update state: TASK_KILLED) I0504 15:43:20.416194 32104 slave.cpp:4860] Shutting down executor '600b509d-caf9-4c9e-a7ec-d0cb49ec2f7a' of framework 2075640b-b7dc-44f0-89b5-b0f9af99be7e-0000 W0504 15:43:20.416224 32104 slave.hpp:863] Unable to send event to executor '600b509d-caf9-4c9e-a7ec-d0cb49ec2f7a' of framework 2075640b-b7dc-44f0-89b5-b0f9af99be7e-0000: unknown connection type ../../src/tests/command_executor_tests.cpp:274: Failure Actual function call count doesn't match EXPECT_CALL(slave, executorTerminated(_, _, _))... Expected: to be called once Actual: never called - unsatisfied and active I0504 15:43:20.417469 32085 master.cpp:7736] Removing task 600b509d-caf9-4c9e-a7ec-d0cb49ec2f7a with resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] of framework 2075640b-b7dc-44f0-89b5-b0f9af99be7e-0000 on agent 2075640b-b7dc-44f0-89b5-b0f9af99be7e-S0 at (1)@172.17.0.3:41866 (41c61dc99119) I0504 15:43:20.417841 32097 hierarchical.cpp:1018] Recovered cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] (total: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000], allocated: {}) on agent 2075640b-b7dc-44f0-89b5-b0f9af99be7e-S0 from framework 2075640b-b7dc-44f0-89b5-b0f9af99be7e-0000 *** Aborted at 1493912600 (unix time) try "date -d @1493912600" if you are using GNU date *** I0504 15:43:20.418335 32096 hierarchical.cpp:337] Removed framework 2075640b-b7dc-44f0-89b5-b0f9af99be7e-0000 PC: @ 0x0 (unknown) *** SIGSEGV (@0x0) received by PID 32064 (TID 0x7fd8a6934700) from PID 0; stack trace: *** @ 0x7fd8b3f2f370 (unknown) On Thu, May 4, 2017 at 3:07 AM, Alex Rukletsov <a...@mesosphere.com> wrote: > Hi all, > > Please vote on releasing the following candidate as Apache Mesos 1.1.2. > > 1.1.2 includes the following: > ------------------------------------------------------------ > -------------------- > ** Bug > * [MESOS-2537] - AC_ARG_ENABLED checks are broken. > * [MESOS-5028] - Copy provisioner cannot replace directory with symlink. > * [MESOS-5172] - Registry puller cannot fetch blobs correctly from http > Redirect 3xx urls. > * [MESOS-6327] - Large docker images causes container launch failures: > Too many levels of symbolic links. > * [MESOS-7057] - Consider using the relink functionality of libprocess in > the executor driver. > * [MESOS-7119] - Mesos master crash while accepting inverse offer. > * [MESOS-7152] - The agent may be flapping after the machine reboots due > to provisioner recover. > * [MESOS-7197] - Requesting tiny amount of CPU crashes master. > * [MESOS-7210] - HTTP health check doesn't work when mesos runs with > --docker_mesos_image. > * [MESOS-7237] - Enabling cgroups_limit_swap can lead to "invalid > argument" error. > * [MESOS-7265] - Containerizer startup may cause sensitive data to leak > into sandbox logs. > * [MESOS-7350] - Failed to pull image from Nexus Registry due to > signature missing. > * [MESOS-7366] - Agent sandbox gc could accidentally delete the entire > persistent volume content. > * [MESOS-7383] - Docker executor logs possibly sensitive parameters. > * [MESOS-7422] - Docker containerizer should not leak possibly sensitive > data to agent log. > > The CHANGELOG for the release is available at: > https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_ > plain;f=CHANGELOG;hb=1.1.2-rc1 > ------------------------------------------------------------ > -------------------- > > The candidate for Mesos 1.1.2 release is available at: > https://dist.apache.org/repos/dist/dev/mesos/1.1.2-rc1/mesos-1.1.2.tar.gz > > The tag to be voted on is 1.1.2-rc1: > https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=1.1.2-rc1 > > The MD5 checksum of the tarball can be found at: > https://dist.apache.org/repos/dist/dev/mesos/1.1.2-rc1/ > mesos-1.1.2.tar.gz.md5 > > The signature of the tarball can be found at: > https://dist.apache.org/repos/dist/dev/mesos/1.1.2-rc1/ > mesos-1.1.2.tar.gz.asc > > The PGP key used to sign the release is here: > https://dist.apache.org/repos/dist/release/mesos/KEYS > > The JAR is up in Maven in a staging repository here: > https://repository.apache.org/content/repositories/orgapachemesos-1188 > > Please vote on releasing this package as Apache Mesos 1.1.2! > > The vote is open until Tue May 9 12:12:12 CEST 2017 and passes if a > majority of at least 3 +1 PMC votes are cast. > > [ ] +1 Release this package as Apache Mesos 1.1.2 > [ ] -1 Do not release this package because ... > > Thanks, > Alex & Till >
