[
https://issues.apache.org/jira/browse/MESOS-6135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15844112#comment-15844112
]
Greg Mann commented on MESOS-6135:
----------------------------------
This same failure mode was observed again yesterday on ASF CI.
> ContainerLoggerTest.LOGROTATE_RotateInSandbox is flaky
> ------------------------------------------------------
>
> Key: MESOS-6135
> URL: https://issues.apache.org/jira/browse/MESOS-6135
> Project: Mesos
> Issue Type: Bug
> Affects Versions: 1.0.1
> Environment: Ubuntu 14, libev, non-SSL
> Reporter: Greg Mann
> Labels: logging, mesosphere
>
> Observed in our internal CI:
> {code}
> [19:53:51] : [Step 10/10] [ RUN ]
> ContainerLoggerTest.LOGROTATE_RotateInSandbox
> [19:53:51]W: [Step 10/10] I0906 19:53:51.460055 23729 cluster.cpp:157]
> Creating default 'local' authorizer
> [19:53:51]W: [Step 10/10] I0906 19:53:51.468907 23729 leveldb.cpp:174]
> Opened db in 8.730166ms
> [19:53:51]W: [Step 10/10] I0906 19:53:51.472470 23729 leveldb.cpp:181]
> Compacted db in 3.544028ms
> [19:53:51]W: [Step 10/10] I0906 19:53:51.472491 23729 leveldb.cpp:196]
> Created db iterator in 3678ns
> [19:53:51]W: [Step 10/10] I0906 19:53:51.472496 23729 leveldb.cpp:202]
> Seeked to beginning of db in 673ns
> [19:53:51]W: [Step 10/10] I0906 19:53:51.472499 23729 leveldb.cpp:271]
> Iterated through 0 keys in the db in 256ns
> [19:53:51]W: [Step 10/10] I0906 19:53:51.472510 23729 replica.cpp:776]
> Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned
> [19:53:51]W: [Step 10/10] I0906 19:53:51.472709 23744 recover.cpp:451]
> Starting replica recovery
> [19:53:51]W: [Step 10/10] I0906 19:53:51.472820 23748 recover.cpp:477]
> Replica is in EMPTY status
> [19:53:51]W: [Step 10/10] I0906 19:53:51.473059 23748 replica.cpp:673]
> Replica in EMPTY status received a broadcasted recover request from
> __req_res__(177)@172.30.2.89:44578
> [19:53:51]W: [Step 10/10] I0906 19:53:51.473146 23746 recover.cpp:197]
> Received a recover response from a replica in EMPTY status
> [19:53:51]W: [Step 10/10] I0906 19:53:51.473234 23745 recover.cpp:568]
> Updating replica status to STARTING
> [19:53:51]W: [Step 10/10] I0906 19:53:51.473629 23747 master.cpp:379]
> Master 6d1b2727-f42d-446b-b2f8-a9f7e7667340 (ip-172-30-2-89.mesosphere.io)
> started on 172.30.2.89:44578
> [19:53:51]W: [Step 10/10] I0906 19:53:51.473644 23747 master.cpp:381] Flags
> at startup: --acls="" --agent_ping_timeout="15secs"
> --agent_reregister_timeout="10mins" --allocation_interval="1secs"
> --allocator="HierarchicalDRF" --authenticate_agents="true"
> --authenticate_frameworks="true" --authenticate_http_frameworks="true"
> --authenticate_http_readonly="true" --authenticate_http_readwrite="true"
> --authenticators="crammd5" --authorizers="local"
> --credentials="/tmp/ceLmd7/credentials" --framework_sorter="drf"
> --help="false" --hostname_lookup="true" --http_authenticators="basic"
> --http_framework_authenticators="basic" --initialize_driver_logging="true"
> --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO"
> --max_agent_ping_timeouts="5" --max_completed_frameworks="50"
> --max_completed_tasks_per_framework="1000" --quiet="false"
> --recovery_agent_removal_limit="100%" --registry="replicated_log"
> --registry_fetch_timeout="1mins" --registry_store_timeout="100secs"
> --registry_strict="true" --root_submissions="true" --user_sorter="drf"
> --version="false" --webui_dir="/usr/local/share/mesos/webui"
> --work_dir="/tmp/ceLmd7/master" --zk_session_timeout="10secs"
> [19:53:51]W: [Step 10/10] I0906 19:53:51.473832 23747 master.cpp:431]
> Master only allowing authenticated frameworks to register
> [19:53:51]W: [Step 10/10] I0906 19:53:51.473844 23747 master.cpp:445]
> Master only allowing authenticated agents to register
> [19:53:51]W: [Step 10/10] I0906 19:53:51.473850 23747 master.cpp:458]
> Master only allowing authenticated HTTP frameworks to register
> [19:53:51]W: [Step 10/10] I0906 19:53:51.473856 23747 credentials.hpp:37]
> Loading credentials for authentication from '/tmp/ceLmd7/credentials'
> [19:53:51]W: [Step 10/10] I0906 19:53:51.473975 23747 master.cpp:503] Using
> default 'crammd5' authenticator
> [19:53:51]W: [Step 10/10] I0906 19:53:51.474028 23747 http.cpp:883] Using
> default 'basic' HTTP authenticator for realm 'mesos-master-readonly'
> [19:53:51]W: [Step 10/10] I0906 19:53:51.474097 23747 http.cpp:883] Using
> default 'basic' HTTP authenticator for realm 'mesos-master-readwrite'
> [19:53:51]W: [Step 10/10] I0906 19:53:51.474161 23747 http.cpp:883] Using
> default 'basic' HTTP authenticator for realm 'mesos-master-scheduler'
> [19:53:51]W: [Step 10/10] I0906 19:53:51.474242 23747 master.cpp:583]
> Authorization enabled
> [19:53:51]W: [Step 10/10] I0906 19:53:51.474308 23744 hierarchical.cpp:149]
> Initialized hierarchical allocator process
> [19:53:51]W: [Step 10/10] I0906 19:53:51.474308 23750
> whitelist_watcher.cpp:77] No whitelist given
> [19:53:51]W: [Step 10/10] I0906 19:53:51.474840 23745 master.cpp:1850]
> Elected as the leading master!
> [19:53:51]W: [Step 10/10] I0906 19:53:51.474850 23745 master.cpp:1551]
> Recovering from registrar
> [19:53:51]W: [Step 10/10] I0906 19:53:51.474906 23749 registrar.cpp:332]
> Recovering registrar
> [19:53:51]W: [Step 10/10] I0906 19:53:51.475926 23746 leveldb.cpp:304]
> Persisting metadata (8 bytes) to leveldb took 2.650889ms
> [19:53:51]W: [Step 10/10] I0906 19:53:51.475941 23746 replica.cpp:320]
> Persisted replica status to STARTING
> [19:53:51]W: [Step 10/10] I0906 19:53:51.476001 23746 recover.cpp:477]
> Replica is in STARTING status
> [19:53:51]W: [Step 10/10] I0906 19:53:51.476291 23748 replica.cpp:673]
> Replica in STARTING status received a broadcasted recover request from
> __req_res__(178)@172.30.2.89:44578
> [19:53:51]W: [Step 10/10] I0906 19:53:51.476382 23745 recover.cpp:197]
> Received a recover response from a replica in STARTING status
> [19:53:51]W: [Step 10/10] I0906 19:53:51.476506 23748 recover.cpp:568]
> Updating replica status to VOTING
> [19:53:51]W: [Step 10/10] I0906 19:53:51.478842 23744 leveldb.cpp:304]
> Persisting metadata (8 bytes) to leveldb took 2273us
> [19:53:51]W: [Step 10/10] I0906 19:53:51.478868 23744 replica.cpp:320]
> Persisted replica status to VOTING
> [19:53:51]W: [Step 10/10] I0906 19:53:51.478919 23744 recover.cpp:582]
> Successfully joined the Paxos group
> [19:53:51]W: [Step 10/10] I0906 19:53:51.478989 23744 recover.cpp:466]
> Recover process terminated
> [19:53:51]W: [Step 10/10] I0906 19:53:51.479161 23746 log.cpp:553]
> Attempting to start the writer
> [19:53:51]W: [Step 10/10] I0906 19:53:51.479598 23748 replica.cpp:493]
> Replica received implicit promise request from
> __req_res__(179)@172.30.2.89:44578 with proposal 1
> [19:53:51]W: [Step 10/10] I0906 19:53:51.481842 23748 leveldb.cpp:304]
> Persisting metadata (8 bytes) to leveldb took 2.221933ms
> [19:53:51]W: [Step 10/10] I0906 19:53:51.481863 23748 replica.cpp:342]
> Persisted promised to 1
> [19:53:51]W: [Step 10/10] I0906 19:53:51.482148 23746 coordinator.cpp:238]
> Coordinator attempting to fill missing positions
> [19:53:51]W: [Step 10/10] I0906 19:53:51.482622 23743 replica.cpp:388]
> Replica received explicit promise request from
> __req_res__(180)@172.30.2.89:44578 for position 0 with proposal 2
> [19:53:51]W: [Step 10/10] I0906 19:53:51.484884 23743 leveldb.cpp:341]
> Persisting action (8 bytes) to leveldb took 2.243019ms
> [19:53:51]W: [Step 10/10] I0906 19:53:51.484900 23743 replica.cpp:708]
> Persisted action NOP at position 0
> [19:53:51]W: [Step 10/10] I0906 19:53:51.485437 23745 replica.cpp:537]
> Replica received write request for position 0 from
> __req_res__(181)@172.30.2.89:44578
> [19:53:51]W: [Step 10/10] I0906 19:53:51.485460 23745 leveldb.cpp:436]
> Reading position from leveldb took 10101ns
> [19:53:51]W: [Step 10/10] I0906 19:53:51.486137 23745 leveldb.cpp:341]
> Persisting action (14 bytes) to leveldb took 665496ns
> [19:53:51]W: [Step 10/10] I0906 19:53:51.486146 23745 replica.cpp:708]
> Persisted action NOP at position 0
> [19:53:51]W: [Step 10/10] I0906 19:53:51.486418 23744 replica.cpp:691]
> Replica received learned notice for position 0 from @0.0.0.0:0
> [19:53:51]W: [Step 10/10] I0906 19:53:51.488697 23744 leveldb.cpp:341]
> Persisting action (16 bytes) to leveldb took 2.262057ms
> [19:53:51]W: [Step 10/10] I0906 19:53:51.488710 23744 replica.cpp:708]
> Persisted action NOP at position 0
> [19:53:51]W: [Step 10/10] I0906 19:53:51.489017 23743 log.cpp:569] Writer
> started with ending position 0
> [19:53:51]W: [Step 10/10] I0906 19:53:51.489305 23750 leveldb.cpp:436]
> Reading position from leveldb took 12279ns
> [19:53:51]W: [Step 10/10] I0906 19:53:51.489542 23749 registrar.cpp:365]
> Successfully fetched the registry (0B) in 14.611968ms
> [19:53:51]W: [Step 10/10] I0906 19:53:51.489579 23749 registrar.cpp:464]
> Applied 1 operations in 3299ns; attempting to update the registry
> [19:53:51]W: [Step 10/10] I0906 19:53:51.489758 23744 log.cpp:577]
> Attempting to append 205 bytes to the log
> [19:53:51]W: [Step 10/10] I0906 19:53:51.489815 23745 coordinator.cpp:348]
> Coordinator attempting to write APPEND action at position 1
> [19:53:51]W: [Step 10/10] I0906 19:53:51.490074 23744 replica.cpp:537]
> Replica received write request for position 1 from
> __req_res__(182)@172.30.2.89:44578
> [19:53:51]W: [Step 10/10] I0906 19:53:51.490757 23744 leveldb.cpp:341]
> Persisting action (224 bytes) to leveldb took 667475ns
> [19:53:51]W: [Step 10/10] I0906 19:53:51.490767 23744 replica.cpp:708]
> Persisted action APPEND at position 1
> [19:53:51]W: [Step 10/10] I0906 19:53:51.491016 23747 replica.cpp:691]
> Replica received learned notice for position 1 from @0.0.0.0:0
> [19:53:51]W: [Step 10/10] I0906 19:53:51.493211 23747 leveldb.cpp:341]
> Persisting action (226 bytes) to leveldb took 2.179043ms
> [19:53:51]W: [Step 10/10] I0906 19:53:51.493224 23747 replica.cpp:708]
> Persisted action APPEND at position 1
> [19:53:51]W: [Step 10/10] I0906 19:53:51.493430 23748 registrar.cpp:509]
> Successfully updated the registry in 3.832832ms
> [19:53:51]W: [Step 10/10] I0906 19:53:51.493512 23748 registrar.cpp:395]
> Successfully recovered registrar
> [19:53:51]W: [Step 10/10] I0906 19:53:51.493535 23750 log.cpp:596]
> Attempting to truncate the log to 1
> [19:53:51]W: [Step 10/10] I0906 19:53:51.493640 23746 coordinator.cpp:348]
> Coordinator attempting to write TRUNCATE action at position 2
> [19:53:51]W: [Step 10/10] I0906 19:53:51.493644 23749 master.cpp:1659]
> Recovered 0 agents from the registry (166B); allowing 10mins for agents to
> re-register
> [19:53:51]W: [Step 10/10] I0906 19:53:51.493677 23744 hierarchical.cpp:176]
> Skipping recovery of hierarchical allocator: nothing to recover
> [19:53:51]W: [Step 10/10] I0906 19:53:51.493969 23743 replica.cpp:537]
> Replica received write request for position 2 from
> __req_res__(183)@172.30.2.89:44578
> [19:53:51]W: [Step 10/10] I0906 19:53:51.494638 23743 leveldb.cpp:341]
> Persisting action (16 bytes) to leveldb took 650007ns
> [19:53:51]W: [Step 10/10] I0906 19:53:51.494652 23743 replica.cpp:708]
> Persisted action TRUNCATE at position 2
> [19:53:51]W: [Step 10/10] I0906 19:53:51.494916 23744 replica.cpp:691]
> Replica received learned notice for position 2 from @0.0.0.0:0
> [19:53:51]W: [Step 10/10] I0906 19:53:51.497236 23744 leveldb.cpp:341]
> Persisting action (18 bytes) to leveldb took 2.303939ms
> [19:53:51]W: [Step 10/10] I0906 19:53:51.497257 23744 leveldb.cpp:399]
> Deleting ~1 keys from leveldb took 9463ns
> [19:53:51]W: [Step 10/10] I0906 19:53:51.497263 23744 replica.cpp:708]
> Persisted action TRUNCATE at position 2
> [19:53:51]W: [Step 10/10] I0906 19:53:51.498100 23729
> containerizer.cpp:221] Using isolation:
> posix/cpu,posix/mem,filesystem/posix,network/cni
> [19:53:51]W: [Step 10/10] I0906 19:53:51.506593 23729
> linux_launcher.cpp:104] Using /sys/fs/cgroup/freezer as the freezer hierarchy
> for the Linux launcher
> [19:53:51]W: [Step 10/10] I0906 19:53:51.507963 23729 cluster.cpp:436]
> Creating default 'local' authorizer
> [19:53:51]W: [Step 10/10] I0906 19:53:51.508497 23747 slave.cpp:205] Mesos
> agent started on (10)@172.30.2.89:44578
> [19:53:51]W: [Step 10/10] I0906 19:53:51.508512 23747 slave.cpp:206] Flags
> at startup: --acls="" --appc_simple_discovery_uri_prefix="http://";
> --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="true"
> --authenticate_http_readwrite="true" --authenticatee="crammd5"
> --authentication_backoff_factor="1secs" --authorizer="local"
> --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false"
> --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false"
> --cgroups_root="mesos" --container_disk_watch_interval="15secs"
> --container_logger="org_apache_mesos_LogrotateContainerLogger"
> --containerizers="mesos"
> --credential="/mnt/teamcity/temp/buildTmp/ContainerLoggerTest_LOGROTATE_RotateInSandbox_6K6XIS/credential"
> --default_role="*" --disk_watch_interval="1mins" --docker="docker"
> --docker_kill_orphans="true" --docker_registry="https://registry-1.docker.io";
> --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock"
> --docker_stop_timeout="0ns" --docker_store_dir="/tmp/mesos/store/docker"
> --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume"
> --enforce_container_disk_quota="false"
> --executor_registration_timeout="1mins"
> --executor_shutdown_grace_period="5secs"
> --fetcher_cache_dir="/mnt/teamcity/temp/buildTmp/ContainerLoggerTest_LOGROTATE_RotateInSandbox_6K6XIS/fetch"
> --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks"
> --gc_disk_headroom="0.1" --hadoop_home="" --help="false"
> --hostname_lookup="true" --http_authenticators="basic"
> --http_command_executor="false"
> --http_credentials="/mnt/teamcity/temp/buildTmp/ContainerLoggerTest_LOGROTATE_RotateInSandbox_6K6XIS/http_credentials"
> --image_provisioner_backend="copy" --initialize_driver_logging="true"
> --isolation="posix/cpu,posix/mem"
> --launcher_dir="/mnt/teamcity/work/4240ba9ddd0997c3/build/src"
> --logbufsecs="0" --logging_level="INFO"
> --oversubscribed_resources_interval="15secs" --perf_duration="10secs"
> --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false"
> --recover="reconnect" --recovery_timeout="15mins"
> --registration_backoff_factor="10ms"
> --resources="cpus:2;gpus:0;mem:1024;disk:1024;ports:[31000-32000]"
> --revocable_cpu_low_priority="true"
> --runtime_dir="/mnt/teamcity/temp/buildTmp/ContainerLoggerTest_LOGROTATE_RotateInSandbox_6K6XIS"
> --sandbox_directory="/mnt/mesos/sandbox" --strict="true"
> --switch_user="true" --systemd_enable_support="true"
> --systemd_runtime_directory="/run/systemd/system" --version="false"
> --work_dir="/mnt/teamcity/temp/buildTmp/ContainerLoggerTest_LOGROTATE_RotateInSandbox_Rv39XZ"
> [19:53:51]W: [Step 10/10] I0906 19:53:51.508811 23747 credentials.hpp:86]
> Loading credential for authentication from
> '/mnt/teamcity/temp/buildTmp/ContainerLoggerTest_LOGROTATE_RotateInSandbox_6K6XIS/credential'
> [19:53:51]W: [Step 10/10] I0906 19:53:51.508883 23747 slave.cpp:343] Agent
> using credential for: test-principal
> [19:53:51]W: [Step 10/10] I0906 19:53:51.508893 23747 credentials.hpp:37]
> Loading credentials for authentication from
> '/mnt/teamcity/temp/buildTmp/ContainerLoggerTest_LOGROTATE_RotateInSandbox_6K6XIS/http_credentials'
> [19:53:51]W: [Step 10/10] I0906 19:53:51.508954 23747 http.cpp:883] Using
> default 'basic' HTTP authenticator for realm 'mesos-agent-readonly'
> [19:53:51]W: [Step 10/10] I0906 19:53:51.508992 23747 http.cpp:883] Using
> default 'basic' HTTP authenticator for realm 'mesos-agent-readwrite'
> [19:53:51]W: [Step 10/10] I0906 19:53:51.509233 23747 slave.cpp:526] Agent
> resources: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000]
> [19:53:51]W: [Step 10/10] I0906 19:53:51.509258 23747 slave.cpp:534] Agent
> attributes: [ ]
> [19:53:51]W: [Step 10/10] I0906 19:53:51.509261 23747 slave.cpp:539] Agent
> hostname: ip-172-30-2-89.mesosphere.io
> [19:53:51]W: [Step 10/10] I0906 19:53:51.509600 23745 state.cpp:57]
> Recovering state from
> '/mnt/teamcity/temp/buildTmp/ContainerLoggerTest_LOGROTATE_RotateInSandbox_Rv39XZ/meta'
> [19:53:51]W: [Step 10/10] I0906 19:53:51.509735 23748
> status_update_manager.cpp:203] Recovering status update manager
> [19:53:51]W: [Step 10/10] I0906 19:53:51.509821 23744
> containerizer.cpp:568] Recovering containerizer
> [19:53:51]W: [Step 10/10] I0906 19:53:51.510553 23743 provisioner.cpp:254]
> Provisioner recovery complete
> [19:53:51]W: [Step 10/10] I0906 19:53:51.510684 23749 slave.cpp:4887]
> Finished recovery
> [19:53:51]W: [Step 10/10] I0906 19:53:51.510848 23749 slave.cpp:5059]
> Querying resource estimator for oversubscribable resources
> [19:53:51]W: [Step 10/10] I0906 19:53:51.510987 23743
> status_update_manager.cpp:177] Pausing sending status updates
> [19:53:51]W: [Step 10/10] I0906 19:53:51.510984 23747 slave.cpp:902] New
> master detected at [email protected]:44578
> [19:53:51]W: [Step 10/10] I0906 19:53:51.511018 23747 slave.cpp:961]
> Authenticating with master [email protected]:44578
> [19:53:51]W: [Step 10/10] I0906 19:53:51.511034 23747 slave.cpp:972] Using
> default CRAM-MD5 authenticatee
> [19:53:51]W: [Step 10/10] I0906 19:53:51.511075 23747 slave.cpp:934]
> Detecting new master
> [19:53:51]W: [Step 10/10] I0906 19:53:51.511099 23749
> authenticatee.cpp:121] Creating new client SASL connection
> [19:53:51]W: [Step 10/10] I0906 19:53:51.511181 23747 slave.cpp:5073]
> Received oversubscribable resources from the resource estimator
> [19:53:51]W: [Step 10/10] I0906 19:53:51.511318 23749 master.cpp:6167]
> Authenticating slave(10)@172.30.2.89:44578
> [19:53:51]W: [Step 10/10] I0906 19:53:51.511368 23749
> authenticator.cpp:414] Starting authentication session for
> crammd5-authenticatee(34)@172.30.2.89:44578
> [19:53:51]W: [Step 10/10] I0906 19:53:51.511490 23743 authenticator.cpp:98]
> Creating new server SASL connection
> [19:53:51]W: [Step 10/10] I0906 19:53:51.511656 23743
> authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5
> [19:53:51]W: [Step 10/10] I0906 19:53:51.511670 23743
> authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5'
> [19:53:51]W: [Step 10/10] I0906 19:53:51.511709 23743
> authenticator.cpp:204] Received SASL authentication start
> [19:53:51]W: [Step 10/10] I0906 19:53:51.511735 23743
> authenticator.cpp:326] Authentication requires more steps
> [19:53:51]W: [Step 10/10] I0906 19:53:51.511770 23743
> authenticatee.cpp:259] Received SASL authentication step
> [19:53:51]W: [Step 10/10] I0906 19:53:51.511821 23743
> authenticator.cpp:232] Received SASL authentication step
> [19:53:51]W: [Step 10/10] I0906 19:53:51.511838 23743 auxprop.cpp:109]
> Request to lookup properties for user: 'test-principal' realm:
> 'ip-172-30-2-89.mesosphere.io' server FQDN: 'ip-172-30-2-89.mesosphere.io'
> SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false
> SASL_AUXPROP_AUTHZID: false
> [19:53:51]W: [Step 10/10] I0906 19:53:51.511844 23743 auxprop.cpp:181]
> Looking up auxiliary property '*userPassword'
> [19:53:51]W: [Step 10/10] I0906 19:53:51.511852 23743 auxprop.cpp:181]
> Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
> [19:53:51]W: [Step 10/10] I0906 19:53:51.511857 23743 auxprop.cpp:109]
> Request to lookup properties for user: 'test-principal' realm:
> 'ip-172-30-2-89.mesosphere.io' server FQDN: 'ip-172-30-2-89.mesosphere.io'
> SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false
> SASL_AUXPROP_AUTHZID: true
> [19:53:51]W: [Step 10/10] I0906 19:53:51.511862 23743 auxprop.cpp:131]
> Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
> [19:53:51]W: [Step 10/10] I0906 19:53:51.511864 23743 auxprop.cpp:131]
> Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since
> SASL_AUXPROP_AUTHZID == true
> [19:53:51]W: [Step 10/10] I0906 19:53:51.511873 23743
> authenticator.cpp:318] Authentication success
> [19:53:51]W: [Step 10/10] I0906 19:53:51.511916 23749
> authenticatee.cpp:299] Authentication success
> [19:53:51]W: [Step 10/10] I0906 19:53:51.511937 23743 master.cpp:6197]
> Successfully authenticated principal 'test-principal' at
> slave(10)@172.30.2.89:44578
> [19:53:51]W: [Step 10/10] I0906 19:53:51.511970 23744
> authenticator.cpp:432] Authentication session cleanup for
> crammd5-authenticatee(34)@172.30.2.89:44578
> [19:53:51]W: [Step 10/10] I0906 19:53:51.512110 23749 slave.cpp:1056]
> Successfully authenticated with master [email protected]:44578
> [19:53:51]W: [Step 10/10] I0906 19:53:51.512187 23749 slave.cpp:1462] Will
> retry registration in 7.351616ms if necessary
> [19:53:51]W: [Step 10/10] I0906 19:53:51.512264 23747 master.cpp:4819]
> Registering agent at slave(10)@172.30.2.89:44578
> (ip-172-30-2-89.mesosphere.io) with id 6d1b2727-f42d-446b-b2f8-a9f7e7667340-S0
> [19:53:51]W: [Step 10/10] I0906 19:53:51.512388 23746 registrar.cpp:464]
> Applied 1 operations in 21440ns; attempting to update the registry
> [19:53:51]W: [Step 10/10] I0906 19:53:51.512578 23749 log.cpp:577]
> Attempting to append 390 bytes to the log
> [19:53:51]W: [Step 10/10] I0906 19:53:51.512627 23743 coordinator.cpp:348]
> Coordinator attempting to write APPEND action at position 3
> [19:53:51]W: [Step 10/10] I0906 19:53:51.512907 23744 replica.cpp:537]
> Replica received write request for position 3 from
> __req_res__(184)@172.30.2.89:44578
> [19:53:51]W: [Step 10/10] I0906 19:53:51.513802 23744 leveldb.cpp:341]
> Persisting action (409 bytes) to leveldb took 875033ns
> [19:53:51]W: [Step 10/10] I0906 19:53:51.513813 23744 replica.cpp:708]
> Persisted action APPEND at position 3
> [19:53:51]W: [Step 10/10] I0906 19:53:51.514133 23744 replica.cpp:691]
> Replica received learned notice for position 3 from @0.0.0.0:0
> [19:53:51]W: [Step 10/10] I0906 19:53:51.516587 23744 leveldb.cpp:341]
> Persisting action (411 bytes) to leveldb took 2.432739ms
> [19:53:51]W: [Step 10/10] I0906 19:53:51.516607 23744 replica.cpp:708]
> Persisted action APPEND at position 3
> [19:53:51]W: [Step 10/10] I0906 19:53:51.516921 23750 registrar.cpp:509]
> Successfully updated the registry in 4.501248ms
> [19:53:51]W: [Step 10/10] I0906 19:53:51.516964 23743 log.cpp:596]
> Attempting to truncate the log to 3
> [19:53:51]W: [Step 10/10] I0906 19:53:51.517045 23746 coordinator.cpp:348]
> Coordinator attempting to write TRUNCATE action at position 4
> [19:53:51]W: [Step 10/10] I0906 19:53:51.517127 23745 master.cpp:4889]
> Registered agent 6d1b2727-f42d-446b-b2f8-a9f7e7667340-S0 at
> slave(10)@172.30.2.89:44578 (ip-172-30-2-89.mesosphere.io) with cpus(*):2;
> mem(*):1024; disk(*):1024; ports(*):[31000-32000]
> [19:53:51]W: [Step 10/10] I0906 19:53:51.517148 23744 slave.cpp:3754]
> Received ping from slave-observer(9)@172.30.2.89:44578
> [19:53:51]W: [Step 10/10] I0906 19:53:51.517236 23746 hierarchical.cpp:476]
> Added agent 6d1b2727-f42d-446b-b2f8-a9f7e7667340-S0
> (ip-172-30-2-89.mesosphere.io) with cpus(*):2; mem(*):1024; disk(*):1024;
> ports(*):[31000-32000] (allocated: )
> [19:53:51]W: [Step 10/10] I0906 19:53:51.517287 23746
> hierarchical.cpp:1550] No allocations performed
> [19:53:51]W: [Step 10/10] I0906 19:53:51.517313 23746
> hierarchical.cpp:1217] Performed allocation for agent
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-S0 in 50937ns
> [19:53:51]W: [Step 10/10] I0906 19:53:51.517287 23744 slave.cpp:1102]
> Registered with master [email protected]:44578; given agent ID
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-S0
> [19:53:51]W: [Step 10/10] I0906 19:53:51.517343 23744 fetcher.cpp:86]
> Clearing fetcher cache
> [19:53:51]W: [Step 10/10] I0906 19:53:51.517403 23750 replica.cpp:537]
> Replica received write request for position 4 from
> __req_res__(185)@172.30.2.89:44578
> [19:53:51]W: [Step 10/10] I0906 19:53:51.517416 23746
> status_update_manager.cpp:184] Resuming sending status updates
> [19:53:51]W: [Step 10/10] I0906 19:53:51.517513 23744 slave.cpp:1125]
> Checkpointing SlaveInfo to
> '/mnt/teamcity/temp/buildTmp/ContainerLoggerTest_LOGROTATE_RotateInSandbox_Rv39XZ/meta/slaves/6d1b2727-f42d-446b-b2f8-a9f7e7667340-S0/slave.info'
> [19:53:51]W: [Step 10/10] I0906 19:53:51.517623 23744 slave.cpp:1162]
> Forwarding total oversubscribed resources
> [19:53:51]W: [Step 10/10] I0906 19:53:51.517675 23744 master.cpp:5273]
> Received update of agent 6d1b2727-f42d-446b-b2f8-a9f7e7667340-S0 at
> slave(10)@172.30.2.89:44578 (ip-172-30-2-89.mesosphere.io) with total
> oversubscribed resources
> [19:53:51]W: [Step 10/10] I0906 19:53:51.517781 23747 hierarchical.cpp:540]
> Agent 6d1b2727-f42d-446b-b2f8-a9f7e7667340-S0 (ip-172-30-2-89.mesosphere.io)
> updated with oversubscribed resources (total: cpus(*):2; mem(*):1024;
> disk(*):1024; ports(*):[31000-32000], allocated: )
> [19:53:51]W: [Step 10/10] I0906 19:53:51.517809 23747
> hierarchical.cpp:1550] No allocations performed
> [19:53:51]W: [Step 10/10] I0906 19:53:51.517818 23747
> hierarchical.cpp:1217] Performed allocation for agent
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-S0 in 20735ns
> [19:53:51]W: [Step 10/10] I0906 19:53:51.518247 23729 sched.cpp:226]
> Version: 1.1.0
> [19:53:51]W: [Step 10/10] I0906 19:53:51.518407 23743 sched.cpp:330] New
> master detected at [email protected]:44578
> [19:53:51]W: [Step 10/10] I0906 19:53:51.518429 23743 sched.cpp:396]
> Authenticating with master [email protected]:44578
> [19:53:51]W: [Step 10/10] I0906 19:53:51.518436 23743 sched.cpp:403] Using
> default CRAM-MD5 authenticatee
> [19:53:51]W: [Step 10/10] I0906 19:53:51.518501 23747
> authenticatee.cpp:121] Creating new client SASL connection
> [19:53:51]W: [Step 10/10] I0906 19:53:51.518657 23747 master.cpp:6167]
> Authenticating
> [email protected]:44578
> [19:53:51]W: [Step 10/10] I0906 19:53:51.518707 23749
> authenticator.cpp:414] Starting authentication session for
> crammd5-authenticatee(35)@172.30.2.89:44578
> [19:53:51]W: [Step 10/10] I0906 19:53:51.518766 23749 authenticator.cpp:98]
> Creating new server SASL connection
> [19:53:51]W: [Step 10/10] I0906 19:53:51.518921 23749
> authenticatee.cpp:213] Received SASL authentication mechanisms: CRAM-MD5
> [19:53:51]W: [Step 10/10] I0906 19:53:51.518955 23749
> authenticatee.cpp:239] Attempting to authenticate with mechanism 'CRAM-MD5'
> [19:53:51]W: [Step 10/10] I0906 19:53:51.519013 23748
> authenticator.cpp:204] Received SASL authentication start
> [19:53:51]W: [Step 10/10] I0906 19:53:51.519037 23748
> authenticator.cpp:326] Authentication requires more steps
> [19:53:51]W: [Step 10/10] I0906 19:53:51.519068 23748
> authenticatee.cpp:259] Received SASL authentication step
> [19:53:51]W: [Step 10/10] I0906 19:53:51.519120 23748
> authenticator.cpp:232] Received SASL authentication step
> [19:53:51]W: [Step 10/10] I0906 19:53:51.519137 23748 auxprop.cpp:109]
> Request to lookup properties for user: 'test-principal' realm:
> 'ip-172-30-2-89.mesosphere.io' server FQDN: 'ip-172-30-2-89.mesosphere.io'
> SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false
> SASL_AUXPROP_AUTHZID: false
> [19:53:51]W: [Step 10/10] I0906 19:53:51.519143 23748 auxprop.cpp:181]
> Looking up auxiliary property '*userPassword'
> [19:53:51]W: [Step 10/10] I0906 19:53:51.519150 23748 auxprop.cpp:181]
> Looking up auxiliary property '*cmusaslsecretCRAM-MD5'
> [19:53:51]W: [Step 10/10] I0906 19:53:51.519155 23748 auxprop.cpp:109]
> Request to lookup properties for user: 'test-principal' realm:
> 'ip-172-30-2-89.mesosphere.io' server FQDN: 'ip-172-30-2-89.mesosphere.io'
> SASL_AUXPROP_VERIFY_AGAINST_HASH: false SASL_AUXPROP_OVERRIDE: false
> SASL_AUXPROP_AUTHZID: true
> [19:53:51]W: [Step 10/10] I0906 19:53:51.519158 23748 auxprop.cpp:131]
> Skipping auxiliary property '*userPassword' since SASL_AUXPROP_AUTHZID == true
> [19:53:51]W: [Step 10/10] I0906 19:53:51.519160 23748 auxprop.cpp:131]
> Skipping auxiliary property '*cmusaslsecretCRAM-MD5' since
> SASL_AUXPROP_AUTHZID == true
> [19:53:51]W: [Step 10/10] I0906 19:53:51.519167 23748
> authenticator.cpp:318] Authentication success
> [19:53:51]W: [Step 10/10] I0906 19:53:51.519212 23748
> authenticatee.cpp:299] Authentication success
> [19:53:51]W: [Step 10/10] I0906 19:53:51.519225 23745
> authenticator.cpp:432] Authentication session cleanup for
> crammd5-authenticatee(35)@172.30.2.89:44578
> [19:53:51]W: [Step 10/10] I0906 19:53:51.519248 23749 master.cpp:6197]
> Successfully authenticated principal 'test-principal' at
> [email protected]:44578
> [19:53:51]W: [Step 10/10] I0906 19:53:51.519332 23748 sched.cpp:502]
> Successfully authenticated with master [email protected]:44578
> [19:53:51]W: [Step 10/10] I0906 19:53:51.519342 23748 sched.cpp:820]
> Sending SUBSCRIBE call to [email protected]:44578
> [19:53:51]W: [Step 10/10] I0906 19:53:51.519382 23748 sched.cpp:853] Will
> retry registration in 1.70770977secs if necessary
> [19:53:51]W: [Step 10/10] I0906 19:53:51.519426 23747 master.cpp:2424]
> Received SUBSCRIBE call for framework 'default' at
> [email protected]:44578
> [19:53:51]W: [Step 10/10] I0906 19:53:51.519446 23747 master.cpp:1886]
> Authorizing framework principal 'test-principal' to receive offers for role
> '*'
> [19:53:51]W: [Step 10/10] I0906 19:53:51.519567 23748 master.cpp:2500]
> Subscribing framework default with checkpointing disabled and capabilities [
> ]
> [19:53:51]W: [Step 10/10] I0906 19:53:51.519693 23744 hierarchical.cpp:269]
> Added framework 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000
> [19:53:51]W: [Step 10/10] I0906 19:53:51.519758 23748 sched.cpp:743]
> Framework registered with 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000
> [19:53:51]W: [Step 10/10] I0906 19:53:51.519790 23748 sched.cpp:757]
> Scheduler::registered took 20727ns
> [19:53:51]W: [Step 10/10] I0906 19:53:51.519798 23744
> hierarchical.cpp:1645] No inverse offers to send out!
> [19:53:51]W: [Step 10/10] I0906 19:53:51.519822 23744
> hierarchical.cpp:1194] Performed allocation for 1 agents in 116286ns
> [19:53:51]W: [Step 10/10] I0906 19:53:51.519897 23750 leveldb.cpp:341]
> Persisting action (16 bytes) to leveldb took 2.474336ms
> [19:53:51]W: [Step 10/10] I0906 19:53:51.519912 23750 replica.cpp:708]
> Persisted action TRUNCATE at position 4
> [19:53:51]W: [Step 10/10] I0906 19:53:51.520023 23747 master.cpp:5996]
> Sending 1 offers to framework 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000
> (default) at [email protected]:44578
> [19:53:51]W: [Step 10/10] I0906 19:53:51.520211 23748 replica.cpp:691]
> Replica received learned notice for position 4 from @0.0.0.0:0
> [19:53:51]W: [Step 10/10] I0906 19:53:51.520220 23750 sched.cpp:917]
> Scheduler::resourceOffers took 45787ns
> [19:53:51]W: [Step 10/10] I0906 19:53:51.520685 23749 master.cpp:3355]
> Processing ACCEPT call for offers: [ 6d1b2727-f42d-446b-b2f8-a9f7e7667340-O0
> ] on agent 6d1b2727-f42d-446b-b2f8-a9f7e7667340-S0 at
> slave(10)@172.30.2.89:44578 (ip-172-30-2-89.mesosphere.io) for framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000 (default) at
> [email protected]:44578
> [19:53:51]W: [Step 10/10] I0906 19:53:51.520710 23749 master.cpp:2980]
> Authorizing framework principal 'test-principal' to launch task
> 86fbf841-e93f-4995-a72c-7a6cd5a7f2dd
> [19:53:51]W: [Step 10/10] I0906 19:53:51.521106 23746 master.cpp:7736]
> Adding task 86fbf841-e93f-4995-a72c-7a6cd5a7f2dd with resources cpus(*):2;
> mem(*):1024; disk(*):1024; ports(*):[31000-32000] on agent
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-S0 (ip-172-30-2-89.mesosphere.io)
> [19:53:51]W: [Step 10/10] I0906 19:53:51.521152 23746 master.cpp:3911]
> Launching task 86fbf841-e93f-4995-a72c-7a6cd5a7f2dd of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000 (default) at
> [email protected]:44578 with
> resources cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000] on
> agent 6d1b2727-f42d-446b-b2f8-a9f7e7667340-S0 at slave(10)@172.30.2.89:44578
> (ip-172-30-2-89.mesosphere.io)
> [19:53:51]W: [Step 10/10] I0906 19:53:51.521273 23747 slave.cpp:1502] Got
> assigned task 86fbf841-e93f-4995-a72c-7a6cd5a7f2dd for framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000
> [19:53:51]W: [Step 10/10] I0906 19:53:51.521555 23747 slave.cpp:1621]
> Launching task 86fbf841-e93f-4995-a72c-7a6cd5a7f2dd for framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000
> [19:53:51]W: [Step 10/10] I0906 19:53:51.521551 23749 hierarchical.cpp:685]
> Updated allocation of framework 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000 on
> agent 6d1b2727-f42d-446b-b2f8-a9f7e7667340-S0 from cpus(*):2; mem(*):1024;
> disk(*):1024; ports(*):[31000-32000] to cpus(*):2; mem(*):1024; disk(*):1024;
> ports(*):[31000-32000]
> [19:53:51]W: [Step 10/10] I0906 19:53:51.521811 23747 paths.cpp:536] Trying
> to chown
> '/mnt/teamcity/temp/buildTmp/ContainerLoggerTest_LOGROTATE_RotateInSandbox_Rv39XZ/slaves/6d1b2727-f42d-446b-b2f8-a9f7e7667340-S0/frameworks/6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000/executors/86fbf841-e93f-4995-a72c-7a6cd5a7f2dd/runs/e637fed1-ded9-4038-ac0f-915864195bf6'
> to user 'root'
> [19:53:51]W: [Step 10/10] I0906 19:53:51.522692 23748 leveldb.cpp:341]
> Persisting action (18 bytes) to leveldb took 2.443692ms
> [19:53:51]W: [Step 10/10] I0906 19:53:51.522761 23748 leveldb.cpp:399]
> Deleting ~2 keys from leveldb took 35372ns
> [19:53:51]W: [Step 10/10] I0906 19:53:51.522778 23748 replica.cpp:708]
> Persisted action TRUNCATE at position 4
> [19:53:51]W: [Step 10/10] I0906 19:53:51.524709 23747 slave.cpp:5779]
> Launching executor '86fbf841-e93f-4995-a72c-7a6cd5a7f2dd' of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000 with resources cpus(*):0.1;
> mem(*):32 in work directory
> '/mnt/teamcity/temp/buildTmp/ContainerLoggerTest_LOGROTATE_RotateInSandbox_Rv39XZ/slaves/6d1b2727-f42d-446b-b2f8-a9f7e7667340-S0/frameworks/6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000/executors/86fbf841-e93f-4995-a72c-7a6cd5a7f2dd/runs/e637fed1-ded9-4038-ac0f-915864195bf6'
> [19:53:51]W: [Step 10/10] I0906 19:53:51.524901 23747 slave.cpp:1847]
> Queuing task '86fbf841-e93f-4995-a72c-7a6cd5a7f2dd' for executor
> '86fbf841-e93f-4995-a72c-7a6cd5a7f2dd' of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000
> [19:53:51]W: [Step 10/10] I0906 19:53:51.524904 23749
> containerizer.cpp:828] Starting container
> e637fed1-ded9-4038-ac0f-915864195bf6 for executor
> '86fbf841-e93f-4995-a72c-7a6cd5a7f2dd' of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000
> [19:53:51]W: [Step 10/10] I0906 19:53:51.524953 23747 slave.cpp:855]
> Successfully attached file
> '/mnt/teamcity/temp/buildTmp/ContainerLoggerTest_LOGROTATE_RotateInSandbox_Rv39XZ/slaves/6d1b2727-f42d-446b-b2f8-a9f7e7667340-S0/frameworks/6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000/executors/86fbf841-e93f-4995-a72c-7a6cd5a7f2dd/runs/e637fed1-ded9-4038-ac0f-915864195bf6'
> [19:53:51]W: [Step 10/10] I0906 19:53:51.527631 23746
> containerizer.cpp:1239] Launching 'mesos-containerizer' with flags
> '--command="{"arguments":["mesos-executor","--launcher_dir=\/mnt\/teamcity\/work\/4240ba9ddd0997c3\/build\/src"],"shell":false,"value":"\/mnt\/teamcity\/work\/4240ba9ddd0997c3\/build\/src\/mesos-executor"}"
> --help="false" --pipe_read="8" --pipe_write="13" --pre_exec_commands="[]"
> --unshare_namespace_mnt="false" --user="root"
> --working_directory="/mnt/teamcity/temp/buildTmp/ContainerLoggerTest_LOGROTATE_RotateInSandbox_Rv39XZ/slaves/6d1b2727-f42d-446b-b2f8-a9f7e7667340-S0/frameworks/6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000/executors/86fbf841-e93f-4995-a72c-7a6cd5a7f2dd/runs/e637fed1-ded9-4038-ac0f-915864195bf6"'
> [19:53:51]W: [Step 10/10] I0906 19:53:51.527694 23746
> linux_launcher.cpp:284] Cloning child process with flags =
> [19:53:51]W: [Step 10/10] I0906 19:53:51.598599 23748 slave.cpp:2888] Got
> registration for executor '86fbf841-e93f-4995-a72c-7a6cd5a7f2dd' of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000 from executor(1)@172.30.2.89:48161
> [19:53:51]W: [Step 10/10] I0906 19:53:51.599298 23750 slave.cpp:2012]
> Sending queued task '86fbf841-e93f-4995-a72c-7a6cd5a7f2dd' to executor
> '86fbf841-e93f-4995-a72c-7a6cd5a7f2dd' of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000 at executor(1)@172.30.2.89:48161
> [19:53:51]W: [Step 10/10] I0906 19:53:51.603513 23750 slave.cpp:3271]
> Handling status update TASK_RUNNING (UUID:
> ef933446-f6c5-4978-9471-62b82e13a689) for task
> 86fbf841-e93f-4995-a72c-7a6cd5a7f2dd of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000 from executor(1)@172.30.2.89:48161
> [19:53:51]W: [Step 10/10] I0906 19:53:51.604051 23746
> status_update_manager.cpp:323] Received status update TASK_RUNNING (UUID:
> ef933446-f6c5-4978-9471-62b82e13a689) for task
> 86fbf841-e93f-4995-a72c-7a6cd5a7f2dd of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000
> [19:53:51]W: [Step 10/10] I0906 19:53:51.604069 23746
> status_update_manager.cpp:500] Creating StatusUpdate stream for task
> 86fbf841-e93f-4995-a72c-7a6cd5a7f2dd of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000
> [19:53:51]W: [Step 10/10] I0906 19:53:51.604217 23746
> status_update_manager.cpp:377] Forwarding update TASK_RUNNING (UUID:
> ef933446-f6c5-4978-9471-62b82e13a689) for task
> 86fbf841-e93f-4995-a72c-7a6cd5a7f2dd of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000 to the agent
> [19:53:51]W: [Step 10/10] I0906 19:53:51.604322 23743 slave.cpp:3672]
> Forwarding the update TASK_RUNNING (UUID:
> ef933446-f6c5-4978-9471-62b82e13a689) for task
> 86fbf841-e93f-4995-a72c-7a6cd5a7f2dd of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000 to [email protected]:44578
> [19:53:51]W: [Step 10/10] I0906 19:53:51.604390 23743 slave.cpp:3566]
> Status update manager successfully handled status update TASK_RUNNING (UUID:
> ef933446-f6c5-4978-9471-62b82e13a689) for task
> 86fbf841-e93f-4995-a72c-7a6cd5a7f2dd of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000
> [19:53:51]W: [Step 10/10] I0906 19:53:51.604411 23743 slave.cpp:3582]
> Sending acknowledgement for status update TASK_RUNNING (UUID:
> ef933446-f6c5-4978-9471-62b82e13a689) for task
> 86fbf841-e93f-4995-a72c-7a6cd5a7f2dd of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000 to executor(1)@172.30.2.89:48161
> [19:53:51]W: [Step 10/10] I0906 19:53:51.604457 23750 master.cpp:5408]
> Status update TASK_RUNNING (UUID: ef933446-f6c5-4978-9471-62b82e13a689) for
> task 86fbf841-e93f-4995-a72c-7a6cd5a7f2dd of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000 from agent
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-S0 at slave(10)@172.30.2.89:44578
> (ip-172-30-2-89.mesosphere.io)
> [19:53:51]W: [Step 10/10] I0906 19:53:51.604487 23750 master.cpp:5470]
> Forwarding status update TASK_RUNNING (UUID:
> ef933446-f6c5-4978-9471-62b82e13a689) for task
> 86fbf841-e93f-4995-a72c-7a6cd5a7f2dd of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000
> [19:53:51]W: [Step 10/10] I0906 19:53:51.604557 23750 master.cpp:7114]
> Updating the state of task 86fbf841-e93f-4995-a72c-7a6cd5a7f2dd of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000 (latest state: TASK_RUNNING, status
> update state: TASK_RUNNING)
> [19:53:51]W: [Step 10/10] I0906 19:53:51.604676 23750 sched.cpp:1025]
> Scheduler::statusUpdate took 49191ns
> [19:53:51]W: [Step 10/10] I0906 19:53:51.604796 23750 master.cpp:4531]
> Processing ACKNOWLEDGE call ef933446-f6c5-4978-9471-62b82e13a689 for task
> 86fbf841-e93f-4995-a72c-7a6cd5a7f2dd of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000 (default) at
> [email protected]:44578 on agent
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-S0
> [19:53:51]W: [Step 10/10] I0906 19:53:51.604964 23750
> status_update_manager.cpp:395] Received status update acknowledgement (UUID:
> ef933446-f6c5-4978-9471-62b82e13a689) for task
> 86fbf841-e93f-4995-a72c-7a6cd5a7f2dd of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000
> [19:53:51]W: [Step 10/10] I0906 19:53:51.605056 23750 slave.cpp:2657]
> Status update manager successfully handled status update acknowledgement
> (UUID: ef933446-f6c5-4978-9471-62b82e13a689) for task
> 86fbf841-e93f-4995-a72c-7a6cd5a7f2dd of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000
> [19:53:51]W: [Step 10/10] WARNING: Logging before InitGoogleLogging() is
> written to STDERR
> [19:53:51]W: [Step 10/10] I0906 19:53:51.990226 23947 process.cpp:2388]
> Overriding default number of worker threads 8, using the value
> LIBPROCESS_NUM_WORKER_THREADS=8 instead
> [19:53:51]W: [Step 10/10] WARNING: Logging before InitGoogleLogging() is
> written to STDERR
> [19:53:51]W: [Step 10/10] I0906 19:53:51.990226 23946 process.cpp:2388]
> Overriding default number of worker threads 8, using the value
> LIBPROCESS_NUM_WORKER_THREADS=8 instead
> [19:53:51]W: [Step 10/10] I0906 19:53:51.991104 23946 process.cpp:1070]
> libprocess is initialized on 172.30.2.89:52014 with 8 worker threads
> [19:53:51]W: [Step 10/10] I0906 19:53:51.991348 23947 process.cpp:1070]
> libprocess is initialized on 172.30.2.89:50572 with 8 worker threads
> [19:53:52]W: [Step 10/10] I0906 19:53:52.474897 23744
> hierarchical.cpp:1550] No allocations performed
> [19:53:52]W: [Step 10/10] I0906 19:53:52.474920 23744
> hierarchical.cpp:1645] No inverse offers to send out!
> [19:53:52]W: [Step 10/10] I0906 19:53:52.474933 23744
> hierarchical.cpp:1194] Performed allocation for 1 agents in 89472ns
> [19:53:52]W: [Step 10/10] I0906 19:53:52.506855 23746 slave.cpp:3271]
> Handling status update TASK_FINISHED (UUID:
> c23db77c-a775-44fb-98f6-7ae2135013db) for task
> 86fbf841-e93f-4995-a72c-7a6cd5a7f2dd of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000 from executor(1)@172.30.2.89:48161
> [19:53:52]W: [Step 10/10] I0906 19:53:52.507603 23743
> status_update_manager.cpp:323] Received status update TASK_FINISHED (UUID:
> c23db77c-a775-44fb-98f6-7ae2135013db) for task
> 86fbf841-e93f-4995-a72c-7a6cd5a7f2dd of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000
> [19:53:52]W: [Step 10/10] I0906 19:53:52.507650 23743
> status_update_manager.cpp:377] Forwarding update TASK_FINISHED (UUID:
> c23db77c-a775-44fb-98f6-7ae2135013db) for task
> 86fbf841-e93f-4995-a72c-7a6cd5a7f2dd of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000 to the agent
> [19:53:52]W: [Step 10/10] I0906 19:53:52.507740 23750 slave.cpp:3672]
> Forwarding the update TASK_FINISHED (UUID:
> c23db77c-a775-44fb-98f6-7ae2135013db) for task
> 86fbf841-e93f-4995-a72c-7a6cd5a7f2dd of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000 to [email protected]:44578
> [19:53:52]W: [Step 10/10] I0906 19:53:52.507849 23750 slave.cpp:3566]
> Status update manager successfully handled status update TASK_FINISHED (UUID:
> c23db77c-a775-44fb-98f6-7ae2135013db) for task
> 86fbf841-e93f-4995-a72c-7a6cd5a7f2dd of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000
> [19:53:52]W: [Step 10/10] I0906 19:53:52.507863 23750 slave.cpp:3582]
> Sending acknowledgement for status update TASK_FINISHED (UUID:
> c23db77c-a775-44fb-98f6-7ae2135013db) for task
> 86fbf841-e93f-4995-a72c-7a6cd5a7f2dd of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000 to executor(1)@172.30.2.89:48161
> [19:53:52]W: [Step 10/10] I0906 19:53:52.507868 23744 master.cpp:5408]
> Status update TASK_FINISHED (UUID: c23db77c-a775-44fb-98f6-7ae2135013db) for
> task 86fbf841-e93f-4995-a72c-7a6cd5a7f2dd of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000 from agent
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-S0 at slave(10)@172.30.2.89:44578
> (ip-172-30-2-89.mesosphere.io)
> [19:53:52]W: [Step 10/10] I0906 19:53:52.507892 23744 master.cpp:5470]
> Forwarding status update TASK_FINISHED (UUID:
> c23db77c-a775-44fb-98f6-7ae2135013db) for task
> 86fbf841-e93f-4995-a72c-7a6cd5a7f2dd of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000
> [19:53:52]W: [Step 10/10] I0906 19:53:52.507943 23744 master.cpp:7114]
> Updating the state of task 86fbf841-e93f-4995-a72c-7a6cd5a7f2dd of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000 (latest state: TASK_FINISHED,
> status update state: TASK_FINISHED)
> [19:53:52]W: [Step 10/10] I0906 19:53:52.508044 23747 sched.cpp:1025]
> Scheduler::statusUpdate took 36993ns
> [19:53:52]W: [Step 10/10] I0906 19:53:52.508149 23744 master.cpp:4531]
> Processing ACKNOWLEDGE call c23db77c-a775-44fb-98f6-7ae2135013db for task
> 86fbf841-e93f-4995-a72c-7a6cd5a7f2dd of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000 (default) at
> [email protected]:44578 on agent
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-S0
> [19:53:52]W: [Step 10/10] I0906 19:53:52.508157 23748 hierarchical.cpp:926]
> Recovered cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000]
> (total: cpus(*):2; mem(*):1024; disk(*):1024; ports(*):[31000-32000],
> allocated: ) on agent 6d1b2727-f42d-446b-b2f8-a9f7e7667340-S0 from framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000
> [19:53:52]W: [Step 10/10] I0906 19:53:52.508167 23744 master.cpp:7210]
> Removing task 86fbf841-e93f-4995-a72c-7a6cd5a7f2dd with resources cpus(*):2;
> mem(*):1024; disk(*):1024; ports(*):[31000-32000] of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000 on agent
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-S0 at slave(10)@172.30.2.89:44578
> (ip-172-30-2-89.mesosphere.io)
> [19:53:52]W: [Step 10/10] I0906 19:53:52.508307 23746
> status_update_manager.cpp:395] Received status update acknowledgement (UUID:
> c23db77c-a775-44fb-98f6-7ae2135013db) for task
> 86fbf841-e93f-4995-a72c-7a6cd5a7f2dd of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000
> [19:53:52]W: [Step 10/10] I0906 19:53:52.508358 23746
> status_update_manager.cpp:531] Cleaning up status update stream for task
> 86fbf841-e93f-4995-a72c-7a6cd5a7f2dd of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000
> [19:53:52]W: [Step 10/10] I0906 19:53:52.508388 23729 sched.cpp:1987] Asked
> to stop the driver
> [19:53:52]W: [Step 10/10] I0906 19:53:52.508435 23744 sched.cpp:1187]
> Stopping framework 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000
> [19:53:52]W: [Step 10/10] I0906 19:53:52.508507 23743 master.cpp:6572]
> Processing TEARDOWN call for framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000 (default) at
> [email protected]:44578
> [19:53:52]W: [Step 10/10] I0906 19:53:52.508509 23750 slave.cpp:2657]
> Status update manager successfully handled status update acknowledgement
> (UUID: c23db77c-a775-44fb-98f6-7ae2135013db) for task
> 86fbf841-e93f-4995-a72c-7a6cd5a7f2dd of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000
> [19:53:52]W: [Step 10/10] I0906 19:53:52.508527 23743 master.cpp:6584]
> Removing framework 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000 (default) at
> [email protected]:44578
> [19:53:52]W: [Step 10/10] I0906 19:53:52.508532 23750 slave.cpp:6127]
> Completing task 86fbf841-e93f-4995-a72c-7a6cd5a7f2dd
> [19:53:52]W: [Step 10/10] I0906 19:53:52.510607 23745 hierarchical.cpp:380]
> Deactivated framework 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000
> [19:53:52]W: [Step 10/10] I0906 19:53:52.510612 23750 slave.cpp:2225] Asked
> to shut down framework 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000 by
> [email protected]:44578
> [19:53:52]W: [Step 10/10] I0906 19:53:52.510633 23750 slave.cpp:2250]
> Shutting down framework 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000
> [19:53:52]W: [Step 10/10] I0906 19:53:52.510645 23750 slave.cpp:4475]
> Shutting down executor '86fbf841-e93f-4995-a72c-7a6cd5a7f2dd' of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000 at executor(1)@172.30.2.89:48161
> [19:53:52]W: [Step 10/10] I0906 19:53:52.510679 23745 hierarchical.cpp:331]
> Removed framework 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000
> [19:53:52]W: [Step 10/10] I0906 19:53:52.514551 23749
> hierarchical.cpp:1550] No allocations performed
> [19:53:52]W: [Step 10/10] I0906 19:53:52.514575 23749
> hierarchical.cpp:1194] Performed allocation for 1 agents in 93321ns
> [19:53:52]W: [Step 10/10] I0906 19:53:52.614637 23746
> hierarchical.cpp:1550] No allocations performed
> [19:53:52]W: [Step 10/10] I0906 19:53:52.614660 23746
> hierarchical.cpp:1194] Performed allocation for 1 agents in 88246ns
> [19:53:52]W: [Step 10/10] I0906 19:53:52.714848 23747
> hierarchical.cpp:1550] No allocations performed
> [19:53:52]W: [Step 10/10] I0906 19:53:52.714865 23747
> hierarchical.cpp:1194] Performed allocation for 1 agents in 36499ns
> [19:53:52]W: [Step 10/10] I0906 19:53:52.814941 23744
> hierarchical.cpp:1550] No allocations performed
> [19:53:52]W: [Step 10/10] I0906 19:53:52.814956 23744
> hierarchical.cpp:1194] Performed allocation for 1 agents in 40007ns
> [19:53:52]W: [Step 10/10] I0906 19:53:52.915081 23745 slave.cpp:4548]
> Killing executor '86fbf841-e93f-4995-a72c-7a6cd5a7f2dd' of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000 at executor(1)@172.30.2.89:48161
> [19:53:52]W: [Step 10/10] I0906 19:53:52.915137 23743
> hierarchical.cpp:1550] No allocations performed
> [19:53:52]W: [Step 10/10] I0906 19:53:52.915149 23745
> containerizer.cpp:1599] Destroying container
> e637fed1-ded9-4038-ac0f-915864195bf6
> [19:53:52]W: [Step 10/10] I0906 19:53:52.915158 23743
> hierarchical.cpp:1194] Performed allocation for 1 agents in 67679ns
> [19:53:52]W: [Step 10/10] I0906 19:53:52.916584 23746 cgroups.cpp:2678]
> Freezing cgroup
> /sys/fs/cgroup/freezer/mesos/e637fed1-ded9-4038-ac0f-915864195bf6
> [19:53:52]W: [Step 10/10] I0906 19:53:52.917534 23743 cgroups.cpp:1412]
> Successfully froze cgroup
> /sys/fs/cgroup/freezer/mesos/e637fed1-ded9-4038-ac0f-915864195bf6 after
> 925952ns
> [19:53:52]W: [Step 10/10] I0906 19:53:52.918895 23745 cgroups.cpp:2696]
> Thawing cgroup
> /sys/fs/cgroup/freezer/mesos/e637fed1-ded9-4038-ac0f-915864195bf6
> [19:53:52]W: [Step 10/10] I0906 19:53:52.919890 23744 cgroups.cpp:1441]
> Successfully thawed cgroup
> /sys/fs/cgroup/freezer/mesos/e637fed1-ded9-4038-ac0f-915864195bf6 after
> 977152ns
> [19:53:52]W: [Step 10/10] I0906 19:53:52.921178 23744 slave.cpp:3800] Got
> exited event for executor(1)@172.30.2.89:48161
> [19:53:53]W: [Step 10/10] I0906 19:53:53.015213 23747
> hierarchical.cpp:1550] No allocations performed
> [19:53:53]W: [Step 10/10] I0906 19:53:53.015242 23747
> hierarchical.cpp:1194] Performed allocation for 1 agents in 62640ns
> [19:53:53]W: [Step 10/10] I0906 19:53:53.015311 23745
> containerizer.cpp:1823] Executor for container
> e637fed1-ded9-4038-ac0f-915864195bf6 has exited
> [19:53:53]W: [Step 10/10] I0906 19:53:53.015323 23745
> containerizer.cpp:1595] Destroy has already been initiated for
> e637fed1-ded9-4038-ac0f-915864195bf6
> [19:53:53]W: [Step 10/10] I0906 19:53:53.016587 23743 provisioner.cpp:414]
> Ignoring destroy request for unknown container
> e637fed1-ded9-4038-ac0f-915864195bf6
> [19:53:53]W: [Step 10/10] I0906 19:53:53.016708 23749 slave.cpp:4157]
> Executor '86fbf841-e93f-4995-a72c-7a6cd5a7f2dd' of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000 terminated with signal Killed
> [19:53:53]W: [Step 10/10] I0906 19:53:53.016731 23749 slave.cpp:4261]
> Cleaning up executor '86fbf841-e93f-4995-a72c-7a6cd5a7f2dd' of framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000 at executor(1)@172.30.2.89:48161
> [19:53:53]W: [Step 10/10] I0906 19:53:53.016863 23749 slave.cpp:4349]
> Cleaning up framework 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000
> [19:53:53]W: [Step 10/10] I0906 19:53:53.016852 23744 gc.cpp:55] Scheduling
> '/mnt/teamcity/temp/buildTmp/ContainerLoggerTest_LOGROTATE_RotateInSandbox_Rv39XZ/slaves/6d1b2727-f42d-446b-b2f8-a9f7e7667340-S0/frameworks/6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000/executors/86fbf841-e93f-4995-a72c-7a6cd5a7f2dd/runs/e637fed1-ded9-4038-ac0f-915864195bf6'
> for gc 6.9999998054637days in the future
> [19:53:53]W: [Step 10/10] I0906 19:53:53.016916 23744 gc.cpp:55] Scheduling
> '/mnt/teamcity/temp/buildTmp/ContainerLoggerTest_LOGROTATE_RotateInSandbox_Rv39XZ/slaves/6d1b2727-f42d-446b-b2f8-a9f7e7667340-S0/frameworks/6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000/executors/86fbf841-e93f-4995-a72c-7a6cd5a7f2dd'
> for gc 6.99999980495407days in the future
> [19:53:53]W: [Step 10/10] I0906 19:53:53.016942 23745
> status_update_manager.cpp:285] Closing status update streams for framework
> 6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000
> [19:53:53]W: [Step 10/10] I0906 19:53:53.016950 23744 gc.cpp:55] Scheduling
> '/mnt/teamcity/temp/buildTmp/ContainerLoggerTest_LOGROTATE_RotateInSandbox_Rv39XZ/slaves/6d1b2727-f42d-446b-b2f8-a9f7e7667340-S0/frameworks/6d1b2727-f42d-446b-b2f8-a9f7e7667340-0000'
> for gc 6.99999980399111days in the future
> [19:53:53] : [Step 10/10] ../../src/tests/container_logger_tests.cpp:505:
> Failure
> [19:53:53] : [Step 10/10] Expected: (1050u) >= (stdoutSize->kilobytes()),
> actual: 1050 vs 1051
> [19:53:53]W: [Step 10/10] I0906 19:53:53.114886 23747 slave.cpp:774] Agent
> terminating
> [19:53:53]W: [Step 10/10] I0906 19:53:53.114982 23748 master.cpp:1249]
> Agent 6d1b2727-f42d-446b-b2f8-a9f7e7667340-S0 at slave(10)@172.30.2.89:44578
> (ip-172-30-2-89.mesosphere.io) disconnected
> [19:53:53]W: [Step 10/10] I0906 19:53:53.114996 23748 master.cpp:2784]
> Disconnecting agent 6d1b2727-f42d-446b-b2f8-a9f7e7667340-S0 at
> slave(10)@172.30.2.89:44578 (ip-172-30-2-89.mesosphere.io)
> [19:53:53]W: [Step 10/10] I0906 19:53:53.115025 23748 master.cpp:2803]
> Deactivating agent 6d1b2727-f42d-446b-b2f8-a9f7e7667340-S0 at
> slave(10)@172.30.2.89:44578 (ip-172-30-2-89.mesosphere.io)
> [19:53:53]W: [Step 10/10] I0906 19:53:53.115083 23745 hierarchical.cpp:569]
> Agent 6d1b2727-f42d-446b-b2f8-a9f7e7667340-S0 deactivated
> [19:53:53]W: [Step 10/10] I0906 19:53:53.116849 23729 master.cpp:1096]
> Master terminating
> [19:53:53]W: [Step 10/10] I0906 19:53:53.116998 23747 hierarchical.cpp:508]
> Removed agent 6d1b2727-f42d-446b-b2f8-a9f7e7667340-S0
> [19:53:53] : [Step 10/10] [ FAILED ]
> ContainerLoggerTest.LOGROTATE_RotateInSandbox (1659 ms)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
