[ https://issues.apache.org/jira/browse/MESOS-3273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14727689#comment-14727689 ]
Vinod Kone commented on MESOS-3273: ----------------------------------- This is still happening :( https://builds.apache.org/job/Mesos/COMPILER=clang,CONFIGURATION=--verbose%20--enable-libevent%20--enable-ssl,OS=ubuntu:14.04,label_exp=docker%7C%7CHadoop/762/consoleFull {code} [ RUN ] ExamplesTest.EventCallFramework II0902 00:30:52.212740 28492 exec.cpp:443] Ignoring exited event because the driver is aborted! I0902 00:30:52.212781 28534 exec.cpp:443] Ignoring exited event because the driver is aborted! 0902 00:30:52.212716 28513 exec.cpp:443] Ignoring exited event because the driver is aborted! I0902 00:30:52.212926 28551 exec.cpp:443] Ignoring exited event because the driver is aborted! Using temporary directory '/tmp/ExamplesTest_EventCallFramework_JN7PJN' I0902 00:30:52.213004 28576 exec.cpp:443] Ignoring exited event because the driver is aborted! Shutting down Sending SIGTERM to process tree at pid 28560 Killing the following process trees: [ ] Shutting down Sending SIGTERM to process tree at pid 28561 Killing the following process trees: [ ] Shutting down Sending SIGTERM to process tree at pid 28562 Shutting down Sending SIGTERM to process tree at pid 28580 Killing the following process trees: [ ] Killing the following process trees: [ ] Shutting down Sending SIGTERM to process tree at pid 28581 Killing the following process trees: [ ] WARNING: Logging before InitGoogleLogging() is written to STDERR I0902 00:30:53.636021 28582 process.cpp:1012] libprocess is initialized on 172.17.3.73:36681 for 16 cpus I0902 00:30:53.636329 28582 logging.cpp:177] Logging to STDERR I0902 00:30:53.637977 28582 scheduler.cpp:157] Version: 0.25.0 I0902 00:30:53.653961 28582 leveldb.cpp:176] Opened db in 4.138502ms I0902 00:30:53.655617 28582 leveldb.cpp:183] Compacted db in 1.629758ms I0902 00:30:53.655752 28582 leveldb.cpp:198] Created db iterator in 80178ns I0902 00:30:53.655781 28582 leveldb.cpp:204] Seeked to beginning of db in 10727ns I0902 00:30:53.655792 28582 leveldb.cpp:273] Iterated through 0 keys in the db in 6510ns I0902 00:30:53.655972 28582 replica.cpp:744] Replica recovered with log positions 0 -> 0 with 1 holes and 0 unlearned I0902 00:30:53.658758 28612 recover.cpp:449] Starting replica recovery I0902 00:30:53.659834 28612 recover.cpp:475] Replica is in EMPTY status I0902 00:30:53.660012 28582 local.cpp:241] Using 'local' authorizer I0902 00:30:53.662570 28605 replica.cpp:641] Replica in EMPTY status received a broadcasted recover request I0902 00:30:53.663496 28611 master.cpp:378] Master 20150902-003053-1224937900-36681-28582 (441f06bc837a) started on 172.17.3.73:36681 I0902 00:30:53.663532 28611 master.cpp:380] Flags at startup: --acls="permissive: false register_frameworks { principals { type: SOME values: "test-principal" } roles { type: SOME values: "*" } } run_tasks { principals { type: SOME values: "test-principal" } users { type: SOME values: "mesos" } } " --allocation_interval="1secs" --allocator="HierarchicalDRF" --authenticate="false" --authenticate_slaves="false" --authenticators="crammd5" --authorizers="local" --credentials="/tmp/ExamplesTest_EventCallFramework_JN7PJN/credentials" --framework_sorter="drf" --help="false" --initialize_driver_logging="true" --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" --max_slave_ping_timeouts="5" --quiet="false" --recovery_slave_removal_limit="100%" --registry="replicated_log" --registry_fetch_timeout="1mins" --registry_store_timeout="5secs" --registry_strict="false" --root_submissions="true" --slave_ping_timeout="15secs" --slave_reregister_timeout="10mins" --user_sorter="drf" --version="false" --webui_dir="/mesos/mesos-0.25.0/src/webui" --work_dir="/tmp/mesos-s1Q2uY" --zk_session_timeout="10secs" I0902 00:30:53.664083 28611 master.cpp:427] Master allowing unauthenticated frameworks to register I0902 00:30:53.664096 28611 master.cpp:432] Master allowing unauthenticated slaves to register I0902 00:30:53.664104 28611 credentials.hpp:37] Loading credentials for authentication from '/tmp/ExamplesTest_EventCallFramework_JN7PJN/credentials' W0902 00:30:53.664187 28611 credentials.hpp:52] Permissions on credentials file '/tmp/ExamplesTest_EventCallFramework_JN7PJN/credentials' are too open. It is recommended that your credentials file is NOT accessible by others. I0902 00:30:53.664290 28610 recover.cpp:195] Received a recover response from a replica in EMPTY status I0902 00:30:53.664329 28611 master.cpp:469] Using default 'crammd5' authenticator I0902 00:30:53.664505 28611 authenticator.cpp:512] Initializing server SASL I0902 00:30:53.664973 28612 recover.cpp:566] Updating replica status to STARTING I0902 00:30:53.665920 28619 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 622741ns I0902 00:30:53.665946 28619 replica.cpp:323] Persisted replica status to STARTING I0902 00:30:53.666154 28611 auxprop.cpp:66] Initialized in-memory auxiliary property plugin I0902 00:30:53.666316 28610 recover.cpp:475] Replica is in STARTING status I0902 00:30:53.666332 28611 master.cpp:506] Authorization enabled I0902 00:30:53.666404 28582 containerizer.cpp:160] Using isolation: filesystem/posix,posix/cpu,posix/mem I0902 00:30:53.666851 28619 whitelist_watcher.cpp:79] No whitelist given I0902 00:30:53.667111 28610 hierarchical.hpp:346] Initialized hierarchical allocator process I0902 00:30:53.667134 28609 replica.cpp:641] Replica in STARTING status received a broadcasted recover request I0902 00:30:53.667367 28609 recover.cpp:195] Received a recover response from a replica in STARTING status I0902 00:30:53.667743 28605 recover.cpp:566] Updating replica status to VOTING I0902 00:30:53.668181 28619 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 300973ns I0902 00:30:53.668211 28619 replica.cpp:323] Persisted replica status to VOTING I0902 00:30:53.669222 28615 recover.cpp:580] Successfully joined the Paxos group I0902 00:30:53.669587 28615 recover.cpp:464] Recover process terminated I0902 00:30:53.671195 28616 slave.cpp:190] Slave started on 1)@172.17.3.73:36681 I0902 00:30:53.671234 28616 slave.cpp:191] Flags at startup: --appc_store_dir="/tmp/mesos/store/appc" --authenticatee="crammd5" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --initialize_driver_logging="true" --isolation="filesystem/posix,posix/cpu,posix/mem" --launcher_dir="/mesos/mesos-0.25.0/_build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --resource_monitoring_interval="1secs" --resources="cpus:2;mem:10240" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --version="false" --work_dir="/tmp/mesos-s1Q2uY/0" I0902 00:30:53.672694 28616 slave.cpp:354] Slave resources: cpus(*):2; mem(*):10240; disk(*):3.70122e+06; ports(*):[31000-32000] I0902 00:30:53.672760 28582 containerizer.cpp:160] Using isolation: filesystem/posix,posix/cpu,posix/mem I0902 00:30:53.672828 28616 slave.cpp:384] Slave hostname: 441f06bc837a I0902 00:30:53.672844 28616 slave.cpp:389] Slave checkpoint: true I0902 00:30:53.674482 28605 master.cpp:1559] The newly elected leader is master@172.17.3.73:36681 with id 20150902-003053-1224937900-36681-28582 I0902 00:30:53.674528 28605 master.cpp:1572] Elected as the leading master! I0902 00:30:53.674546 28605 master.cpp:1332] Recovering from registrar I0902 00:30:53.674584 28612 state.cpp:54] Recovering state from '/tmp/mesos-s1Q2uY/0/meta' I0902 00:30:53.674712 28615 registrar.cpp:311] Recovering registrar I0902 00:30:53.675217 28605 status_update_manager.cpp:202] Recovering status update manager I0902 00:30:53.675644 28618 containerizer.cpp:396] Recovering containerizer I0902 00:30:53.676468 28614 log.cpp:661] Attempting to start the writer I0902 00:30:53.676507 28619 slave.cpp:190] Slave started on 2)@172.17.3.73:36681 I0902 00:30:53.676807 28619 slave.cpp:191] Flags at startup: --appc_store_dir="/tmp/mesos/store/appc" --authenticatee="crammd5" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --initialize_driver_logging="true" --isolation="filesystem/posix,posix/cpu,posix/mem" --launcher_dir="/mesos/mesos-0.25.0/_build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --resource_monitoring_interval="1secs" --resources="cpus:2;mem:10240" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --version="false" --work_dir="/tmp/mesos-s1Q2uY/1" I0902 00:30:53.677983 28619 slave.cpp:354] Slave resources: cpus(*):2; mem(*):10240; disk(*):3.70122e+06; ports(*):[31000-32000] I0902 00:30:53.678058 28611 slave.cpp:4069] Finished recovery I0902 00:30:53.678170 28619 slave.cpp:384] Slave hostname: 441f06bc837a I0902 00:30:53.678203 28619 slave.cpp:389] Slave checkpoint: true I0902 00:30:53.678493 28582 containerizer.cpp:160] Using isolation: filesystem/posix,posix/cpu,posix/mem I0902 00:30:53.678817 28611 slave.cpp:4226] Querying resource estimator for oversubscribable resources I0902 00:30:53.679097 28619 state.cpp:54] Recovering state from '/tmp/mesos-s1Q2uY/1/meta' I0902 00:30:53.679438 28619 status_update_manager.cpp:202] Recovering status update manager I0902 00:30:53.679734 28616 status_update_manager.cpp:176] Pausing sending status updates I0902 00:30:53.679805 28611 slave.cpp:684] New master detected at master@172.17.3.73:36681 I0902 00:30:53.680107 28611 slave.cpp:709] No credentials provided. Attempting to register without authentication I0902 00:30:53.680172 28606 containerizer.cpp:396] Recovering containerizer I0902 00:30:53.680227 28611 slave.cpp:720] Detecting new master I0902 00:30:53.680311 28620 replica.cpp:477] Replica received implicit promise request with proposal 1 I0902 00:30:53.680373 28611 slave.cpp:4240] Received oversubscribable resources from the resource estimator I0902 00:30:53.680917 28620 leveldb.cpp:306] Persisting metadata (8 bytes) to leveldb took 458464ns I0902 00:30:53.681080 28620 replica.cpp:345] Persisted promised to 1 I0902 00:30:53.682653 28613 slave.cpp:4069] Finished recovery I0902 00:30:53.682688 28620 coordinator.cpp:231] Coordinator attemping to fill missing position I0902 00:30:53.683275 28613 slave.cpp:4226] Querying resource estimator for oversubscribable resources I0902 00:30:53.683584 28607 status_update_manager.cpp:176] Pausing sending status updates I0902 00:30:53.683601 28615 slave.cpp:684] New master detected at master@172.17.3.73:36681 I0902 00:30:53.683730 28615 slave.cpp:709] No credentials provided. Attempting to register without authentication I0902 00:30:53.683789 28615 slave.cpp:720] Detecting new master I0902 00:30:53.684020 28615 slave.cpp:4240] Received oversubscribable resources from the resource estimator I0902 00:30:53.684445 28619 slave.cpp:190] Slave started on 3)@172.17.3.73:36681 I0902 00:30:53.684656 28605 replica.cpp:378] Replica received explicit promise request for position 0 with proposal 2 I0902 00:30:53.684484 28619 slave.cpp:191] Flags at startup: --appc_store_dir="/tmp/mesos/store/appc" --authenticatee="crammd5" --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false" --cgroups_root="mesos" --container_disk_watch_interval="15secs" --containerizers="mesos" --default_role="*" --disk_watch_interval="1mins" --docker="docker" --docker_kill_orphans="true" --docker_remove_delay="6hrs" --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" --enforce_container_disk_quota="false" --executor_registration_timeout="1mins" --executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" --hadoop_home="" --help="false" --initialize_driver_logging="true" --isolation="filesystem/posix,posix/cpu,posix/mem" --launcher_dir="/mesos/mesos-0.25.0/_build/src" --logbufsecs="0" --logging_level="INFO" --oversubscribed_resources_interval="15secs" --perf_duration="10secs" --perf_interval="1mins" --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" --recovery_timeout="15mins" --registration_backoff_factor="1secs" --resource_monitoring_interval="1secs" --resources="cpus:2;mem:10240" --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true" --version="false" --work_dir="/tmp/mesos-s1Q2uY/2" I0902 00:30:53.685191 28605 leveldb.cpp:343] Persisting action (8 bytes) to leveldb took 495897ns I0902 00:30:53.685223 28605 replica.cpp:679] Persisted action at 0 I0902 00:30:53.685535 28619 slave.cpp:354] Slave resources: cpus(*):2; mem(*):10240; disk(*):3.70122e+06; ports(*):[31000-32000] I0902 00:30:53.685657 28619 slave.cpp:384] Slave hostname: 441f06bc837a I0902 00:30:53.685678 28619 slave.cpp:389] Slave checkpoint: true I0902 00:30:53.686754 28605 state.cpp:54] Recovering state from '/tmp/mesos-s1Q2uY/2/meta' I0902 00:30:53.687188 28616 status_update_manager.cpp:202] Recovering status update manager I0902 00:30:53.687327 28605 replica.cpp:511] Replica received write request for position 0 I0902 00:30:53.687433 28605 leveldb.cpp:438] Reading position from leveldb took 77420ns I0902 00:30:53.687559 28620 scheduler.cpp:240] New master detected at master@172.17.3.73:36681 I0902 00:30:53.688076 28605 leveldb.cpp:343] Persisting action (14 bytes) to leveldb took 546957ns I0902 00:30:53.688114 28605 replica.cpp:679] Persisted action at 0 I0902 00:30:53.687638 28619 containerizer.cpp:396] Recovering containerizer I0902 00:30:53.689191 28607 replica.cpp:658] Replica received learned notice for position 0 I0902 00:30:53.689815 28607 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 607808ns I0902 00:30:53.689865 28607 replica.cpp:679] Persisted action at 0 I0902 00:30:53.689914 28607 replica.cpp:664] Replica learned NOP action at position 0 I0902 00:30:53.690187 28620 slave.cpp:4069] Finished recovery I0902 00:30:53.690893 28620 slave.cpp:4226] Querying resource estimator for oversubscribable resources I0902 00:30:53.691164 28611 log.cpp:677] Writer started with ending position 0 I0902 00:30:53.691167 28616 status_update_manager.cpp:176] Pausing sending status updates I0902 00:30:53.691202 28620 slave.cpp:684] New master detected at master@172.17.3.73:36681 I0902 00:30:53.691305 28620 slave.cpp:709] No credentials provided. Attempting to register without authentication I0902 00:30:53.691411 28620 slave.cpp:720] Detecting new master I0902 00:30:53.691535 28620 slave.cpp:4240] Received oversubscribable resources from the resource estimator I0902 00:30:53.693686 28609 leveldb.cpp:438] Reading position from leveldb took 55603ns I0902 00:30:53.697943 28611 registrar.cpp:344] Successfully fetched the registry (0B) in 22.782976ms I0902 00:30:53.698328 28611 registrar.cpp:443] Applied 1 operations in 137518ns; attempting to update the 'registry' I0902 00:30:53.701880 28607 log.cpp:685] Attempting to append 176 bytes to the log I0902 00:30:53.702030 28606 coordinator.cpp:341] Coordinator attempting to write APPEND action at position 1 I0902 00:30:53.702842 28605 replica.cpp:511] Replica received write request for position 1 I0902 00:30:53.703675 28605 leveldb.cpp:343] Persisting action (195 bytes) to leveldb took 810079ns I0902 00:30:53.703706 28605 replica.cpp:679] Persisted action at 1 I0902 00:30:53.704337 28608 replica.cpp:658] Replica received learned notice for position 1 I0902 00:30:53.705054 28608 leveldb.cpp:343] Persisting action (197 bytes) to leveldb took 702560ns I0902 00:30:53.705088 28608 replica.cpp:679] Persisted action at 1 I0902 00:30:53.705116 28608 replica.cpp:664] Replica learned APPEND action at position 1 I0902 00:30:53.706897 28608 registrar.cpp:488] Successfully updated the 'registry' in 8.417024ms I0902 00:30:53.707123 28608 registrar.cpp:374] Successfully recovered registrar I0902 00:30:53.707484 28616 log.cpp:704] Attempting to truncate the log to 1 I0902 00:30:53.707660 28608 coordinator.cpp:341] Coordinator attempting to write TRUNCATE action at position 2 I0902 00:30:53.707659 28614 master.cpp:1369] Recovered 0 slaves from the Registry (137B) ; allowing 10mins for slaves to re-register I0902 00:30:53.708447 28612 replica.cpp:511] Replica received write request for position 2 I0902 00:30:53.709228 28612 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 749671ns I0902 00:30:53.709259 28612 replica.cpp:679] Persisted action at 2 I0902 00:30:53.709920 28616 replica.cpp:658] Replica received learned notice for position 2 I0902 00:30:53.710574 28616 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 640764ns I0902 00:30:53.710654 28616 leveldb.cpp:401] Deleting ~1 keys from leveldb took 59488ns I0902 00:30:53.710682 28616 replica.cpp:679] Persisted action at 2 I0902 00:30:53.710728 28616 replica.cpp:664] Replica learned TRUNCATE action at position 2 I0902 00:30:53.741991 28619 slave.cpp:1209] Will retry registration in 1.091568855secs if necessary I0902 00:30:53.742467 28609 master.cpp:3670] Registering slave at slave(3)@172.17.3.73:36681 (441f06bc837a) with id 20150902-003053-1224937900-36681-28582-S0 I0902 00:30:53.743149 28614 registrar.cpp:443] Applied 1 operations in 170710ns; attempting to update the 'registry' I0902 00:30:53.745995 28608 log.cpp:685] Attempting to append 347 bytes to the log I0902 00:30:53.746286 28620 coordinator.cpp:341] Coordinator attempting to write APPEND action at position 3 I0902 00:30:53.747166 28619 replica.cpp:511] Replica received write request for position 3 I0902 00:30:53.747927 28619 leveldb.cpp:343] Persisting action (366 bytes) to leveldb took 738551ns I0902 00:30:53.747967 28619 replica.cpp:679] Persisted action at 3 I0902 00:30:53.748620 28605 replica.cpp:658] Replica received learned notice for position 3 I0902 00:30:53.749323 28605 leveldb.cpp:343] Persisting action (368 bytes) to leveldb took 570424ns I0902 00:30:53.749356 28605 replica.cpp:679] Persisted action at 3 I0902 00:30:53.749390 28605 replica.cpp:664] Replica learned APPEND action at position 3 I0902 00:30:53.750951 28615 registrar.cpp:488] Successfully updated the 'registry' in 7.702016ms I0902 00:30:53.751273 28607 log.cpp:704] Attempting to truncate the log to 3 I0902 00:30:53.751399 28619 coordinator.cpp:341] Coordinator attempting to write TRUNCATE action at position 4 I0902 00:30:53.752230 28614 replica.cpp:511] Replica received write request for position 4 I0902 00:30:53.752382 28609 slave.cpp:3058] Received ping from slave-observer(1)@172.17.3.73:36681 I0902 00:30:53.752697 28616 master.cpp:3733] Registered slave 20150902-003053-1224937900-36681-28582-S0 at slave(3)@172.17.3.73:36681 (441f06bc837a) with cpus(*):2; mem(*):10240; disk(*):3.70122e+06; ports(*):[31000-32000] I0902 00:30:53.752836 28619 slave.cpp:859] Registered with master master@172.17.3.73:36681; given slave ID 20150902-003053-1224937900-36681-28582-S0 I0902 00:30:53.752884 28619 fetcher.cpp:77] Clearing fetcher cache I0902 00:30:53.752969 28620 hierarchical.hpp:542] Added slave 20150902-003053-1224937900-36681-28582-S0 (441f06bc837a) with cpus(*):2; mem(*):10240; disk(*):3.70122e+06; ports(*):[31000-32000] (allocated: ) I0902 00:30:53.753084 28613 status_update_manager.cpp:183] Resuming sending status updates I0902 00:30:53.753156 28614 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 906644ns I0902 00:30:53.753195 28614 replica.cpp:679] Persisted action at 4 I0902 00:30:53.753587 28620 hierarchical.hpp:1010] No resources available to allocate! I0902 00:30:53.753640 28620 hierarchical.hpp:928] Performed allocation for slave 20150902-003053-1224937900-36681-28582-S0 in 617892ns I0902 00:30:53.753787 28619 slave.cpp:882] Checkpointing SlaveInfo to '/tmp/mesos-s1Q2uY/2/meta/slaves/20150902-003053-1224937900-36681-28582-S0/slave.info' I0902 00:30:53.753886 28605 replica.cpp:658] Replica received learned notice for position 4 I0902 00:30:53.754436 28619 slave.cpp:918] Forwarding total oversubscribed resources I0902 00:30:53.754580 28605 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 677038ns I0902 00:30:53.754675 28605 leveldb.cpp:401] Deleting ~2 keys from leveldb took 72858ns I0902 00:30:53.754695 28608 master.cpp:4032] Received update of slave 20150902-003053-1224937900-36681-28582-S0 at slave(3)@172.17.3.73:36681 (441f06bc837a) with total oversubscribed resources I0902 00:30:53.754710 28605 replica.cpp:679] Persisted action at 4 I0902 00:30:53.754761 28605 replica.cpp:664] Replica learned TRUNCATE action at position 4 I0902 00:30:53.755244 28619 hierarchical.hpp:602] Slave 20150902-003053-1224937900-36681-28582-S0 (441f06bc837a) updated with oversubscribed resources (total: cpus(*):2; mem(*):10240; disk(*):3.70122e+06; ports(*):[31000-32000], allocated: ) I0902 00:30:53.755473 28619 hierarchical.hpp:1010] No resources available to allocate! I0902 00:30:53.755507 28619 hierarchical.hpp:928] Performed allocation for slave 20150902-003053-1224937900-36681-28582-S0 in 217181ns I0902 00:30:54.464862 28617 slave.cpp:1209] Will retry registration in 1.181298785secs if necessary I0902 00:30:54.464993 28609 master.cpp:3670] Registering slave at slave(1)@172.17.3.73:36681 (441f06bc837a) with id 20150902-003053-1224937900-36681-28582-S1 I0902 00:30:54.465430 28619 registrar.cpp:443] Applied 1 operations in 113891ns; attempting to update the 'registry' I0902 00:30:54.468351 28613 log.cpp:685] Attempting to append 515 bytes to the log I0902 00:30:54.468456 28618 coordinator.cpp:341] Coordinator attempting to write APPEND action at position 5 I0902 00:30:54.469163 28605 replica.cpp:511] Replica received write request for position 5 I0902 00:30:54.469934 28605 leveldb.cpp:343] Persisting action (534 bytes) to leveldb took 741179ns I0902 00:30:54.469966 28605 replica.cpp:679] Persisted action at 5 I0902 00:30:54.470788 28614 replica.cpp:658] Replica received learned notice for position 5 I0902 00:30:54.471417 28614 leveldb.cpp:343] Persisting action (536 bytes) to leveldb took 615160ns I0902 00:30:54.471451 28614 replica.cpp:679] Persisted action at 5 I0902 00:30:54.471477 28614 replica.cpp:664] Replica learned APPEND action at position 5 I0902 00:30:54.473342 28607 registrar.cpp:488] Successfully updated the 'registry' in 7.836928ms I0902 00:30:54.473642 28616 log.cpp:704] Attempting to truncate the log to 5 I0902 00:30:54.474309 28617 slave.cpp:3058] Received ping from slave-observer(2)@172.17.3.73:36681 I0902 00:30:54.474455 28608 master.cpp:3733] Registered slave 20150902-003053-1224937900-36681-28582-S1 at slave(1)@172.17.3.73:36681 (441f06bc837a) with cpus(*):2; mem(*):10240; disk(*):3.70122e+06; ports(*):[31000-32000] I0902 00:30:54.474581 28617 slave.cpp:859] Registered with master master@172.17.3.73:36681; given slave ID 20150902-003053-1224937900-36681-28582-S1 I0902 00:30:54.474618 28617 fetcher.cpp:77] Clearing fetcher cache I0902 00:30:54.474635 28610 coordinator.cpp:341] Coordinator attempting to write TRUNCATE action at position 6 I0902 00:30:54.474769 28605 status_update_manager.cpp:183] Resuming sending status updates I0902 00:30:54.474747 28609 hierarchical.hpp:542] Added slave 20150902-003053-1224937900-36681-28582-S1 (441f06bc837a) with cpus(*):2; mem(*):10240; disk(*):3.70122e+06; ports(*):[31000-32000] (allocated: ) I0902 00:30:54.475209 28617 slave.cpp:882] Checkpointing SlaveInfo to '/tmp/mesos-s1Q2uY/0/meta/slaves/20150902-003053-1224937900-36681-28582-S1/slave.info' I0902 00:30:54.475317 28609 hierarchical.hpp:1010] No resources available to allocate! I0902 00:30:54.475446 28609 hierarchical.hpp:928] Performed allocation for slave 20150902-003053-1224937900-36681-28582-S1 in 463257ns I0902 00:30:54.475603 28607 replica.cpp:511] Replica received write request for position 6 I0902 00:30:54.475780 28617 slave.cpp:918] Forwarding total oversubscribed resources I0902 00:30:54.475937 28617 master.cpp:4032] Received update of slave 20150902-003053-1224937900-36681-28582-S1 at slave(1)@172.17.3.73:36681 (441f06bc837a) with total oversubscribed resources I0902 00:30:54.476305 28610 hierarchical.hpp:602] Slave 20150902-003053-1224937900-36681-28582-S1 (441f06bc837a) updated with oversubscribed resources (total: cpus(*):2; mem(*):10240; disk(*):3.70122e+06; ports(*):[31000-32000], allocated: ) I0902 00:30:54.476480 28607 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 772408ns I0902 00:30:54.476511 28607 replica.cpp:679] Persisted action at 6 I0902 00:30:54.476541 28610 hierarchical.hpp:1010] No resources available to allocate! I0902 00:30:54.476567 28610 hierarchical.hpp:928] Performed allocation for slave 20150902-003053-1224937900-36681-28582-S1 in 214977ns I0902 00:30:54.477264 28620 replica.cpp:658] Replica received learned notice for position 6 I0902 00:30:54.477743 28620 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 458654ns I0902 00:30:54.477814 28620 leveldb.cpp:401] Deleting ~2 keys from leveldb took 51876ns I0902 00:30:54.477840 28620 replica.cpp:679] Persisted action at 6 I0902 00:30:54.477867 28620 replica.cpp:664] Replica learned TRUNCATE action at position 6 I0902 00:30:54.596190 28611 slave.cpp:1209] Will retry registration in 217.215177ms if necessary I0902 00:30:54.596360 28608 master.cpp:3670] Registering slave at slave(2)@172.17.3.73:36681 (441f06bc837a) with id 20150902-003053-1224937900-36681-28582-S2 I0902 00:30:54.596786 28611 registrar.cpp:443] Applied 1 operations in 117272ns; attempting to update the 'registry' I0902 00:30:54.599691 28613 log.cpp:685] Attempting to append 683 bytes to the log I0902 00:30:54.599798 28618 coordinator.cpp:341] Coordinator attempting to write APPEND action at position 7 I0902 00:30:54.600631 28616 replica.cpp:511] Replica received write request for position 7 I0902 00:30:54.600955 28616 leveldb.cpp:343] Persisting action (702 bytes) to leveldb took 298371ns I0902 00:30:54.600988 28616 replica.cpp:679] Persisted action at 7 I0902 00:30:54.601665 28609 replica.cpp:658] Replica received learned notice for position 7 I0902 00:30:54.602424 28609 leveldb.cpp:343] Persisting action (704 bytes) to leveldb took 743909ns I0902 00:30:54.602457 28609 replica.cpp:679] Persisted action at 7 I0902 00:30:54.602481 28609 replica.cpp:664] Replica learned APPEND action at position 7 I0902 00:30:54.604488 28615 registrar.cpp:488] Successfully updated the 'registry' in 7.63008ms I0902 00:30:54.604832 28617 log.cpp:704] Attempting to truncate the log to 7 I0902 00:30:54.605005 28613 coordinator.cpp:341] Coordinator attempting to write TRUNCATE action at position 8 I0902 00:30:54.605356 28615 slave.cpp:3058] Received ping from slave-observer(3)@172.17.3.73:36681 I0902 00:30:54.605599 28615 slave.cpp:859] Registered with master master@172.17.3.73:36681; given slave ID 20150902-003053-1224937900-36681-28582-S2 I0902 00:30:54.605530 28606 master.cpp:3733] Registered slave 20150902-003053-1224937900-36681-28582-S2 at slave(2)@172.17.3.73:36681 (441f06bc837a) with cpus(*):2; mem(*):10240; disk(*):3.70122e+06; ports(*):[31000-32000] I0902 00:30:54.605690 28615 fetcher.cpp:77] Clearing fetcher cache I0902 00:30:54.605751 28609 hierarchical.hpp:542] Added slave 20150902-003053-1224937900-36681-28582-S2 (441f06bc837a) with cpus(*):2; mem(*):10240; disk(*):3.70122e+06; ports(*):[31000-32000] (allocated: ) I0902 00:30:54.605989 28605 status_update_manager.cpp:183] Resuming sending status updates I0902 00:30:54.606201 28609 hierarchical.hpp:1010] No resources available to allocate! I0902 00:30:54.606220 28607 replica.cpp:511] Replica received write request for position 8 I0902 00:30:54.606238 28609 hierarchical.hpp:928] Performed allocation for slave 20150902-003053-1224937900-36681-28582-S2 in 348187ns I0902 00:30:54.606287 28615 slave.cpp:882] Checkpointing SlaveInfo to '/tmp/mesos-s1Q2uY/1/meta/slaves/20150902-003053-1224937900-36681-28582-S2/slave.info' I0902 00:30:54.607102 28615 slave.cpp:918] Forwarding total oversubscribed resources I0902 00:30:54.607115 28607 leveldb.cpp:343] Persisting action (16 bytes) to leveldb took 872303ns I0902 00:30:54.607148 28607 replica.cpp:679] Persisted action at 8 I0902 00:30:54.607295 28615 master.cpp:4032] Received update of slave 20150902-003053-1224937900-36681-28582-S2 at slave(2)@172.17.3.73:36681 (441f06bc837a) with total oversubscribed resources I0902 00:30:54.607785 28620 hierarchical.hpp:602] Slave 20150902-003053-1224937900-36681-28582-S2 (441f06bc837a) updated with oversubscribed resources (total: cpus(*):2; mem(*):10240; disk(*):3.70122e+06; ports(*):[31000-32000], allocated: ) I0902 00:30:54.607882 28618 replica.cpp:658] Replica received learned notice for position 8 I0902 00:30:54.608000 28620 hierarchical.hpp:1010] No resources available to allocate! I0902 00:30:54.608031 28620 hierarchical.hpp:928] Performed allocation for slave 20150902-003053-1224937900-36681-28582-S2 in 197321ns I0902 00:30:54.608292 28618 leveldb.cpp:343] Persisting action (18 bytes) to leveldb took 389254ns I0902 00:30:54.608384 28618 leveldb.cpp:401] Deleting ~2 keys from leveldb took 70973ns I0902 00:30:54.608415 28618 replica.cpp:679] Persisted action at 8 I0902 00:30:54.608456 28618 replica.cpp:664] Replica learned TRUNCATE action at position 8 I0902 00:30:54.668705 28614 hierarchical.hpp:1010] No resources available to allocate! I0902 00:30:54.668745 28614 hierarchical.hpp:910] Performed allocation for 3 slaves in 557805ns I0902 00:30:55.669527 28619 hierarchical.hpp:1010] No resources available to allocate! I0902 00:30:55.669566 28619 hierarchical.hpp:910] Performed allocation for 3 slaves in 465684ns I0902 00:30:56.671480 28617 hierarchical.hpp:1010] No resources available to allocate! I0902 00:30:56.671515 28617 hierarchical.hpp:910] Performed allocation for 3 slaves in 485449ns I0902 00:30:57.673353 28607 hierarchical.hpp:1010] No resources available to allocate! I0902 00:30:57.673391 28607 hierarchical.hpp:910] Performed allocation for 3 slaves in 518196ns I0902 00:30:58.675276 28618 hierarchical.hpp:1010] No resources available to allocate! I0902 00:30:58.675312 28618 hierarchical.hpp:910] Performed allocation for 3 slaves in 503347ns I0902 00:30:59.677129 28610 hierarchical.hpp:1010] No resources available to allocate! I0902 00:30:59.677163 28610 hierarchical.hpp:910] Performed allocation for 3 slaves in 468757ns I0902 00:31:00.678947 28613 hierarchical.hpp:1010] No resources available to allocate! I0902 00:31:00.678983 28613 hierarchical.hpp:910] Performed allocation for 3 slaves in 497686ns I0902 00:31:01.680778 28620 hierarchical.hpp:1010] No resources available to allocate! I0902 00:31:01.680814 28620 hierarchical.hpp:910] Performed allocation for 3 slaves in 433630ns I0902 00:31:02.682585 28609 hierarchical.hpp:1010] No resources available to allocate! I0902 00:31:02.682620 28609 hierarchical.hpp:910] Performed allocation for 3 slaves in 507390ns I0902 00:31:03.683991 28611 hierarchical.hpp:1010] No resources available to allocate! I0902 00:31:03.684037 28611 hierarchical.hpp:910] Performed allocation for 3 slaves in 694017ns I0902 00:31:04.686070 28619 hierarchical.hpp:1010] No resources available to allocate! I0902 00:31:04.686112 28619 hierarchical.hpp:910] Performed allocation for 3 slaves in 490512ns I0902 00:31:05.687798 28618 hierarchical.hpp:1010] No resources available to allocate! I0902 00:31:05.687832 28618 hierarchical.hpp:910] Performed allocation for 3 slaves in 421718ns I0902 00:31:06.689510 28607 hierarchical.hpp:1010] No resources available to allocate! I0902 00:31:06.689548 28607 hierarchical.hpp:910] Performed allocation for 3 slaves in 448215ns I0902 00:31:07.691249 28610 hierarchical.hpp:1010] No resources available to allocate! I0902 00:31:07.691285 28610 hierarchical.hpp:910] Performed allocation for 3 slaves in 469345ns I0902 00:31:08.683100 28615 slave.cpp:4226] Querying resource estimator for oversubscribable resources I0902 00:31:08.683322 28614 slave.cpp:4240] Received oversubscribable resources from the resource estimator I0902 00:31:08.685286 28606 slave.cpp:4226] Querying resource estimator for oversubscribable resources I0902 00:31:08.685518 28616 slave.cpp:4240] Received oversubscribable resources from the resource estimator I0902 00:31:08.692482 28608 slave.cpp:4226] Querying resource estimator for oversubscribable resources {code} [~ijimenez] Are you up for triaging this? > EventCall Test Framework is flaky > --------------------------------- > > Key: MESOS-3273 > URL: https://issues.apache.org/jira/browse/MESOS-3273 > Project: Mesos > Issue Type: Bug > Components: HTTP API > Affects Versions: 0.24.0 > Environment: > https://builds.apache.org/job/Mesos/705/COMPILER=clang,CONFIGURATION=--verbose,OS=ubuntu:14.04,label_exp=docker%7C%7CHadoop/consoleFull > Reporter: Vinod Kone > Labels: flaky-test, tech-debt, twitter > > Observed this on ASF CI. h/t [~haosd...@gmail.com] > Looks like the HTTP scheduler never sent a SUBSCRIBE request to the master. > {code} > [ RUN ] ExamplesTest.EventCallFramework > Using temporary directory '/tmp/ExamplesTest_EventCallFramework_k4vXkx' > I0813 19:55:15.643579 26085 exec.cpp:443] Ignoring exited event because the > driver is aborted! > Shutting down > Sending SIGTERM to process tree at pid 26061 > Killing the following process trees: > [ > ] > Shutting down > Sending SIGTERM to process tree at pid 26062 > Shutting down > Killing the following process trees: > [ > ] > Sending SIGTERM to process tree at pid 26063 > Killing the following process trees: > [ > ] > Shutting down > Sending SIGTERM to process tree at pid 26098 > Killing the following process trees: > [ > ] > Shutting down > Sending SIGTERM to process tree at pid 26099 > Killing the following process trees: > [ > ] > WARNING: Logging before InitGoogleLogging() is written to STDERR > I0813 19:55:17.161726 26100 process.cpp:1012] libprocess is initialized on > 172.17.2.10:60249 for 16 cpus > I0813 19:55:17.161888 26100 logging.cpp:177] Logging to STDERR > I0813 19:55:17.163625 26100 scheduler.cpp:157] Version: 0.24.0 > I0813 19:55:17.175302 26100 leveldb.cpp:176] Opened db in 3.167446ms > I0813 19:55:17.176393 26100 leveldb.cpp:183] Compacted db in 1.047996ms > I0813 19:55:17.176496 26100 leveldb.cpp:198] Created db iterator in 77155ns > I0813 19:55:17.176518 26100 leveldb.cpp:204] Seeked to beginning of db in > 8429ns > I0813 19:55:17.176527 26100 leveldb.cpp:273] Iterated through 0 keys in the > db in 4219ns > I0813 19:55:17.176708 26100 replica.cpp:744] Replica recovered with log > positions 0 -> 0 with 1 holes and 0 unlearned > I0813 19:55:17.178951 26136 recover.cpp:449] Starting replica recovery > I0813 19:55:17.179934 26136 recover.cpp:475] Replica is in EMPTY status > I0813 19:55:17.181970 26126 master.cpp:378] Master > 20150813-195517-167907756-60249-26100 (297daca2d01a) started on > 172.17.2.10:60249 > I0813 19:55:17.182317 26126 master.cpp:380] Flags at startup: > --acls="permissive: false > register_frameworks { > principals { > type: SOME > values: "test-principal" > } > roles { > type: SOME > values: "*" > } > } > run_tasks { > principals { > type: SOME > values: "test-principal" > } > users { > type: SOME > values: "mesos" > } > } > " --allocation_interval="1secs" --allocator="HierarchicalDRF" > --authenticate="false" --authenticate_slaves="false" > --authenticators="crammd5" > --credentials="/tmp/ExamplesTest_EventCallFramework_k4vXkx/credentials" > --framework_sorter="drf" --help="false" --initialize_driver_logging="true" > --log_auto_initialize="true" --logbufsecs="0" --logging_level="INFO" > --max_slave_ping_timeouts="5" --quiet="false" > --recovery_slave_removal_limit="100%" --registry="replicated_log" > --registry_fetch_timeout="1mins" --registry_store_timeout="5secs" > --registry_strict="false" --root_submissions="true" > --slave_ping_timeout="15secs" --slave_reregister_timeout="10mins" > --user_sorter="drf" --version="false" > --webui_dir="/mesos/mesos-0.24.0/src/webui" --work_dir="/tmp/mesos-II8Gua" > --zk_session_timeout="10secs" > I0813 19:55:17.183475 26126 master.cpp:427] Master allowing unauthenticated > frameworks to register > I0813 19:55:17.183536 26126 master.cpp:432] Master allowing unauthenticated > slaves to register > I0813 19:55:17.183615 26126 credentials.hpp:37] Loading credentials for > authentication from '/tmp/ExamplesTest_EventCallFramework_k4vXkx/credentials' > W0813 19:55:17.183859 26126 credentials.hpp:52] Permissions on credentials > file '/tmp/ExamplesTest_EventCallFramework_k4vXkx/credentials' are too open. > It is recommended that your credentials file is NOT accessible by others. > I0813 19:55:17.183969 26123 replica.cpp:641] Replica in EMPTY status received > a broadcasted recover request > I0813 19:55:17.184306 26126 master.cpp:469] Using default 'crammd5' > authenticator > I0813 19:55:17.184661 26126 authenticator.cpp:512] Initializing server SASL > I0813 19:55:17.185104 26138 recover.cpp:195] Received a recover response from > a replica in EMPTY status > I0813 19:55:17.185972 26100 containerizer.cpp:143] Using isolation: > posix/cpu,posix/mem,filesystem/posix > I0813 19:55:17.186058 26135 recover.cpp:566] Updating replica status to > STARTING > I0813 19:55:17.187001 26138 leveldb.cpp:306] Persisting metadata (8 bytes) to > leveldb took 654586ns > I0813 19:55:17.187037 26138 replica.cpp:323] Persisted replica status to > STARTING > I0813 19:55:17.187499 26134 recover.cpp:475] Replica is in STARTING status > I0813 19:55:17.187605 26126 auxprop.cpp:66] Initialized in-memory auxiliary > property plugin > I0813 19:55:17.187710 26126 master.cpp:506] Authorization enabled > I0813 19:55:17.188657 26138 replica.cpp:641] Replica in STARTING status > received a broadcasted recover request > I0813 19:55:17.188853 26131 hierarchical.hpp:346] Initialized hierarchical > allocator process > I0813 19:55:17.189252 26132 whitelist_watcher.cpp:79] No whitelist given > I0813 19:55:17.189321 26134 recover.cpp:195] Received a recover response from > a replica in STARTING status > I0813 19:55:17.190001 26125 recover.cpp:566] Updating replica status to VOTING > I0813 19:55:17.190696 26124 leveldb.cpp:306] Persisting metadata (8 bytes) to > leveldb took 357331ns > I0813 19:55:17.190775 26124 replica.cpp:323] Persisted replica status to > VOTING > I0813 19:55:17.190970 26133 recover.cpp:580] Successfully joined the Paxos > group > I0813 19:55:17.192183 26129 recover.cpp:464] Recover process terminated > I0813 19:55:17.192699 26123 slave.cpp:190] Slave started on > 1)@172.17.2.10:60249 > I0813 19:55:17.192741 26123 slave.cpp:191] Flags at startup: > --authenticatee="crammd5" --cgroups_cpu_enable_pids_and_tids_count="false" > --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" > --cgroups_limit_swap="false" --cgroups_root="mesos" > --container_disk_watch_interval="15secs" --containerizers="mesos" > --default_role="*" --disk_watch_interval="1mins" --docker="docker" > --docker_kill_orphans="true" --docker_remove_delay="6hrs" > --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" > --enforce_container_disk_quota="false" > --executor_registration_timeout="1mins" > --executor_shutdown_grace_period="5secs" > --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" > --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" > --hadoop_home="" --help="false" --initialize_driver_logging="true" > --isolation="posix/cpu,posix/mem" > --launcher_dir="/mesos/mesos-0.24.0/_build/src" --logbufsecs="0" > --logging_level="INFO" --oversubscribed_resources_interval="15secs" > --perf_duration="10secs" --perf_interval="1mins" > --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" > --recovery_timeout="15mins" --registration_backoff_factor="1secs" > --resource_monitoring_interval="1secs" --resources="cpus:2;mem:10240" > --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" > --strict="true" --switch_user="true" --version="false" > --work_dir="/tmp/mesos-II8Gua/0" > I0813 19:55:17.194514 26100 containerizer.cpp:143] Using isolation: > posix/cpu,posix/mem,filesystem/posix > I0813 19:55:17.194658 26123 slave.cpp:354] Slave resources: cpus(*):2; > mem(*):10240; disk(*):3.70122e+06; ports(*):[31000-32000] > I0813 19:55:17.194854 26123 slave.cpp:384] Slave hostname: 297daca2d01a > I0813 19:55:17.194877 26123 slave.cpp:389] Slave checkpoint: true > I0813 19:55:17.196751 26132 master.cpp:1524] The newly elected leader is > master@172.17.2.10:60249 with id 20150813-195517-167907756-60249-26100 > I0813 19:55:17.196797 26132 master.cpp:1537] Elected as the leading master! > I0813 19:55:17.196815 26132 master.cpp:1307] Recovering from registrar > I0813 19:55:17.197032 26138 registrar.cpp:311] Recovering registrar > I0813 19:55:17.197845 26132 slave.cpp:190] Slave started on > 2)@172.17.2.10:60249 > I0813 19:55:17.198420 26125 log.cpp:661] Attempting to start the writer > I0813 19:55:17.197948 26132 slave.cpp:191] Flags at startup: > --authenticatee="crammd5" --cgroups_cpu_enable_pids_and_tids_count="false" > --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" > --cgroups_limit_swap="false" --cgroups_root="mesos" > --container_disk_watch_interval="15secs" --containerizers="mesos" > --default_role="*" --disk_watch_interval="1mins" --docker="docker" > --docker_kill_orphans="true" --docker_remove_delay="6hrs" > --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" > --enforce_container_disk_quota="false" > --executor_registration_timeout="1mins" > --executor_shutdown_grace_period="5secs" > --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" > --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" > --hadoop_home="" --help="false" --initialize_driver_logging="true" > --isolation="posix/cpu,posix/mem" > --launcher_dir="/mesos/mesos-0.24.0/_build/src" --logbufsecs="0" > --logging_level="INFO" --oversubscribed_resources_interval="15secs" > --perf_duration="10secs" --perf_interval="1mins" > --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" > --recovery_timeout="15mins" --registration_backoff_factor="1secs" > --resource_monitoring_interval="1secs" --resources="cpus:2;mem:10240" > --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" > --strict="true" --switch_user="true" --version="false" > --work_dir="/tmp/mesos-II8Gua/1" > I0813 19:55:17.199121 26132 slave.cpp:354] Slave resources: cpus(*):2; > mem(*):10240; disk(*):3.70122e+06; ports(*):[31000-32000] > I0813 19:55:17.199235 26138 state.cpp:54] Recovering state from > '/tmp/mesos-II8Gua/0/meta' > I0813 19:55:17.199322 26132 slave.cpp:384] Slave hostname: 297daca2d01a > I0813 19:55:17.199345 26132 slave.cpp:389] Slave checkpoint: true > I0813 19:55:17.199676 26100 containerizer.cpp:143] Using isolation: > posix/cpu,posix/mem,filesystem/posix > I0813 19:55:17.200085 26135 state.cpp:54] Recovering state from > '/tmp/mesos-II8Gua/1/meta' > I0813 19:55:17.200317 26132 status_update_manager.cpp:202] Recovering status > update manager > I0813 19:55:17.200371 26129 status_update_manager.cpp:202] Recovering status > update manager > I0813 19:55:17.202003 26129 replica.cpp:477] Replica received implicit > promise request with proposal 1 > I0813 19:55:17.202585 26131 slave.cpp:190] Slave started on > 3)@172.17.2.10:60249 > I0813 19:55:17.202596 26129 leveldb.cpp:306] Persisting metadata (8 bytes) to > leveldb took 523191ns > I0813 19:55:17.202756 26129 replica.cpp:345] Persisted promised to 1 > I0813 19:55:17.202770 26132 containerizer.cpp:379] Recovering containerizer > I0813 19:55:17.203061 26135 containerizer.cpp:379] Recovering containerizer > I0813 19:55:17.202663 26131 slave.cpp:191] Flags at startup: > --authenticatee="crammd5" --cgroups_cpu_enable_pids_and_tids_count="false" > --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" > --cgroups_limit_swap="false" --cgroups_root="mesos" > --container_disk_watch_interval="15secs" --containerizers="mesos" > --default_role="*" --disk_watch_interval="1mins" --docker="docker" > --docker_kill_orphans="true" --docker_remove_delay="6hrs" > --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns" > --enforce_container_disk_quota="false" > --executor_registration_timeout="1mins" > --executor_shutdown_grace_period="5secs" > --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" > --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" > --hadoop_home="" --help="false" --initialize_driver_logging="true" > --isolation="posix/cpu,posix/mem" > --launcher_dir="/mesos/mesos-0.24.0/_build/src" --logbufsecs="0" > --logging_level="INFO" --oversubscribed_resources_interval="15secs" > --perf_duration="10secs" --perf_interval="1mins" > --qos_correction_interval_min="0ns" --quiet="false" --recover="reconnect" > --recovery_timeout="15mins" --registration_backoff_factor="1secs" > --resource_monitoring_interval="1secs" --resources="cpus:2;mem:10240" > --revocable_cpu_low_priority="true" --sandbox_directory="/mnt/mesos/sandbox" > --strict="true" --switch_user="true" --version="false" > --work_dir="/tmp/mesos-II8Gua/2" > I0813 19:55:17.203819 26131 slave.cpp:354] Slave resources: cpus(*):2; > mem(*):10240; disk(*):3.70122e+06; ports(*):[31000-32000] > I0813 19:55:17.203930 26131 slave.cpp:384] Slave hostname: 297daca2d01a > I0813 19:55:17.203948 26131 slave.cpp:389] Slave checkpoint: true > I0813 19:55:17.204674 26137 state.cpp:54] Recovering state from > '/tmp/mesos-II8Gua/2/meta' > I0813 19:55:17.205178 26135 status_update_manager.cpp:202] Recovering status > update manager > I0813 19:55:17.205323 26135 containerizer.cpp:379] Recovering containerizer > I0813 19:55:17.205521 26136 slave.cpp:4069] Finished recovery > I0813 19:55:17.206074 26136 slave.cpp:4226] Querying resource estimator for > oversubscribable resources > I0813 19:55:17.206424 26128 slave.cpp:4069] Finished recovery > I0813 19:55:17.206722 26137 status_update_manager.cpp:176] Pausing sending > status updates > I0813 19:55:17.206858 26136 slave.cpp:684] New master detected at > master@172.17.2.10:60249 > I0813 19:55:17.206902 26138 slave.cpp:4069] Finished recovery > I0813 19:55:17.206962 26128 slave.cpp:4226] Querying resource estimator for > oversubscribable resources > I0813 19:55:17.208312 26134 scheduler.cpp:272] New master detected at > master@172.17.2.10:60249 > I0813 19:55:17.208364 26136 slave.cpp:709] No credentials provided. > Attempting to register without authentication > I0813 19:55:17.208608 26136 slave.cpp:720] Detecting new master > I0813 19:55:17.208839 26138 slave.cpp:4226] Querying resource estimator for > oversubscribable resources > I0813 19:55:17.209216 26123 coordinator.cpp:231] Coordinator attemping to > fill missing position > I0813 19:55:17.209247 26127 status_update_manager.cpp:176] Pausing sending > status updates > I0813 19:55:17.209259 26128 slave.cpp:684] New master detected at > master@172.17.2.10:60249 > I0813 19:55:17.209322 26127 status_update_manager.cpp:176] Pausing sending > status updates > I0813 19:55:17.209364 26128 slave.cpp:709] No credentials provided. > Attempting to register without authentication > I0813 19:55:17.209344 26138 slave.cpp:684] New master detected at > master@172.17.2.10:60249 > I0813 19:55:17.209455 26128 slave.cpp:720] Detecting new master > I0813 19:55:17.209492 26138 slave.cpp:709] No credentials provided. > Attempting to register without authentication > I0813 19:55:17.209573 26128 slave.cpp:4240] Received oversubscribable > resources from the resource estimator > I0813 19:55:17.209601 26138 slave.cpp:720] Detecting new master > I0813 19:55:17.209730 26138 slave.cpp:4240] Received oversubscribable > resources from the resource estimator > I0813 19:55:17.209883 26136 slave.cpp:4240] Received oversubscribable > resources from the resource estimator > I0813 19:55:17.211266 26136 replica.cpp:378] Replica received explicit > promise request for position 0 with proposal 2 > I0813 19:55:17.211771 26136 leveldb.cpp:343] Persisting action (8 bytes) to > leveldb took 462128ns > I0813 19:55:17.211797 26136 replica.cpp:679] Persisted action at 0 > I0813 19:55:17.212980 26130 replica.cpp:511] Replica received write request > for position 0 > I0813 19:55:17.213124 26130 leveldb.cpp:438] Reading position from leveldb > took 67075ns > I0813 19:55:17.213580 26130 leveldb.cpp:343] Persisting action (14 bytes) to > leveldb took 301649ns > I0813 19:55:17.213603 26130 replica.cpp:679] Persisted action at 0 > I0813 19:55:17.214284 26123 replica.cpp:658] Replica received learned notice > for position 0 > I0813 19:55:17.214622 26123 leveldb.cpp:343] Persisting action (16 bytes) to > leveldb took 284547ns > I0813 19:55:17.214648 26123 replica.cpp:679] Persisted action at 0 > I0813 19:55:17.214675 26123 replica.cpp:664] Replica learned NOP action at > position 0 > I0813 19:55:17.215420 26136 log.cpp:677] Writer started with ending position 0 > I0813 19:55:17.217463 26133 leveldb.cpp:438] Reading position from leveldb > took 47943ns > I0813 19:55:17.220762 26125 registrar.cpp:344] Successfully fetched the > registry (0B) in 23.649024ms > I0813 19:55:17.221081 26125 registrar.cpp:443] Applied 1 operations in > 136902ns; attempting to update the 'registry' > I0813 19:55:17.223667 26133 log.cpp:685] Attempting to append 174 bytes to > the log > I0813 19:55:17.223778 26125 coordinator.cpp:341] Coordinator attempting to > write APPEND action at position 1 > I0813 19:55:17.224516 26127 replica.cpp:511] Replica received write request > for position 1 > I0813 19:55:17.225009 26127 leveldb.cpp:343] Persisting action (193 bytes) to > leveldb took 466230ns > I0813 19:55:17.225042 26127 replica.cpp:679] Persisted action at 1 > I0813 19:55:17.225653 26126 replica.cpp:658] Replica received learned notice > for position 1 > I0813 19:55:17.225953 26126 leveldb.cpp:343] Persisting action (195 bytes) to > leveldb took 286966ns > I0813 19:55:17.225975 26126 replica.cpp:679] Persisted action at 1 > I0813 19:55:17.226013 26126 replica.cpp:664] Replica learned APPEND action at > position 1 > I0813 19:55:17.227545 26137 registrar.cpp:488] Successfully updated the > 'registry' in 6.328064ms > I0813 19:55:17.227722 26137 registrar.cpp:374] Successfully recovered > registrar > I0813 19:55:17.227918 26124 log.cpp:704] Attempting to truncate the log to 1 > I0813 19:55:17.228024 26133 coordinator.cpp:341] Coordinator attempting to > write TRUNCATE action at position 2 > I0813 19:55:17.228193 26131 master.cpp:1334] Recovered 0 slaves from the > Registry (135B) ; allowing 10mins for slaves to re-register > I0813 19:55:17.228659 26127 replica.cpp:511] Replica received write request > for position 2 > I0813 19:55:17.228972 26127 leveldb.cpp:343] Persisting action (16 bytes) to > leveldb took 297903ns > I0813 19:55:17.229004 26127 replica.cpp:679] Persisted action at 2 > I0813 19:55:17.229565 26127 replica.cpp:658] Replica received learned notice > for position 2 > I0813 19:55:17.229837 26127 leveldb.cpp:343] Persisting action (18 bytes) to > leveldb took 260326ns > I0813 19:55:17.229899 26127 leveldb.cpp:401] Deleting ~1 keys from leveldb > took 48697ns > I0813 19:55:17.229923 26127 replica.cpp:679] Persisted action at 2 > I0813 19:55:17.229956 26127 replica.cpp:664] Replica learned TRUNCATE action > at position 2 > I0813 19:55:17.325634 26138 slave.cpp:1209] Will retry registration in > 445.955946ms if necessary > I0813 19:55:17.326088 26124 master.cpp:3635] Registering slave at > slave(2)@172.17.2.10:60249 (297daca2d01a) with id > 20150813-195517-167907756-60249-26100-S0 > I0813 19:55:17.327446 26124 registrar.cpp:443] Applied 1 operations in > 231072ns; attempting to update the 'registry' > I0813 19:55:17.330252 26136 log.cpp:685] Attempting to append 344 bytes to > the log > I0813 19:55:17.330407 26132 coordinator.cpp:341] Coordinator attempting to > write APPEND action at position 3 > I0813 19:55:17.331418 26128 replica.cpp:511] Replica received write request > for position 3 > I0813 19:55:17.331753 26128 leveldb.cpp:343] Persisting action (363 bytes) to > leveldb took 264140ns > I0813 19:55:17.331778 26128 replica.cpp:679] Persisted action at 3 > I0813 19:55:17.332324 26133 replica.cpp:658] Replica received learned notice > for position 3 > I0813 19:55:17.332809 26133 leveldb.cpp:343] Persisting action (365 bytes) to > leveldb took 313064ns > I0813 19:55:17.332834 26133 replica.cpp:679] Persisted action at 3 > I0813 19:55:17.332865 26133 replica.cpp:664] Replica learned APPEND action at > position 3 > I0813 19:55:17.334211 26132 registrar.cpp:488] Successfully updated the > 'registry' in 6.668032ms > I0813 19:55:17.334430 26127 log.cpp:704] Attempting to truncate the log to 3 > I0813 19:55:17.334566 26132 coordinator.cpp:341] Coordinator attempting to > write TRUNCATE action at position 4 > I0813 19:55:17.335283 26129 replica.cpp:511] Replica received write request > for position 4 > I0813 19:55:17.335615 26127 slave.cpp:3058] Received ping from > slave-observer(1)@172.17.2.10:60249 > I0813 19:55:17.335816 26129 leveldb.cpp:343] Persisting action (16 bytes) to > leveldb took 458268ns > I0813 19:55:17.335908 26137 master.cpp:3698] Registered slave > 20150813-195517-167907756-60249-26100-S0 at slave(2)@172.17.2.10:60249 > (297daca2d01a) with cpus(*):2; mem(*):10240; disk(*):3.70122e+06; > ports(*):[31000-32000] > I0813 19:55:17.335983 26129 replica.cpp:679] Persisted action at 4 > I0813 19:55:17.336019 26136 slave.cpp:859] Registered with master > master@172.17.2.10:60249; given slave ID > 20150813-195517-167907756-60249-26100-S0 > I0813 19:55:17.336073 26136 fetcher.cpp:77] Clearing fetcher cache > I0813 19:55:17.336220 26127 status_update_manager.cpp:183] Resuming sending > status updates > I0813 19:55:17.336328 26128 hierarchical.hpp:540] Added slave > 20150813-195517-167907756-60249-26100-S0 (297daca2d01a) with cpus(*):2; > mem(*):10240; disk(*):3.70122e+06; ports(*):[31000-32000] (allocated: ) > I0813 19:55:17.336599 26138 replica.cpp:658] Replica received learned notice > for position 4 > I0813 19:55:17.336910 26128 hierarchical.hpp:1008] No resources available to > allocate! > I0813 19:55:17.336957 26128 hierarchical.hpp:926] Performed allocation for > slave 20150813-195517-167907756-60249-26100-S0 in 580663ns > I0813 19:55:17.337016 26136 slave.cpp:882] Checkpointing SlaveInfo to > '/tmp/mesos-II8Gua/1/meta/slaves/20150813-195517-167907756-60249-26100-S0/slave.info' > I0813 19:55:17.337035 26138 leveldb.cpp:343] Persisting action (18 bytes) to > leveldb took 403607ns > I0813 19:55:17.337138 26138 leveldb.cpp:401] Deleting ~2 keys from leveldb > took 77040ns > I0813 19:55:17.337167 26138 replica.cpp:679] Persisted action at 4 > I0813 19:55:17.337208 26138 replica.cpp:664] Replica learned TRUNCATE action > at position 4 > I0813 19:55:17.337514 26136 slave.cpp:918] Forwarding total oversubscribed > resources > I0813 19:55:17.337745 26131 master.cpp:3997] Received update of slave > 20150813-195517-167907756-60249-26100-S0 at slave(2)@172.17.2.10:60249 > (297daca2d01a) with total oversubscribed resources > I0813 19:55:17.338240 26131 hierarchical.hpp:600] Slave > 20150813-195517-167907756-60249-26100-S0 (297daca2d01a) updated with > oversubscribed resources (total: cpus(*):2; mem(*):10240; > disk(*):3.70122e+06; ports(*):[31000-32000], allocated: ) > I0813 19:55:17.338479 26131 hierarchical.hpp:1008] No resources available to > allocate! > I0813 19:55:17.338505 26131 hierarchical.hpp:926] Performed allocation for > slave 20150813-195517-167907756-60249-26100-S0 in 216259ns > I0813 19:55:17.504086 26124 slave.cpp:1209] Will retry registration in > 1.92618421secs if necessary > I0813 19:55:17.504408 26124 master.cpp:3635] Registering slave at > slave(3)@172.17.2.10:60249 (297daca2d01a) with id > 20150813-195517-167907756-60249-26100-S1 > I0813 19:55:17.505203 26124 registrar.cpp:443] Applied 1 operations in > 144314ns; attempting to update the 'registry' > I0813 19:55:17.507616 26124 log.cpp:685] Attempting to append 511 bytes to > the log > I0813 19:55:17.507796 26132 coordinator.cpp:341] Coordinator attempting to > write APPEND action at position 5 > I0813 19:55:17.508735 26128 replica.cpp:511] Replica received write request > for position 5 > I0813 19:55:17.509291 26128 leveldb.cpp:343] Persisting action (530 bytes) to > leveldb took 527776ns > I0813 19:55:17.509328 26128 replica.cpp:679] Persisted action at 5 > I0813 19:55:17.509945 26124 replica.cpp:658] Replica received learned notice > for position 5 > I0813 19:55:17.510393 26124 leveldb.cpp:343] Persisting action (532 bytes) to > leveldb took 438543ns > I0813 19:55:17.510416 26124 replica.cpp:679] Persisted action at 5 > I0813 19:55:17.510437 26124 replica.cpp:664] Replica learned APPEND action at > position 5 > I0813 19:55:17.511907 26125 registrar.cpp:488] Successfully updated the > 'registry' in 6624us > I0813 19:55:17.512225 26138 log.cpp:704] Attempting to truncate the log to 5 > I0813 19:55:17.512305 26136 coordinator.cpp:341] Coordinator attempting to > write TRUNCATE action at position 6 > I0813 19:55:17.513066 26133 slave.cpp:3058] Received ping from > slave-observer(2)@172.17.2.10:60249 > I0813 19:55:17.513242 26133 slave.cpp:859] Registered with master > master@172.17.2.10:60249; given slave ID > 20150813-195517-167907756-60249-26100-S1 > I0813 19:55:17.513221 26126 master.cpp:3698] Registered slave > 20150813-195517-167907756-60249-26100-S1 at slave(3)@172.17.2.10:60249 > (297daca2d01a) with cpus(*):2; mem(*):10240; disk(*):3.70122e+06; > ports(*):[31000-32000] > I0813 19:55:17.513089 26129 replica.cpp:511] Replica received write request > for position 6 > I0813 19:55:17.513393 26133 fetcher.cpp:77] Clearing fetcher cache > I0813 19:55:17.513380 26138 hierarchical.hpp:540] Added slave > 20150813-195517-167907756-60249-26100-S1 (297daca2d01a) with cpus(*):2; > mem(*):10240; disk(*):3.70122e+06; ports(*):[31000-32000] (allocated: ) > I0813 19:55:17.513805 26132 status_update_manager.cpp:183] Resuming sending > status updates > I0813 19:55:17.513949 26129 leveldb.cpp:343] Persisting action (16 bytes) to > leveldb took 340511ns > I0813 19:55:17.514046 26138 hierarchical.hpp:1008] No resources available to > allocate! > I0813 19:55:17.514050 26129 replica.cpp:679] Persisted action at 6 > I0813 19:55:17.514195 26133 slave.cpp:882] Checkpointing SlaveInfo to > '/tmp/mesos-II8Gua/2/meta/slaves/20150813-195517-167907756-60249-26100-S1/slave.info' > I0813 19:55:17.514140 26138 hierarchical.hpp:926] Performed allocation for > slave 20150813-195517-167907756-60249-26100-S1 in 417609ns > I0813 19:55:17.514704 26133 slave.cpp:918] Forwarding total oversubscribed > resources > I0813 19:55:17.514708 26138 replica.cpp:658] Replica received learned notice > for position 6 > I0813 19:55:17.514880 26133 master.cpp:3997] Received update of slave > 20150813-195517-167907756-60249-26100-S1 at slave(3)@172.17.2.10:60249 > (297daca2d01a) with total oversubscribed resources > I0813 19:55:17.515244 26127 hierarchical.hpp:600] Slave > 20150813-195517-167907756-60249-26100-S1 (297daca2d01a) updated with > oversubscribed resources (total: cpus(*):2; mem(*):10240; > disk(*):3.70122e+06; ports(*):[31000-32000], allocated: ) > I0813 19:55:17.515454 26138 leveldb.cpp:343] Persisting action (18 bytes) to > leveldb took 640882ns > I0813 19:55:17.515522 26138 leveldb.cpp:401] Deleting ~2 keys from leveldb > took 56550ns > I0813 19:55:17.515547 26138 replica.cpp:679] Persisted action at 6 > I0813 19:55:17.515581 26138 replica.cpp:664] Replica learned TRUNCATE action > at position 6 > I0813 19:55:17.515802 26127 hierarchical.hpp:1008] No resources available to > allocate! > I0813 19:55:17.515866 26127 hierarchical.hpp:926] Performed allocation for > slave 20150813-195517-167907756-60249-26100-S1 in 591007ns > I0813 19:55:17.984196 26135 slave.cpp:1209] Will retry registration in > 1.542495291secs if necessary > I0813 19:55:17.984391 26138 master.cpp:3635] Registering slave at > slave(1)@172.17.2.10:60249 (297daca2d01a) with id > 20150813-195517-167907756-60249-26100-S2 > I0813 19:55:17.985170 26133 registrar.cpp:443] Applied 1 operations in > 202126ns; attempting to update the 'registry' > I0813 19:55:17.987498 26133 log.cpp:685] Attempting to append 678 bytes to > the log > I0813 19:55:17.987656 26123 coordinator.cpp:341] Coordinator attempting to > write APPEND action at position 7 > I0813 19:55:17.988704 26138 replica.cpp:511] Replica received write request > for position 7 > I0813 19:55:17.989223 26138 leveldb.cpp:343] Persisting action (697 bytes) to > leveldb took 490422ns > I0813 19:55:17.989248 26138 replica.cpp:679] Persisted action at 7 > I0813 19:55:17.989972 26126 replica.cpp:658] Replica received learned notice > for position 7 > I0813 19:55:17.990401 26126 leveldb.cpp:343] Persisting action (699 bytes) to > leveldb took 404333ns > I0813 19:55:17.990420 26126 replica.cpp:679] Persisted action at 7 > I0813 19:55:17.990440 26126 replica.cpp:664] Replica learned APPEND action at > position 7 > I0813 19:55:17.994066 26123 registrar.cpp:488] Successfully updated the > 'registry' in 8.788224ms > I0813 19:55:17.994436 26134 log.cpp:704] Attempting to truncate the log to 7 > I0813 19:55:17.994575 26123 coordinator.cpp:341] Coordinator attempting to > write TRUNCATE action at position 8 > I0813 19:55:17.995070 26134 slave.cpp:3058] Received ping from > slave-observer(3)@172.17.2.10:60249 > I0813 19:55:17.995291 26134 slave.cpp:859] Registered with master > master@172.17.2.10:60249; given slave ID > 20150813-195517-167907756-60249-26100-S2 > I0813 19:55:17.995319 26134 fetcher.cpp:77] Clearing fetcher cache > I0813 19:55:17.995246 26129 master.cpp:3698] Registered slave > 20150813-195517-167907756-60249-26100-S2 at slave(1)@172.17.2.10:60249 > (297daca2d01a) with cpus(*):2; mem(*):10240; disk(*):3.70122e+06; > ports(*):[31000-32000] > I0813 19:55:17.995565 26123 status_update_manager.cpp:183] Resuming sending > status updates > I0813 19:55:17.995579 26129 replica.cpp:511] Replica received write request > for position 8 > I0813 19:55:17.996016 26134 slave.cpp:882] Checkpointing SlaveInfo to > '/tmp/mesos-II8Gua/0/meta/slaves/20150813-195517-167907756-60249-26100-S2/slave.info' > I0813 19:55:17.996039 26129 leveldb.cpp:343] Persisting action (16 bytes) to > leveldb took 440511ns > I0813 19:55:17.996067 26129 replica.cpp:679] Persisted action at 8 > I0813 19:55:17.996294 26128 hierarchical.hpp:540] Added slave > 20150813-195517-167907756-60249-26100-S2 (297daca2d01a) with cpus(*):2; > mem(*):10240; disk(*):3.70122e+06; ports(*):[31000-32000] (allocated: ) > I0813 19:55:17.996556 26134 slave.cpp:918] Forwarding total oversubscribed > resources > I0813 19:55:17.996623 26133 replica.cpp:658] Replica received learned notice > for position 8 > I0813 19:55:17.997095 26134 master.cpp:3997] Received update of slave > 20150813-195517-167907756-60249-26100-S2 at slave(1)@172.17.2.10:60249 > (297daca2d01a) with total oversubscribed resources > I0813 19:55:17.997263 26133 leveldb.cpp:343] Persisting action (18 bytes) to > leveldb took 442619ns > I0813 19:55:17.997385 26133 leveldb.cpp:401] Deleting ~2 keys from leveldb > took 95741ns > I0813 19:55:17.997413 26133 replica.cpp:679] Persisted action at 8 > I0813 19:55:17.997465 26133 replica.cpp:664] Replica learned TRUNCATE action > at position 8 > I0813 19:55:17.997756 26128 hierarchical.hpp:1008] No resources available to > allocate! > I0813 19:55:17.997925 26128 hierarchical.hpp:926] Performed allocation for > slave 20150813-195517-167907756-60249-26100-S2 in 1.14489ms > I0813 19:55:17.998159 26128 hierarchical.hpp:600] Slave > 20150813-195517-167907756-60249-26100-S2 (297daca2d01a) updated with > oversubscribed resources (total: cpus(*):2; mem(*):10240; > disk(*):3.70122e+06; ports(*):[31000-32000], allocated: ) > I0813 19:55:17.998445 26128 hierarchical.hpp:1008] No resources available to > allocate! > I0813 19:55:17.998471 26128 hierarchical.hpp:926] Performed allocation for > slave 20150813-195517-167907756-60249-26100-S2 in 218856ns > I0813 19:55:18.190146 26133 hierarchical.hpp:1008] No resources available to > allocate! > I0813 19:55:18.190217 26133 hierarchical.hpp:908] Performed allocation for 3 > slaves in 637042ns > I0813 19:55:19.191346 26131 hierarchical.hpp:1008] No resources available to > allocate! > I0813 19:55:19.191915 26131 hierarchical.hpp:908] Performed allocation for 3 > slaves in 1.215355ms > I0813 19:55:20.193631 26135 hierarchical.hpp:1008] No resources available to > allocate! > I0813 19:55:20.193709 26135 hierarchical.hpp:908] Performed allocation for 3 > slaves in 834491ns > I0813 19:55:21.194805 26134 hierarchical.hpp:1008] No resources available to > allocate! > I0813 19:55:21.194870 26134 hierarchical.hpp:908] Performed allocation for 3 > slaves in 536547ns > I0813 19:55:22.196143 26137 hierarchical.hpp:1008] No resources available to > allocate! > I0813 19:55:22.196216 26137 hierarchical.hpp:908] Performed allocation for 3 > slaves in 755140ns > I0813 19:55:23.197412 26132 hierarchical.hpp:1008] No resources available to > allocate! > I0813 19:55:23.197979 26132 hierarchical.hpp:908] Performed allocation for 3 > slaves in 1.223984ms > I0813 19:55:24.199429 26132 hierarchical.hpp:1008] No resources available to > allocate! > I0813 19:55:24.199735 26132 hierarchical.hpp:908] Performed allocation for 3 > slaves in 904654ns > I0813 19:55:25.200978 26127 hierarchical.hpp:1008] No resources available to > allocate! > I0813 19:55:25.201206 26127 hierarchical.hpp:908] Performed allocation for 3 > slaves in 939979ns > I0813 19:55:26.203023 26132 hierarchical.hpp:1008] No resources available to > allocate! > I0813 19:55:26.203101 26132 hierarchical.hpp:908] Performed allocation for 3 > slaves in 721178ns > I0813 19:55:27.204815 26126 hierarchical.hpp:1008] No resources available to > allocate! > I0813 19:55:27.204888 26126 hierarchical.hpp:908] Performed allocation for 3 > slaves in 767983ns > I0813 19:55:28.206374 26126 hierarchical.hpp:1008] No resources available to > allocate! > I0813 19:55:28.206444 26126 hierarchical.hpp:908] Performed allocation for 3 > slaves in 745214ns > I0813 19:55:29.207515 26124 hierarchical.hpp:1008] No resources available to > allocate! > I0813 19:55:29.207579 26124 hierarchical.hpp:908] Performed allocation for 3 > slaves in 551217ns > I0813 19:55:30.208966 26136 hierarchical.hpp:1008] No resources available to > allocate! > I0813 19:55:30.209053 26136 hierarchical.hpp:908] Performed allocation for 3 > slaves in 649887ns > I0813 19:55:31.210078 26123 hierarchical.hpp:1008] No resources available to > allocate! > I0813 19:55:31.210144 26123 hierarchical.hpp:908] Performed allocation for 3 > slaves in 558919ns > I0813 19:55:32.211027 26130 slave.cpp:4226] Querying resource estimator for > oversubscribable resources > I0813 19:55:32.211045 26129 slave.cpp:4226] Querying resource estimator for > oversubscribable resources > I0813 19:55:32.211084 26132 slave.cpp:4226] Querying resource estimator for > oversubscribable resources > I0813 19:55:32.211386 26129 slave.cpp:4240] Received oversubscribable > resources from the resource estimator > I0813 19:55:32.211688 26132 slave.cpp:4240] Received oversubscribable > resources from the resource estimator > I0813 19:55:32.211853 26133 hierarchical.hpp:1008] No resources available to > allocate! > I0813 19:55:32.212035 26133 hierarchical.hpp:908] Performed allocation for 3 > slaves in 898985ns > I0813 19:55:32.212169 26133 slave.cpp:4240] Received oversubscribable > resources from the resource estimator > I0813 19:55:32.336745 26135 slave.cpp:3058] Received ping from > slave-observer(1)@172.17.2.10:60249 > I0813 19:55:32.514333 26129 slave.cpp:3058] Received ping from > slave-observer(2)@172.17.2.10:60249 > I0813 19:55:32.996134 26128 slave.cpp:3058] Received ping from > slave-observer(3)@172.17.2.10:60249 > I0813 19:55:33.213248 26128 hierarchical.hpp:1008] No resources available to > allocate! > I0813 19:55:33.213326 26128 hierarchical.hpp:908] Performed allocation for 3 > slaves in 827511ns > I0813 19:55:34.214326 26125 hierarchical.hpp:1008] No resources available to > allocate! > I0813 19:55:34.214391 26125 hierarchical.hpp:908] Performed allocation for 3 > slaves in 546422ns > I0813 19:55:35.215909 26123 hierarchical.hpp:1008] No resources available to > allocate! > I0813 19:55:35.215973 26123 hierarchical.hpp:908] Performed allocation for 3 > slaves in 627190ns > I0813 19:55:36.217156 26134 hierarchical.hpp:1008] No resources available to > allocate! > I0813 19:55:36.217339 26134 hierarchical.hpp:908] Performed allocation for 3 > slaves in 906249ns > I0813 19:55:37.218739 26132 hierarchical.hpp:1008] No resources available to > allocate! > I0813 19:55:37.219169 26132 hierarchical.hpp:908] Performed allocation for 3 > slaves in 1.102465ms > I0813 19:55:38.220641 26133 hierarchical.hpp:1008] No resources available to > allocate! > I0813 19:55:38.220711 26133 hierarchical.hpp:908] Performed allocation for 3 > slaves in 643146ns > I0813 19:55:39.221976 26133 hierarchical.hpp:1008] No resources available to > allocate! > I0813 19:55:39.222118 26133 hierarchical.hpp:908] Performed allocation for 3 > slaves in 845334ns > I0813 19:55:40.223338 26129 hierarchical.hpp:1008] No resources available to > allocate! > I0813 19:55:40.223546 26129 hierarchical.hpp:908] Performed allocation for 3 > slaves in 849995ns > I0813 19:55:41.225558 26138 hierarchical.hpp:1008] No resources available to > allocate! > I0813 19:55:41.225752 26138 hierarchical.hpp:908] Performed allocation for 3 > slaves in 958480ns > I0813 19:55:42.227176 26131 hierarchical.hpp:1008] No resources available to > allocate! > I0813 19:55:42.227378 26131 hierarchical.hpp:908] Performed allocation for 3 > slaves in 927048ns > I0813 19:55:43.228813 26137 hierarchical.hpp:1008] No resources available to > allocate! > I0813 19:55:43.229441 26137 hierarchical.hpp:908] Performed allocation for 3 > slaves in 1.310118ms > I0813 19:55:44.230828 26135 hierarchical.hpp:1008] No resources available to > allocate! > I0813 19:55:44.231142 26135 hierarchical.hpp:908] Performed allocation for 3 > slaves in 896369ns > I0813 19:55:45.232656 26135 hierarchical.hpp:1008] No resources available to > allocate! > I0813 19:55:45.232903 26135 hierarchical.hpp:908] Performed allocation for 3 > slaves in 1.357693ms > I0813 19:55:46.234973 26137 hierarchical.hpp:1008 > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)