[
https://issues.apache.org/jira/browse/MESOS-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kasim updated MESOS-6205:
-------------------------
Description:
I follow this
[doc][https://open.mesosphere.com/getting-started/install/#verifying-installation]
to setup mesos cluster.
There are three vm(ubuntu 12, centos 6.5, centos 7.2).
$ cat /etc/hosts
10.142.55.190 zk1
10.142.55.196 zk2
10.142.55.202 zk3
config in each mathine:
$ cat /etc/mesos/zk
zk://10.142.55.190:2181,10.142.55.196:2181,10.142.55.202:2181/mesos
After start zookeeper, mesos-master and mesos-slave in three vm, I can view the
mesos webui(10.142.55.190:5050), but agents count is 0.
After a little time, mesos page get error:
Failed to connect to 10.142.55.190:5050!
Retrying in 16 seconds...
(I found that zookeeper would elect a new leader in a short interval)
mesos-master cmd:
```
mesos-master --agent_ping_timeout="15secs" --agent_reregister_timeout="10mins"
--allocation_interval="1secs" --allocator="HierarchicalDRF"
--authenticate_agents="false" --authenticate_frameworks="false"
--authenticate_http_frameworks="false" --authenticate_http_readonly="false"
--authenticate_http_readwrite="false" --authenticators="crammd5"
--authorizers="local" --framework_sorter="drf" --help="false"
--hostname_lookup="true" --http_authenticators="basic"
--initialize_driver_logging="true" --ip="10.142.55.190"
--log_auto_initialize="true" --log_dir="/var/log/mesos" --logbufsecs="0"
--logging_level="INFO" --max_agent_ping_timeouts="5"
--max_completed_frameworks="50" --max_completed_tasks_per_framework="1000"
--port="5050" --quiet="false" --quorum="2"
--recovery_agent_removal_limit="100%" --registry="replicated_log"
--registry_fetch_timeout="1mins" --registry_store_timeout="20secs"
--registry_strict="false" --root_submissions="true" --user_sorter="drf"
--version="false" --webui_dir="/usr/share/mesos/webui"
--work_dir="/var/lib/mesos"
--zk="zk://10.142.55.190:2181,10.142.55.196:2181,10.142.55.202:2181/mesos"
```
mesos-slave cmd:
```
mesos-slave --appc_simple_discovery_uri_prefix="http://";
--appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="false"
--authenticate_http_readwrite="false" --authenticatee="crammd5"
--authentication_backoff_factor="1secs" --authorizer="local"
--cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false"
--cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false"
--cgroups_root="mesos" --container_disk_watch_interval="15secs"
--containerizers="mesos" --default_role="*" --disk_watch_interval="1mins"
--docker="docker" --docker_kill_orphans="true"
--docker_registry="https://registry-1.docker.io"; --docker_remove_delay="6hrs"
--docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns"
--docker_store_dir="/tmp/mesos/store/docker"
--docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume"
--enforce_container_disk_quota="false" --executor_registration_timeout="1mins"
--executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch"
--fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks"
--gc_disk_headroom="0.1" --hadoop_home="" --help="false"
--hostname="10.142.55.190" --hostname_lookup="true"
--http_authenticators="basic" --http_command_executor="false"
--image_provisioner_backend="copy" --initialize_driver_logging="true"
--ip="10.142.55.190" --isolation="posix/cpu,posix/mem" --launcher="posix"
--launcher_dir="/usr/libexec/mesos" --log_dir="/var/log/mesos" --logbufsecs="0"
--logging_level="INFO"
--master="zk://10.142.55.190:2181,10.142.55.196:2181,10.142.55.202:2181/mesos"
--oversubscribed_resources_interval="15secs" --perf_duration="10secs"
--perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns"
--quiet="false" --recover="reconnect" --recovery_timeout="15mins"
--registration_backoff_factor="1secs" --revocable_cpu_low_priority="true"
--sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true"
--systemd_enable_support="true"
--systemd_runtime_directory="/run/systemd/system" --version="false"
--work_dir="/var/lib/mesos"
```
When I run mesos-master from command-line, I got
```
I0919 17:20:19.286264 17550 replica.cpp:673] Replica in VOTING status received
a broadcasted recover request from (583)@10.142.55.202:5050
F0919 17:20:20.009371 17556 master.cpp:1536] Recovery failed: Failed to recover
registrar: Failed to perform fetch within 1mins
*** Check failure stack trace: ***
@ 0x7f9db78458dd google::LogMessage::Fail()
@ 0x7f9db784771d google::LogMessage::SendToLog()
@ 0x7f9db78454cc google::LogMessage::Flush()
@ 0x7f9db7848019 google::LogMessageFatal::~LogMessageFatal()
@ 0x7f9db6e2dbbc mesos::internal::master::fail()
@ 0x7f9db6e75b20
_ZNSt17_Function_handlerIFvRKSsEZNK7process6FutureI7NothingE8onFailedISt5_BindIFPFvS1_S1_EPKcSt12_PlaceholderILi1EE
EEvEERKS6_OT_NS6_6PreferEEUlS1_E_E9_M_invokeERKSt9_Any_dataS1_
@ 0x42a116 process::Future<>::fail()
@ 0x7f9db6e9f705 process::internal::thenf<>()
@ 0x7f9db6efd016
_ZN7process8internal3runISt8functionIFvRKNS_6FutureIN5mesos8internal8RegistryEEEEEJRS7_EEEvRKSt6vectorIT_SaISE_EEDp
OT0_
I0919 17:20:20.025172 17553 replica.cpp:673] Replica in VOTING status received
a broadcasted recover request from (212)@10.142.55.196:5050
@ 0x7f9db6f100de process::Future<>::fail()
@ 0x7f9db6c57e86 process::internal::run<>()
@ 0x7f9db6f100cb process::Future<>::fail()
@ 0x7f9db6ef2d34 mesos::internal::master::RegistrarProcess::_recover()
@ 0x7f9db77d5171 process::ProcessManager::resume()
@ 0x7f9db77d5477
_ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUt_vEEE6_M_runEv
@ 0x7f9db5e439c0 (unknown)
@ 0x7f9db568ae9a start_thread
@ 0x7f9db53b836d (unknown)
[1] 17548 abort (core dumped) mesos-master --agent_ping_timeout="15secs"
--agent_reregister_timeout="10mins
```
master info log:
I0919 15:54:59.677438 13281 http.cpp:2022] Redirecting request for
/master/state?jsonp=angular.callbacks._1x to the leading master zk3
I0919 15:55:00.098667 13281 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (768)@10.142.55.202:5050
I0919 15:55:00.385279 13281 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (185)@10.142.55.196:5050
I0919 15:55:00.711119 13281 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (771)@10.142.55.202:5050
I0919 15:55:01.347291 13284 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (188)@10.142.55.196:5050
I0919 15:55:01.597682 13284 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (774)@10.142.55.202:5050
I0919 15:55:02.257159 13282 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (191)@10.142.55.196:5050
I0919 15:55:02.370692 13287 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (777)@10.142.55.202:5050
I0919 15:55:03.205920 13285 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (780)@10.142.55.202:5050
I0919 15:55:03.260007 13281 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (194)@10.142.55.196:5050
I0919 15:55:03.929611 13283 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (783)@10.142.55.202:5050
I0919 15:55:04.033308 13287 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (197)@10.142.55.196:5050
I0919 15:55:04.591275 13284 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (200)@10.142.55.196:5050
I0919 15:55:04.608211 13283 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (786)@10.142.55.202:5050
I0919 15:55:05.184682 13280 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (789)@10.142.55.202:5050
I0919 15:55:05.268277 13280 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (203)@10.142.55.196:5050
I0919 15:55:05.775377 13281 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (206)@10.142.55.196:5050
I0919 15:55:05.916445 13285 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (792)@10.142.55.202:5050
I0919 15:55:06.744927 13280 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (209)@10.142.55.196:5050
I0919 15:55:07.378521 13283 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (5)@10.142.55.202:5050
I0919 15:55:07.393311 13285 network.hpp:430] ZooKeeper group memberships
changed
I0919 15:55:07.393427 13285 group.cpp:706] Trying to get
'/mesos/log_replicas/0000000709' in ZooKeeper
I0919 15:55:07.393985 13285 group.cpp:706] Trying to get
'/mesos/log_replicas/0000000711' in ZooKeeper
I0919 15:55:07.394394 13285 group.cpp:706] Trying to get
'/mesos/log_replicas/0000000714' in ZooKeeper
I0919 15:55:07.394843 13285 group.cpp:706] Trying to get
'/mesos/log_replicas/0000000715' in ZooKeeper
I0919 15:55:07.395418 13285 network.hpp:478] ZooKeeper group PIDs: {
log-replica(1)@10.142.55.190:5050, log-replica(1)@10.142.55.196:5050,
log-replica(1)@10.142.55.202:5050 }
I0919 15:55:08.178272 13280 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (14)@10.142.55.202:5050
I0919 15:55:09.059562 13282 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (21)@10.142.55.202:5050
I0919 15:55:09.700711 13286 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (24)@10.142.55.202:5050
I0919 15:55:09.742185 13287 http.cpp:381] HTTP GET for /master/state from
10.142.50.94:59987 with User-Agent='Mozilla/5.0 (Windows NT 6.2; WOW64;
rv:47.0) Gecko/20100101 Firefox/47.0'
I0919 15:55:09.742359 13287 http.cpp:2022] Redirecting request for
/master/state?jsonp=angular.callbacks._1y to the leading master zk3
I0919 15:55:10.660789 13280 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (30)@10.142.55.202:5050
I0919 15:55:11.480326 13281 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (34)@10.142.55.202:5050
I0919 15:55:12.386256 13286 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (37)@10.142.55.202:5050
I0919 15:55:12.975137 13287 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (42)@10.142.55.202:5050
I0919 15:55:13.843091 13285 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (47)@10.142.55.202:5050
I0919 15:55:14.373478 13281 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (51)@10.142.55.202:5050
I0919 15:55:14.937181 13280 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (54)@10.142.55.202:5050
I0919 15:55:15.658219 13283 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (58)@10.142.55.202:5050
I0919 15:55:16.007822 13286 network.hpp:430] ZooKeeper group memberships
changed
I0919 15:55:16.007972 13286 group.cpp:706] Trying to get
'/mesos/log_replicas/0000000711' in ZooKeeper
I0919 15:55:16.010170 13286 group.cpp:706] Trying to get
'/mesos/log_replicas/0000000714' in ZooKeeper
I0919 15:55:16.011462 13284 detector.cpp:152] Detected a new leader:
(id='702')
I0919 15:55:16.011556 13284 group.cpp:706] Trying to get
'/mesos/json.info_0000000702' in ZooKeeper
I0919 15:55:16.011968 13286 group.cpp:706] Trying to get
'/mesos/log_replicas/0000000715' in ZooKeeper
I0919 15:55:16.012526 13286 network.hpp:478] ZooKeeper group PIDs: {
log-replica(1)@10.142.55.190:5050, log-replica(1)@10.142.55.196:5050,
log-replica(1)@10.142.55.202:5050 }
I0919 15:55:16.013156 13284 zookeeper.cpp:259] A new leading master
(UPID=master@10.142.55.190:5050) is detected
I0919 15:55:16.013222 13284 master.cpp:1847] The newly elected leader is
master@10.142.55.190:5050 with id 677967bc-f6f0-46b3-a44e-72eed1befd60
I0919 15:55:16.013244 13284 master.cpp:1860] Elected as the leading master!
I0919 15:55:16.013273 13284 master.cpp:1547] Recovering from registrar
I0919 15:55:16.013352 13284 registrar.cpp:332] Recovering registrar
I0919 15:55:16.014081 13280 log.cpp:553] Attempting to start the writer
I0919 15:55:16.014515 13280 replica.cpp:493] Replica received implicit
promise request from (211)@10.142.55.190:5050 with proposal 1204590
I0919 15:55:16.018023 13282 consensus.cpp:360] Aborting implicit promise
request because 2 ignores received
I0919 15:55:16.018028 13280 leveldb.cpp:304] Persisting metadata (10 bytes)
to leveldb took 3.469479ms
I0919 15:55:16.018338 13280 replica.cpp:342] Persisted promised to 1204590
I0919 15:55:16.018508 13282 log.cpp:565] Could not start the writer, but
can be retried
I0919 15:55:16.018645 13282 log.cpp:553] Attempting to start the writer
I0919 15:55:16.018899 13282 replica.cpp:493] Replica received implicit
promise request from (215)@10.142.55.190:5050 with proposal 1204591
I0919 15:55:16.022183 13287 consensus.cpp:360] Aborting implicit promise
request because 2 ignores received
I0919 15:55:16.022367 13280 log.cpp:565] Could not start the writer, but
can be retried
I0919 15:55:16.022510 13280 log.cpp:553] Attempting to start the writer
I0919 15:55:16.028880 13282 leveldb.cpp:304] Persisting metadata (10 bytes)
to leveldb took 9.870818ms
I0919 15:55:16.029024 13282 replica.cpp:342] Persisted promised to 1204591
I0919 15:55:16.029428 13286 replica.cpp:493] Replica received implicit
promise request from (219)@10.142.55.190:5050 with proposal 1204592
I0919 15:55:16.031600 13280 consensus.cpp:360] Aborting implicit promise
request because 2 ignores received
I0919 15:55:16.036208 13283 log.cpp:565] Could not start the writer, but
can be retried
I0919 15:55:16.036454 13283 log.cpp:553] Attempting to start the writer
I0919 15:55:16.040256 13286 leveldb.cpp:304] Persisting metadata (10 bytes)
to leveldb took 10.783237ms
I0919 15:55:16.040339 13286 replica.cpp:342] Persisted promised to 1204592
I0919 15:55:16.040712 13286 replica.cpp:493] Replica received implicit
promise request from (222)@10.142.55.190:5050 with proposal 1204593
I0919 15:55:16.042196 13286 leveldb.cpp:304] Persisting metadata (10 bytes)
to leveldb took 1.435071ms
I0919 15:55:16.042250 13286 replica.cpp:342] Persisted promised to 1204593
I0919 15:55:16.042981 13286 consensus.cpp:360] Aborting implicit promise
request because 2 ignores received
I0919 15:55:16.043099 13286 log.cpp:565] Could not start the writer, but
can be retried
I0919 15:55:16.043303 13283 log.cpp:553] Attempting to start the writer
All later logs are looping :
I0919 15:55:16.043676 13286 replica.cpp:493] Replica received implicit
promise request from (225)@10.142.55.190:5050 with proposal 1204594
I0919 15:55:16.044122 13286 leveldb.cpp:304] Persisting metadata (10 bytes)
to leveldb took 404769ns
I0919 15:55:16.044209 13286 replica.cpp:342] Persisted promised to 1204594
I0919 15:55:16.044837 13281 consensus.cpp:360] Aborting implicit promise
request because 2 ignores received
I0919 15:55:16.044926 13281 log.cpp:565] Could not start the writer, but
can be retried
I0919 15:55:16.045038 13281 log.cpp:553] Attempting to start the writer
slave info log:
Log file created at: 2016/09/19 15:41:16
Running on machine: ubuntu12
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0919 15:41:16.346844 12986 logging.cpp:194] INFO level logging started!
I0919 15:41:16.363313 12986 containerizer.cpp:196] Using isolation:
posix/cpu,posix/mem,filesystem/posix,network/cni
I0919 15:41:16.370334 12986 main.cpp:434] Starting Mesos agent
I0919 15:41:16.371184 12986 slave.cpp:198] Agent started on
1)@127.0.1.1:5051
I0919 15:41:16.371636 12986 slave.cpp:199] Flags at startup:
--appc_simple_discovery_uri_prefix="http://";
--appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="false"
--authenticate_http_readwrite="false" --authenticatee="crammd5"
--authentication_backoff_factor="1secs" --authorizer="local"
--cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false"
--cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false"
--cgroups_root="mesos" --container_disk_watch_interval="15secs"
--containerizers="mesos" --default_role="*" --disk_watch_interval="1mins"
--docker="docker" --docker_kill_orphans="true"
--docker_registry="https://registry-1.docker.io"; --docker_remove_delay="6hrs"
--docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns"
--docker_store_dir="/tmp/mesos/store/docker"
--docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume"
--enforce_container_disk_quota="false" --executor_registration_timeout="1mins"
--executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch"
--fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks"
--gc_disk_headroom="0.1" --hadoop_home="" --help="false"
--hostname_lookup="true" --http_authenticators="basic"
--http_command_executor="false" --image_provisioner_backend="copy"
--initialize_driver_logging="true" --isolation="posix/cpu,posix/mem"
--launcher="posix" --launcher_dir="/usr/libexec/mesos"
--log_dir="/var/log/mesos" --logbufsecs="0" --logging_level="INFO"
--master="zk://10.142.55.190:2181,10.142.55.196:2181,10.142.55.202:2181/mesos"
--oversubscribed_resources_interval="15secs" --perf_duration="10secs"
--perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns"
--quiet="false" --recover="reconnect" --recovery_timeout="15mins"
--registration_backoff_factor="1secs" --revocable_cpu_low_priority="true"
--sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true"
--systemd_enable_support="true"
--systemd_runtime_directory="/run/systemd/system" --version="false"
--work_dir="/var/lib/mesos"
I0919 15:41:16.373072 12986 slave.cpp:519] Agent resources: cpus(*):2;
mem(*):2930; disk(*):4469; ports(*):[31000-32000]
I0919 15:41:16.373291 12986 slave.cpp:527] Agent attributes: [ ]
I0919 15:41:16.373347 12986 slave.cpp:532] Agent hostname: ubuntu12
I0919 15:41:16.379895 13005 state.cpp:57] Recovering state from
'/var/lib/mesos/meta'
I0919 15:41:16.382519 13005 group.cpp:349] Group process
(group(1)@127.0.1.1:5051) connected to ZooKeeper
I0919 15:41:16.382593 13005 group.cpp:837] Syncing group operations: queue
size (joins, cancels, datas) = (0, 0, 0)
I0919 15:41:16.382663 13005 group.cpp:427] Trying to create path '/mesos'
in ZooKeeper
I0919 15:41:16.382910 13009 status_update_manager.cpp:200] Recovering
status update manager
I0919 15:41:16.383419 13009 containerizer.cpp:522] Recovering containerizer
I0919 15:41:16.392206 13004 provisioner.cpp:253] Provisioner recovery
complete
I0919 15:41:16.392354 13004 slave.cpp:4782] Finished recovery
I0919 15:41:16.405709 13004 detector.cpp:152] Detected a new leader:
(id='678')
I0919 15:41:16.406067 13005 group.cpp:706] Trying to get
'/mesos/json.info_0000000678' in ZooKeeper
I0919 15:41:16.407572 13002 zookeeper.cpp:259] A new leading master
(UPID=master@10.142.55.190:5050) is detected
I0919 15:41:16.407977 13002 slave.cpp:895] New master detected at
master@10.142.55.190:5050
I0919 15:41:16.408043 13002 slave.cpp:916] No credentials provided.
Attempting to register without authentication
I0919 15:41:16.408140 13002 slave.cpp:927] Detecting new master
I0919 15:41:16.408223 13005 status_update_manager.cpp:174] Pausing sending
status updates
I0919 15:42:08.418956 13006 slave.cpp:3732] master@10.142.55.190:5050 exited
W0919 15:42:08.419035 13006 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
I0919 15:42:16.374977 13007 slave.cpp:4591] Current disk usage 72.41%. Max
allowed age: 1.231186482451933days
I0919 15:42:20.007169 13007 detector.cpp:152] Detected a new leader:
(id='679')
I0919 15:42:20.007297 13007 group.cpp:706] Trying to get
'/mesos/json.info_0000000679' in ZooKeeper
I0919 15:42:20.008503 13007 zookeeper.cpp:259] A new leading master
(UPID=master@10.142.55.196:5050) is detected
I0919 15:42:20.008587 13007 slave.cpp:895] New master detected at
master@10.142.55.196:5050
I0919 15:42:20.008610 13007 slave.cpp:916] No credentials provided.
Attempting to register without authentication
I0919 15:42:20.008703 13007 slave.cpp:927] Detecting new master
I0919 15:42:20.008750 13007 status_update_manager.cpp:174] Pausing sending
status updates
I0919 15:43:16.387984 13005 slave.cpp:4591] Current disk usage 72.41%. Max
allowed age: 1.231162010606794days
I0919 15:43:20.081272 13005 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:43:20.081374 13005 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
I0919 15:43:26.855154 13005 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:43:26.855315 13005 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:43:26.855159 13010 process.cpp:2105] Failed to shutdown socket
with fd 12: Transport endpoint is not connected
I0919 15:43:32.020196 13002 detector.cpp:152] Detected a new leader:
(id='682')
I0919 15:43:32.020300 13002 group.cpp:706] Trying to get
'/mesos/json.info_0000000682' in ZooKeeper
I0919 15:43:32.022203 13002 zookeeper.cpp:259] A new leading master
(UPID=master@10.142.55.202:5050) is detected
I0919 15:43:32.022302 13002 slave.cpp:895] New master detected at
master@10.142.55.202:5050
I0919 15:43:32.022328 13002 slave.cpp:916] No credentials provided.
Attempting to register without authentication
I0919 15:43:32.022382 13002 slave.cpp:927] Detecting new master
I0919 15:43:32.022423 13002 status_update_manager.cpp:174] Pausing sending
status updates
I0919 15:44:16.389369 13003 slave.cpp:4591] Current disk usage 72.41%. Max
allowed age: 1.231119184877789days
I0919 15:44:32.535347 13003 slave.cpp:3732] master@10.142.55.202:5050 exited
W0919 15:44:32.535522 13003 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
I0919 15:44:42.005375 13002 detector.cpp:152] Detected a new leader:
(id='684')
I0919 15:44:42.005496 13002 group.cpp:706] Trying to get
'/mesos/json.info_0000000684' in ZooKeeper
I0919 15:44:42.006367 13002 zookeeper.cpp:259] A new leading master
(UPID=master@10.142.55.190:5050) is detected
I0919 15:44:42.006492 13002 slave.cpp:895] New master detected at
master@10.142.55.190:5050
I0919 15:44:42.006597 13002 slave.cpp:916] No credentials provided.
Attempting to register without authentication
I0919 15:44:42.006675 13002 slave.cpp:927] Detecting new master
I0919 15:44:42.006577 13008 status_update_manager.cpp:174] Pausing sending
status updates
I0919 15:45:16.400794 13006 slave.cpp:4591] Current disk usage 72.48%. Max
allowed age: 1.226390000804074days
I0919 15:45:42.354790 13005 slave.cpp:3732] master@10.142.55.190:5050 exited
W0919 15:45:42.354857 13005 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
I0919 15:45:54.020563 13002 detector.cpp:152] Detected a new leader:
(id='687')
I0919 15:45:54.020756 13002 group.cpp:706] Trying to get
'/mesos/json.info_0000000687' in ZooKeeper
I0919 15:45:54.023296 13002 zookeeper.cpp:259] A new leading master
(UPID=master@10.142.55.196:5050) is detected
I0919 15:45:54.023455 13002 slave.cpp:895] New master detected at
master@10.142.55.196:5050
I0919 15:45:54.023558 13002 slave.cpp:916] No credentials provided.
Attempting to register without authentication
I0919 15:45:54.023526 13008 status_update_manager.cpp:174] Pausing sending
status updates
I0919 15:45:54.023669 13002 slave.cpp:927] Detecting new master
I0919 15:46:16.402402 13003 slave.cpp:4591] Current disk usage 72.53%. Max
allowed age: 1.223205601954942days
I0919 15:46:54.075505 13007 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:46:54.075592 13007 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:46:56.098012 13010 process.cpp:2105] Failed to shutdown socket
with fd 14: Transport endpoint is not connected
I0919 15:46:56.098016 13007 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:46:56.098253 13007 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:46:56.462254 13010 process.cpp:2105] Failed to shutdown socket
with fd 14: Transport endpoint is not connected
I0919 15:46:56.462260 13005 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:46:56.462540 13005 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
I0919 15:47:02.005637 13009 detector.cpp:152] Detected a new leader:
(id='688')
I0919 15:47:02.005765 13009 group.cpp:706] Trying to get
'/mesos/json.info_0000000688' in ZooKeeper
I0919 15:47:02.006853 13009 zookeeper.cpp:259] A new leading master
(UPID=master@10.142.55.202:5050) is detected
I0919 15:47:02.006959 13009 slave.cpp:895] New master detected at
master@10.142.55.202:5050
I0919 15:47:02.006986 13009 slave.cpp:916] No credentials provided.
Attempting to register without authentication
I0919 15:47:02.007025 13009 slave.cpp:927] Detecting new master
I0919 15:47:02.007061 13009 status_update_manager.cpp:174] Pausing sending
status updates
I0919 15:47:16.406669 13008 slave.cpp:4591] Current disk usage 72.53%. Max
allowed age: 1.223184189090440days
I0919 15:48:02.950891 13005 slave.cpp:3732] master@10.142.55.202:5050 exited
W0919 15:48:02.950994 13005 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
I0919 15:48:12.006634 13005 detector.cpp:152] Detected a new leader:
(id='690')
I0919 15:48:12.006817 13003 group.cpp:706] Trying to get
'/mesos/json.info_0000000690' in ZooKeeper
I0919 15:48:12.007987 13003 zookeeper.cpp:259] A new leading master
(UPID=master@10.142.55.190:5050) is detected
I0919 15:48:12.008126 13003 slave.cpp:895] New master detected at
master@10.142.55.190:5050
I0919 15:48:12.008210 13003 slave.cpp:916] No credentials provided.
Attempting to register without authentication
I0919 15:48:12.008280 13003 slave.cpp:927] Detecting new master
I0919 15:48:12.008191 13008 status_update_manager.cpp:174] Pausing sending
status updates
I0919 15:48:16.409266 13003 slave.cpp:4591] Current disk usage 72.54%. Max
allowed age: 1.222480623542604days
I0919 15:49:12.379010 13009 slave.cpp:3732] master@10.142.55.190:5050 exited
W0919 15:49:12.379149 13009 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:49:12.379233 13010 process.cpp:2105] Failed to shutdown socket
with fd 12: Transport endpoint is not connected
I0919 15:49:16.413767 13007 slave.cpp:4591] Current disk usage 72.64%. Max
allowed age: 1.215032005677465days
I0919 15:49:24.016290 13007 detector.cpp:152] Detected a new leader:
(id='693')
I0919 15:49:24.016417 13007 group.cpp:706] Trying to get
'/mesos/json.info_0000000693' in ZooKeeper
I0919 15:49:24.018273 13007 zookeeper.cpp:259] A new leading master
(UPID=master@10.142.55.196:5050) is detected
I0919 15:49:24.018437 13007 slave.cpp:895] New master detected at
master@10.142.55.196:5050
I0919 15:49:24.018523 13007 slave.cpp:916] No credentials provided.
Attempting to register without authentication
I0919 15:49:24.018604 13007 slave.cpp:927] Detecting new master
I0919 15:49:24.018496 13008 status_update_manager.cpp:174] Pausing sending
status updates
I0919 15:50:16.416391 13008 slave.cpp:4591] Current disk usage 72.64%. Max
allowed age: 1.215016710774248days
I0919 15:50:24.065268 13003 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:50:24.065342 13003 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
I0919 15:50:24.485752 13004 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:50:24.485839 13004 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:50:24.485977 13010 process.cpp:2105] Failed to shutdown socket
with fd 14: Transport endpoint is not connected
I0919 15:50:28.343647 13003 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:50:28.343719 13003 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:50:28.343819 13010 process.cpp:2105] Failed to shutdown socket
with fd 14: Transport endpoint is not connected
I0919 15:50:31.545099 13005 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:50:31.545171 13005 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:50:31.545284 13010 process.cpp:2105] Failed to shutdown socket
with fd 14: Transport endpoint is not connected
I0919 15:50:32.007096 13008 detector.cpp:152] Detected a new leader:
(id='694')
I0919 15:50:32.007195 13008 group.cpp:706] Trying to get
'/mesos/json.info_0000000694' in ZooKeeper
I0919 15:50:32.009881 13008 zookeeper.cpp:259] A new leading master
(UPID=master@10.142.55.202:5050) is detected
I0919 15:50:32.009970 13008 slave.cpp:895] New master detected at
master@10.142.55.202:5050
I0919 15:50:32.009994 13008 slave.cpp:916] No credentials provided.
Attempting to register without authentication
I0919 15:50:32.010030 13008 slave.cpp:927] Detecting new master
I0919 15:50:32.010079 13008 status_update_manager.cpp:174] Pausing sending
status updates
I0919 15:51:16.417846 13006 slave.cpp:4591] Current disk usage 72.64%. Max
allowed age: 1.214964708103322days
I0919 15:51:32.560317 13003 slave.cpp:3732] master@10.142.55.202:5050 exited
W0919 15:51:32.560410 13003 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
I0919 15:51:42.005147 13009 detector.cpp:152] Detected a new leader:
(id='696')
I0919 15:51:42.005265 13009 group.cpp:706] Trying to get
'/mesos/json.info_0000000696' in ZooKeeper
I0919 15:51:42.006824 13009 zookeeper.cpp:259] A new leading master
(UPID=master@10.142.55.190:5050) is detected
I0919 15:51:42.006904 13009 slave.cpp:895] New master detected at
master@10.142.55.190:5050
I0919 15:51:42.006928 13009 slave.cpp:916] No credentials provided.
Attempting to register without authentication
I0919 15:51:42.006963 13009 slave.cpp:927] Detecting new master
I0919 15:51:42.006999 13009 status_update_manager.cpp:174] Pausing sending
status updates
I0919 15:52:16.419373 13003 slave.cpp:4591] Current disk usage 72.71%. Max
allowed age: 1.209981628636250days
I0919 15:52:42.336305 13002 slave.cpp:3732] master@10.142.55.190:5050 exited
W0919 15:52:42.336426 13002 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
I0919 15:52:54.005267 13005 detector.cpp:152] Detected a new leader:
(id='699')
I0919 15:52:54.005408 13005 group.cpp:706] Trying to get
'/mesos/json.info_0000000699' in ZooKeeper
I0919 15:52:54.006206 13005 zookeeper.cpp:259] A new leading master
(UPID=master@10.142.55.196:5050) is detected
I0919 15:52:54.006285 13005 slave.cpp:895] New master detected at
master@10.142.55.196:5050
I0919 15:52:54.006309 13005 slave.cpp:916] No credentials provided.
Attempting to register without authentication
I0919 15:52:54.006398 13005 slave.cpp:927] Detecting new master
I0919 15:52:54.006451 13005 status_update_manager.cpp:174] Pausing sending
status updates
I0919 15:53:16.420258 13005 slave.cpp:4591] Current disk usage 72.76%. Max
allowed age: 1.206748286096840days
I0919 15:53:54.071012 13005 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:53:54.071143 13005 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
I0919 15:54:01.105780 13002 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:54:01.105854 13002 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:54:01.105970 13010 process.cpp:2105] Failed to shutdown socket
with fd 15: Transport endpoint is not connected
I0919 15:54:05.733837 13007 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:54:05.733932 13007 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:54:05.734071 13010 process.cpp:2105] Failed to shutdown socket
with fd 15: Transport endpoint is not connected
E0919 15:54:05.818560 13010 process.cpp:2105] Failed to shutdown socket
with fd 15: Transport endpoint is not connected
I0919 15:54:05.818583 13003 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:54:05.818758 13003 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
I0919 15:54:06.004385 13009 detector.cpp:152] Detected a new leader:
(id='700')
I0919 15:54:06.004494 13009 group.cpp:706] Trying to get
'/mesos/json.info_0000000700' in ZooKeeper
I0919 15:54:06.005511 13009 zookeeper.cpp:259] A new leading master
(UPID=master@10.142.55.202:5050) is detected
I0919 15:54:06.005586 13009 slave.cpp:895] New master detected at
master@10.142.55.202:5050
I0919 15:54:06.005609 13009 slave.cpp:916] No credentials provided.
Attempting to register without authentication
I0919 15:54:06.005676 13009 slave.cpp:927] Detecting new master
I0919 15:54:06.005720 13009 status_update_manager.cpp:174] Pausing sending
status updates
I0919 15:54:16.423193 13002 slave.cpp:4591] Current disk usage 72.76%. Max
allowed age: 1.206699342406551days
slave warn log
Log file created at: 2016/09/19 15:42:08
Running on machine: ubuntu12
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
W0919 15:42:08.419035 13006 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
W0919 15:43:20.081374 13005 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
W0919 15:43:26.855315 13005 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:43:26.855159 13010 process.cpp:2105] Failed to shutdown socket
with fd 12: Transport endpoint is not connected
W0919 15:44:32.535522 13003 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
W0919 15:45:42.354857 13005 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
W0919 15:46:54.075592 13007 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:46:56.098012 13010 process.cpp:2105] Failed to shutdown socket
with fd 14: Transport endpoint is not connected
W0919 15:46:56.098253 13007 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:46:56.462254 13010 process.cpp:2105] Failed to shutdown socket
with fd 14: Transport endpoint is not connected
W0919 15:46:56.462540 13005 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
W0919 15:48:02.950994 13005 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
W0919 15:49:12.379149 13009 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:49:12.379233 13010 process.cpp:2105] Failed to shutdown socket
with fd 12: Transport endpoint is not connected
W0919 15:50:24.065342 13003 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
W0919 15:50:24.485839 13004 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:50:24.485977 13010 process.cpp:2105] Failed to shutdown socket
with fd 14: Transport endpoint is not connected
W0919 15:50:28.343719 13003 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:50:28.343819 13010 process.cpp:2105] Failed to shutdown socket
with fd 14: Transport endpoint is not connected
W0919 15:50:31.545171 13005 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:50:31.545284 13010 process.cpp:2105] Failed to shutdown socket
with fd 14: Transport endpoint is not connected
W0919 15:51:32.560410 13003 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
W0919 15:52:42.336426 13002 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
W0919 15:53:54.071143 13005 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
W0919 15:54:01.105854 13002 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:54:01.105970 13010 process.cpp:2105] Failed to shutdown socket
with fd 15: Transport endpoint is not connected
W0919 15:54:05.733932 13007 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:54:05.734071 13010 process.cpp:2105] Failed to shutdown socket
with fd 15: Transport endpoint is not connected
E0919 15:54:05.818560 13010 process.cpp:2105] Failed to shutdown socket
with fd 15: Transport endpoint is not connected
W0919 15:54:05.818758 13003 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
W0919 15:55:06.821486 13009 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
was:
I follow this
[doc][https://open.mesosphere.com/getting-started/install/#verifying-installation]
to setup mesos cluster.
There are three vm(ubuntu 12, centos 6.5, centos 7.2).
$ cat /etc/hosts
10.142.55.190 zk1
10.142.55.196 zk2
10.142.55.202 zk3
config in each mathine:
$ cat /etc/mesos/zk
zk://10.142.55.190:2181,10.142.55.196:2181,10.142.55.202:2181/mesos
After start zookeeper, mesos-master and mesos-slave in three vm, I can view the
mesos webui(10.142.55.190:5050), but agents count is 0.
After a little time, mesos page get error:
Failed to connect to 10.142.55.190:5050!
Retrying in 16 seconds...
(I found that zookeeper would elect a new leader in a short interval)
master info log:
I0919 15:54:59.677438 13281 http.cpp:2022] Redirecting request for
/master/state?jsonp=angular.callbacks._1x to the leading master zk3
I0919 15:55:00.098667 13281 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (768)@10.142.55.202:5050
I0919 15:55:00.385279 13281 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (185)@10.142.55.196:5050
I0919 15:55:00.711119 13281 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (771)@10.142.55.202:5050
I0919 15:55:01.347291 13284 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (188)@10.142.55.196:5050
I0919 15:55:01.597682 13284 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (774)@10.142.55.202:5050
I0919 15:55:02.257159 13282 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (191)@10.142.55.196:5050
I0919 15:55:02.370692 13287 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (777)@10.142.55.202:5050
I0919 15:55:03.205920 13285 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (780)@10.142.55.202:5050
I0919 15:55:03.260007 13281 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (194)@10.142.55.196:5050
I0919 15:55:03.929611 13283 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (783)@10.142.55.202:5050
I0919 15:55:04.033308 13287 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (197)@10.142.55.196:5050
I0919 15:55:04.591275 13284 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (200)@10.142.55.196:5050
I0919 15:55:04.608211 13283 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (786)@10.142.55.202:5050
I0919 15:55:05.184682 13280 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (789)@10.142.55.202:5050
I0919 15:55:05.268277 13280 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (203)@10.142.55.196:5050
I0919 15:55:05.775377 13281 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (206)@10.142.55.196:5050
I0919 15:55:05.916445 13285 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (792)@10.142.55.202:5050
I0919 15:55:06.744927 13280 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (209)@10.142.55.196:5050
I0919 15:55:07.378521 13283 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (5)@10.142.55.202:5050
I0919 15:55:07.393311 13285 network.hpp:430] ZooKeeper group memberships
changed
I0919 15:55:07.393427 13285 group.cpp:706] Trying to get
'/mesos/log_replicas/0000000709' in ZooKeeper
I0919 15:55:07.393985 13285 group.cpp:706] Trying to get
'/mesos/log_replicas/0000000711' in ZooKeeper
I0919 15:55:07.394394 13285 group.cpp:706] Trying to get
'/mesos/log_replicas/0000000714' in ZooKeeper
I0919 15:55:07.394843 13285 group.cpp:706] Trying to get
'/mesos/log_replicas/0000000715' in ZooKeeper
I0919 15:55:07.395418 13285 network.hpp:478] ZooKeeper group PIDs: {
log-replica(1)@10.142.55.190:5050, log-replica(1)@10.142.55.196:5050,
log-replica(1)@10.142.55.202:5050 }
I0919 15:55:08.178272 13280 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (14)@10.142.55.202:5050
I0919 15:55:09.059562 13282 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (21)@10.142.55.202:5050
I0919 15:55:09.700711 13286 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (24)@10.142.55.202:5050
I0919 15:55:09.742185 13287 http.cpp:381] HTTP GET for /master/state from
10.142.50.94:59987 with User-Agent='Mozilla/5.0 (Windows NT 6.2; WOW64;
rv:47.0) Gecko/20100101 Firefox/47.0'
I0919 15:55:09.742359 13287 http.cpp:2022] Redirecting request for
/master/state?jsonp=angular.callbacks._1y to the leading master zk3
I0919 15:55:10.660789 13280 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (30)@10.142.55.202:5050
I0919 15:55:11.480326 13281 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (34)@10.142.55.202:5050
I0919 15:55:12.386256 13286 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (37)@10.142.55.202:5050
I0919 15:55:12.975137 13287 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (42)@10.142.55.202:5050
I0919 15:55:13.843091 13285 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (47)@10.142.55.202:5050
I0919 15:55:14.373478 13281 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (51)@10.142.55.202:5050
I0919 15:55:14.937181 13280 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (54)@10.142.55.202:5050
I0919 15:55:15.658219 13283 replica.cpp:673] Replica in VOTING status
received a broadcasted recover request from (58)@10.142.55.202:5050
I0919 15:55:16.007822 13286 network.hpp:430] ZooKeeper group memberships
changed
I0919 15:55:16.007972 13286 group.cpp:706] Trying to get
'/mesos/log_replicas/0000000711' in ZooKeeper
I0919 15:55:16.010170 13286 group.cpp:706] Trying to get
'/mesos/log_replicas/0000000714' in ZooKeeper
I0919 15:55:16.011462 13284 detector.cpp:152] Detected a new leader:
(id='702')
I0919 15:55:16.011556 13284 group.cpp:706] Trying to get
'/mesos/json.info_0000000702' in ZooKeeper
I0919 15:55:16.011968 13286 group.cpp:706] Trying to get
'/mesos/log_replicas/0000000715' in ZooKeeper
I0919 15:55:16.012526 13286 network.hpp:478] ZooKeeper group PIDs: {
log-replica(1)@10.142.55.190:5050, log-replica(1)@10.142.55.196:5050,
log-replica(1)@10.142.55.202:5050 }
I0919 15:55:16.013156 13284 zookeeper.cpp:259] A new leading master
(UPID=master@10.142.55.190:5050) is detected
I0919 15:55:16.013222 13284 master.cpp:1847] The newly elected leader is
master@10.142.55.190:5050 with id 677967bc-f6f0-46b3-a44e-72eed1befd60
I0919 15:55:16.013244 13284 master.cpp:1860] Elected as the leading master!
I0919 15:55:16.013273 13284 master.cpp:1547] Recovering from registrar
I0919 15:55:16.013352 13284 registrar.cpp:332] Recovering registrar
I0919 15:55:16.014081 13280 log.cpp:553] Attempting to start the writer
I0919 15:55:16.014515 13280 replica.cpp:493] Replica received implicit
promise request from (211)@10.142.55.190:5050 with proposal 1204590
I0919 15:55:16.018023 13282 consensus.cpp:360] Aborting implicit promise
request because 2 ignores received
I0919 15:55:16.018028 13280 leveldb.cpp:304] Persisting metadata (10 bytes)
to leveldb took 3.469479ms
I0919 15:55:16.018338 13280 replica.cpp:342] Persisted promised to 1204590
I0919 15:55:16.018508 13282 log.cpp:565] Could not start the writer, but
can be retried
I0919 15:55:16.018645 13282 log.cpp:553] Attempting to start the writer
I0919 15:55:16.018899 13282 replica.cpp:493] Replica received implicit
promise request from (215)@10.142.55.190:5050 with proposal 1204591
I0919 15:55:16.022183 13287 consensus.cpp:360] Aborting implicit promise
request because 2 ignores received
I0919 15:55:16.022367 13280 log.cpp:565] Could not start the writer, but
can be retried
I0919 15:55:16.022510 13280 log.cpp:553] Attempting to start the writer
I0919 15:55:16.028880 13282 leveldb.cpp:304] Persisting metadata (10 bytes)
to leveldb took 9.870818ms
I0919 15:55:16.029024 13282 replica.cpp:342] Persisted promised to 1204591
I0919 15:55:16.029428 13286 replica.cpp:493] Replica received implicit
promise request from (219)@10.142.55.190:5050 with proposal 1204592
I0919 15:55:16.031600 13280 consensus.cpp:360] Aborting implicit promise
request because 2 ignores received
I0919 15:55:16.036208 13283 log.cpp:565] Could not start the writer, but
can be retried
I0919 15:55:16.036454 13283 log.cpp:553] Attempting to start the writer
I0919 15:55:16.040256 13286 leveldb.cpp:304] Persisting metadata (10 bytes)
to leveldb took 10.783237ms
I0919 15:55:16.040339 13286 replica.cpp:342] Persisted promised to 1204592
I0919 15:55:16.040712 13286 replica.cpp:493] Replica received implicit
promise request from (222)@10.142.55.190:5050 with proposal 1204593
I0919 15:55:16.042196 13286 leveldb.cpp:304] Persisting metadata (10 bytes)
to leveldb took 1.435071ms
I0919 15:55:16.042250 13286 replica.cpp:342] Persisted promised to 1204593
I0919 15:55:16.042981 13286 consensus.cpp:360] Aborting implicit promise
request because 2 ignores received
I0919 15:55:16.043099 13286 log.cpp:565] Could not start the writer, but
can be retried
I0919 15:55:16.043303 13283 log.cpp:553] Attempting to start the writer
All later logs are looping :
I0919 15:55:16.043676 13286 replica.cpp:493] Replica received implicit
promise request from (225)@10.142.55.190:5050 with proposal 1204594
I0919 15:55:16.044122 13286 leveldb.cpp:304] Persisting metadata (10 bytes)
to leveldb took 404769ns
I0919 15:55:16.044209 13286 replica.cpp:342] Persisted promised to 1204594
I0919 15:55:16.044837 13281 consensus.cpp:360] Aborting implicit promise
request because 2 ignores received
I0919 15:55:16.044926 13281 log.cpp:565] Could not start the writer, but
can be retried
I0919 15:55:16.045038 13281 log.cpp:553] Attempting to start the writer
slave info log:
Log file created at: 2016/09/19 15:41:16
Running on machine: ubuntu12
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0919 15:41:16.346844 12986 logging.cpp:194] INFO level logging started!
I0919 15:41:16.363313 12986 containerizer.cpp:196] Using isolation:
posix/cpu,posix/mem,filesystem/posix,network/cni
I0919 15:41:16.370334 12986 main.cpp:434] Starting Mesos agent
I0919 15:41:16.371184 12986 slave.cpp:198] Agent started on
1)@127.0.1.1:5051
I0919 15:41:16.371636 12986 slave.cpp:199] Flags at startup:
--appc_simple_discovery_uri_prefix="http://";
--appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="false"
--authenticate_http_readwrite="false" --authenticatee="crammd5"
--authentication_backoff_factor="1secs" --authorizer="local"
--cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false"
--cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false"
--cgroups_root="mesos" --container_disk_watch_interval="15secs"
--containerizers="mesos" --default_role="*" --disk_watch_interval="1mins"
--docker="docker" --docker_kill_orphans="true"
--docker_registry="https://registry-1.docker.io"; --docker_remove_delay="6hrs"
--docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns"
--docker_store_dir="/tmp/mesos/store/docker"
--docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume"
--enforce_container_disk_quota="false" --executor_registration_timeout="1mins"
--executor_shutdown_grace_period="5secs" --fetcher_cache_dir="/tmp/mesos/fetch"
--fetcher_cache_size="2GB" --frameworks_home="" --gc_delay="1weeks"
--gc_disk_headroom="0.1" --hadoop_home="" --help="false"
--hostname_lookup="true" --http_authenticators="basic"
--http_command_executor="false" --image_provisioner_backend="copy"
--initialize_driver_logging="true" --isolation="posix/cpu,posix/mem"
--launcher="posix" --launcher_dir="/usr/libexec/mesos"
--log_dir="/var/log/mesos" --logbufsecs="0" --logging_level="INFO"
--master="zk://10.142.55.190:2181,10.142.55.196:2181,10.142.55.202:2181/mesos"
--oversubscribed_resources_interval="15secs" --perf_duration="10secs"
--perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns"
--quiet="false" --recover="reconnect" --recovery_timeout="15mins"
--registration_backoff_factor="1secs" --revocable_cpu_low_priority="true"
--sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true"
--systemd_enable_support="true"
--systemd_runtime_directory="/run/systemd/system" --version="false"
--work_dir="/var/lib/mesos"
I0919 15:41:16.373072 12986 slave.cpp:519] Agent resources: cpus(*):2;
mem(*):2930; disk(*):4469; ports(*):[31000-32000]
I0919 15:41:16.373291 12986 slave.cpp:527] Agent attributes: [ ]
I0919 15:41:16.373347 12986 slave.cpp:532] Agent hostname: ubuntu12
I0919 15:41:16.379895 13005 state.cpp:57] Recovering state from
'/var/lib/mesos/meta'
I0919 15:41:16.382519 13005 group.cpp:349] Group process
(group(1)@127.0.1.1:5051) connected to ZooKeeper
I0919 15:41:16.382593 13005 group.cpp:837] Syncing group operations: queue
size (joins, cancels, datas) = (0, 0, 0)
I0919 15:41:16.382663 13005 group.cpp:427] Trying to create path '/mesos'
in ZooKeeper
I0919 15:41:16.382910 13009 status_update_manager.cpp:200] Recovering
status update manager
I0919 15:41:16.383419 13009 containerizer.cpp:522] Recovering containerizer
I0919 15:41:16.392206 13004 provisioner.cpp:253] Provisioner recovery
complete
I0919 15:41:16.392354 13004 slave.cpp:4782] Finished recovery
I0919 15:41:16.405709 13004 detector.cpp:152] Detected a new leader:
(id='678')
I0919 15:41:16.406067 13005 group.cpp:706] Trying to get
'/mesos/json.info_0000000678' in ZooKeeper
I0919 15:41:16.407572 13002 zookeeper.cpp:259] A new leading master
(UPID=master@10.142.55.190:5050) is detected
I0919 15:41:16.407977 13002 slave.cpp:895] New master detected at
master@10.142.55.190:5050
I0919 15:41:16.408043 13002 slave.cpp:916] No credentials provided.
Attempting to register without authentication
I0919 15:41:16.408140 13002 slave.cpp:927] Detecting new master
I0919 15:41:16.408223 13005 status_update_manager.cpp:174] Pausing sending
status updates
I0919 15:42:08.418956 13006 slave.cpp:3732] master@10.142.55.190:5050 exited
W0919 15:42:08.419035 13006 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
I0919 15:42:16.374977 13007 slave.cpp:4591] Current disk usage 72.41%. Max
allowed age: 1.231186482451933days
I0919 15:42:20.007169 13007 detector.cpp:152] Detected a new leader:
(id='679')
I0919 15:42:20.007297 13007 group.cpp:706] Trying to get
'/mesos/json.info_0000000679' in ZooKeeper
I0919 15:42:20.008503 13007 zookeeper.cpp:259] A new leading master
(UPID=master@10.142.55.196:5050) is detected
I0919 15:42:20.008587 13007 slave.cpp:895] New master detected at
master@10.142.55.196:5050
I0919 15:42:20.008610 13007 slave.cpp:916] No credentials provided.
Attempting to register without authentication
I0919 15:42:20.008703 13007 slave.cpp:927] Detecting new master
I0919 15:42:20.008750 13007 status_update_manager.cpp:174] Pausing sending
status updates
I0919 15:43:16.387984 13005 slave.cpp:4591] Current disk usage 72.41%. Max
allowed age: 1.231162010606794days
I0919 15:43:20.081272 13005 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:43:20.081374 13005 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
I0919 15:43:26.855154 13005 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:43:26.855315 13005 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:43:26.855159 13010 process.cpp:2105] Failed to shutdown socket
with fd 12: Transport endpoint is not connected
I0919 15:43:32.020196 13002 detector.cpp:152] Detected a new leader:
(id='682')
I0919 15:43:32.020300 13002 group.cpp:706] Trying to get
'/mesos/json.info_0000000682' in ZooKeeper
I0919 15:43:32.022203 13002 zookeeper.cpp:259] A new leading master
(UPID=master@10.142.55.202:5050) is detected
I0919 15:43:32.022302 13002 slave.cpp:895] New master detected at
master@10.142.55.202:5050
I0919 15:43:32.022328 13002 slave.cpp:916] No credentials provided.
Attempting to register without authentication
I0919 15:43:32.022382 13002 slave.cpp:927] Detecting new master
I0919 15:43:32.022423 13002 status_update_manager.cpp:174] Pausing sending
status updates
I0919 15:44:16.389369 13003 slave.cpp:4591] Current disk usage 72.41%. Max
allowed age: 1.231119184877789days
I0919 15:44:32.535347 13003 slave.cpp:3732] master@10.142.55.202:5050 exited
W0919 15:44:32.535522 13003 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
I0919 15:44:42.005375 13002 detector.cpp:152] Detected a new leader:
(id='684')
I0919 15:44:42.005496 13002 group.cpp:706] Trying to get
'/mesos/json.info_0000000684' in ZooKeeper
I0919 15:44:42.006367 13002 zookeeper.cpp:259] A new leading master
(UPID=master@10.142.55.190:5050) is detected
I0919 15:44:42.006492 13002 slave.cpp:895] New master detected at
master@10.142.55.190:5050
I0919 15:44:42.006597 13002 slave.cpp:916] No credentials provided.
Attempting to register without authentication
I0919 15:44:42.006675 13002 slave.cpp:927] Detecting new master
I0919 15:44:42.006577 13008 status_update_manager.cpp:174] Pausing sending
status updates
I0919 15:45:16.400794 13006 slave.cpp:4591] Current disk usage 72.48%. Max
allowed age: 1.226390000804074days
I0919 15:45:42.354790 13005 slave.cpp:3732] master@10.142.55.190:5050 exited
W0919 15:45:42.354857 13005 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
I0919 15:45:54.020563 13002 detector.cpp:152] Detected a new leader:
(id='687')
I0919 15:45:54.020756 13002 group.cpp:706] Trying to get
'/mesos/json.info_0000000687' in ZooKeeper
I0919 15:45:54.023296 13002 zookeeper.cpp:259] A new leading master
(UPID=master@10.142.55.196:5050) is detected
I0919 15:45:54.023455 13002 slave.cpp:895] New master detected at
master@10.142.55.196:5050
I0919 15:45:54.023558 13002 slave.cpp:916] No credentials provided.
Attempting to register without authentication
I0919 15:45:54.023526 13008 status_update_manager.cpp:174] Pausing sending
status updates
I0919 15:45:54.023669 13002 slave.cpp:927] Detecting new master
I0919 15:46:16.402402 13003 slave.cpp:4591] Current disk usage 72.53%. Max
allowed age: 1.223205601954942days
I0919 15:46:54.075505 13007 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:46:54.075592 13007 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:46:56.098012 13010 process.cpp:2105] Failed to shutdown socket
with fd 14: Transport endpoint is not connected
I0919 15:46:56.098016 13007 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:46:56.098253 13007 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:46:56.462254 13010 process.cpp:2105] Failed to shutdown socket
with fd 14: Transport endpoint is not connected
I0919 15:46:56.462260 13005 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:46:56.462540 13005 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
I0919 15:47:02.005637 13009 detector.cpp:152] Detected a new leader:
(id='688')
I0919 15:47:02.005765 13009 group.cpp:706] Trying to get
'/mesos/json.info_0000000688' in ZooKeeper
I0919 15:47:02.006853 13009 zookeeper.cpp:259] A new leading master
(UPID=master@10.142.55.202:5050) is detected
I0919 15:47:02.006959 13009 slave.cpp:895] New master detected at
master@10.142.55.202:5050
I0919 15:47:02.006986 13009 slave.cpp:916] No credentials provided.
Attempting to register without authentication
I0919 15:47:02.007025 13009 slave.cpp:927] Detecting new master
I0919 15:47:02.007061 13009 status_update_manager.cpp:174] Pausing sending
status updates
I0919 15:47:16.406669 13008 slave.cpp:4591] Current disk usage 72.53%. Max
allowed age: 1.223184189090440days
I0919 15:48:02.950891 13005 slave.cpp:3732] master@10.142.55.202:5050 exited
W0919 15:48:02.950994 13005 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
I0919 15:48:12.006634 13005 detector.cpp:152] Detected a new leader:
(id='690')
I0919 15:48:12.006817 13003 group.cpp:706] Trying to get
'/mesos/json.info_0000000690' in ZooKeeper
I0919 15:48:12.007987 13003 zookeeper.cpp:259] A new leading master
(UPID=master@10.142.55.190:5050) is detected
I0919 15:48:12.008126 13003 slave.cpp:895] New master detected at
master@10.142.55.190:5050
I0919 15:48:12.008210 13003 slave.cpp:916] No credentials provided.
Attempting to register without authentication
I0919 15:48:12.008280 13003 slave.cpp:927] Detecting new master
I0919 15:48:12.008191 13008 status_update_manager.cpp:174] Pausing sending
status updates
I0919 15:48:16.409266 13003 slave.cpp:4591] Current disk usage 72.54%. Max
allowed age: 1.222480623542604days
I0919 15:49:12.379010 13009 slave.cpp:3732] master@10.142.55.190:5050 exited
W0919 15:49:12.379149 13009 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:49:12.379233 13010 process.cpp:2105] Failed to shutdown socket
with fd 12: Transport endpoint is not connected
I0919 15:49:16.413767 13007 slave.cpp:4591] Current disk usage 72.64%. Max
allowed age: 1.215032005677465days
I0919 15:49:24.016290 13007 detector.cpp:152] Detected a new leader:
(id='693')
I0919 15:49:24.016417 13007 group.cpp:706] Trying to get
'/mesos/json.info_0000000693' in ZooKeeper
I0919 15:49:24.018273 13007 zookeeper.cpp:259] A new leading master
(UPID=master@10.142.55.196:5050) is detected
I0919 15:49:24.018437 13007 slave.cpp:895] New master detected at
master@10.142.55.196:5050
I0919 15:49:24.018523 13007 slave.cpp:916] No credentials provided.
Attempting to register without authentication
I0919 15:49:24.018604 13007 slave.cpp:927] Detecting new master
I0919 15:49:24.018496 13008 status_update_manager.cpp:174] Pausing sending
status updates
I0919 15:50:16.416391 13008 slave.cpp:4591] Current disk usage 72.64%. Max
allowed age: 1.215016710774248days
I0919 15:50:24.065268 13003 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:50:24.065342 13003 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
I0919 15:50:24.485752 13004 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:50:24.485839 13004 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:50:24.485977 13010 process.cpp:2105] Failed to shutdown socket
with fd 14: Transport endpoint is not connected
I0919 15:50:28.343647 13003 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:50:28.343719 13003 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:50:28.343819 13010 process.cpp:2105] Failed to shutdown socket
with fd 14: Transport endpoint is not connected
I0919 15:50:31.545099 13005 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:50:31.545171 13005 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:50:31.545284 13010 process.cpp:2105] Failed to shutdown socket
with fd 14: Transport endpoint is not connected
I0919 15:50:32.007096 13008 detector.cpp:152] Detected a new leader:
(id='694')
I0919 15:50:32.007195 13008 group.cpp:706] Trying to get
'/mesos/json.info_0000000694' in ZooKeeper
I0919 15:50:32.009881 13008 zookeeper.cpp:259] A new leading master
(UPID=master@10.142.55.202:5050) is detected
I0919 15:50:32.009970 13008 slave.cpp:895] New master detected at
master@10.142.55.202:5050
I0919 15:50:32.009994 13008 slave.cpp:916] No credentials provided.
Attempting to register without authentication
I0919 15:50:32.010030 13008 slave.cpp:927] Detecting new master
I0919 15:50:32.010079 13008 status_update_manager.cpp:174] Pausing sending
status updates
I0919 15:51:16.417846 13006 slave.cpp:4591] Current disk usage 72.64%. Max
allowed age: 1.214964708103322days
I0919 15:51:32.560317 13003 slave.cpp:3732] master@10.142.55.202:5050 exited
W0919 15:51:32.560410 13003 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
I0919 15:51:42.005147 13009 detector.cpp:152] Detected a new leader:
(id='696')
I0919 15:51:42.005265 13009 group.cpp:706] Trying to get
'/mesos/json.info_0000000696' in ZooKeeper
I0919 15:51:42.006824 13009 zookeeper.cpp:259] A new leading master
(UPID=master@10.142.55.190:5050) is detected
I0919 15:51:42.006904 13009 slave.cpp:895] New master detected at
master@10.142.55.190:5050
I0919 15:51:42.006928 13009 slave.cpp:916] No credentials provided.
Attempting to register without authentication
I0919 15:51:42.006963 13009 slave.cpp:927] Detecting new master
I0919 15:51:42.006999 13009 status_update_manager.cpp:174] Pausing sending
status updates
I0919 15:52:16.419373 13003 slave.cpp:4591] Current disk usage 72.71%. Max
allowed age: 1.209981628636250days
I0919 15:52:42.336305 13002 slave.cpp:3732] master@10.142.55.190:5050 exited
W0919 15:52:42.336426 13002 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
I0919 15:52:54.005267 13005 detector.cpp:152] Detected a new leader:
(id='699')
I0919 15:52:54.005408 13005 group.cpp:706] Trying to get
'/mesos/json.info_0000000699' in ZooKeeper
I0919 15:52:54.006206 13005 zookeeper.cpp:259] A new leading master
(UPID=master@10.142.55.196:5050) is detected
I0919 15:52:54.006285 13005 slave.cpp:895] New master detected at
master@10.142.55.196:5050
I0919 15:52:54.006309 13005 slave.cpp:916] No credentials provided.
Attempting to register without authentication
I0919 15:52:54.006398 13005 slave.cpp:927] Detecting new master
I0919 15:52:54.006451 13005 status_update_manager.cpp:174] Pausing sending
status updates
I0919 15:53:16.420258 13005 slave.cpp:4591] Current disk usage 72.76%. Max
allowed age: 1.206748286096840days
I0919 15:53:54.071012 13005 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:53:54.071143 13005 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
I0919 15:54:01.105780 13002 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:54:01.105854 13002 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:54:01.105970 13010 process.cpp:2105] Failed to shutdown socket
with fd 15: Transport endpoint is not connected
I0919 15:54:05.733837 13007 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:54:05.733932 13007 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:54:05.734071 13010 process.cpp:2105] Failed to shutdown socket
with fd 15: Transport endpoint is not connected
E0919 15:54:05.818560 13010 process.cpp:2105] Failed to shutdown socket
with fd 15: Transport endpoint is not connected
I0919 15:54:05.818583 13003 slave.cpp:3732] master@10.142.55.196:5050 exited
W0919 15:54:05.818758 13003 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
I0919 15:54:06.004385 13009 detector.cpp:152] Detected a new leader:
(id='700')
I0919 15:54:06.004494 13009 group.cpp:706] Trying to get
'/mesos/json.info_0000000700' in ZooKeeper
I0919 15:54:06.005511 13009 zookeeper.cpp:259] A new leading master
(UPID=master@10.142.55.202:5050) is detected
I0919 15:54:06.005586 13009 slave.cpp:895] New master detected at
master@10.142.55.202:5050
I0919 15:54:06.005609 13009 slave.cpp:916] No credentials provided.
Attempting to register without authentication
I0919 15:54:06.005676 13009 slave.cpp:927] Detecting new master
I0919 15:54:06.005720 13009 status_update_manager.cpp:174] Pausing sending
status updates
I0919 15:54:16.423193 13002 slave.cpp:4591] Current disk usage 72.76%. Max
allowed age: 1.206699342406551days
slave warn log
Log file created at: 2016/09/19 15:42:08
Running on machine: ubuntu12
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
W0919 15:42:08.419035 13006 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
W0919 15:43:20.081374 13005 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
W0919 15:43:26.855315 13005 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:43:26.855159 13010 process.cpp:2105] Failed to shutdown socket
with fd 12: Transport endpoint is not connected
W0919 15:44:32.535522 13003 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
W0919 15:45:42.354857 13005 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
W0919 15:46:54.075592 13007 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:46:56.098012 13010 process.cpp:2105] Failed to shutdown socket
with fd 14: Transport endpoint is not connected
W0919 15:46:56.098253 13007 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:46:56.462254 13010 process.cpp:2105] Failed to shutdown socket
with fd 14: Transport endpoint is not connected
W0919 15:46:56.462540 13005 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
W0919 15:48:02.950994 13005 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
W0919 15:49:12.379149 13009 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:49:12.379233 13010 process.cpp:2105] Failed to shutdown socket
with fd 12: Transport endpoint is not connected
W0919 15:50:24.065342 13003 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
W0919 15:50:24.485839 13004 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:50:24.485977 13010 process.cpp:2105] Failed to shutdown socket
with fd 14: Transport endpoint is not connected
W0919 15:50:28.343719 13003 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:50:28.343819 13010 process.cpp:2105] Failed to shutdown socket
with fd 14: Transport endpoint is not connected
W0919 15:50:31.545171 13005 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:50:31.545284 13010 process.cpp:2105] Failed to shutdown socket
with fd 14: Transport endpoint is not connected
W0919 15:51:32.560410 13003 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
W0919 15:52:42.336426 13002 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
W0919 15:53:54.071143 13005 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
W0919 15:54:01.105854 13002 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:54:01.105970 13010 process.cpp:2105] Failed to shutdown socket
with fd 15: Transport endpoint is not connected
W0919 15:54:05.733932 13007 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
E0919 15:54:05.734071 13010 process.cpp:2105] Failed to shutdown socket
with fd 15: Transport endpoint is not connected
E0919 15:54:05.818560 13010 process.cpp:2105] Failed to shutdown socket
with fd 15: Transport endpoint is not connected
W0919 15:54:05.818758 13003 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
W0919 15:55:06.821486 13009 slave.cpp:3737] Master disconnected! Waiting
for a new master to be elected
> mesos-master can not found mesos-slave, and elect a new leader in a short
> interval
> ----------------------------------------------------------------------------------
>
> Key: MESOS-6205
> URL: https://issues.apache.org/jira/browse/MESOS-6205
> Project: Mesos
> Issue Type: Bug
> Components: master
> Environment: ubuntu 12 x64, centos 6.5 x64, centos 7.2 x64
> Reporter: kasim
>
> I follow this
> [doc][https://open.mesosphere.com/getting-started/install/#verifying-installation]
> to setup mesos cluster.
> There are three vm(ubuntu 12, centos 6.5, centos 7.2).
> $ cat /etc/hosts
> 10.142.55.190 zk1
> 10.142.55.196 zk2
> 10.142.55.202 zk3
> config in each mathine:
> $ cat /etc/mesos/zk
> zk://10.142.55.190:2181,10.142.55.196:2181,10.142.55.202:2181/mesos
> After start zookeeper, mesos-master and mesos-slave in three vm, I can view
> the mesos webui(10.142.55.190:5050), but agents count is 0.
> After a little time, mesos page get error:
> Failed to connect to 10.142.55.190:5050!
> Retrying in 16 seconds...
> (I found that zookeeper would elect a new leader in a short interval)
> mesos-master cmd:
> ```
> mesos-master --agent_ping_timeout="15secs"
> --agent_reregister_timeout="10mins" --allocation_interval="1secs"
> --allocator="HierarchicalDRF" --authenticate_agents="false"
> --authenticate_frameworks="false" --authenticate_http_frameworks="false"
> --authenticate_http_readonly="false" --authenticate_http_readwrite="false"
> --authenticators="crammd5" --authorizers="local" --framework_sorter="drf"
> --help="false" --hostname_lookup="true" --http_authenticators="basic"
> --initialize_driver_logging="true" --ip="10.142.55.190"
> --log_auto_initialize="true" --log_dir="/var/log/mesos" --logbufsecs="0"
> --logging_level="INFO" --max_agent_ping_timeouts="5"
> --max_completed_frameworks="50" --max_completed_tasks_per_framework="1000"
> --port="5050" --quiet="false" --quorum="2"
> --recovery_agent_removal_limit="100%" --registry="replicated_log"
> --registry_fetch_timeout="1mins" --registry_store_timeout="20secs"
> --registry_strict="false" --root_submissions="true" --user_sorter="drf"
> --version="false" --webui_dir="/usr/share/mesos/webui"
> --work_dir="/var/lib/mesos"
> --zk="zk://10.142.55.190:2181,10.142.55.196:2181,10.142.55.202:2181/mesos"
> ```
> mesos-slave cmd:
> ```
> mesos-slave --appc_simple_discovery_uri_prefix="http://";
> --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="false"
> --authenticate_http_readwrite="false" --authenticatee="crammd5"
> --authentication_backoff_factor="1secs" --authorizer="local"
> --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false"
> --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false"
> --cgroups_root="mesos" --container_disk_watch_interval="15secs"
> --containerizers="mesos" --default_role="*" --disk_watch_interval="1mins"
> --docker="docker" --docker_kill_orphans="true"
> --docker_registry="https://registry-1.docker.io"; --docker_remove_delay="6hrs"
> --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns"
> --docker_store_dir="/tmp/mesos/store/docker"
> --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume"
> --enforce_container_disk_quota="false"
> --executor_registration_timeout="1mins"
> --executor_shutdown_grace_period="5secs"
> --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB"
> --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1"
> --hadoop_home="" --help="false" --hostname="10.142.55.190"
> --hostname_lookup="true" --http_authenticators="basic"
> --http_command_executor="false" --image_provisioner_backend="copy"
> --initialize_driver_logging="true" --ip="10.142.55.190"
> --isolation="posix/cpu,posix/mem" --launcher="posix"
> --launcher_dir="/usr/libexec/mesos" --log_dir="/var/log/mesos"
> --logbufsecs="0" --logging_level="INFO"
> --master="zk://10.142.55.190:2181,10.142.55.196:2181,10.142.55.202:2181/mesos"
> --oversubscribed_resources_interval="15secs" --perf_duration="10secs"
> --perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns"
> --quiet="false" --recover="reconnect" --recovery_timeout="15mins"
> --registration_backoff_factor="1secs" --revocable_cpu_low_priority="true"
> --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true"
> --systemd_enable_support="true"
> --systemd_runtime_directory="/run/systemd/system" --version="false"
> --work_dir="/var/lib/mesos"
> ```
> When I run mesos-master from command-line, I got
> ```
> I0919 17:20:19.286264 17550 replica.cpp:673] Replica in VOTING status
> received a broadcasted recover request from (583)@10.142.55.202:5050
> F0919 17:20:20.009371 17556 master.cpp:1536] Recovery failed: Failed to
> recover registrar: Failed to perform fetch within 1mins
> *** Check failure stack trace: ***
> @ 0x7f9db78458dd google::LogMessage::Fail()
> @ 0x7f9db784771d google::LogMessage::SendToLog()
> @ 0x7f9db78454cc google::LogMessage::Flush()
> @ 0x7f9db7848019 google::LogMessageFatal::~LogMessageFatal()
> @ 0x7f9db6e2dbbc mesos::internal::master::fail()
> @ 0x7f9db6e75b20
> _ZNSt17_Function_handlerIFvRKSsEZNK7process6FutureI7NothingE8onFailedISt5_BindIFPFvS1_S1_EPKcSt12_PlaceholderILi1EE
> EEvEERKS6_OT_NS6_6PreferEEUlS1_E_E9_M_invokeERKSt9_Any_dataS1_
> @ 0x42a116 process::Future<>::fail()
> @ 0x7f9db6e9f705 process::internal::thenf<>()
> @ 0x7f9db6efd016
> _ZN7process8internal3runISt8functionIFvRKNS_6FutureIN5mesos8internal8RegistryEEEEEJRS7_EEEvRKSt6vectorIT_SaISE_EEDp
> OT0_
> I0919 17:20:20.025172 17553 replica.cpp:673] Replica in VOTING status
> received a broadcasted recover request from (212)@10.142.55.196:5050
> @ 0x7f9db6f100de process::Future<>::fail()
> @ 0x7f9db6c57e86 process::internal::run<>()
> @ 0x7f9db6f100cb process::Future<>::fail()
> @ 0x7f9db6ef2d34
> mesos::internal::master::RegistrarProcess::_recover()
> @ 0x7f9db77d5171 process::ProcessManager::resume()
> @ 0x7f9db77d5477
> _ZNSt6thread5_ImplISt12_Bind_simpleIFZN7process14ProcessManager12init_threadsEvEUt_vEEE6_M_runEv
> @ 0x7f9db5e439c0 (unknown)
> @ 0x7f9db568ae9a start_thread
> @ 0x7f9db53b836d (unknown)
> [1] 17548 abort (core dumped) mesos-master --agent_ping_timeout="15secs"
> --agent_reregister_timeout="10mins
> ```
> master info log:
> I0919 15:54:59.677438 13281 http.cpp:2022] Redirecting request for
> /master/state?jsonp=angular.callbacks._1x to the leading master zk3
> I0919 15:55:00.098667 13281 replica.cpp:673] Replica in VOTING status
> received a broadcasted recover request from (768)@10.142.55.202:5050
> I0919 15:55:00.385279 13281 replica.cpp:673] Replica in VOTING status
> received a broadcasted recover request from (185)@10.142.55.196:5050
> I0919 15:55:00.711119 13281 replica.cpp:673] Replica in VOTING status
> received a broadcasted recover request from (771)@10.142.55.202:5050
> I0919 15:55:01.347291 13284 replica.cpp:673] Replica in VOTING status
> received a broadcasted recover request from (188)@10.142.55.196:5050
> I0919 15:55:01.597682 13284 replica.cpp:673] Replica in VOTING status
> received a broadcasted recover request from (774)@10.142.55.202:5050
> I0919 15:55:02.257159 13282 replica.cpp:673] Replica in VOTING status
> received a broadcasted recover request from (191)@10.142.55.196:5050
> I0919 15:55:02.370692 13287 replica.cpp:673] Replica in VOTING status
> received a broadcasted recover request from (777)@10.142.55.202:5050
> I0919 15:55:03.205920 13285 replica.cpp:673] Replica in VOTING status
> received a broadcasted recover request from (780)@10.142.55.202:5050
> I0919 15:55:03.260007 13281 replica.cpp:673] Replica in VOTING status
> received a broadcasted recover request from (194)@10.142.55.196:5050
> I0919 15:55:03.929611 13283 replica.cpp:673] Replica in VOTING status
> received a broadcasted recover request from (783)@10.142.55.202:5050
> I0919 15:55:04.033308 13287 replica.cpp:673] Replica in VOTING status
> received a broadcasted recover request from (197)@10.142.55.196:5050
> I0919 15:55:04.591275 13284 replica.cpp:673] Replica in VOTING status
> received a broadcasted recover request from (200)@10.142.55.196:5050
> I0919 15:55:04.608211 13283 replica.cpp:673] Replica in VOTING status
> received a broadcasted recover request from (786)@10.142.55.202:5050
> I0919 15:55:05.184682 13280 replica.cpp:673] Replica in VOTING status
> received a broadcasted recover request from (789)@10.142.55.202:5050
> I0919 15:55:05.268277 13280 replica.cpp:673] Replica in VOTING status
> received a broadcasted recover request from (203)@10.142.55.196:5050
> I0919 15:55:05.775377 13281 replica.cpp:673] Replica in VOTING status
> received a broadcasted recover request from (206)@10.142.55.196:5050
> I0919 15:55:05.916445 13285 replica.cpp:673] Replica in VOTING status
> received a broadcasted recover request from (792)@10.142.55.202:5050
> I0919 15:55:06.744927 13280 replica.cpp:673] Replica in VOTING status
> received a broadcasted recover request from (209)@10.142.55.196:5050
> I0919 15:55:07.378521 13283 replica.cpp:673] Replica in VOTING status
> received a broadcasted recover request from (5)@10.142.55.202:5050
> I0919 15:55:07.393311 13285 network.hpp:430] ZooKeeper group memberships
> changed
> I0919 15:55:07.393427 13285 group.cpp:706] Trying to get
> '/mesos/log_replicas/0000000709' in ZooKeeper
> I0919 15:55:07.393985 13285 group.cpp:706] Trying to get
> '/mesos/log_replicas/0000000711' in ZooKeeper
> I0919 15:55:07.394394 13285 group.cpp:706] Trying to get
> '/mesos/log_replicas/0000000714' in ZooKeeper
> I0919 15:55:07.394843 13285 group.cpp:706] Trying to get
> '/mesos/log_replicas/0000000715' in ZooKeeper
> I0919 15:55:07.395418 13285 network.hpp:478] ZooKeeper group PIDs: {
> log-replica(1)@10.142.55.190:5050, log-replica(1)@10.142.55.196:5050,
> log-replica(1)@10.142.55.202:5050 }
> I0919 15:55:08.178272 13280 replica.cpp:673] Replica in VOTING status
> received a broadcasted recover request from (14)@10.142.55.202:5050
> I0919 15:55:09.059562 13282 replica.cpp:673] Replica in VOTING status
> received a broadcasted recover request from (21)@10.142.55.202:5050
> I0919 15:55:09.700711 13286 replica.cpp:673] Replica in VOTING status
> received a broadcasted recover request from (24)@10.142.55.202:5050
> I0919 15:55:09.742185 13287 http.cpp:381] HTTP GET for /master/state from
> 10.142.50.94:59987 with User-Agent='Mozilla/5.0 (Windows NT 6.2; WOW64;
> rv:47.0) Gecko/20100101 Firefox/47.0'
> I0919 15:55:09.742359 13287 http.cpp:2022] Redirecting request for
> /master/state?jsonp=angular.callbacks._1y to the leading master zk3
> I0919 15:55:10.660789 13280 replica.cpp:673] Replica in VOTING status
> received a broadcasted recover request from (30)@10.142.55.202:5050
> I0919 15:55:11.480326 13281 replica.cpp:673] Replica in VOTING status
> received a broadcasted recover request from (34)@10.142.55.202:5050
> I0919 15:55:12.386256 13286 replica.cpp:673] Replica in VOTING status
> received a broadcasted recover request from (37)@10.142.55.202:5050
> I0919 15:55:12.975137 13287 replica.cpp:673] Replica in VOTING status
> received a broadcasted recover request from (42)@10.142.55.202:5050
> I0919 15:55:13.843091 13285 replica.cpp:673] Replica in VOTING status
> received a broadcasted recover request from (47)@10.142.55.202:5050
> I0919 15:55:14.373478 13281 replica.cpp:673] Replica in VOTING status
> received a broadcasted recover request from (51)@10.142.55.202:5050
> I0919 15:55:14.937181 13280 replica.cpp:673] Replica in VOTING status
> received a broadcasted recover request from (54)@10.142.55.202:5050
> I0919 15:55:15.658219 13283 replica.cpp:673] Replica in VOTING status
> received a broadcasted recover request from (58)@10.142.55.202:5050
> I0919 15:55:16.007822 13286 network.hpp:430] ZooKeeper group memberships
> changed
> I0919 15:55:16.007972 13286 group.cpp:706] Trying to get
> '/mesos/log_replicas/0000000711' in ZooKeeper
> I0919 15:55:16.010170 13286 group.cpp:706] Trying to get
> '/mesos/log_replicas/0000000714' in ZooKeeper
> I0919 15:55:16.011462 13284 detector.cpp:152] Detected a new leader:
> (id='702')
> I0919 15:55:16.011556 13284 group.cpp:706] Trying to get
> '/mesos/json.info_0000000702' in ZooKeeper
> I0919 15:55:16.011968 13286 group.cpp:706] Trying to get
> '/mesos/log_replicas/0000000715' in ZooKeeper
> I0919 15:55:16.012526 13286 network.hpp:478] ZooKeeper group PIDs: {
> log-replica(1)@10.142.55.190:5050, log-replica(1)@10.142.55.196:5050,
> log-replica(1)@10.142.55.202:5050 }
> I0919 15:55:16.013156 13284 zookeeper.cpp:259] A new leading master
> (UPID=master@10.142.55.190:5050) is detected
> I0919 15:55:16.013222 13284 master.cpp:1847] The newly elected leader is
> master@10.142.55.190:5050 with id 677967bc-f6f0-46b3-a44e-72eed1befd60
> I0919 15:55:16.013244 13284 master.cpp:1860] Elected as the leading
> master!
> I0919 15:55:16.013273 13284 master.cpp:1547] Recovering from registrar
> I0919 15:55:16.013352 13284 registrar.cpp:332] Recovering registrar
> I0919 15:55:16.014081 13280 log.cpp:553] Attempting to start the writer
> I0919 15:55:16.014515 13280 replica.cpp:493] Replica received implicit
> promise request from (211)@10.142.55.190:5050 with proposal 1204590
> I0919 15:55:16.018023 13282 consensus.cpp:360] Aborting implicit promise
> request because 2 ignores received
> I0919 15:55:16.018028 13280 leveldb.cpp:304] Persisting metadata (10
> bytes) to leveldb took 3.469479ms
> I0919 15:55:16.018338 13280 replica.cpp:342] Persisted promised to 1204590
> I0919 15:55:16.018508 13282 log.cpp:565] Could not start the writer, but
> can be retried
> I0919 15:55:16.018645 13282 log.cpp:553] Attempting to start the writer
> I0919 15:55:16.018899 13282 replica.cpp:493] Replica received implicit
> promise request from (215)@10.142.55.190:5050 with proposal 1204591
> I0919 15:55:16.022183 13287 consensus.cpp:360] Aborting implicit promise
> request because 2 ignores received
> I0919 15:55:16.022367 13280 log.cpp:565] Could not start the writer, but
> can be retried
> I0919 15:55:16.022510 13280 log.cpp:553] Attempting to start the writer
> I0919 15:55:16.028880 13282 leveldb.cpp:304] Persisting metadata (10
> bytes) to leveldb took 9.870818ms
> I0919 15:55:16.029024 13282 replica.cpp:342] Persisted promised to 1204591
> I0919 15:55:16.029428 13286 replica.cpp:493] Replica received implicit
> promise request from (219)@10.142.55.190:5050 with proposal 1204592
> I0919 15:55:16.031600 13280 consensus.cpp:360] Aborting implicit promise
> request because 2 ignores received
> I0919 15:55:16.036208 13283 log.cpp:565] Could not start the writer, but
> can be retried
> I0919 15:55:16.036454 13283 log.cpp:553] Attempting to start the writer
> I0919 15:55:16.040256 13286 leveldb.cpp:304] Persisting metadata (10
> bytes) to leveldb took 10.783237ms
> I0919 15:55:16.040339 13286 replica.cpp:342] Persisted promised to 1204592
> I0919 15:55:16.040712 13286 replica.cpp:493] Replica received implicit
> promise request from (222)@10.142.55.190:5050 with proposal 1204593
> I0919 15:55:16.042196 13286 leveldb.cpp:304] Persisting metadata (10
> bytes) to leveldb took 1.435071ms
> I0919 15:55:16.042250 13286 replica.cpp:342] Persisted promised to 1204593
> I0919 15:55:16.042981 13286 consensus.cpp:360] Aborting implicit promise
> request because 2 ignores received
> I0919 15:55:16.043099 13286 log.cpp:565] Could not start the writer, but
> can be retried
> I0919 15:55:16.043303 13283 log.cpp:553] Attempting to start the writer
> All later logs are looping :
> I0919 15:55:16.043676 13286 replica.cpp:493] Replica received implicit
> promise request from (225)@10.142.55.190:5050 with proposal 1204594
> I0919 15:55:16.044122 13286 leveldb.cpp:304] Persisting metadata (10
> bytes) to leveldb took 404769ns
> I0919 15:55:16.044209 13286 replica.cpp:342] Persisted promised to 1204594
> I0919 15:55:16.044837 13281 consensus.cpp:360] Aborting implicit promise
> request because 2 ignores received
> I0919 15:55:16.044926 13281 log.cpp:565] Could not start the writer, but
> can be retried
> I0919 15:55:16.045038 13281 log.cpp:553] Attempting to start the writer
> slave info log:
> Log file created at: 2016/09/19 15:41:16
> Running on machine: ubuntu12
> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
> I0919 15:41:16.346844 12986 logging.cpp:194] INFO level logging started!
> I0919 15:41:16.363313 12986 containerizer.cpp:196] Using isolation:
> posix/cpu,posix/mem,filesystem/posix,network/cni
> I0919 15:41:16.370334 12986 main.cpp:434] Starting Mesos agent
> I0919 15:41:16.371184 12986 slave.cpp:198] Agent started on
> 1)@127.0.1.1:5051
> I0919 15:41:16.371636 12986 slave.cpp:199] Flags at startup:
> --appc_simple_discovery_uri_prefix="http://";
> --appc_store_dir="/tmp/mesos/store/appc" --authenticate_http_readonly="false"
> --authenticate_http_readwrite="false" --authenticatee="crammd5"
> --authentication_backoff_factor="1secs" --authorizer="local"
> --cgroups_cpu_enable_pids_and_tids_count="false" --cgroups_enable_cfs="false"
> --cgroups_hierarchy="/sys/fs/cgroup" --cgroups_limit_swap="false"
> --cgroups_root="mesos" --container_disk_watch_interval="15secs"
> --containerizers="mesos" --default_role="*" --disk_watch_interval="1mins"
> --docker="docker" --docker_kill_orphans="true"
> --docker_registry="https://registry-1.docker.io"; --docker_remove_delay="6hrs"
> --docker_socket="/var/run/docker.sock" --docker_stop_timeout="0ns"
> --docker_store_dir="/tmp/mesos/store/docker"
> --docker_volume_checkpoint_dir="/var/run/mesos/isolators/docker/volume"
> --enforce_container_disk_quota="false"
> --executor_registration_timeout="1mins"
> --executor_shutdown_grace_period="5secs"
> --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB"
> --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1"
> --hadoop_home="" --help="false" --hostname_lookup="true"
> --http_authenticators="basic" --http_command_executor="false"
> --image_provisioner_backend="copy" --initialize_driver_logging="true"
> --isolation="posix/cpu,posix/mem" --launcher="posix"
> --launcher_dir="/usr/libexec/mesos" --log_dir="/var/log/mesos"
> --logbufsecs="0" --logging_level="INFO"
> --master="zk://10.142.55.190:2181,10.142.55.196:2181,10.142.55.202:2181/mesos"
> --oversubscribed_resources_interval="15secs" --perf_duration="10secs"
> --perf_interval="1mins" --port="5051" --qos_correction_interval_min="0ns"
> --quiet="false" --recover="reconnect" --recovery_timeout="15mins"
> --registration_backoff_factor="1secs" --revocable_cpu_low_priority="true"
> --sandbox_directory="/mnt/mesos/sandbox" --strict="true" --switch_user="true"
> --systemd_enable_support="true"
> --systemd_runtime_directory="/run/systemd/system" --version="false"
> --work_dir="/var/lib/mesos"
> I0919 15:41:16.373072 12986 slave.cpp:519] Agent resources: cpus(*):2;
> mem(*):2930; disk(*):4469; ports(*):[31000-32000]
> I0919 15:41:16.373291 12986 slave.cpp:527] Agent attributes: [ ]
> I0919 15:41:16.373347 12986 slave.cpp:532] Agent hostname: ubuntu12
> I0919 15:41:16.379895 13005 state.cpp:57] Recovering state from
> '/var/lib/mesos/meta'
> I0919 15:41:16.382519 13005 group.cpp:349] Group process
> (group(1)@127.0.1.1:5051) connected to ZooKeeper
> I0919 15:41:16.382593 13005 group.cpp:837] Syncing group operations:
> queue size (joins, cancels, datas) = (0, 0, 0)
> I0919 15:41:16.382663 13005 group.cpp:427] Trying to create path '/mesos'
> in ZooKeeper
> I0919 15:41:16.382910 13009 status_update_manager.cpp:200] Recovering
> status update manager
> I0919 15:41:16.383419 13009 containerizer.cpp:522] Recovering
> containerizer
> I0919 15:41:16.392206 13004 provisioner.cpp:253] Provisioner recovery
> complete
> I0919 15:41:16.392354 13004 slave.cpp:4782] Finished recovery
> I0919 15:41:16.405709 13004 detector.cpp:152] Detected a new leader:
> (id='678')
> I0919 15:41:16.406067 13005 group.cpp:706] Trying to get
> '/mesos/json.info_0000000678' in ZooKeeper
> I0919 15:41:16.407572 13002 zookeeper.cpp:259] A new leading master
> (UPID=master@10.142.55.190:5050) is detected
> I0919 15:41:16.407977 13002 slave.cpp:895] New master detected at
> master@10.142.55.190:5050
> I0919 15:41:16.408043 13002 slave.cpp:916] No credentials provided.
> Attempting to register without authentication
> I0919 15:41:16.408140 13002 slave.cpp:927] Detecting new master
> I0919 15:41:16.408223 13005 status_update_manager.cpp:174] Pausing
> sending status updates
> I0919 15:42:08.418956 13006 slave.cpp:3732] master@10.142.55.190:5050
> exited
> W0919 15:42:08.419035 13006 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> I0919 15:42:16.374977 13007 slave.cpp:4591] Current disk usage 72.41%.
> Max allowed age: 1.231186482451933days
> I0919 15:42:20.007169 13007 detector.cpp:152] Detected a new leader:
> (id='679')
> I0919 15:42:20.007297 13007 group.cpp:706] Trying to get
> '/mesos/json.info_0000000679' in ZooKeeper
> I0919 15:42:20.008503 13007 zookeeper.cpp:259] A new leading master
> (UPID=master@10.142.55.196:5050) is detected
> I0919 15:42:20.008587 13007 slave.cpp:895] New master detected at
> master@10.142.55.196:5050
> I0919 15:42:20.008610 13007 slave.cpp:916] No credentials provided.
> Attempting to register without authentication
> I0919 15:42:20.008703 13007 slave.cpp:927] Detecting new master
> I0919 15:42:20.008750 13007 status_update_manager.cpp:174] Pausing
> sending status updates
> I0919 15:43:16.387984 13005 slave.cpp:4591] Current disk usage 72.41%.
> Max allowed age: 1.231162010606794days
> I0919 15:43:20.081272 13005 slave.cpp:3732] master@10.142.55.196:5050
> exited
> W0919 15:43:20.081374 13005 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> I0919 15:43:26.855154 13005 slave.cpp:3732] master@10.142.55.196:5050
> exited
> W0919 15:43:26.855315 13005 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> E0919 15:43:26.855159 13010 process.cpp:2105] Failed to shutdown socket
> with fd 12: Transport endpoint is not connected
> I0919 15:43:32.020196 13002 detector.cpp:152] Detected a new leader:
> (id='682')
> I0919 15:43:32.020300 13002 group.cpp:706] Trying to get
> '/mesos/json.info_0000000682' in ZooKeeper
> I0919 15:43:32.022203 13002 zookeeper.cpp:259] A new leading master
> (UPID=master@10.142.55.202:5050) is detected
> I0919 15:43:32.022302 13002 slave.cpp:895] New master detected at
> master@10.142.55.202:5050
> I0919 15:43:32.022328 13002 slave.cpp:916] No credentials provided.
> Attempting to register without authentication
> I0919 15:43:32.022382 13002 slave.cpp:927] Detecting new master
> I0919 15:43:32.022423 13002 status_update_manager.cpp:174] Pausing
> sending status updates
> I0919 15:44:16.389369 13003 slave.cpp:4591] Current disk usage 72.41%.
> Max allowed age: 1.231119184877789days
> I0919 15:44:32.535347 13003 slave.cpp:3732] master@10.142.55.202:5050
> exited
> W0919 15:44:32.535522 13003 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> I0919 15:44:42.005375 13002 detector.cpp:152] Detected a new leader:
> (id='684')
> I0919 15:44:42.005496 13002 group.cpp:706] Trying to get
> '/mesos/json.info_0000000684' in ZooKeeper
> I0919 15:44:42.006367 13002 zookeeper.cpp:259] A new leading master
> (UPID=master@10.142.55.190:5050) is detected
> I0919 15:44:42.006492 13002 slave.cpp:895] New master detected at
> master@10.142.55.190:5050
> I0919 15:44:42.006597 13002 slave.cpp:916] No credentials provided.
> Attempting to register without authentication
> I0919 15:44:42.006675 13002 slave.cpp:927] Detecting new master
> I0919 15:44:42.006577 13008 status_update_manager.cpp:174] Pausing
> sending status updates
> I0919 15:45:16.400794 13006 slave.cpp:4591] Current disk usage 72.48%.
> Max allowed age: 1.226390000804074days
> I0919 15:45:42.354790 13005 slave.cpp:3732] master@10.142.55.190:5050
> exited
> W0919 15:45:42.354857 13005 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> I0919 15:45:54.020563 13002 detector.cpp:152] Detected a new leader:
> (id='687')
> I0919 15:45:54.020756 13002 group.cpp:706] Trying to get
> '/mesos/json.info_0000000687' in ZooKeeper
> I0919 15:45:54.023296 13002 zookeeper.cpp:259] A new leading master
> (UPID=master@10.142.55.196:5050) is detected
> I0919 15:45:54.023455 13002 slave.cpp:895] New master detected at
> master@10.142.55.196:5050
> I0919 15:45:54.023558 13002 slave.cpp:916] No credentials provided.
> Attempting to register without authentication
> I0919 15:45:54.023526 13008 status_update_manager.cpp:174] Pausing
> sending status updates
> I0919 15:45:54.023669 13002 slave.cpp:927] Detecting new master
> I0919 15:46:16.402402 13003 slave.cpp:4591] Current disk usage 72.53%.
> Max allowed age: 1.223205601954942days
> I0919 15:46:54.075505 13007 slave.cpp:3732] master@10.142.55.196:5050
> exited
> W0919 15:46:54.075592 13007 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> E0919 15:46:56.098012 13010 process.cpp:2105] Failed to shutdown socket
> with fd 14: Transport endpoint is not connected
> I0919 15:46:56.098016 13007 slave.cpp:3732] master@10.142.55.196:5050
> exited
> W0919 15:46:56.098253 13007 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> E0919 15:46:56.462254 13010 process.cpp:2105] Failed to shutdown socket
> with fd 14: Transport endpoint is not connected
> I0919 15:46:56.462260 13005 slave.cpp:3732] master@10.142.55.196:5050
> exited
> W0919 15:46:56.462540 13005 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> I0919 15:47:02.005637 13009 detector.cpp:152] Detected a new leader:
> (id='688')
> I0919 15:47:02.005765 13009 group.cpp:706] Trying to get
> '/mesos/json.info_0000000688' in ZooKeeper
> I0919 15:47:02.006853 13009 zookeeper.cpp:259] A new leading master
> (UPID=master@10.142.55.202:5050) is detected
> I0919 15:47:02.006959 13009 slave.cpp:895] New master detected at
> master@10.142.55.202:5050
> I0919 15:47:02.006986 13009 slave.cpp:916] No credentials provided.
> Attempting to register without authentication
> I0919 15:47:02.007025 13009 slave.cpp:927] Detecting new master
> I0919 15:47:02.007061 13009 status_update_manager.cpp:174] Pausing
> sending status updates
> I0919 15:47:16.406669 13008 slave.cpp:4591] Current disk usage 72.53%.
> Max allowed age: 1.223184189090440days
> I0919 15:48:02.950891 13005 slave.cpp:3732] master@10.142.55.202:5050
> exited
> W0919 15:48:02.950994 13005 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> I0919 15:48:12.006634 13005 detector.cpp:152] Detected a new leader:
> (id='690')
> I0919 15:48:12.006817 13003 group.cpp:706] Trying to get
> '/mesos/json.info_0000000690' in ZooKeeper
> I0919 15:48:12.007987 13003 zookeeper.cpp:259] A new leading master
> (UPID=master@10.142.55.190:5050) is detected
> I0919 15:48:12.008126 13003 slave.cpp:895] New master detected at
> master@10.142.55.190:5050
> I0919 15:48:12.008210 13003 slave.cpp:916] No credentials provided.
> Attempting to register without authentication
> I0919 15:48:12.008280 13003 slave.cpp:927] Detecting new master
> I0919 15:48:12.008191 13008 status_update_manager.cpp:174] Pausing
> sending status updates
> I0919 15:48:16.409266 13003 slave.cpp:4591] Current disk usage 72.54%.
> Max allowed age: 1.222480623542604days
> I0919 15:49:12.379010 13009 slave.cpp:3732] master@10.142.55.190:5050
> exited
> W0919 15:49:12.379149 13009 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> E0919 15:49:12.379233 13010 process.cpp:2105] Failed to shutdown socket
> with fd 12: Transport endpoint is not connected
> I0919 15:49:16.413767 13007 slave.cpp:4591] Current disk usage 72.64%.
> Max allowed age: 1.215032005677465days
> I0919 15:49:24.016290 13007 detector.cpp:152] Detected a new leader:
> (id='693')
> I0919 15:49:24.016417 13007 group.cpp:706] Trying to get
> '/mesos/json.info_0000000693' in ZooKeeper
> I0919 15:49:24.018273 13007 zookeeper.cpp:259] A new leading master
> (UPID=master@10.142.55.196:5050) is detected
> I0919 15:49:24.018437 13007 slave.cpp:895] New master detected at
> master@10.142.55.196:5050
> I0919 15:49:24.018523 13007 slave.cpp:916] No credentials provided.
> Attempting to register without authentication
> I0919 15:49:24.018604 13007 slave.cpp:927] Detecting new master
> I0919 15:49:24.018496 13008 status_update_manager.cpp:174] Pausing
> sending status updates
> I0919 15:50:16.416391 13008 slave.cpp:4591] Current disk usage 72.64%.
> Max allowed age: 1.215016710774248days
> I0919 15:50:24.065268 13003 slave.cpp:3732] master@10.142.55.196:5050
> exited
> W0919 15:50:24.065342 13003 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> I0919 15:50:24.485752 13004 slave.cpp:3732] master@10.142.55.196:5050
> exited
> W0919 15:50:24.485839 13004 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> E0919 15:50:24.485977 13010 process.cpp:2105] Failed to shutdown socket
> with fd 14: Transport endpoint is not connected
> I0919 15:50:28.343647 13003 slave.cpp:3732] master@10.142.55.196:5050
> exited
> W0919 15:50:28.343719 13003 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> E0919 15:50:28.343819 13010 process.cpp:2105] Failed to shutdown socket
> with fd 14: Transport endpoint is not connected
> I0919 15:50:31.545099 13005 slave.cpp:3732] master@10.142.55.196:5050
> exited
> W0919 15:50:31.545171 13005 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> E0919 15:50:31.545284 13010 process.cpp:2105] Failed to shutdown socket
> with fd 14: Transport endpoint is not connected
> I0919 15:50:32.007096 13008 detector.cpp:152] Detected a new leader:
> (id='694')
> I0919 15:50:32.007195 13008 group.cpp:706] Trying to get
> '/mesos/json.info_0000000694' in ZooKeeper
> I0919 15:50:32.009881 13008 zookeeper.cpp:259] A new leading master
> (UPID=master@10.142.55.202:5050) is detected
> I0919 15:50:32.009970 13008 slave.cpp:895] New master detected at
> master@10.142.55.202:5050
> I0919 15:50:32.009994 13008 slave.cpp:916] No credentials provided.
> Attempting to register without authentication
> I0919 15:50:32.010030 13008 slave.cpp:927] Detecting new master
> I0919 15:50:32.010079 13008 status_update_manager.cpp:174] Pausing
> sending status updates
> I0919 15:51:16.417846 13006 slave.cpp:4591] Current disk usage 72.64%.
> Max allowed age: 1.214964708103322days
> I0919 15:51:32.560317 13003 slave.cpp:3732] master@10.142.55.202:5050
> exited
> W0919 15:51:32.560410 13003 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> I0919 15:51:42.005147 13009 detector.cpp:152] Detected a new leader:
> (id='696')
> I0919 15:51:42.005265 13009 group.cpp:706] Trying to get
> '/mesos/json.info_0000000696' in ZooKeeper
> I0919 15:51:42.006824 13009 zookeeper.cpp:259] A new leading master
> (UPID=master@10.142.55.190:5050) is detected
> I0919 15:51:42.006904 13009 slave.cpp:895] New master detected at
> master@10.142.55.190:5050
> I0919 15:51:42.006928 13009 slave.cpp:916] No credentials provided.
> Attempting to register without authentication
> I0919 15:51:42.006963 13009 slave.cpp:927] Detecting new master
> I0919 15:51:42.006999 13009 status_update_manager.cpp:174] Pausing
> sending status updates
> I0919 15:52:16.419373 13003 slave.cpp:4591] Current disk usage 72.71%.
> Max allowed age: 1.209981628636250days
> I0919 15:52:42.336305 13002 slave.cpp:3732] master@10.142.55.190:5050
> exited
> W0919 15:52:42.336426 13002 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> I0919 15:52:54.005267 13005 detector.cpp:152] Detected a new leader:
> (id='699')
> I0919 15:52:54.005408 13005 group.cpp:706] Trying to get
> '/mesos/json.info_0000000699' in ZooKeeper
> I0919 15:52:54.006206 13005 zookeeper.cpp:259] A new leading master
> (UPID=master@10.142.55.196:5050) is detected
> I0919 15:52:54.006285 13005 slave.cpp:895] New master detected at
> master@10.142.55.196:5050
> I0919 15:52:54.006309 13005 slave.cpp:916] No credentials provided.
> Attempting to register without authentication
> I0919 15:52:54.006398 13005 slave.cpp:927] Detecting new master
> I0919 15:52:54.006451 13005 status_update_manager.cpp:174] Pausing
> sending status updates
> I0919 15:53:16.420258 13005 slave.cpp:4591] Current disk usage 72.76%.
> Max allowed age: 1.206748286096840days
> I0919 15:53:54.071012 13005 slave.cpp:3732] master@10.142.55.196:5050
> exited
> W0919 15:53:54.071143 13005 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> I0919 15:54:01.105780 13002 slave.cpp:3732] master@10.142.55.196:5050
> exited
> W0919 15:54:01.105854 13002 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> E0919 15:54:01.105970 13010 process.cpp:2105] Failed to shutdown socket
> with fd 15: Transport endpoint is not connected
> I0919 15:54:05.733837 13007 slave.cpp:3732] master@10.142.55.196:5050
> exited
> W0919 15:54:05.733932 13007 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> E0919 15:54:05.734071 13010 process.cpp:2105] Failed to shutdown socket
> with fd 15: Transport endpoint is not connected
> E0919 15:54:05.818560 13010 process.cpp:2105] Failed to shutdown socket
> with fd 15: Transport endpoint is not connected
> I0919 15:54:05.818583 13003 slave.cpp:3732] master@10.142.55.196:5050
> exited
> W0919 15:54:05.818758 13003 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> I0919 15:54:06.004385 13009 detector.cpp:152] Detected a new leader:
> (id='700')
> I0919 15:54:06.004494 13009 group.cpp:706] Trying to get
> '/mesos/json.info_0000000700' in ZooKeeper
> I0919 15:54:06.005511 13009 zookeeper.cpp:259] A new leading master
> (UPID=master@10.142.55.202:5050) is detected
> I0919 15:54:06.005586 13009 slave.cpp:895] New master detected at
> master@10.142.55.202:5050
> I0919 15:54:06.005609 13009 slave.cpp:916] No credentials provided.
> Attempting to register without authentication
> I0919 15:54:06.005676 13009 slave.cpp:927] Detecting new master
> I0919 15:54:06.005720 13009 status_update_manager.cpp:174] Pausing
> sending status updates
> I0919 15:54:16.423193 13002 slave.cpp:4591] Current disk usage 72.76%.
> Max allowed age: 1.206699342406551days
> slave warn log
> Log file created at: 2016/09/19 15:42:08
> Running on machine: ubuntu12
> Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
> W0919 15:42:08.419035 13006 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> W0919 15:43:20.081374 13005 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> W0919 15:43:26.855315 13005 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> E0919 15:43:26.855159 13010 process.cpp:2105] Failed to shutdown socket
> with fd 12: Transport endpoint is not connected
> W0919 15:44:32.535522 13003 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> W0919 15:45:42.354857 13005 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> W0919 15:46:54.075592 13007 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> E0919 15:46:56.098012 13010 process.cpp:2105] Failed to shutdown socket
> with fd 14: Transport endpoint is not connected
> W0919 15:46:56.098253 13007 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> E0919 15:46:56.462254 13010 process.cpp:2105] Failed to shutdown socket
> with fd 14: Transport endpoint is not connected
> W0919 15:46:56.462540 13005 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> W0919 15:48:02.950994 13005 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> W0919 15:49:12.379149 13009 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> E0919 15:49:12.379233 13010 process.cpp:2105] Failed to shutdown socket
> with fd 12: Transport endpoint is not connected
> W0919 15:50:24.065342 13003 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> W0919 15:50:24.485839 13004 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> E0919 15:50:24.485977 13010 process.cpp:2105] Failed to shutdown socket
> with fd 14: Transport endpoint is not connected
> W0919 15:50:28.343719 13003 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> E0919 15:50:28.343819 13010 process.cpp:2105] Failed to shutdown socket
> with fd 14: Transport endpoint is not connected
> W0919 15:50:31.545171 13005 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> E0919 15:50:31.545284 13010 process.cpp:2105] Failed to shutdown socket
> with fd 14: Transport endpoint is not connected
> W0919 15:51:32.560410 13003 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> W0919 15:52:42.336426 13002 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> W0919 15:53:54.071143 13005 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> W0919 15:54:01.105854 13002 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> E0919 15:54:01.105970 13010 process.cpp:2105] Failed to shutdown socket
> with fd 15: Transport endpoint is not connected
> W0919 15:54:05.733932 13007 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> E0919 15:54:05.734071 13010 process.cpp:2105] Failed to shutdown socket
> with fd 15: Transport endpoint is not connected
> E0919 15:54:05.818560 13010 process.cpp:2105] Failed to shutdown socket
> with fd 15: Transport endpoint is not connected
> W0919 15:54:05.818758 13003 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
> W0919 15:55:06.821486 13009 slave.cpp:3737] Master disconnected! Waiting
> for a new master to be elected
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
