Hi, everyone.
Recently, when I was doing some stress test, one of the monitors of my ceph
cluster was marked down, and all the monitors repeatedly call new election and
the I/O can be finished. There were three monitors in my cluster: rg3-ceph36,
rg3-ceph40, rg3-ceph45. It was rg3-ceph40 that was always marked down, but it
was running.
Here is rg3-ceph40’s monitor log with debug_mon and debug_paxos set to 20/0.
2017-02-03 16:33:25.738290 7f4f301ca700 5
mon.rg3-ceph40@1(electing).elector(7009) handle_ack from mon.2
2017-02-03 16:33:25.738294 7f4f301ca700 5
mon.rg3-ceph40@1(electing).elector(7009) so far i have
{1=37154696925806591,2=37154696925806591}
2017-02-03 16:33:28.033563 7f4f309cb700 11 mon.rg3-ceph40@1(electing) e1 tick
2017-02-03 16:33:28.033584 7f4f309cb700 20 mon.rg3-ceph40@1(electing) e1
sync_trim_providers
2017-02-03 16:33:30.737928 7f4f309cb700 5
mon.rg3-ceph40@1(electing).elector(7009) election timer expired
2017-02-03 16:33:30.737953 7f4f309cb700 10
mon.rg3-ceph40@1(electing).elector(7009) bump_epoch 7009 to 7010
2017-02-03 16:33:30.740784 7f4f309cb700 10 mon.rg3-ceph40@1(electing) e1
join_election
2017-02-03 16:33:30.740802 7f4f309cb700 10 mon.rg3-ceph40@1(electing) e1 _reset
2017-02-03 16:33:30.740805 7f4f309cb700 10 mon.rg3-ceph40@1(electing) e1
cancel_probe_timeout (none scheduled)
2017-02-03 16:33:30.740807 7f4f309cb700 10 mon.rg3-ceph40@1(electing) e1
timecheck_finish
2017-02-03 16:33:30.740810 7f4f309cb700 15 mon.rg3-ceph40@1(electing) e1
health_tick_stop
2017-02-03 16:33:30.740812 7f4f309cb700 15 mon.rg3-ceph40@1(electing) e1
health_interval_stop
2017-02-03 16:33:30.740814 7f4f309cb700 10 mon.rg3-ceph40@1(electing) e1
scrub_reset
2017-02-03 16:33:30.740816 7f4f309cb700 10
mon.rg3-ceph40@1(electing).paxos(paxos recovering c 200550..201284) restart --
canceling timeouts
2017-02-03 16:33:30.740823 7f4f309cb700 10
mon.rg3-ceph40@1(electing).paxosservice(pgmap 85794..86329) restart
2017-02-03 16:33:30.740827 7f4f309cb700 10
mon.rg3-ceph40@1(electing).paxosservice(mdsmap 1..1) restart
2017-02-03 16:33:30.740830 7f4f309cb700 10
mon.rg3-ceph40@1(electing).paxosservice(osdmap 25125..25724) restart
2017-02-03 16:33:30.740832 7f4f309cb700 10
mon.rg3-ceph40@1(electing).paxosservice(logm 98223..98787) restart
2017-02-03 16:33:30.740834 7f4f309cb700 10
mon.rg3-ceph40@1(electing).paxosservice(monmap 1..1) restart
2017-02-03 16:33:30.740836 7f4f309cb700 10
mon.rg3-ceph40@1(electing).paxosservice(auth 501..623) restart
2017-02-03 16:33:30.740872 7f4f309cb700 10 mon.rg3-ceph40@1(electing) e1
win_election epoch 7010 quorum 1,2 features 37154696925806591
2017-02-03 16:33:30.740889 7f4f309cb700 0 log_channel(cluster) log [INF] :
mon.rg3-ceph40@1 won leader election with quorum 1,2
2017-02-03 16:33:30.741119 7f4f309cb700 10 mon.rg3-ceph40@1(leader).paxos(paxos
recovering c 200550..201284) leader_init -- starting paxos recovery
2017-02-03 16:33:30.742301 7f4f309cb700 10 mon.rg3-ceph40@1(leader).paxos(paxos
recovering c 200550..201284) get_new_proposal_number = 350501
2017-02-03 16:33:30.742315 7f4f309cb700 10 mon.rg3-ceph40@1(leader).paxos(paxos
recovering c 200550..201284) collect with pn 350501
2017-02-03 16:33:30.742328 7f4f309cb700 10
mon.rg3-ceph40@1(leader).paxosservice(monmap 1..1) election_finished
2017-02-03 16:33:30.742332 7f4f309cb700 10
mon.rg3-ceph40@1(leader).paxosservice(monmap 1..1) _active - not active
2017-02-03 16:33:30.742334 7f4f309cb700 10
mon.rg3-ceph40@1(leader).paxosservice(pgmap 85794..86329) election_finished
2017-02-03 16:33:30.742336 7f4f309cb700 10
mon.rg3-ceph40@1(leader).paxosservice(pgmap 85794..86329) _active - not active
2017-02-03 16:33:30.742338 7f4f309cb700 10
mon.rg3-ceph40@1(leader).paxosservice(mdsmap 1..1) election_finished
2017-02-03 16:33:30.742340 7f4f309cb700 10
mon.rg3-ceph40@1(leader).paxosservice(mdsmap 1..1) _active - not active
2017-02-03 16:33:30.742341 7f4f309cb700 10
mon.rg3-ceph40@1(leader).paxosservice(osdmap 25125..25724) election_finished
2017-02-03 16:33:30.742343 7f4f309cb700 10
mon.rg3-ceph40@1(leader).paxosservice(osdmap 25125..25724) _active - not active
2017-02-03 16:33:30.742345 7f4f309cb700 10
mon.rg3-ceph40@1(leader).paxosservice(logm 98223..98787) election_finished
2017-02-03 16:33:30.742346 7f4f309cb700 10
mon.rg3-ceph40@1(leader).paxosservice(logm 98223..98787) _active - not active
2017-02-03 16:33:30.742348 7f4f309cb700 10
mon.rg3-ceph40@1(leader).paxosservice(auth 501..623) election_finished
2017-02-03 16:33:30.742350 7f4f309cb700 10
mon.rg3-ceph40@1(leader).paxosservice(auth 501..623) _active - not active
2017-02-03 16:33:30.742352 7f4f309cb700 10
mon.rg3-ceph40@1(leader).data_health(7010) start_epoch epoch 7010
2017-02-03 16:33:30.742361 7f4f309cb700 10 mon.rg3-ceph40@1(leader) e1
timecheck_finish
2017-02-03 16:33:30.742363 7f4f309cb700 10 mon.rg3-ceph40@1(leader) e1
resend_routed_requests
2017-02-03 16:33:30.742377 7f4f309cb700 10 mon.rg3-ceph40@1(leader) e1 requeue
for self tid 3383 log(1 entries) v1
2017-02-03 16:33:30.742383 7f4f309cb700 20 mon.rg3-ceph40@1(leader) e1 have
connection
2017-02-03 16:33:30.742386 7f4f309cb700 20 mon.rg3-ceph40@1(leader) e1
ms_dispatch existing session MonSession: mon.1 10.205.198.85:6789/0 is
openallow * for mon.1 10.205.198.85:6789/0
2017-02-03 16:33:30.742396 7f4f309cb700 20 mon.rg3-ceph40@1(leader) e1 caps
allow *
2017-02-03 16:33:30.742400 7f4f309cb700 10
mon.rg3-ceph40@1(leader).paxosservice(logm 98223..98787) dispatch log(1
entries) v1 from mon.1 10.205.198.85:6789/0
2017-02-03 16:33:30.742407 7f4f309cb700 5 mon.rg3-ceph40@1(leader).paxos(paxos
recovering c 200550..201284) is_readable = 0 - now=2017-02-03 16:33:30.742408
lease_expire=0.000000 has v0 lc 201284
2017-02-03 16:33:30.742420 7f4f309cb700 10
mon.rg3-ceph40@1(leader).paxosservice(logm 98223..98787) waiting for paxos ->
readable (v0)
2017-02-03 16:33:30.742423 7f4f309cb700 5 mon.rg3-ceph40@1(leader).paxos(paxos
recovering c 200550..201284) is_readable = 0 - now=2017-02-03 16:33:30.742424
lease_expire=0.000000 has v0 lc 201284
2017-02-03 16:33:30.742432 7f4f309cb700 10 mon.rg3-ceph40@1(leader) e1
register_cluster_logger
2017-02-03 16:33:30.742438 7f4f309cb700 10 mon.rg3-ceph40@1(leader) e1
timecheck_start
2017-02-03 16:33:30.742440 7f4f309cb700 10 mon.rg3-ceph40@1(leader) e1
timecheck_start_round curr 0
2017-02-03 16:33:30.742442 7f4f309cb700 10 mon.rg3-ceph40@1(leader) e1
timecheck_start_round new 1
2017-02-03 16:33:30.742444 7f4f309cb700 10 mon.rg3-ceph40@1(leader) e1 timecheck
2017-02-03 16:33:30.742445 7f4f309cb700 10 mon.rg3-ceph40@1(leader) e1
timecheck start timecheck epoch 7010 round 1
2017-02-03 16:33:30.742452 7f4f309cb700 10 mon.rg3-ceph40@1(leader) e1
timecheck send time_check( ping e 7010 r 1 ) v1 to mon.2 10.205.198.149:6789/0
2017-02-03 16:33:30.742462 7f4f309cb700 10 mon.rg3-ceph40@1(leader) e1
timecheck_start_round setting up next event
2017-02-03 16:33:30.742467 7f4f309cb700 15 mon.rg3-ceph40@1(leader) e1
health_tick_start
2017-02-03 16:33:30.742469 7f4f309cb700 15 mon.rg3-ceph40@1(leader) e1
health_tick_stop
2017-02-03 16:33:30.742472 7f4f309cb700 10 mon.rg3-ceph40@1(leader) e1
do_health_to_clog_interval
2017-02-03 16:33:30.742474 7f4f309cb700 10 mon.rg3-ceph40@1(leader) e1
do_health_to_clog (force)
2017-02-03 16:33:30.743637 7f4f309cb700 10
mon.rg3-ceph40@1(leader).data_health(7010) get_health
2017-02-03 16:33:30.743661 7f4f309cb700 0 log_channel(cluster) log [INF] :
HEALTH_WARN; 1 mons down, quorum 1,2 rg3-ceph40,rg3-ceph45
2017-02-03 16:33:30.743701 7f4f309cb700 15 mon.rg3-ceph40@1(leader) e1
health_interval_start
2017-02-03 16:33:30.743704 7f4f309cb700 15 mon.rg3-ceph40@1(leader) e1
health_interval_stop
2017-02-03 16:33:30.743705 7f4f309cb700 20 mon.rg3-ceph40@1(leader) e1
health_interval_calc_next_update now: 2017-02-03 16:33:30.743705, next:
2017-02-03 17:00:00.000000, interval: 3600
2017-02-03 16:33:30.743724 7f4f301ca700 20 mon.rg3-ceph40@1(leader) e1 have
connection
2017-02-03 16:33:30.743732 7f4f301ca700 20 mon.rg3-ceph40@1(leader) e1
ms_dispatch existing session MonSession: mon.1 10.205.198.85:6789/0 is
openallow * for mon.1 10.205.198.85:6789/0
2017-02-03 16:33:30.743744 7f4f301ca700 20 mon.rg3-ceph40@1(leader) e1 caps
allow *
2017-02-03 16:33:30.743747 7f4f301ca700 10
mon.rg3-ceph40@1(leader).paxosservice(logm 98223..98787) dispatch log(1
entries) v1 from mon.1 10.205.198.85:6789/0
2017-02-03 16:33:30.743756 7f4f301ca700 5 mon.rg3-ceph40@1(leader).paxos(paxos
recovering c 200550..201284) is_readable = 0 - now=2017-02-03 16:33:30.743756
lease_expire=0.000000 has v0 lc 201284
2017-02-03 16:33:30.743766 7f4f301ca700 10
mon.rg3-ceph40@1(leader).paxosservice(logm 98223..98787) waiting for paxos ->
readable (v0)
2017-02-03 16:33:30.743770 7f4f301ca700 5 mon.rg3-ceph40@1(leader).paxos(paxos
recovering c 200550..201284) is_readable = 0 - now=2017-02-03 16:33:30.743775
lease_expire=0.000000 has v0 lc 201284
2017-02-03 16:33:30.743786 7f4f301ca700 20 mon.rg3-ceph40@1(leader) e1 have
connection
2017-02-03 16:33:30.743789 7f4f301ca700 20 mon.rg3-ceph40@1(leader) e1
ms_dispatch existing session MonSession: mon.1 10.205.198.85:6789/0 is
openallow * for mon.1 10.205.198.85:6789/0
2017-02-03 16:33:30.743796 7f4f301ca700 20 mon.rg3-ceph40@1(leader) e1 caps
allow *
2017-02-03 16:33:30.743798 7f4f301ca700 10
mon.rg3-ceph40@1(leader).paxosservice(logm 98223..98787) dispatch log(1
entries) v1 from mon.1 10.205.198.85:6789/0
2017-02-03 16:33:30.743803 7f4f301ca700 5 mon.rg3-ceph40@1(leader).paxos(paxos
recovering c 200550..201284) is_readable = 0 - now=2017-02-03 16:33:30.743804
lease_expire=0.000000 has v0 lc 201284
2017-02-03 16:33:30.743809 7f4f301ca700 10
mon.rg3-ceph40@1(leader).paxosservice(logm 98223..98787) waiting for paxos ->
readable (v0)
2017-02-03 16:33:30.743812 7f4f301ca700 5 mon.rg3-ceph40@1(leader).paxos(paxos
recovering c 200550..201284) is_readable = 0 - now=2017-02-03 16:33:30.743814
lease_expire=0.000000 has v0 lc 201284
2017-02-03 16:33:30.789898 7f4f301ca700 20 mon.rg3-ceph40@1(leader) e1 have
connection
2017-02-03 16:33:30.789910 7f4f301ca700 20 mon.rg3-ceph40@1(leader) e1
ms_dispatch existing session MonSession: mon.2 10.205.198.149:6789/0 is
openallow * for mon.2 10.205.198.149:6789/0
I turned down rg3-ceph40 by /etc/init.d/ceph stop mon, the other two monitors
stopped calling new election, however, the I/O were still stuck. The following
the monitor log of rg3-ceph36:
2017-02-03 17:03:02.279230 7fe0be819700 10 mon.rg3-ceph36@0(leader).pg v86564
encode_pending v 86565
2017-02-03 17:03:02.337531 7fe0be819700 10 mon.rg3-ceph36@0(leader).log v99039
encode_full log v 99039
2017-02-03 17:03:02.337612 7fe0be819700 10 mon.rg3-ceph36@0(leader).log v99039
encode_pending v99040
2017-02-03 17:03:02.354173 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader) e1
refresh_from_paxos
2017-02-03 17:03:02.354261 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader).pg v86564
update_from_paxos read_incremental
2017-02-03 17:03:02.354766 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader).pg v86565
read_pgmap_meta
2017-02-03 17:03:02.354874 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader).pg v86565
map_pg_creates to 0 pgs -- no change
2017-02-03 17:03:02.354881 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader).pg v86565
send_pg_creates to 0 pgs
2017-02-03 17:03:02.354885 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader).pg v86565
update_logger
2017-02-03 17:03:02.355051 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader).log v99039
update_from_paxos
2017-02-03 17:03:02.355061 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader).log v99039
update_from_paxos version 99039 summary v 99039
2017-02-03 17:03:02.355180 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader).auth v625
update_from_paxos
2017-02-03 17:03:02.355188 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader).pg v86565
map_pg_creates to 0 pgs -- no change
2017-02-03 17:03:02.355190 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader).pg v86565
send_pg_creates to 0 pgs
2017-02-03 17:03:02.355234 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader).pg v86565
create_pending v 86566
2017-02-03 17:03:02.355238 7fe0bfc1f700 7 mon.rg3-ceph36@0(leader).pg v86565
_updated_stats for osd.67 10.205.198.148:6812/99322
2017-02-03 17:03:02.355259 7fe0bfc1f700 7 mon.rg3-ceph36@0(leader).pg v86565
_updated_stats for osd.12 10.205.198.147:6802/96650
2017-02-03 17:03:02.355272 7fe0bfc1f700 7 mon.rg3-ceph36@0(leader).pg v86565
_updated_stats for osd.80 10.205.198.147:6816/99711
2017-02-03 17:03:02.355283 7fe0bfc1f700 7 mon.rg3-ceph36@0(leader).pg v86565
_updated_stats for osd.72 10.205.198.82:6804/2609
2017-02-03 17:03:02.355292 7fe0bfc1f700 7 mon.rg3-ceph36@0(leader).pg v86565
_updated_stats for osd.17 10.205.198.82:6802/2275
2017-02-03 17:03:02.355302 7fe0bfc1f700 7 mon.rg3-ceph36@0(leader).pg v86565
_updated_stats for osd.22 10.205.198.149:6802/11553
2017-02-03 17:03:02.355324 7fe0bfc1f700 7 mon.rg3-ceph36@0(leader).pg v86565
_updated_stats for osd.83 10.205.198.145:6816/83116
2017-02-03 17:03:02.355335 7fe0bfc1f700 7 mon.rg3-ceph36@0(leader).pg v86565
_updated_stats for osd.86 10.205.198.146:6806/92691
2017-02-03 17:03:02.355362 7fe0bfc1f700 7 mon.rg3-ceph36@0(leader).pg v86565
_updated_stats for osd.71 10.205.198.82:6808/3461
2017-02-03 17:03:02.355377 7fe0bfc1f700 7 mon.rg3-ceph36@0(leader).pg v86565
_updated_stats for osd.32 10.205.198.81:6806/48193
2017-02-03 17:03:02.355395 7fe0bfc1f700 7 mon.rg3-ceph36@0(leader).pg v86565
_updated_stats for osd.30 10.205.198.83:6812/1188
2017-02-03 17:03:02.355418 7fe0bfc1f700 7 mon.rg3-ceph36@0(leader).pg v86565
_updated_stats for osd.65 10.205.198.145:6810/81841
2017-02-03 17:03:02.355448 7fe0bfc1f700 7 mon.rg3-ceph36@0(leader).pg v86565
_updated_stats for osd.33 10.205.198.149:6808/12810
2017-02-03 17:03:02.355466 7fe0bfc1f700 7 mon.rg3-ceph36@0(leader).pg v86565
_updated_stats for osd.34 10.205.198.149:6816/14873
2017-02-03 17:03:02.355477 7fe0bfc1f700 7 mon.rg3-ceph36@0(leader).pg v86565
_updated_stats for osd.3 10.205.198.81:6802/47424
2017-02-03 17:03:02.355488 7fe0bfc1f700 7 mon.rg3-ceph36@0(leader).pg v86565
_updated_stats for osd.20 10.205.198.83:6806/101823
2017-02-03 17:03:02.355498 7fe0bfc1f700 7 mon.rg3-ceph36@0(leader).pg v86565
_updated_stats for osd.55 10.205.198.148:6816/100258
2017-02-03 17:03:02.355509 7fe0bfc1f700 7 mon.rg3-ceph36@0(leader).pg v86565
_updated_stats for osd.9 10.205.198.145:6806/80779
2017-02-03 17:03:02.355519 7fe0bfc1f700 7 mon.rg3-ceph36@0(leader).pg v86565
_updated_stats for osd.84 10.205.198.146:6804/92333
2017-02-03 17:03:02.355528 7fe0bfc1f700 7 mon.rg3-ceph36@0(leader).pg v86565
_updated_stats for osd.54 10.205.198.145:6814/82758
2017-02-03 17:03:02.355538 7fe0bfc1f700 7 mon.rg3-ceph36@0(leader).pg v86565
_updated_stats for osd.87 10.205.198.146:6812/94054
2017-02-03 17:03:02.355563 7fe0bfc1f700 7 mon.rg3-ceph36@0(leader).pg v86565
_updated_stats for osd.68 10.205.198.148:6802/97249
2017-02-03 17:03:02.355603 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader).pg v86565
check_osd_map already seen 25728 >= 25728
2017-02-03 17:03:02.355611 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader).pg v86565
update_logger
2017-02-03 17:03:02.355635 7fe0bfc1f700 0 log_channel(cluster) log [INF] :
pgmap v86565: 4096 pgs: 4096 active+clean; 296 MB data, 14135 MB used, 290 TB /
306 TB avail; 5268 B/s rd, 6 op/s
2017-02-03 17:03:02.379079 7fe0be018700 10 mon.rg3-ceph36@0(leader).log v99039
preprocess_query log(1 entries) v1 from mon.0 10.205.198.81:6789/0
2017-02-03 17:03:02.379089 7fe0be018700 10 mon.rg3-ceph36@0(leader).log v99039
preprocess_log log(1 entries) v1 from mon.0
2017-02-03 17:03:02.412442 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader) e1
refresh_from_paxos
2017-02-03 17:03:02.412681 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader).log v99040
update_from_paxos
2017-02-03 17:03:02.412689 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader).log v99040
update_from_paxos version 99040 summary v 99039
2017-02-03 17:03:02.412709 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader).log v99040
update_from_paxos latest full 99039
2017-02-03 17:03:02.412743 7fe0bfc1f700 7 mon.rg3-ceph36@0(leader).log v99040
update_from_paxos applying incremental log 99040 2017-02-03 17:03:01.280909
mon.0 10.205.198.81:6789/0 13620 : cluster [INF] pgmap v86564: 4096 pgs: 4096
active+clean; 296 MB data, 13947 MB used, 290 TB / 306 TB avail
2017-02-03 17:03:02.412822 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader).log v99040
check_subs
2017-02-03 17:03:02.412962 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader).auth v625
update_from_paxos
2017-02-03 17:03:02.412969 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader).pg v86565
map_pg_creates to 0 pgs -- no change
2017-02-03 17:03:02.412972 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader).pg v86565
send_pg_creates to 0 pgs
2017-02-03 17:03:02.413011 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader).log v99040
create_pending v 99041
2017-02-03 17:03:02.413015 7fe0bfc1f700 7 mon.rg3-ceph36@0(leader).log v99040
_updated_log for mon.0 10.205.198.81:6789/0
2017-02-03 17:03:02.413042 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader).log v99040
preprocess_query log(1 entries) v1 from mon.0 10.205.198.81:6789/0
2017-02-03 17:03:02.413048 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader).log v99040
preprocess_log log(1 entries) v1 from mon.0
2017-02-03 17:03:02.413055 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader).log v99040
prepare_update log(1 entries) v1 from mon.0 10.205.198.81:6789/0
2017-02-03 17:03:02.413059 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader).log v99040
prepare_log log(1 entries) v1 from mon.0
2017-02-03 17:03:02.413063 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader).log v99040
logging 2017-02-03 17:03:02.355639 mon.0 10.205.198.81:6789/0 13621 : cluster
[INF] pgmap v86565: 4096 pgs: 4096 active+clean; 296 MB data, 14135 MB used,
290 TB / 306 TB avail; 5268 B/s rd, 6 op/s
2017-02-03 17:03:03.412444 7fe0be819700 10 mon.rg3-ceph36@0(leader).log v99040
encode_full log v 99040
2017-02-03 17:03:03.412548 7fe0be819700 10 mon.rg3-ceph36@0(leader).log v99040
encode_pending v99041
2017-02-03 17:03:03.479045 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader) e1
refresh_from_paxos
2017-02-03 17:03:03.479292 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader).log v99041
update_from_paxos
2017-02-03 17:03:03.479299 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader).log v99041
update_from_paxos version 99041 summary v 99040
2017-02-03 17:03:03.479320 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader).log v99041
update_from_paxos latest full 99040
2017-02-03 17:03:03.479348 7fe0bfc1f700 7 mon.rg3-ceph36@0(leader).log v99041
update_from_paxos applying incremental log 99041 2017-02-03 17:03:02.355639
mon.0 10.205.198.81:6789/0 13621 : cluster [INF] pgmap v86565: 4096 pgs: 4096
active+clean; 296 MB data, 14135 MB used, 290 TB / 306 TB avail; 5268 B/s rd, 6
op/s
2017-02-03 17:03:03.479423 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader).log v99041
check_subs
2017-02-03 17:03:03.479572 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader).auth v625
update_from_paxos
2017-02-03 17:03:03.479581 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader).pg v86565
map_pg_creates to 0 pgs -- no change
2017-02-03 17:03:03.479584 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader).pg v86565
send_pg_creates to 0 pgs
2017-02-03 17:03:03.479631 7fe0bfc1f700 10 mon.rg3-ceph36@0(leader).log v99041
create_pending v 99042
2017-02-03 17:03:03.479638 7fe0bfc1f700 7 mon.rg3-ceph36@0(leader).log v99041
_updated_log for mon.0 10.205.198.81:6789/0
2017-02-03 17:03:04.489266 7fe0be819700 10 mon.rg3-ceph36@0(leader).pg v86565
check_down_pgs
2017-02-03 17:03:04.489440 7fe0be819700 10 mon.rg3-ceph36@0(leader).pg v86565
v86565: 4096 pgs: 4096 active+clean; 296 MB data, 14135 MB used, 290 TB / 306
TB avail; 5268 B/s rd, 6 op/s
2017-02-03 17:03:04.489473 7fe0be819700 10 mon.rg3-ceph36@0(leader).osd e25728
e25728: 90 osds: 90 up, 90 in
2017-02-03 17:03:04.489530 7fe0be819700 10 mon.rg3-ceph36@0(leader).osd e25728
min_last_epoch_clean 25728
2017-02-03 17:03:04.489533 7fe0be819700 10 mon.rg3-ceph36@0(leader).log v99041
log
2017-02-03 17:03:04.489538 7fe0be819700 10 mon.rg3-ceph36@0(leader).auth v625
auth
2017-02-03 17:03:05.814732 7fe0be018700 10 mon.rg3-ceph36@0(leader) e1
handle_subscribe mon_subscribe({monmap=2+,osd_pg_creates=0}) v2
2017-02-03 17:03:05.814760 7fe0be018700 10 mon.rg3-ceph36@0(leader) e1
check_sub monmap next 2 have 1
2017-02-03 17:03:08.567850 7fe0be018700 10 mon.rg3-ceph36@0(leader) e1
handle_subscribe mon_subscribe({monmap=2+,osd_pg_creates=0}) v2
2017-02-03 17:03:08.567880 7fe0be018700 10 mon.rg3-ceph36@0(leader) e1
check_sub monmap next 2 have 1
2017-02-03 17:03:09.489665 7fe0be819700 10 mon.rg3-ceph36@0(leader).pg v86565
v86565: 4096 pgs: 4096 active+clean; 296 MB data, 14135 MB used, 290 TB / 306
TB avail; 5268 B/s rd, 6 op/s
2017-02-03 17:03:09.489708 7fe0be819700 10 mon.rg3-ceph36@0(leader).osd e25728
e25728: 90 osds: 90 up, 90 in
2017-02-03 17:03:09.489750 7fe0be819700 10 mon.rg3-ceph36@0(leader).osd e25728
min_last_epoch_clean 25728
2017-02-03 17:03:09.489753 7fe0be819700 10 mon.rg3-ceph36@0(leader).log v99041
log
2017-02-03 17:03:09.489758 7fe0be819700 10 mon.rg3-ceph36@0(leader).auth v625
auth
2017-02-03 17:03:10.940004 7fe0be018700 10 mon.rg3-ceph36@0(leader) e1 received
forwarded message from osd.29 10.205.198.82:6812/4405 via mon.2
10.205.198.149:6789/0
2017-02-03 17:03:10.940036 7fe0be018700 10 mon.rg3-ceph36@0(leader) e1 caps
are allow *
2017-02-03 17:03:10.940040 7fe0be018700 10 mon.rg3-ceph36@0(leader) e1 entity
name 'osd.29' type 4
2017-02-03 17:03:10.940042 7fe0be018700 10 mon.rg3-ceph36@0(leader) e1 mesg
0x7fe0bb1a4c00 from 10.205.198.149:6789/0
2017-02-03 17:03:10.940065 7fe0be018700 10 mon.rg3-ceph36@0(leader).pg v86565
preprocess_query pg_stats(0 pgs tid 16094 v 0) v1 from osd.29
10.205.198.82:6812/4405
2017-02-03 17:03:10.940082 7fe0be018700 10 mon.rg3-ceph36@0(leader).pg v86565
prepare_update pg_stats(0 pgs tid 16094 v 0) v1 from osd.29
10.205.198.82:6812/4405
2017-02-03 17:03:10.940088 7fe0be018700 10 mon.rg3-ceph36@0(leader).pg v86565
prepare_pg_stats pg_stats(0 pgs tid 16094 v 0) v1 from osd.29
2017-02-03 17:03:10.940101 7fe0be018700 10 mon.rg3-ceph36@0(leader).pg v86565
message contains no new osd|pg stats
2017-02-03 17:03:10.986247 7fe0be018700 10 mon.rg3-ceph36@0(leader) e1
handle_subscribe mon_subscribe({monmap=2+,osd_pg_creates=0}) v2
2017-02-03 17:03:10.986265 7fe0be018700 10 mon.rg3-ceph36@0(leader) e1
check_sub monmap next 2 have 1
2017-02-03 17:03:11.344862 7fe0be018700 10 mon.rg3-ceph36@0(leader) e1 received
forwarded message from osd.77 10.205.198.145:6812/82199 via mon.2
10.205.198.149:6789/0
2017-02-03 17:03:11.344881 7fe0be018700 10 mon.rg3-ceph36@0(leader) e1 caps
are allow *
2017-02-03 17:03:11.344885 7fe0be018700 10 mon.rg3-ceph36@0(leader) e1 entity
name 'osd.77' type 4
2017-02-03 17:03:11.344887 7fe0be018700 10 mon.rg3-ceph36@0(leader) e1 mesg
0x7fe0bb1a5b00 from 10.205.198.149:6789/0
2017-02-03 17:03:11.344904 7fe0be018700 10 mon.rg3-ceph36@0(leader).pg v86565
preprocess_query pg_stats(0 pgs tid 16272 v 0) v1 from osd.77
10.205.198.145:6812/82199
2017-02-03 17:03:11.344926 7fe0be018700 10 mon.rg3-ceph36@0(leader).pg v86565
prepare_update pg_stats(0 pgs tid 16272 v 0) v1 from osd.77
10.205.198.145:6812/82199
2017-02-03 17:03:11.344931 7fe0be018700 10 mon.rg3-ceph36@0(leader).pg v86565
prepare_pg_stats pg_stats(0 pgs tid 16272 v 0) v1 from osd.77
2017-02-03 17:03:11.344941 7fe0be018700 10 mon.rg3-ceph36@0(leader).pg v86565
message contains no new osd|pg stats
2017-02-03 17:03:11.833360 7fe0be018700 10 mon.rg3-ceph36@0(leader).pg v86565
preprocess_query pg_stats(0 pgs tid 17368 v 0) v1 from osd.49
10.205.198.85:6812/13922
2017-02-03 17:03:11.833385 7fe0be018700 10 mon.rg3-ceph36@0(leader).pg v86565
prepare_update pg_stats(0 pgs tid 17368 v 0) v1 from osd.49
10.205.198.85:6812/13922
2017-02-03 17:03:11.833390 7fe0be018700 10 mon.rg3-ceph36@0(leader).pg v86565
prepare_pg_stats pg_stats(0 pgs tid 17368 v 0) v1 from osd.49
2017-02-03 17:03:11.833398 7fe0be018700 10 mon.rg3-ceph36@0(leader).pg v86565
message contains no new osd|pg stats
I’m using hammer version, 0.94.5. Please help me, thank you.
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
