Hello,
In one of our cluster set up, there is frequent monitor elections
happening.
In the logs of one of the monitor, there is "lease_timeout" message before
that happens. Can anyone help me to figure it out ?
(When this happens, the Ceph Dashboard GUI gets stuck and we have to
restart the manager daemon to make it work again)
Ceph version : Luminous 12.2.2
Log :
=========================
2018-01-16 16:33:08.001937 7f0cfbaad700 4 rocksdb:
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUILD/ceph-12.2.2/src/rocksdb/db/compaction_job.cc:1173]
[default] [JOB 885] Compacted 1@0 + 1@1 files to L1 => 20046585 bytes
2018-01-16 16:33:08.015891 7f0cfbaad700 4 rocksdb: (Original Log Time
2018/01/16-16:33:08.015826)
[/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.2/rpm/el7/BUILD/ceph-12.2.2/src/rocksdb/db/compaction_job.cc:621]
[default] compacted to: base level 1 max bytes base 268435456 files[0 1 0 0
0 0 0] max score 0.07, MB/sec: 32.7 rd, 30.9 wr, level 1, files in(1, 1)
out(1) MB in(1.3, 18.9) out(19.1), read-write-amplify(31.0)
write-amplify(15.1) OK, records in: 4305, records dropped: 515
2018-01-16 16:33:08.015897 7f0cfbaad700 4 rocksdb: (Original Log Time
2018/01/16-16:33:08.015840) EVENT_LOG_v1 {"time_micros": 1516149188015833,
"job": 885, "event": "compaction_finished", "compaction_time_micros":
647876, "output_level": 1, "num_output_files": 1, "total_output_size":
20046585, "num_input_records": 4305, "num_output_records": 3790,
"num_subcompactions": 1, "num_single_delete_mismatches": 0,
"num_single_delete_fallthrough": 0, "lsm_state": [0, 1, 0, 0, 0, 0, 0]}
2018-01-16 16:33:08.016131 7f0cfbaad700 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1516149188016128, "job": 885, "event":
"table_file_deletion", "file_number": 2419}
2018-01-16 16:33:08.018147 7f0cfbaad700 4 rocksdb: EVENT_LOG_v1
{"time_micros": 1516149188018146, "job": 885, "event":
"table_file_deletion", "file_number": 2417}
2018-01-16 16:33:11.051010 7f0d042be700 0
mon.ceph-mon3@2(peon).data_health(436)
update_stats avail 84% total 20918 MB, used 2179 MB, avail 17653 MB
2018-01-16 16:33:17.269954 7f0d042be700 1 mon.ceph-mon3@2(peon).paxos(paxos
active c 84337..84838) lease_timeout -- calling new election
2018-01-16 16:33:17.291096 7f0d01ab9700 0 log_channel(cluster) log [INF] :
mon.ceph-sgp-mon3 calling new monitor election
2018-01-16 16:33:17.291182 7f0d01ab9700 1
mon.ceph-mon3@2(electing).elector(436)
init, last seen epoch 436
2018-01-16 16:33:20.834853 7f0d01ab9700 1 mon.ceph-mon3@2(peon).log v23189
check_sub sending message to client.65755 10.255.0.95:0/2603001850 with 8
entries (version 23189)
Karun
_______________________________________________
ceph-users mailing list
[email protected]
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
