Hi, Mike I also think the patch is not related. just wanted to let someone know the version which I use.
I checked out with this command "git fetch http://gerrit.cloudera.org:8080/kudu refs/changes/25/6925/11 && git checkout FETCH_HEAD" I've included more logs. I0605 15:04:30.276859 138639 maintenance_manager.cc:436] P 125dbd2ffb8a401bb7e4fd982995ccf8: UndoDeltaBlockGCOp(f50aac31d00a4ea6b8716f205a1e8f40) metrics: {"cfile_init":1,"lbm_read_time_us":6,"lbm_reads_lt_1ms":4} I0605 15:04:30.279415 138637 tablet.cc:1512] T f50aac31d00a4ea6b8716f205a1e8f40 P 125dbd2ffb8a401bb7e4fd982995ccf8: Compaction: stage 1 complete, picked 9 rowsets to compact I0605 15:04:30.279464 138637 compaction.cc:881] Selected 9 rowsets to compact: I0605 15:04:30.279474 138637 compaction.cc:884] RowSet(524)(current size on disk: ~23945114 bytes) I0605 15:04:30.279479 138637 compaction.cc:884] RowSet(525)(current size on disk: ~24539818 bytes) I0605 15:04:30.279482 138637 compaction.cc:884] RowSet(526)(current size on disk: ~24827238 bytes) I0605 15:04:30.279485 138637 compaction.cc:884] RowSet(527)(current size on disk: ~24766186 bytes) I0605 15:04:30.279489 138637 compaction.cc:884] RowSet(528)(current size on disk: ~24862612 bytes) I0605 15:04:30.279495 138637 compaction.cc:884] RowSet(529)(current size on disk: ~24750447 bytes) I0605 15:04:30.279498 138637 compaction.cc:884] RowSet(530)(current size on disk: ~24760393 bytes) I0605 15:04:30.279502 138637 compaction.cc:884] RowSet(531)(current size on disk: ~15210250 bytes) I0605 15:04:30.279505 138637 compaction.cc:884] RowSet(532)(current size on disk: ~1125877 bytes) I0605 15:04:30.279516 138637 tablet.cc:1298] T f50aac31d00a4ea6b8716f205a1e8f40 P 125dbd2ffb8a401bb7e4fd982995ccf8: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6130248128052830208 or (T in {6130248128052830208})}] I0605 15:04:30.280143 138637 multi_column_writer.cc:89] Opened CFile writers for 8 column(s) I0605 15:04:30.939551 138640 multi_column_writer.cc:89] Opened CFile writers for 8 column(s) I0605 15:04:31.252856 138637 multi_column_writer.cc:89] Opened CFile writers for 8 column(s) I0605 15:04:31.967231 138640 multi_column_writer.cc:89] Opened CFile writers for 8 column(s) I0605 15:04:32.265857 138637 multi_column_writer.cc:89] Opened CFile writers for 8 column(s) I0605 15:04:33.017536 138640 multi_column_writer.cc:89] Opened CFile writers for 8 column(s) I0605 15:04:33.309141 138637 multi_column_writer.cc:89] Opened CFile writers for 8 column(s) I0605 15:04:33.361656 138635 tablet.cc:1039] T 3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8: Flush: entering stage 1 (old memrowset already frozen for inserts) I0605 15:04:33.361706 138635 compaction.cc:881] Selected 1 rowsets to compact: I0605 15:04:33.361712 138635 compaction.cc:884] memrowset(current size on disk: ~0 bytes) I0605 15:04:33.361716 138635 tablet.cc:1041] T 3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8: Memstore in-memory size: 763 bytes I0605 15:04:33.361727 138635 tablet.cc:1298] T 3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8: Flush: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6130248381986324480 or (T in {6130248381986324480})}] I0605 15:04:33.361981 138635 multi_column_writer.cc:89] Opened CFile writers for 6 column(s) I0605 15:04:33.363195 138635 tablet.cc:1383] T 3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8: Flush: entering phase 2 (starting to duplicate updates in new rowsets) I0605 15:04:33.363463 138635 tablet.cc:1438] T 3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8: Flush Phase 2: carrying over any updates which arrived during Phase 1 I0605 15:04:33.363481 138635 tablet.cc:1440] T 3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6130248381986324480 or (T in {6130248381986324480})}] I0605 15:04:33.365365 138635 tablet.cc:1482] T 3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8: Flush successful on 2 rows (5123 bytes) I0605 15:04:33.365437 138635 maintenance_manager.cc:430] Time spent running FlushMRSOp(3cca831acf744e1daee72582b8e16dc4): real 0.004s user 0.002s sys 0.001s I0605 15:04:33.365506 138635 maintenance_manager.cc:436] P 125dbd2ffb8a401bb7e4fd982995ccf8: FlushMRSOp(3cca831acf744e1daee72582b8e16dc4) metrics: {"cfile_init":1,"data dir 0.queue_time_us":9,"data dir 0.run_cpu_time_us":60,"data dir 0.run_wall_time_us":97,"fdatasync":29,"fdatasync_us":522,"lbm_read_time_us":5,"lbm_reads_lt_1ms":4,"lbm_write_time_us":290,"lbm_writes_lt_1ms":40,"tcmalloc_contention_cycles":198912,"thread_start_us":200,"threads_started":1,"wal-append.queue_time_us":210} I0605 15:04:33.370317 138638 tablet.cc:1512] T 3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8: Compaction: stage 1 complete, picked 3 rowsets to compact I0605 15:04:33.370349 138638 compaction.cc:881] Selected 3 rowsets to compact: I0605 15:04:33.370360 138638 compaction.cc:884] RowSet(59)(current size on disk: ~3640 bytes) I0605 15:04:33.370368 138638 compaction.cc:884] RowSet(60)(current size on disk: ~700 bytes) I0605 15:04:33.370371 138638 compaction.cc:884] RowSet(150)(current size on disk: ~1020 bytes) I0605 15:04:33.370379 138638 tablet.cc:1298] T 3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8: Compaction: entering phase 1 (flushing snapshot). Phase 1 snapshot: MvccSnapshot[committed={T|T < 6130248381986324480 or (T in {6130248381986324480})}] I0605 15:04:33.370710 138638 multi_column_writer.cc:89] Opened CFile writers for 6 column(s) I0605 15:04:33.372845 138638 tablet.cc:1383] T 3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8: Compaction: entering phase 2 (starting to duplicate updates in new rowsets) I0605 15:04:33.373126 138638 tablet.cc:1438] T 3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8: Compaction Phase 2: carrying over any updates which arrived during Phase 1 I0605 15:04:33.373152 138638 tablet.cc:1440] T 3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8: Phase 2 snapshot: MvccSnapshot[committed={T|T < 6130248381986324480 or (T in {6130248381986324480})}] I0605 15:04:33.375113 138638 tablet.cc:1482] T 3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8: Compaction successful on 73 rows (8585 bytes) I0605 15:04:33.375154 138638 maintenance_manager.cc:430] Time spent running CompactRowSetsOp(3cca831acf744e1daee72582b8e16dc4): real 0.005s user 0.004s sys 0.000s I0605 15:04:33.375205 138638 maintenance_manager.cc:436] P 125dbd2ffb8a401bb7e4fd982995ccf8: CompactRowSetsOp(3cca831acf744e1daee72582b8e16dc4) metrics: {"cfile_cache_hit":28,"cfile_cache_hit_bytes":3300,"cfile_cache_miss":66,"cfile_cache_miss_bytes":5524,"cfile_init":14,"data dir 0.queue_time_us":8,"data dir 0.run_cpu_time_us":47,"data dir 0.run_wall_time_us":101,"delta_iterators_relevant":2,"fdatasync":28,"fdatasync_us":259,"lbm_read_time_us":388,"lbm_reads_lt_1ms":122,"lbm_write_time_us":347,"lbm_writes_lt_1ms":40,"spinlock_wait_cycles":4864} E0605 15:04:33.376554 138642 tablet.cc:1219] T 3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8: Rowset selected for compaction but not available anymore: RowSet(150) E0605 15:04:33.376605 138642 tablet.cc:1219] T 3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8: Rowset selected for compaction but not available anymore: RowSet(59) E0605 15:04:33.376615 138642 tablet.cc:1219] T 3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8: Rowset selected for compaction but not available anymore: RowSet(60) F0605 15:04:33.377100 138642 tablet.cc:1222] T 3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8: Was unable to find all rowsets selected for compaction I0605 15:04:33.748011 138636 maintenance_manager.cc:430] Time spent running UndoDeltaBlockGCOp(3cca831acf744e1daee72582b8e16dc4): real 0.006s user 0.000s sys 0.003s I0605 15:04:33.748826 138636 maintenance_manager.cc:436] P 125dbd2ffb8a401bb7e4fd982995ccf8: UndoDeltaBlockGCOp(3cca831acf744e1daee72582b8e16dc4) metrics: {"cfile_init":1,"lbm_read_time_us":27,"lbm_reads_lt_1ms":4,"tcmalloc_contention_cycles":136192} I0605 15:04:35.059636 140969 consensus_queue.cc:709] T 55ef51757f1a45889e16f9ff3fe1139c P 125dbd2ffb8a401bb7e4fd982995ccf8 [LEADER]: Peer df7fc4cfb2d44e329e4cff76ccb64987 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1 I0605 15:04:35.059742 140969 consensus_queue.cc:733] T 55ef51757f1a45889e16f9ff3fe1139c P 125dbd2ffb8a401bb7e4fd982995ccf8 [LEADER]: Peer responded invalid term: Peer: df7fc4cfb2d44e329e4cff76ccb64987, Is new: false, Last received: 1.1, Next index: 2, Last known committed idx: 1, Last exchange result: ERROR, Needs tablet copy: false I0605 15:04:35.064334 74955 raft_consensus.cc:2354] T 55ef51757f1a45889e16f9ff3fe1139c P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 LEADER]: Stepping down as leader of term 1 I0605 15:04:35.064350 140925 raft_consensus.cc:412] T 87e5a254f3aa4070bf138a6d08ed611d P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dac8ac3bc0754eb6b40d0e1697d0050b) I0605 15:04:35.064354 141001 raft_consensus.cc:412] T 881134030ff347d992791e1d8db93632 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b846822ae026439faa359d75183c4b54) I0605 15:04:35.064427 141044 raft_consensus.cc:412] T 48cd3f9450ce4d2a8477b4fc17e73d1c P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b846822ae026439faa359d75183c4b54) I0605 15:04:35.064491 141851 raft_consensus.cc:412] T 26f4871c17624227b8157b1a01e5e2cf P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b846822ae026439faa359d75183c4b54) I0605 15:04:35.064463 140607 raft_consensus.cc:412] T 4203d61ed3aa4008955cf2cf023c2132 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader bcde875557b54a70b5724ed5ac988caa) I0605 15:04:35.064522 139580 raft_consensus.cc:412] T d766a7aac6294db28bc5202a064c9835 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader dac8ac3bc0754eb6b40d0e1697d0050b) I0605 15:04:35.064555 141851 raft_consensus.cc:2315] T 26f4871c17624227b8157b1a01e5e2cf P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.521s I0605 15:04:35.064543 140646 raft_consensus.cc:412] T 351835030fc54ff79a605b20ee7741e5 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b846822ae026439faa359d75183c4b54) I0605 15:04:35.064580 141671 raft_consensus.cc:412] T 63396a9239184c3799d53d93f0ce5d49 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader df7fc4cfb2d44e329e4cff76ccb64987) I0605 15:04:35.065110 74955 raft_consensus.cc:549] T 55ef51757f1a45889e16f9ff3fe1139c P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 125dbd2ffb8a401bb7e4fd982995ccf8, State: 1, Role: LEADER I0605 15:04:35.066002 138766 raft_consensus.cc:1861] T 19e5232aec6047209f91cd42786ad053 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate df7fc4cfb2d44e329e4cff76ccb64987 for term 4 because replica is either leader or believes a valid leader to be alive. I0605 15:04:35.070186 140637 raft_consensus.cc:412] T d0bb69948a5d4278b22943c7d79e8d19 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dac8ac3bc0754eb6b40d0e1697d0050b) I0605 15:04:35.075618 141001 raft_consensus.cc:2315] T 881134030ff347d992791e1d8db93632 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.506s I0605 15:04:35.077335 140925 raft_consensus.cc:2315] T 87e5a254f3aa4070bf138a6d08ed611d P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.775s I0605 15:04:35.078053 141671 raft_consensus.cc:2315] T 63396a9239184c3799d53d93f0ce5d49 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.676s I0605 15:04:35.078641 139580 raft_consensus.cc:2315] T d766a7aac6294db28bc5202a064c9835 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.982s I0605 15:04:35.080402 141044 raft_consensus.cc:2315] T 48cd3f9450ce4d2a8477b4fc17e73d1c P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.970s I0605 15:04:35.081475 140637 raft_consensus.cc:2315] T d0bb69948a5d4278b22943c7d79e8d19 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.557s I0605 15:04:35.084607 140607 raft_consensus.cc:2315] T 4203d61ed3aa4008955cf2cf023c2132 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.721s I0605 15:04:35.086302 141851 raft_consensus.cc:436] T 26f4871c17624227b8157b1a01e5e2cf P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c7963f2bfe04e9c9a687879fcf81e51" member_type: VOTER last_known_addr { host: "host25.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid: "125dbd2ffb8a401bb7e4fd982995ccf8" member_type: VOTER last_known_addr { host: "host29.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid: "b846822ae026439faa359d75183c4b54" member_type: VOTER last_known_addr { host: "host22.domain.nhnsystem.com" port: 7050 } } I0605 15:04:35.086418 140925 raft_consensus.cc:436] T 87e5a254f3aa4070bf138a6d08ed611d P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff831fe6da7f4388aa633dc5d3f811c6" member_type: VOTER last_known_addr { host: "host26.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid: "125dbd2ffb8a401bb7e4fd982995ccf8" member_type: VOTER last_known_addr { host: "host29.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid: "dac8ac3bc0754eb6b40d0e1697d0050b" member_type: VOTER last_known_addr { host: "host28.domain.nhnsystem.com" port: 7050 } } I0605 15:04:35.087604 140925 leader_election.cc:216] T 87e5a254f3aa4070bf138a6d08ed611d P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer ff831fe6da7f4388aa633dc5d3f811c6 I0605 15:04:35.087857 139580 raft_consensus.cc:436] T d766a7aac6294db28bc5202a064c9835 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125dbd2ffb8a401bb7e4fd982995ccf8" member_type: VOTER last_known_addr { host: "host29.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid: "dac8ac3bc0754eb6b40d0e1697d0050b" member_type: VOTER last_known_addr { host: "host28.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid: "bcde875557b54a70b5724ed5ac988caa" member_type: VOTER last_known_addr { host: "host23.domain.nhnsystem.com" port: 7050 } } I0605 15:04:35.087862 140646 raft_consensus.cc:2315] T 351835030fc54ff79a605b20ee7741e5 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.081s I0605 15:04:35.087882 141671 raft_consensus.cc:436] T 63396a9239184c3799d53d93f0ce5d49 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df7fc4cfb2d44e329e4cff76ccb64987" member_type: VOTER last_known_addr { host: "host27.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid: "125dbd2ffb8a401bb7e4fd982995ccf8" member_type: VOTER last_known_addr { host: "host29.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid: "dac8ac3bc0754eb6b40d0e1697d0050b" member_type: VOTER last_known_addr { host: "host28.domain.nhnsystem.com" port: 7050 } } I0605 15:04:35.088114 141044 raft_consensus.cc:436] T 48cd3f9450ce4d2a8477b4fc17e73d1c P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "125dbd2ffb8a401bb7e4fd982995ccf8" member_type: VOTER last_known_addr { host: "host29.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid: "0c7963f2bfe04e9c9a687879fcf81e51" member_type: VOTER last_known_addr { host: "host25.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid: "b846822ae026439faa359d75183c4b54" member_type: VOTER last_known_addr { host: "host22.domain.nhnsystem.com" port: 7050 } } I0605 15:04:35.088122 140637 raft_consensus.cc:436] T d0bb69948a5d4278b22943c7d79e8d19 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dac8ac3bc0754eb6b40d0e1697d0050b" member_type: VOTER last_known_addr { host: "host28.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid: "b846822ae026439faa359d75183c4b54" member_type: VOTER last_known_addr { host: "host22.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid: "125dbd2ffb8a401bb7e4fd982995ccf8" member_type: VOTER last_known_addr { host: "host29.domain.nhnsystem.com" port: 7050 } } I0605 15:04:35.088127 141001 raft_consensus.cc:436] T 881134030ff347d992791e1d8db93632 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b846822ae026439faa359d75183c4b54" member_type: VOTER last_known_addr { host: "host22.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid: "125dbd2ffb8a401bb7e4fd982995ccf8" member_type: VOTER last_known_addr { host: "host29.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid: "df7fc4cfb2d44e329e4cff76ccb64987" member_type: VOTER last_known_addr { host: "host27.domain.nhnsystem.com" port: 7050 } } I0605 15:04:35.088165 140607 raft_consensus.cc:436] T 4203d61ed3aa4008955cf2cf023c2132 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff831fe6da7f4388aa633dc5d3f811c6" member_type: VOTER last_known_addr { host: "host26.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid: "bcde875557b54a70b5724ed5ac988caa" member_type: VOTER last_known_addr { host: "host23.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid: "125dbd2ffb8a401bb7e4fd982995ccf8" member_type: VOTER last_known_addr { host: "host29.domain.nhnsystem.com" port: 7050 } } I0605 15:04:35.088374 140646 raft_consensus.cc:436] T 351835030fc54ff79a605b20ee7741e5 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b846822ae026439faa359d75183c4b54" member_type: VOTER last_known_addr { host: "host22.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid: "df7fc4cfb2d44e329e4cff76ccb64987" member_type: VOTER last_known_addr { host: "host27.domain.nhnsystem.com" port: 7050 } } peers { permanent_uuid: "125dbd2ffb8a401bb7e4fd982995ccf8" member_type: VOTER last_known_addr { host: "host29.domain.nhnsystem.com" port: 7050 } } I0605 15:04:35.090010 140637 leader_election.cc:216] T d0bb69948a5d4278b22943c7d79e8d19 P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer dac8ac3bc0754eb6b40d0e1697d0050b I0605 15:04:35.098152 141851 leader_election.cc:216] T 26f4871c17624227b8157b1a01e5e2cf P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 0c7963f2bfe04e9c9a687879fcf81e51 I0605 15:04:35.099189 141001 leader_election.cc:216] T 881134030ff347d992791e1d8db93632 P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer b846822ae026439faa359d75183c4b54 I0605 15:04:35.099319 141044 leader_election.cc:216] T 48cd3f9450ce4d2a8477b4fc17e73d1c P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer 0c7963f2bfe04e9c9a687879fcf81e51 I0605 15:04:35.106904 141001 leader_election.cc:216] T 881134030ff347d992791e1d8db93632 P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer df7fc4cfb2d44e329e4cff76ccb64987 I0605 15:04:35.107388 140637 leader_election.cc:216] T d0bb69948a5d4278b22943c7d79e8d19 P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer b846822ae026439faa359d75183c4b54 I0605 15:04:35.108178 141044 leader_election.cc:216] T 48cd3f9450ce4d2a8477b4fc17e73d1c P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer b846822ae026439faa359d75183c4b54 I0605 15:04:35.109321 140646 leader_election.cc:216] T 351835030fc54ff79a605b20ee7741e5 P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer b846822ae026439faa359d75183c4b54 I0605 15:04:35.110162 141851 leader_election.cc:216] T 26f4871c17624227b8157b1a01e5e2cf P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer b846822ae026439faa359d75183c4b54 I0605 15:04:35.111464 74955 consensus_queue.cc:184] T 55ef51757f1a45889e16f9ff3fe1139c P 125dbd2ffb8a401bb7e4fd982995ccf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 1, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 1, Mode: NON_LEADER I0605 15:04:35.111796 141292 consensus_peers.cc:187] T 55ef51757f1a45889e16f9ff3fe1139c P 125dbd2ffb8a401bb7e4fd982995ccf8 -> Peer df7fc4cfb2d44e329e4cff76ccb64987 (host27.domain.nhnsystem.com:7050): Could not obtain request from queue for peer: df7fc4cfb2d44e329e4cff76ccb64987. Status: Not found: Peer not tracked or queue not in leader mode. I0605 15:04:35.111865 139580 leader_election.cc:216] T d766a7aac6294db28bc5202a064c9835 P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer dac8ac3bc0754eb6b40d0e1697d0050b I0605 15:04:35.111913 140925 leader_election.cc:216] T 87e5a254f3aa4070bf138a6d08ed611d P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer dac8ac3bc0754eb6b40d0e1697d0050b I0605 15:04:35.111917 139580 leader_election.cc:216] T d766a7aac6294db28bc5202a064c9835 P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 4 pre-election: Requesting pre-vote from peer bcde875557b54a70b5724ed5ac988caa I0605 15:04:35.112076 141671 leader_election.cc:216] T 63396a9239184c3799d53d93f0ce5d49 P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer df7fc4cfb2d44e329e4cff76ccb64987 I0605 15:04:35.112323 140607 leader_election.cc:216] T 4203d61ed3aa4008955cf2cf023c2132 P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer ff831fe6da7f4388aa633dc5d3f811c6 I0605 15:04:35.112344 140646 leader_election.cc:216] T 351835030fc54ff79a605b20ee7741e5 P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer df7fc4cfb2d44e329e4cff76ccb64987 I0605 15:04:35.113402 141671 leader_election.cc:216] T 63396a9239184c3799d53d93f0ce5d49 P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer dac8ac3bc0754eb6b40d0e1697d0050b I0605 15:04:35.113420 140607 leader_election.cc:216] T 4203d61ed3aa4008955cf2cf023c2132 P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Requesting pre-vote from peer bcde875557b54a70b5724ed5ac988caa I0605 15:04:35.114395 138647 leader_election.cc:362] T 881134030ff347d992791e1d8db93632 P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Vote denied by peer df7fc4cfb2d44e329e4cff76ccb64987. Message: Invalid argument: T 881134030ff347d992791e1d8db93632 P df7fc4cfb2d44e329e4cff76ccb64987 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 125dbd2ffb8a401bb7e4fd982995ccf8 for term 2 because replica is either leader or believes a valid leader to be alive. I0605 15:04:35.115078 138647 leader_election.cc:362] T 63396a9239184c3799d53d93f0ce5d49 P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Vote denied by peer df7fc4cfb2d44e329e4cff76ccb64987. Message: Invalid argument: T 63396a9239184c3799d53d93f0ce5d49 P df7fc4cfb2d44e329e4cff76ccb64987 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 125dbd2ffb8a401bb7e4fd982995ccf8 for term 2 because replica is either leader or believes a valid leader to be alive. I0605 15:04:35.115154 138647 leader_election.cc:362] T 351835030fc54ff79a605b20ee7741e5 P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Vote denied by peer df7fc4cfb2d44e329e4cff76ccb64987. Message: Invalid argument: T 351835030fc54ff79a605b20ee7741e5 P df7fc4cfb2d44e329e4cff76ccb64987 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 125dbd2ffb8a401bb7e4fd982995ccf8 for term 2 because replica is either leader or believes a valid leader to be alive. I0605 15:04:35.115432 75021 consensus_queue.cc:733] T 55ef51757f1a45889e16f9ff3fe1139c P 125dbd2ffb8a401bb7e4fd982995ccf8 [NON_LEADER]: Peer responded invalid term: Peer: bcde875557b54a70b5724ed5ac988caa, Is new: false, Last received: 1.1, Next index: 2, Last known committed idx: 1, Last exchange result: ERROR, Needs tablet copy: false I0605 15:04:35.115860 75045 consensus_queue.cc:733] T fafc91cf33974e5baeb266492a9057b4 P 125dbd2ffb8a401bb7e4fd982995ccf8 [LEADER]: Peer responded invalid term: Peer: dac8ac3bc0754eb6b40d0e1697d0050b, Is new: false, Last received: 3.3, Next index: 4, Last known committed idx: 3, Last exchange result: ERROR, Needs tablet copy: false I0605 15:04:35.116771 138796 raft_consensus.cc:1861] T cdfb5dcd98734ab3bdaeac2937b1006b P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate bcde875557b54a70b5724ed5ac988caa for term 2 because replica is either leader or believes a valid leader to be alive. I0605 15:04:35.116952 138783 raft_consensus.cc:1861] T fafc91cf33974e5baeb266492a9057b4 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate bcde875557b54a70b5724ed5ac988caa for term 4 because replica is either leader or believes a valid leader to be alive. I0605 15:04:35.117341 138646 leader_election.cc:362] T 26f4871c17624227b8157b1a01e5e2cf P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 0c7963f2bfe04e9c9a687879fcf81e51. Message: Invalid argument: T 26f4871c17624227b8157b1a01e5e2cf P 0c7963f2bfe04e9c9a687879fcf81e51 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 125dbd2ffb8a401bb7e4fd982995ccf8 for term 2 because replica is either leader or believes a valid leader to be alive. I0605 15:04:35.117452 74955 consensus_peers.cc:389] T 55ef51757f1a45889e16f9ff3fe1139c P 125dbd2ffb8a401bb7e4fd982995ccf8 -> Peer bcde875557b54a70b5724ed5ac988caa (host23.domain.nhnsystem.com:7050): Closing peer: bcde875557b54a70b5724ed5ac988caa I0605 15:04:35.117959 138646 leader_election.cc:362] T 48cd3f9450ce4d2a8477b4fc17e73d1c P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 0c7963f2bfe04e9c9a687879fcf81e51. Message: Invalid argument: T 48cd3f9450ce4d2a8477b4fc17e73d1c P 0c7963f2bfe04e9c9a687879fcf81e51 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 125dbd2ffb8a401bb7e4fd982995ccf8 for term 2 because replica is either leader or believes a valid leader to be alive. I0605 15:04:35.118019 138646 leader_election.cc:362] T 881134030ff347d992791e1d8db93632 P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Vote denied by peer b846822ae026439faa359d75183c4b54. Message: Invalid argument: T 881134030ff347d992791e1d8db93632 P b846822ae026439faa359d75183c4b54 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 125dbd2ffb8a401bb7e4fd982995ccf8 for term 2 because replica is either leader or believes a valid leader to be alive. I0605 15:04:35.118037 138646 leader_election.cc:243] T 881134030ff347d992791e1d8db93632 P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0605 15:04:35.118077 74955 consensus_peers.cc:389] T 55ef51757f1a45889e16f9ff3fe1139c P 125dbd2ffb8a401bb7e4fd982995ccf8 -> Peer df7fc4cfb2d44e329e4cff76ccb64987 (host27.domain.nhnsystem.com:7050): Closing peer: df7fc4cfb2d44e329e4cff76ccb64987 I0605 15:04:35.118149 74955 raft_consensus.cc:2359] T 55ef51757f1a45889e16f9ff3fe1139c P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Advancing to term 2 I0605 15:04:35.118224 138778 raft_consensus.cc:1861] T 270218692311420c8b6596fa7647010e P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate dac8ac3bc0754eb6b40d0e1697d0050b for term 2 because replica is either leader or believes a valid leader to be alive. I0605 15:04:35.118348 75047 raft_consensus.cc:2315] T 881134030ff347d992791e1d8db93632 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.601s I0605 15:04:35.118381 75047 raft_consensus.cc:2061] T 881134030ff347d992791e1d8db93632 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0605 15:04:35.118518 138755 raft_consensus.cc:1861] T 96cd3b27b43c42d7b3f5cca0757486f9 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate dac8ac3bc0754eb6b40d0e1697d0050b for term 2 because replica is either leader or believes a valid leader to be alive. I0605 15:04:35.118790 138646 leader_election.cc:362] T 351835030fc54ff79a605b20ee7741e5 P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Vote denied by peer b846822ae026439faa359d75183c4b54. Message: Invalid argument: T 351835030fc54ff79a605b20ee7741e5 P b846822ae026439faa359d75183c4b54 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 125dbd2ffb8a401bb7e4fd982995ccf8 for term 2 because replica is either leader or believes a valid leader to be alive. I0605 15:04:35.118806 138646 leader_election.cc:243] T 351835030fc54ff79a605b20ee7741e5 P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0605 15:04:35.119104 138756 raft_consensus.cc:1861] T 5fa2b7db1bc54795856ef0fef58d2770 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate dac8ac3bc0754eb6b40d0e1697d0050b for term 4 because replica is either leader or believes a valid leader to be alive. I0605 15:04:35.119427 138646 leader_election.cc:362] T 48cd3f9450ce4d2a8477b4fc17e73d1c P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Vote denied by peer b846822ae026439faa359d75183c4b54. Message: Invalid argument: T 48cd3f9450ce4d2a8477b4fc17e73d1c P b846822ae026439faa359d75183c4b54 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 125dbd2ffb8a401bb7e4fd982995ccf8 for term 2 because replica is either leader or believes a valid leader to be alive. I0605 15:04:35.119443 138646 leader_election.cc:243] T 48cd3f9450ce4d2a8477b4fc17e73d1c P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0605 15:04:35.119434 138766 raft_consensus.cc:1875] T fafc91cf33974e5baeb266492a9057b4 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 3 LEADER]: Leader election vote request: Denying vote to candidate bcde875557b54a70b5724ed5ac988caa for term 4 because replica is already servicing an update from a current leader or another vote. I0605 15:04:35.119475 75048 raft_consensus.cc:2315] T 351835030fc54ff79a605b20ee7741e5 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.911s I0605 15:04:35.119493 75048 raft_consensus.cc:2061] T 351835030fc54ff79a605b20ee7741e5 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0605 15:04:35.119493 138648 leader_election.cc:362] T 4203d61ed3aa4008955cf2cf023c2132 P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Vote denied by peer ff831fe6da7f4388aa633dc5d3f811c6. Message: Invalid argument: T 4203d61ed3aa4008955cf2cf023c2132 P ff831fe6da7f4388aa633dc5d3f811c6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 125dbd2ffb8a401bb7e4fd982995ccf8 for term 2 because replica is either leader or believes a valid leader to be alive. I0605 15:04:35.119614 75046 raft_consensus.cc:2354] T fafc91cf33974e5baeb266492a9057b4 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 3 LEADER]: Stepping down as leader of term 3 I0605 15:04:35.119638 75046 raft_consensus.cc:549] T fafc91cf33974e5baeb266492a9057b4 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 125dbd2ffb8a401bb7e4fd982995ccf8, State: 1, Role: LEADER I0605 15:04:35.119695 75046 consensus_queue.cc:184] T fafc91cf33974e5baeb266492a9057b4 P 125dbd2ffb8a401bb7e4fd982995ccf8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 3, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 1, Mode: NON_LEADER I0605 15:04:35.119776 138648 leader_election.cc:362] T 87e5a254f3aa4070bf138a6d08ed611d P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Vote denied by peer ff831fe6da7f4388aa633dc5d3f811c6. Message: Invalid argument: T 87e5a254f3aa4070bf138a6d08ed611d P ff831fe6da7f4388aa633dc5d3f811c6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 125dbd2ffb8a401bb7e4fd982995ccf8 for term 2 because replica is either leader or believes a valid leader to be alive. I0605 15:04:35.119892 75049 raft_consensus.cc:2315] T 48cd3f9450ce4d2a8477b4fc17e73d1c P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.579s I0605 15:04:35.119910 75049 raft_consensus.cc:2061] T 48cd3f9450ce4d2a8477b4fc17e73d1c P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0605 15:04:35.119997 75046 consensus_peers.cc:389] T fafc91cf33974e5baeb266492a9057b4 P 125dbd2ffb8a401bb7e4fd982995ccf8 -> Peer bcde875557b54a70b5724ed5ac988caa (host23.domain.nhnsystem.com:7050): Closing peer: bcde875557b54a70b5724ed5ac988caa I0605 15:04:35.120003 138646 leader_election.cc:362] T d0bb69948a5d4278b22943c7d79e8d19 P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Vote denied by peer b846822ae026439faa359d75183c4b54. Message: Invalid argument: T d0bb69948a5d4278b22943c7d79e8d19 P b846822ae026439faa359d75183c4b54 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 125dbd2ffb8a401bb7e4fd982995ccf8 for term 2 because replica is either leader or believes a valid leader to be alive. I0605 15:04:35.120048 138646 leader_election.cc:362] T 26f4871c17624227b8157b1a01e5e2cf P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Vote denied by peer b846822ae026439faa359d75183c4b54. Message: Invalid argument: T 26f4871c17624227b8157b1a01e5e2cf P b846822ae026439faa359d75183c4b54 [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 125dbd2ffb8a401bb7e4fd982995ccf8 for term 2 because replica is either leader or believes a valid leader to be alive. I0605 15:04:35.120059 138646 leader_election.cc:243] T 26f4871c17624227b8157b1a01e5e2cf P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0605 15:04:35.120159 75046 consensus_peers.cc:389] T fafc91cf33974e5baeb266492a9057b4 P 125dbd2ffb8a401bb7e4fd982995ccf8 -> Peer dac8ac3bc0754eb6b40d0e1697d0050b (host28.domain.nhnsystem.com:7050): Closing peer: dac8ac3bc0754eb6b40d0e1697d0050b I0605 15:04:35.120218 75046 raft_consensus.cc:2359] T fafc91cf33974e5baeb266492a9057b4 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 3 FOLLOWER]: Advancing to term 4 I0605 15:04:35.120309 75050 raft_consensus.cc:2315] T 26f4871c17624227b8157b1a01e5e2cf P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.594s I0605 15:04:35.120324 75050 raft_consensus.cc:2061] T 26f4871c17624227b8157b1a01e5e2cf P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0605 15:04:35.120527 138648 leader_election.cc:362] T d0bb69948a5d4278b22943c7d79e8d19 P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Vote denied by peer dac8ac3bc0754eb6b40d0e1697d0050b. Message: Invalid argument: T d0bb69948a5d4278b22943c7d79e8d19 P dac8ac3bc0754eb6b40d0e1697d0050b [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 125dbd2ffb8a401bb7e4fd982995ccf8 for term 2 because replica is either leader or believes a valid leader to be alive. I0605 15:04:35.120545 138648 leader_election.cc:243] T d0bb69948a5d4278b22943c7d79e8d19 P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0605 15:04:35.120784 75051 raft_consensus.cc:2315] T d0bb69948a5d4278b22943c7d79e8d19 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.516s I0605 15:04:35.120793 75051 raft_consensus.cc:2061] T d0bb69948a5d4278b22943c7d79e8d19 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0605 15:04:35.120863 138648 leader_election.cc:362] T 4203d61ed3aa4008955cf2cf023c2132 P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Vote denied by peer bcde875557b54a70b5724ed5ac988caa. Message: Invalid argument: T 4203d61ed3aa4008955cf2cf023c2132 P bcde875557b54a70b5724ed5ac988caa [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 125dbd2ffb8a401bb7e4fd982995ccf8 for term 2 because replica is either leader or believes a valid leader to be alive. I0605 15:04:35.120875 138648 leader_election.cc:243] T 4203d61ed3aa4008955cf2cf023c2132 P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0605 15:04:35.120986 75052 raft_consensus.cc:2315] T 4203d61ed3aa4008955cf2cf023c2132 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.730s I0605 15:04:35.120995 75052 raft_consensus.cc:2061] T 4203d61ed3aa4008955cf2cf023c2132 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0605 15:04:35.121003 138648 leader_election.cc:362] T d766a7aac6294db28bc5202a064c9835 P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 4 pre-election: Vote denied by peer dac8ac3bc0754eb6b40d0e1697d0050b. Message: Invalid argument: T d766a7aac6294db28bc5202a064c9835 P dac8ac3bc0754eb6b40d0e1697d0050b [term 3 LEADER]: Leader pre-election vote request: Denying vote to candidate 125dbd2ffb8a401bb7e4fd982995ccf8 for term 4 because replica is either leader or believes a valid leader to be alive. I0605 15:04:35.121055 138648 leader_election.cc:362] T d766a7aac6294db28bc5202a064c9835 P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 4 pre-election: Vote denied by peer bcde875557b54a70b5724ed5ac988caa. Message: Invalid argument: T d766a7aac6294db28bc5202a064c9835 P bcde875557b54a70b5724ed5ac988caa [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 125dbd2ffb8a401bb7e4fd982995ccf8 for term 4 because replica is either leader or believes a valid leader to be alive. I0605 15:04:35.121064 138648 leader_election.cc:243] T d766a7aac6294db28bc5202a064c9835 P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. I0605 15:04:35.121155 138806 raft_consensus.cc:1875] T 55ef51757f1a45889e16f9ff3fe1139c P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate df7fc4cfb2d44e329e4cff76ccb64987 for term 2 because replica is already servicing an update from a current leader or another vote. I0605 15:04:35.121177 75053 raft_consensus.cc:2315] T d766a7aac6294db28bc5202a064c9835 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 3 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 2.129s I0605 15:04:35.121183 75053 raft_consensus.cc:2061] T d766a7aac6294db28bc5202a064c9835 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority I0605 15:04:35.121188 138648 leader_election.cc:362] T 63396a9239184c3799d53d93f0ce5d49 P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Vote denied by peer dac8ac3bc0754eb6b40d0e1697d0050b. Message: Invalid argument: T 63396a9239184c3799d53d93f0ce5d49 P dac8ac3bc0754eb6b40d0e1697d0050b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 125dbd2ffb8a401bb7e4fd982995ccf8 for term 2 because replica is either leader or believes a valid leader to be alive. I0605 15:04:35.121201 138648 leader_election.cc:243] T 63396a9239184c3799d53d93f0ce5d49 P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0605 15:04:35.121321 75054 raft_consensus.cc:2315] T 63396a9239184c3799d53d93f0ce5d49 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.513s I0605 15:04:35.121332 138648 leader_election.cc:362] T 87e5a254f3aa4070bf138a6d08ed611d P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Vote denied by peer dac8ac3bc0754eb6b40d0e1697d0050b. Message: Invalid argument: T 87e5a254f3aa4070bf138a6d08ed611d P dac8ac3bc0754eb6b40d0e1697d0050b [term 1 LEADER]: Leader pre-election vote request: Denying vote to candidate 125dbd2ffb8a401bb7e4fd982995ccf8 for term 2 because replica is either leader or believes a valid leader to be alive. I0605 15:04:35.121336 75054 raft_consensus.cc:2061] T 63396a9239184c3799d53d93f0ce5d49 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0605 15:04:35.121361 138648 leader_election.cc:243] T 87e5a254f3aa4070bf138a6d08ed611d P 125dbd2ffb8a401bb7e4fd982995ccf8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. I0605 15:04:35.121372 138797 raft_consensus.cc:2315] T 55ef51757f1a45889e16f9ff3fe1139c P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 2 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.662s I0605 15:04:35.121544 138797 raft_consensus.cc:1900] T 55ef51757f1a45889e16f9ff3fe1139c P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df7fc4cfb2d44e329e4cff76ccb64987 in term 2. I0605 15:04:35.121592 75055 raft_consensus.cc:2315] T 87e5a254f3aa4070bf138a6d08ed611d P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Snoozing failure detection for election timeout plus an additional 1.566s I0605 15:04:35.121601 75055 raft_consensus.cc:2061] T 87e5a254f3aa4070bf138a6d08ed611d P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority I0605 15:04:35.445930 138806 raft_consensus.cc:921] T 55ef51757f1a45889e16f9ff3fe1139c P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 2 FOLLOWER]: Refusing update from remote peer df7fc4cfb2d44e329e4cff76ccb64987: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch) I0605 15:04:35.549202 138806 raft_consensus.cc:921] T fafc91cf33974e5baeb266492a9057b4 P 125dbd2ffb8a401bb7e4fd982995ccf8 [term 4 FOLLOWER]: Refusing update from remote peer bcde875557b54a70b5724ed5ac988caa: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch) I0605 15:04:35.732771 138640 multi_column_writer.cc:89] Opened CFile writers for 8 column(s) <End of Log> 2017-06-06 11:38 GMT+09:00 Mike Percy <[email protected]>: > It seems unrelated. What release did you cherry-pick on top of? > > INFO log from around that timeframe would be useful. > > Mike > > On Mon, Jun 5, 2017 at 6:38 PM, Jason Heo <[email protected]> wrote: > >> Hello. >> >> I'm using this patch https://gerrit.cloudera.org/#/c/6925/ >> >> One of tservers died suddenly. Here is ERROR and FATAL log. >> >> E0605 15:04:33.376554 138642 tablet.cc:1219] T >> 3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8: >> Rowset selected for compaction but not available anymore: RowSet(150) >> >> E0605 15:04:33.376605 138642 tablet.cc:1219] T >> 3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8: >> Rowset selected for compaction but not available anymore: RowSet(59) >> >> E0605 15:04:33.376615 138642 tablet.cc:1219] T >> 3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8: >> Rowset selected for compaction but not available anymore: RowSet(60) >> >> F0605 15:04:33.377100 138642 tablet.cc:1222] T >> 3cca831acf744e1daee72582b8e16dc4 P 125dbd2ffb8a401bb7e4fd982995ccf8: Was >> unable to find all rowsets selected for compaction >> >> >> <End of Log> >> >> >> Could I know what's the problem? Feel free to ask any information to >> resolve it. >> >> >> Thank, >> >> >> Jason >> > >
