Hi: I used CDH5.14.0,kudu version is 1.6.0 (3 kudu master,7 kudu ts), Kudu ts and yarn mix and deploy on the same machine. 8 SAS on each TS machine (1T capacity)
When I query the kudu table with impala, the kudu tsserver appears unavailable. The tsserver log: Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg E0919 16:40:20.339299 139726 consensus_queue.cc:618] T 92d642ac75fe4fa0bdb85fd879a1e725 P 162f275784fa4fbfa49ad8a2639f87c4 [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 400 is ahead of the local log (next sequential op: 400). Destination peer: Peer: 8907a006b28a4d52afcc66ff48e11faf, Status: INVALID_TERM, Last received: 455.400, Next index: 401, Last known committed idx: 400, Time since last communication: 0.084s E0919 16:40:20.630578 140247 consensus_queue.cc:618] T 7f65df8fb13e467493a257aae698f219 P 162f275784fa4fbfa49ad8a2639f87c4 [NON_LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 7925 is ahead of the local log (next sequential op: 7925). Destination peer: Peer: 739b33f810844ebcaef489e0b83c3eba, Status: INVALID_TERM, Last received: 1.7924, Next index: 7926, Last known committed idx: 7925, Time since last communication: 0.001s E0925 10:45:18.919159 33765 consensus_queue.cc:618] T a3854738cf284426bea0943184c0a4d5 P 162f275784fa4fbfa49ad8a2639f87c4 [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 380 is ahead of the local log (next sequential op: 380). Destination peer: Peer: 739b33f810844ebcaef489e0b83c3eba, Status: INVALID_TERM, Last received: 443.380, Next index: 381, Last known committed idx: 380, Time since last communication: 0.042s E0925 10:45:18.908238 33741 consensus_queue.cc:618] T be346255d6344e6ba12f18b1084dd2f4 P 162f275784fa4fbfa49ad8a2639f87c4 [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 74419 is ahead of the local log (next sequential op: 74419). Destination peer: Peer: 739b33f810844ebcaef489e0b83c3eba, Status: INVALID_TERM, Last received: 346.74419, Next index: 74420, Last known committed idx: 74419, Time since last communication: 0.019s E0925 10:45:18.918596 33759 consensus_queue.cc:618] T 98faf1901e2848a985afb974167e0582 P 162f275784fa4fbfa49ad8a2639f87c4 [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 324 is ahead of the local log (next sequential op: 324). Destination peer: Peer: 739b33f810844ebcaef489e0b83c3eba, Status: INVALID_TERM, Last received: 375.324, Next index: 325, Last known committed idx: 324, Time since last communication: 0.043s E0929 16:41:43.993786 191493 consensus_queue.cc:618] T 142db0b84a2d4920a2ce5248319da3b1 P 162f275784fa4fbfa49ad8a2639f87c4 [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 49664 is ahead of the local log (next sequential op: 49664). Destination peer: Peer: 739b33f810844ebcaef489e0b83c3eba, Status: INVALID_TERM, Last received: 1120.49663, Next index: 49665, Last known committed idx: 49664, Time since last communication: 0.536s E0929 16:43:05.334996 196311 consensus_queue.cc:618] T 142db0b84a2d4920a2ce5248319da3b1 P 162f275784fa4fbfa49ad8a2639f87c4 [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 49664 is ahead of the local log (next sequential op: 49664). Destination peer: Peer: 739b33f810844ebcaef489e0b83c3eba, Status: INVALID_TERM, Last received: 1120.49663, Next index: 49665, Last known committed idx: 49664, Time since last communication: 81.877s E0929 16:43:05.337278 196314 consensus_queue.cc:618] T 142db0b84a2d4920a2ce5248319da3b1 P 162f275784fa4fbfa49ad8a2639f87c4 [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 49664 is ahead of the local log (next sequential op: 49664). Destination peer: Peer: 739b33f810844ebcaef489e0b83c3eba, Status: INVALID_TERM, Last received: 1120.49663, Next index: 49665, Last known committed idx: 49664, Time since last communication: 81.879s E0929 16:47:05.313355 7456 consensus_queue.cc:618] T 83105158520f487d9122666b06ff8d34 P 162f275784fa4fbfa49ad8a2639f87c4 [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 295 is ahead of the local log (next sequential op: 295). Destination peer: Peer: 8907a006b28a4d52afcc66ff48e11faf, Status: INVALID_TERM, Last received: 371.294, Next index: 296, Last known committed idx: 295, Time since last communication: 0.606s E0929 16:47:05.822614 12011 consensus_queue.cc:618] T 83105158520f487d9122666b06ff8d34 P 162f275784fa4fbfa49ad8a2639f87c4 [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 295 is ahead of the local log (next sequential op: 295). Destination peer: Peer: 8907a006b28a4d52afcc66ff48e11faf, Status: INVALID_TERM, Last received: 371.294, Next index: 296, Last known committed idx: 295, Time since last communication: 1.115s E0929 16:47:06.278580 12300 consensus_queue.cc:618] T 83105158520f487d9122666b06ff8d34 P 162f275784fa4fbfa49ad8a2639f87c4 [LEADER]: Error trying to read ahead of the log while preparing peer request: Incomplete: Op with index 295 is ahead of the local log (next sequential op: 295). Destination peer: Peer: 8907a006b28a4d52afcc66ff48e11faf, Status: INVALID_TERM, Last received: 371.294, Next index: 296, Last known committed idx: 295, Time since last communication: 1.571s At this point, I use the ksck tool to output as follows: Errors: ================== error fetching info from tablet servers: Network error: Not all Tablet Servers are reachable table consistency check error: Corruption: 193 out of 197 table(s) are bad FAILED Runtime error: ksck discovered errors But in less than five minutes, kudu automatically returns to normal. When kudu has TS anomalies, I use spark streaming to read Kafka data and write to kudu. When kudu returns to normal, there is a certain probability that the written data will be lost. Is there any good advice to help me, or to help me by modifying the parameters of kudu, or to be careful when writing code that the data spark streaming writes is not lost 优速物流有限公司 大数据中心 冯宝利 Mobil：15050552430 Email：[email protected]
