I think those messages are harmless and indicative of an underlying issue of leader elections, perhaps caused by load. The full ksck output would be helpful to understand more.
This shouldn't cause writes to be lost. More likely the data was not written at all- are you checking for the success of the write operations in your code? -Will On Wed, Oct 10, 2018 at 8:12 PM fengba...@uce.cn <fengba...@uce.cn> wrote: > Hi: > > I used CDH5.14.0,kudu version is 1.6.0 (3 kudu master,7 kudu ts), Kudu > ts and yarn mix and deploy on the same machine. > 8 SAS on each TS machine (1T capacity) > > > When I query the kudu table with impala, the kudu tsserver appears > unavailable. The tsserver log: > Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg > > E0919 16:40:20.339299 139726 consensus_queue.cc:618] T > 92d642ac75fe4fa0bdb85fd879a1e725 P 162f275784fa4fbfa49ad8a2639f87c4 [LEADER]: > Error trying to read ahead of the log while preparing peer request: > Incomplete: Op with index 400 is ahead of the local log (next sequential op: > 400). Destination peer: Peer: 8907a006b28a4d52afcc66ff48e11faf, Status: > INVALID_TERM, Last received: 455.400, Next index: 401, Last known committed > idx: 400, Time since last communication: 0.084s > > E0919 16:40:20.630578 140247 consensus_queue.cc:618] T > 7f65df8fb13e467493a257aae698f219 P 162f275784fa4fbfa49ad8a2639f87c4 > [NON_LEADER]: Error trying to read ahead of the log while preparing peer > request: Incomplete: Op with index 7925 is ahead of the local log (next > sequential op: 7925). Destination peer: Peer: > 739b33f810844ebcaef489e0b83c3eba, Status: INVALID_TERM, Last received: > 1.7924, Next index: 7926, Last known committed idx: 7925, Time since last > communication: 0.001s > > E0925 10:45:18.919159 33765 consensus_queue.cc:618] T > a3854738cf284426bea0943184c0a4d5 P 162f275784fa4fbfa49ad8a2639f87c4 [LEADER]: > Error trying to read ahead of the log while preparing peer request: > Incomplete: Op with index 380 is ahead of the local log (next sequential op: > 380). Destination peer: Peer: 739b33f810844ebcaef489e0b83c3eba, Status: > INVALID_TERM, Last received: 443.380, Next index: 381, Last known committed > idx: 380, Time since last communication: 0.042s > > E0925 10:45:18.908238 33741 consensus_queue.cc:618] T > be346255d6344e6ba12f18b1084dd2f4 P 162f275784fa4fbfa49ad8a2639f87c4 [LEADER]: > Error trying to read ahead of the log while preparing peer request: > Incomplete: Op with index 74419 is ahead of the local log (next sequential > op: 74419). Destination peer: Peer: 739b33f810844ebcaef489e0b83c3eba, Status: > INVALID_TERM, Last received: 346.74419, Next index: 74420, Last known > committed idx: 74419, Time since last communication: 0.019s > > E0925 10:45:18.918596 33759 consensus_queue.cc:618] T > 98faf1901e2848a985afb974167e0582 P 162f275784fa4fbfa49ad8a2639f87c4 [LEADER]: > Error trying to read ahead of the log while preparing peer request: > Incomplete: Op with index 324 is ahead of the local log (next sequential op: > 324). Destination peer: Peer: 739b33f810844ebcaef489e0b83c3eba, Status: > INVALID_TERM, Last received: 375.324, Next index: 325, Last known committed > idx: 324, Time since last communication: 0.043s > > E0929 16:41:43.993786 191493 consensus_queue.cc:618] T > 142db0b84a2d4920a2ce5248319da3b1 P 162f275784fa4fbfa49ad8a2639f87c4 [LEADER]: > Error trying to read ahead of the log while preparing peer request: > Incomplete: Op with index 49664 is ahead of the local log (next sequential > op: 49664). Destination peer: Peer: 739b33f810844ebcaef489e0b83c3eba, Status: > INVALID_TERM, Last received: 1120.49663, Next index: 49665, Last known > committed idx: 49664, Time since last communication: 0.536s > > E0929 16:43:05.334996 196311 consensus_queue.cc:618] T > 142db0b84a2d4920a2ce5248319da3b1 P 162f275784fa4fbfa49ad8a2639f87c4 [LEADER]: > Error trying to read ahead of the log while preparing peer request: > Incomplete: Op with index 49664 is ahead of the local log (next sequential > op: 49664). Destination peer: Peer: 739b33f810844ebcaef489e0b83c3eba, Status: > INVALID_TERM, Last received: 1120.49663, Next index: 49665, Last known > committed idx: 49664, Time since last communication: 81.877s > > E0929 16:43:05.337278 196314 consensus_queue.cc:618] T > 142db0b84a2d4920a2ce5248319da3b1 P 162f275784fa4fbfa49ad8a2639f87c4 [LEADER]: > Error trying to read ahead of the log while preparing peer request: > Incomplete: Op with index 49664 is ahead of the local log (next sequential > op: 49664). Destination peer: Peer: 739b33f810844ebcaef489e0b83c3eba, Status: > INVALID_TERM, Last received: 1120.49663, Next index: 49665, Last known > committed idx: 49664, Time since last communication: 81.879s > > E0929 16:47:05.313355 7456 consensus_queue.cc:618] T > 83105158520f487d9122666b06ff8d34 P 162f275784fa4fbfa49ad8a2639f87c4 [LEADER]: > Error trying to read ahead of the log while preparing peer request: > Incomplete: Op with index 295 is ahead of the local log (next sequential op: > 295). Destination peer: Peer: 8907a006b28a4d52afcc66ff48e11faf, Status: > INVALID_TERM, Last received: 371.294, Next index: 296, Last known committed > idx: 295, Time since last communication: 0.606s > > E0929 16:47:05.822614 12011 consensus_queue.cc:618] T > 83105158520f487d9122666b06ff8d34 P 162f275784fa4fbfa49ad8a2639f87c4 [LEADER]: > Error trying to read ahead of the log while preparing peer request: > Incomplete: Op with index 295 is ahead of the local log (next sequential op: > 295). Destination peer: Peer: 8907a006b28a4d52afcc66ff48e11faf, Status: > INVALID_TERM, Last received: 371.294, Next index: 296, Last known committed > idx: 295, Time since last communication: 1.115s > > E0929 16:47:06.278580 12300 consensus_queue.cc:618] T > 83105158520f487d9122666b06ff8d34 P 162f275784fa4fbfa49ad8a2639f87c4 [LEADER]: > Error trying to read ahead of the log while preparing peer request: > Incomplete: Op with index 295 is ahead of the local log (next sequential op: > 295). Destination peer: Peer: 8907a006b28a4d52afcc66ff48e11faf, Status: > INVALID_TERM, Last received: 371.294, Next index: 296, Last known committed > idx: 295, Time since last communication: 1.571s > > At this point, I use the ksck tool to output as follows: > Errors: > ================== > > error fetching info from tablet servers: Network error: Not all Tablet > Servers are reachable > table consistency check error: Corruption: 193 out of 197 table(s) are bad > > FAILED > Runtime error: ksck discovered errors > > > > But in less than five minutes, kudu automatically returns to normal. When > kudu has TS anomalies, I use spark streaming to read Kafka data and write to > kudu. When kudu returns to normal, there is a certain probability that the > written data will be lost. > > > > Is there any good advice to help me, or to help me by modifying the > parameters of kudu, or to be careful when writing code that the data spark > streaming writes is not lost > > > ------------------------------ > [image: 说明: logo1] > 优速物流有限公司 > 大数据中心 冯宝利 > *Mobil*：15050552430 > *Email*：fengba...@uce.cn >
