Hi Lee, It seems this email never got a response, sorry about that. Did you figure out a solution that you can share with the group?
Thanks -Todd On Wed, Sep 6, 2017 at 6:11 AM, Lee King <[email protected]> wrote: > I use kudu command line tool cluster ksck, the error detail: > Fetched info from all 10 Tablet Servers > Tablet 2758e5c68e974b92a3060db8575f3621 of table 'hwx_log' is > under-replicated: 1 replica(s) not RUNNING > 067e1e7245154f0fb2720dec6c77feec (cloud-ocean-kudu-04:7050): bad state > State: FAILED > Data state: TABLET_DATA_READY > Last status: Service unavailable: Transaction failed, tablet > 2758e5c68e974b92a3060db8575f3621 transaction memory consumption > (67031036) has exceeded its limit (67108864) or the limit of an ancestral > tracker > ad1ea284caff4b07a705c9156b0811cd (cloud-ocean-kudu-01:7050): RUNNING > c4ed5cb73f5644a8804d3abc976d02f8 (cloud-ocean-kudu-02:7050): RUNNING > [LEADER] > > 3 replicas' active configs differ from the master's. > All the peers reported by the master and tablet servers are: > A = 067e1e7245154f0fb2720dec6c77feec > B = ad1ea284caff4b07a705c9156b0811cd > C = c4ed5cb73f5644a8804d3abc976d02f8 > > The consensus matrix is: > Config source | Voters | Current term | Config index | Committed? > ---------------+--------------+--------------+--------------+------------ > master | A B C* | | | Yes > A | A B C | 10 | 13049 | Yes > B | A B C | 10 | 13049 | Yes > C | A C | 10 | 13692 | No > Tablet 5f996309d53445769c7db89abaa05845 of table 'hwx_log' is > under-replicated: 1 replica(s) not RUNNING > 067e1e7245154f0fb2720dec6c77feec (cloud-ocean-kudu-04:7050): RUNNING > [LEADER] > 43d450fd58794f008caca58d44a35ff8 (cloud-ocean-kudu-03:7050): bad state > State: FAILED > Data state: TABLET_DATA_READY > Last status: Service unavailable: Transaction failed, tablet > 5f996309d53445769c7db89abaa05845 transaction memory consumption > (66844571) has exceeded its limit (67108864) or the limit of an ancestral > tracker > c4ed5cb73f5644a8804d3abc976d02f8 (cloud-ocean-kudu-02:7050): RUNNING > > 1 replicas' active configs differ from the master's. > All the peers reported by the master and tablet servers are: > A = 067e1e7245154f0fb2720dec6c77feec > B = 43d450fd58794f008caca58d44a35ff8 > C = c4ed5cb73f5644a8804d3abc976d02f8 > > The consensus matrix is: > Config source | Voters | Current term | Config index | Committed? > ---------------+--------------+--------------+--------------+------------ > master | A* B C | | | Yes > A | A* B C | 12 | 7432 | Yes > B | A B C | 11 | 7432 | Yes > C | A* B C | 12 | 7432 | Yes > Table hwx_log has 2 under-replicated tablet(s) > > the tablet cann't be add_replica or remove_replica, and I try to delete > the data of current partition，error also occur： > WARNINGS: > Kudu error(s) reported, first error: Timed out: Failed to write batch of > 76249 ops to tablet 2758e5c68e974b92a3060db8575f3621 after 545 > attempt(s): Failed to write to server: (no server available): Write(tablet: > 2758e5c68e974b92a3060db8575f3621, num_ops: 76249, num_attempts: 545) > passed its deadline: Illegal state: Replica c4ed5cb73f5644a8804d3abc976d02f8 > is not leader of this config. Role: FOLLOWER. Consensus state: > current_term: 10 leader_uuid: "" committed_config { opid_index: 13049 > OBSOLETE_local: false peers { permanent_uuid: " > ad1ea284caff4b07a705c9156b0811cd" member_type: VOTER last_known_addr { > host: "cloud-ocean-kudu-01" port: 7050 } } peers { permanent_uuid: " > c4ed5cb73f5644a8804d3abc976d02f8" member_type: VOTER last_known_addr { > host: "cloud-ocean-kudu-02" port: 7050 } } peers { permanent_uuid: " > 067e1e7245154f0fb2720dec6c77feec" member_type: VOTER last_known_addr { > host: "cloud-ocean-kudu-04" port: 7050 } } } pending_config { opid_index: > 13692 OBSOLETE_local: false peers { permanent_uuid: " > c4ed5cb73f5644a8804d3abc976d02f8" member_type: VOTER last_known_addr { > host: "cloud-ocean-kudu-02" port: 7050 } } peers { permanent_uuid: " > 067e1e7245154f0fb2720dec6c77feec" member_type: VOTER last_known_addr { > host: "cloud-ocean-kudu-04" port: 7050 } } } > > > > Kudu error(s) reported, first error: Timed out: Failed to write batch of > 76249 ops to tablet 2758e5c68e974b92a3060db8575f3621 after 545 > attempt(s): Failed to write to server: (no server available): Write(tablet: > 2758e5c68e974b92a3060db8575f3621, num_ops: 76249, num_attempts: 545) > passed its deadline: Illegal state: Replica c4ed5cb73f5644a8804d3abc976d02f8 > is not leader of this config. Role: FOLLOWER. Consensus state: > current_term: 10 leader_uuid: "" committed_config { opid_index: 13049 > OBSOLETE_local: false peers { permanent_uuid: " > ad1ea284caff4b07a705c9156b0811cd" member_type: VOTER last_known_addr { > host: "cloud-ocean-kudu-01" port: 7050 } } peers { permanent_uuid: " > c4ed5cb73f5644a8804d3abc976d02f8" member_type: VOTER last_known_addr { > host: "cloud-ocean-kudu-02" port: 7050 } } peers { permanent_uuid: " > 067e1e7245154f0fb2720dec6c77feec" member_type: VOTER last_known_addr { > host: "cloud-ocean-kudu-04" port: 7050 } } } pending_config { opid_index: > 13692 OBSOLETE_local: false peers { permanent_uuid: " > c4ed5cb73f5644a8804d3abc976d02f8" member_type: VOTER last_known_addr { > host: "cloud-ocean-kudu-02" port: 7050 } } peers { permanent_uuid: " > 067e1e7245154f0fb2720dec6c77feec" member_type: VOTER last_known_addr { > host: "cloud-ocean-kudu-04" port: 7050 } } } > Error in Kudu table 'hwx_log': Timed out: Failed to write batch of 76249 > ops to tablet 2758e5c68e974b92a3060db8575f3621 after 545 attempt(s): > Failed to write to server: (no server available): Write(tablet: > 2758e5c68e974b92a3060db8575f3621, num_ops: 76249, num_attempts: 545) > passed its deadline: Illegal state: Replica c4ed5cb73f5644a8804d3abc976d02f8 > is not leader of this config. Role: FOLLOWER. Consensus state: > current_term: 10 leader_uuid: "" committed_config { opid_index: 13049 > OBSOLETE_local: false peers { permanent_uuid: " > ad1ea284caff4b07a705c9156b0811cd" member_type: VOTER last_known_addr { > host: "cloud-ocean-kudu-01" port: 7050 } } peers { permanent_uuid: " > c4ed5cb73f5644a8804d3abc976d02f8" member_type: VOTER last_known_addr { > host: "cloud-ocean-kudu-02" port: 7050 } } peers { permanent_uuid: " > 067e1e7245154f0fb2720dec6c77feec" member_type: VOTER last_known_addr { > host: "cloud-ocean-kudu-04" port: 7050 } } } pending_config { opid_index: > 13692 OBSOLETE_local: false peers { permanent_uuid: " > c4ed5cb73f5644a8804d3abc976d02f8" member_type: VOTER last_known_addr { > host: "cloud-ocean-kudu-02" port: 7050 } } peers { permanent_uuid: " > 067e1e7245154f0fb2720dec6c77feec" member_type: VOTER last_known_addr { > host: "cloud-ocean-kudu-04" port: 7050 } } } (1 of 76249 similar) > > 2017-09-06 14:04 GMT+08:00 Lee King <[email protected]>: > >> We got an error about :Service unavailable: Transaction failed, tablet >> 2758e5c68e974b92a3060db8575f3621 transaction memory consumption >> (67031036) has exceeded its limit (67108864) or the limit of an ancestral >> tracker.It looks like https://issues.apache.org/jira/browse/KUDU-1912. >> and the bug will be fix at 1.5,but out version is 1.4,Is there any affect >> for kudu stablity or data consistency? >> > > -- Todd Lipcon Software Engineer, Cloudera
