[
https://issues.apache.org/jira/browse/KUDU-1568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15427892#comment-15427892
]
zhangsong edited comment on KUDU-1568 at 8/19/16 9:32 AM:
----------------------------------------------------------
in kudu-tserver.INFO of the leader node , i found such logs
"T ed0d8b3a835e4c27afe695252ad0b8f5 P 3a77ef5039f447d29db5a44c92279a7a -> Peer
515ab1adcbd64081b646a86133f5f60d (follower_ip:7052): Could not obtain request
from queue for peer: 515ab1adcbd64081b646a86133f5f60d. Status: Not found:
Failed to read ops 1..390376: Segment 1 which contained index 1 has been GCed
I0819 16:54:29.290879 11359 raft_consensus.cc:630] T
ed0d8b3a835e4c27afe695252ad0b8f5 P 3a77ef5039f447d29db5a44c92279a7a [term 2
LEADER]: Processing failure of peer 515ab1adcbd64081b646a86133f5f60d in term 2
(The logs necessary to catch up peer 515ab1adcbd64081b646a86133f5f60d have been
garbage collected. The follower will never be able to catch up (Not found:
Failed to read ops 1..390376: Segment 1 which contained index 1 has been
GCed)): There is already a config change operation in progress. Unable to evict
follower until it completes. Doing nothing."
Here 515ab1adcbd64081b646a86133f5f60d is the follower , i have deleted tablet
ed0d8b3a835e4c27afe695252ad0b8f5 from it ( using kudu-ts-cli delete_tablet ) .
was (Author: brucesz):
in kudu-tserver.INFO of the leader node , i found such logs
"T ed0d8b3a835e4c27afe695252ad0b8f5 P 3a77ef5039f447d29db5a44c92279a7a -> Peer
515ab1adcbd64081b646a86133f5f60d (follower_ip:7052): Could not obtain request
from queue for peer: 515ab1adcbd64081b646a86133f5f60d. Status: Not found:
Failed to read ops 1..390376: Segment 1 which contained index 1 has been GCed
I0819 16:54:29.290879 11359 raft_consensus.cc:630] T
ed0d8b3a835e4c27afe695252ad0b8f5 P 3a77ef5039f447d29db5a44c92279a7a [term 2
LEADER]: Processing failure of peer 515ab1adcbd64081b646a86133f5f60d in term 2
(The logs necessary to catch up peer 515ab1adcbd64081b646a86133f5f60d have been
garbage collected. The follower will never be able to catch up (Not found:
Failed to read ops 1..390376: Segment 1 which contained index 1 has been
GCed)): There is already a config change operation in progress. Unable to evict
follower until it completes. Doing nothing."
> kudu-tserver cannot change config manually in some case.
> --------------------------------------------------------
>
> Key: KUDU-1568
> URL: https://issues.apache.org/jira/browse/KUDU-1568
> Project: Kudu
> Issue Type: Bug
> Reporter: zhangsong
>
> Today, my cluster experience 4 node crash.
> After restart kudu-server on this, i found a table is unable to accept write.
> After some search , i found on both of followers node there are messages :
> " T ed0d8b3a835e4c27afe695252ad0b8f5 P 3a77ef5039f447d29db5a44c92279a7a ->
> Peer 515ab1adcbd64081b646a86133f5f60d (follower_ip:7052): Couldn't send
> request to peer 515ab1adcbd64081b646a86133f5f60d for tablet
> ed0d8b3a835e4c27afe695252ad0b8f5. Error code: TABLET_NOT_RUNNING (12).
> Status: Illegal state: Tablet not RUNNING: FAILED: Corruption: Could not open
> LogReader. Reason: Unable to initialize log reader: Segment sequence numbers
> are not consecutive. Previous segment: seqno 0, path
> /export/servers/kudu/tserver_wal_data_7052/wals/ed0d8b3a835e4c27afe695252ad0b8f5.recovery/wal-000000018;
> Current segment: seqno 17, path
> /export/servers/kudu/tserver_wal_data_7052/wals/ed0d8b3a835e4c27afe695252ad0b8f5.recovery/wal-000000017.
> Retrying in the next heartbeat period. Already tried 26398 times."
> I want to use kudu-admin to add change the raft-config manually, but it
> didn't succeed :
> "Pending config: local: false peers { permanent_uuid:
> "515ab1adcbd64081b646a86133f5f60d" member_type: VOTER last_known_addr { host:
> "follower_ip" port: 7052 } } peers { permanent_uuid:
> "3a77ef5039f447d29db5a44c92279a7a" member_type: VOTER last_known_addr { host:
> "leader_ip" port: 7052 } }"
> So i delete the tablet on this two followers but on master ui (using
> kudu-ts-cli), the raft config does not change ... and i could not add new
> voter to raft_config using kudu-admin .
> now i am facing a dangerous situation where only one leader is service the
> tablet... (i create a new table and load the old table into the new one as
> work around)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
