Ivan Andika created HDDS-16028:
----------------------------------
Summary: SCM follower suggests itself as leader
Key: HDDS-16028
URL: https://issues.apache.org/jira/browse/HDDS-16028
Project: Apache Ozone
Issue Type: Bug
Reporter: Ivan Andika
Assignee: ChenXi
We have another major incident part because of SCM follower kept suggesting
itself as a leader
{code:java}
scm15:9894:FOLLOWER:2059f536-5846-4573-b81d-274dc495c727:<redacted>
scm16:9894:LEADER:d0b9f1ed-7b61-4ac1-89d6-db3bc6fcb9df:<redacted>
scm17:9894:FOLLOWER:23ef2ecf-888c-4a83-b318-35fc5714ea8b:<redacted>{code}
{code:java}
com.google.protobuf.ServiceException:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdds.ratis.ServerNotLeaderException):
Server:23ef2ecf-888c-4a83-b318-35fc5714ea8b is not the leader. Suggested
leader is Server:scm17:9863.
at
org.apache.hadoop.hdds.ratis.ServerNotLeaderException.convertToNotLeaderException(ServerNotLeaderException.java:107)
at
org.apache.hadoop.hdds.scm.ha.RatisUtil.checkRatisException(RatisUtil.java:250)
at
org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.send(ScmBlockLocationProtocolServerSideTranslatorPB.java:117)
at
org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos$ScmBlockLocationProtocolService$2.callBlockingMethod(ScmBlockLocationProtocolProtos.java:16819)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server.processCall(ProtobufRpcEngine.java:495)
at
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:615)
at
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:593)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1153)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1336)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1229)
at
java.base/java.security.AccessController.doPrivileged(AccessController.java:714)
at java.base/javax.security.auth.Subject.doAs(Subject.java:525)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:2031)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:3622)
, while invoking $Proxy34.send over nodeId=scm17,nodeAddress=scm17:9863. Trying
to failover after sleeping for 2000ms. {code}
Ratis RaftServerImpl#generateNotLeaderException logic should not have allowed
it, see below
{code:java}
NotLeaderException generateNotLeaderException() {
if (!lifeCycle.getCurrentState().isRunning()) {
return new NotLeaderException(getMemberId(), null, null);
}
RaftPeerId leaderId = state.getLeaderId();
if (leaderId == null || leaderId.equals(getId())) {
// No idea about who is the current leader. Or the peer is the current
// leader, but it is about to step down. set the suggested leader as null.
leaderId = null;
}
final RaftConfigurationImpl conf = getRaftConf();
Collection<RaftPeer> peers = conf.getAllPeers();
return new NotLeaderException(getMemberId(), conf.getPeer(leaderId), peers);
} {code}
But because Ozone (annoyingly) reinvented a (buggy) wheel of the
NotLeaderException, a non-leader SCM NotLeaderException with itself as the
suggested leader
{code:java}
public NotLeaderException triggerNotLeaderException() {
ByteString leaderId =
division.getInfo().getRoleInfoProto().getFollowerInfo().getLeaderInfo()
.getId().getId();
RaftPeer suggestedLeader = leaderId.isEmpty() ?
null :
division.getRaftConf().getPeer(RaftPeerId.valueOf(leaderId));
return new NotLeaderException(division.getMemberId(),
suggestedLeader,
division.getGroup().getPeers());
} {code}
We should use the correct Ratis generatedNotLeaderException.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
