[
https://issues.apache.org/jira/browse/HDDS-16028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Andika updated HDDS-16028:
-------------------------------
Description:
We have another major incident partly because of SCM follower kept suggesting
itself as a leader
{code:java}
scm15:9894:FOLLOWER:2059f536-5846-4573-b81d-274dc495c727:<redacted>
scm16:9894:LEADER:d0b9f1ed-7b61-4ac1-89d6-db3bc6fcb9df:<redacted>
scm17:9894:FOLLOWER:23ef2ecf-888c-4a83-b318-35fc5714ea8b:<redacted>{code}
{code:java}
com.google.protobuf.ServiceException:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdds.ratis.ServerNotLeaderException):
Server:23ef2ecf-888c-4a83-b318-35fc5714ea8b is not the leader. Suggested
leader is Server:scm17:9863.
at
org.apache.hadoop.hdds.ratis.ServerNotLeaderException.convertToNotLeaderException(ServerNotLeaderException.java:107)
at
org.apache.hadoop.hdds.scm.ha.RatisUtil.checkRatisException(RatisUtil.java:250)
at
org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.send(ScmBlockLocationProtocolServerSideTranslatorPB.java:117)
at
org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos$ScmBlockLocationProtocolService$2.callBlockingMethod(ScmBlockLocationProtocolProtos.java:16819)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server.processCall(ProtobufRpcEngine.java:495)
at
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:615)
at
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:593)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1153)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1336)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1229)
at
java.base/java.security.AccessController.doPrivileged(AccessController.java:714)
at java.base/javax.security.auth.Subject.doAs(Subject.java:525)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:2031)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:3622)
, while invoking $Proxy34.send over nodeId=scm17,nodeAddress=scm17:9863. Trying
to failover after sleeping for 2000ms. {code}
Ratis RaftServerImpl#generateNotLeaderException logic should not have allowed
it, see below
{code:java}
NotLeaderException generateNotLeaderException() {
if (!lifeCycle.getCurrentState().isRunning()) {
return new NotLeaderException(getMemberId(), null, null);
}
RaftPeerId leaderId = state.getLeaderId();
if (leaderId == null || leaderId.equals(getId())) {
// No idea about who is the current leader. Or the peer is the current
// leader, but it is about to step down. set the suggested leader as null.
leaderId = null;
}
final RaftConfigurationImpl conf = getRaftConf();
Collection<RaftPeer> peers = conf.getAllPeers();
return new NotLeaderException(getMemberId(), conf.getPeer(leaderId), peers);
} {code}
But because Ozone (annoyingly) reinvented a (buggy) wheel of NotLeaderException
generation, a non-leader SCM NotLeaderException with itself as the suggested
leader
{code:java}
public NotLeaderException triggerNotLeaderException() {
ByteString leaderId =
division.getInfo().getRoleInfoProto().getFollowerInfo().getLeaderInfo()
.getId().getId();
RaftPeer suggestedLeader = leaderId.isEmpty() ?
null :
division.getRaftConf().getPeer(RaftPeerId.valueOf(leaderId));
return new NotLeaderException(division.getMemberId(),
suggestedLeader,
division.getGroup().getPeers());
} {code}
We should use the correct Ratis generatedNotLeaderException.
This also apply to OM OzoneManagerRatisServer#newOMNotLeaderException, but
thankfully OM suggested leadership mechanism is broken anyway so it will not be
triggered (HDDS-14769).
I will revisit all the Ozone Ratis usage and check if there are some similar
issues. Most Ozone usage of Ratis need to be moved to Ratis. We will create new
Ratis API to access it if necessary. If this causes Ozone to wait Ratis release
every time, so be it.
was:
We have another major incident partly because of SCM follower kept suggesting
itself as a leader
{code:java}
scm15:9894:FOLLOWER:2059f536-5846-4573-b81d-274dc495c727:<redacted>
scm16:9894:LEADER:d0b9f1ed-7b61-4ac1-89d6-db3bc6fcb9df:<redacted>
scm17:9894:FOLLOWER:23ef2ecf-888c-4a83-b318-35fc5714ea8b:<redacted>{code}
{code:java}
com.google.protobuf.ServiceException:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdds.ratis.ServerNotLeaderException):
Server:23ef2ecf-888c-4a83-b318-35fc5714ea8b is not the leader. Suggested
leader is Server:scm17:9863.
at
org.apache.hadoop.hdds.ratis.ServerNotLeaderException.convertToNotLeaderException(ServerNotLeaderException.java:107)
at
org.apache.hadoop.hdds.scm.ha.RatisUtil.checkRatisException(RatisUtil.java:250)
at
org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.send(ScmBlockLocationProtocolServerSideTranslatorPB.java:117)
at
org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos$ScmBlockLocationProtocolService$2.callBlockingMethod(ScmBlockLocationProtocolProtos.java:16819)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server.processCall(ProtobufRpcEngine.java:495)
at
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:615)
at
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:593)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1153)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1336)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1229)
at
java.base/java.security.AccessController.doPrivileged(AccessController.java:714)
at java.base/javax.security.auth.Subject.doAs(Subject.java:525)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:2031)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:3622)
, while invoking $Proxy34.send over nodeId=scm17,nodeAddress=scm17:9863. Trying
to failover after sleeping for 2000ms. {code}
Ratis RaftServerImpl#generateNotLeaderException logic should not have allowed
it, see below
{code:java}
NotLeaderException generateNotLeaderException() {
if (!lifeCycle.getCurrentState().isRunning()) {
return new NotLeaderException(getMemberId(), null, null);
}
RaftPeerId leaderId = state.getLeaderId();
if (leaderId == null || leaderId.equals(getId())) {
// No idea about who is the current leader. Or the peer is the current
// leader, but it is about to step down. set the suggested leader as null.
leaderId = null;
}
final RaftConfigurationImpl conf = getRaftConf();
Collection<RaftPeer> peers = conf.getAllPeers();
return new NotLeaderException(getMemberId(), conf.getPeer(leaderId), peers);
} {code}
But because Ozone (annoyingly) reinvented a (buggy) wheel of NotLeaderException
generation, a non-leader SCM NotLeaderException with itself as the suggested
leader
{code:java}
public NotLeaderException triggerNotLeaderException() {
ByteString leaderId =
division.getInfo().getRoleInfoProto().getFollowerInfo().getLeaderInfo()
.getId().getId();
RaftPeer suggestedLeader = leaderId.isEmpty() ?
null :
division.getRaftConf().getPeer(RaftPeerId.valueOf(leaderId));
return new NotLeaderException(division.getMemberId(),
suggestedLeader,
division.getGroup().getPeers());
} {code}
We should use the correct Ratis generatedNotLeaderException.
This also apply to OM OzoneManagerRatisServer#newOMNotLeaderException, but
thankfully OM suggested leadership mechanism is broken anyway so it will not be
triggered (HDDS-14769).
I will revisit all the Ozone Ratis usage and check if there are some similar
issues. Most Ozone usage of Ratis need to be moved to Ratis. We will create new
Ratis API to access it if necessary. If this causes Ozone to wait Ratis upgrade
every time, so be it.
> SCM follower suggests itself as leader can cause OM to be stuck
> ---------------------------------------------------------------
>
> Key: HDDS-16028
> URL: https://issues.apache.org/jira/browse/HDDS-16028
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Ivan Andika
> Assignee: ChenXi
> Priority: Critical
>
> We have another major incident partly because of SCM follower kept suggesting
> itself as a leader
> {code:java}
> scm15:9894:FOLLOWER:2059f536-5846-4573-b81d-274dc495c727:<redacted>
> scm16:9894:LEADER:d0b9f1ed-7b61-4ac1-89d6-db3bc6fcb9df:<redacted>
> scm17:9894:FOLLOWER:23ef2ecf-888c-4a83-b318-35fc5714ea8b:<redacted>{code}
> {code:java}
> com.google.protobuf.ServiceException:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdds.ratis.ServerNotLeaderException):
> Server:23ef2ecf-888c-4a83-b318-35fc5714ea8b is not the leader. Suggested
> leader is Server:scm17:9863.
> at
> org.apache.hadoop.hdds.ratis.ServerNotLeaderException.convertToNotLeaderException(ServerNotLeaderException.java:107)
> at
> org.apache.hadoop.hdds.scm.ha.RatisUtil.checkRatisException(RatisUtil.java:250)
> at
> org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.send(ScmBlockLocationProtocolServerSideTranslatorPB.java:117)
> at
> org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos$ScmBlockLocationProtocolService$2.callBlockingMethod(ScmBlockLocationProtocolProtos.java:16819)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server.processCall(ProtobufRpcEngine.java:495)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:615)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:593)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1153)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1336)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1229)
> at
> java.base/java.security.AccessController.doPrivileged(AccessController.java:714)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:525)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:2031)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:3622)
> , while invoking $Proxy34.send over nodeId=scm17,nodeAddress=scm17:9863.
> Trying to failover after sleeping for 2000ms. {code}
> Ratis RaftServerImpl#generateNotLeaderException logic should not have allowed
> it, see below
> {code:java}
> NotLeaderException generateNotLeaderException() {
> if (!lifeCycle.getCurrentState().isRunning()) {
> return new NotLeaderException(getMemberId(), null, null);
> }
> RaftPeerId leaderId = state.getLeaderId();
> if (leaderId == null || leaderId.equals(getId())) {
> // No idea about who is the current leader. Or the peer is the current
> // leader, but it is about to step down. set the suggested leader as null.
> leaderId = null;
> }
> final RaftConfigurationImpl conf = getRaftConf();
> Collection<RaftPeer> peers = conf.getAllPeers();
> return new NotLeaderException(getMemberId(), conf.getPeer(leaderId), peers);
> } {code}
> But because Ozone (annoyingly) reinvented a (buggy) wheel of
> NotLeaderException generation, a non-leader SCM NotLeaderException with
> itself as the suggested leader
> {code:java}
> public NotLeaderException triggerNotLeaderException() {
> ByteString leaderId =
> division.getInfo().getRoleInfoProto().getFollowerInfo().getLeaderInfo()
> .getId().getId();
> RaftPeer suggestedLeader = leaderId.isEmpty() ?
> null :
> division.getRaftConf().getPeer(RaftPeerId.valueOf(leaderId));
> return new NotLeaderException(division.getMemberId(),
> suggestedLeader,
> division.getGroup().getPeers());
> } {code}
> We should use the correct Ratis generatedNotLeaderException.
> This also apply to OM OzoneManagerRatisServer#newOMNotLeaderException, but
> thankfully OM suggested leadership mechanism is broken anyway so it will not
> be triggered (HDDS-14769).
> I will revisit all the Ozone Ratis usage and check if there are some similar
> issues. Most Ozone usage of Ratis need to be moved to Ratis. We will create
> new Ratis API to access it if necessary. If this causes Ozone to wait Ratis
> release every time, so be it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
