Swaminathan Balachandran created HDDS-7292:
----------------------------------------------
Summary: [Ozone EC] SCM went down with replica index mismatch in
replicaSet
Key: HDDS-7292
URL: https://issues.apache.org/jira/browse/HDDS-7292
Project: Apache Ozone
Issue Type: Bug
Components: EC
Reporter: Swaminathan Balachandran
Assignee: Swaminathan Balachandran
steps taken :
1) Write tpc-ds hive dataset.
2) shutdown DNs for online reconstruction.
3) Initially queries were successful.
4) SCMs went down with java.lang.IllegalArgumentException: Replica Index in
replicaSet for containerID 4012must be between 1 and 5. But the given index is: 0
ozone-scm.log:
{noformat}
2022-09-22 20:17:42,722 INFO
org.apache.hadoop.hdds.scm.container.replication.LegacyReplicationManager:
Sending delete container command for container #5070 to datanode
507a0a6a-0745-49e7-9b42-c9bfd68eace8{ip: 172.27.14.11, host:
quasar-nkmwlh-1.quasar-nkmwlh.root.hwx.site, ports: [REPLICATION=9886,
RATIS=9858, RATIS_ADMIN=9857, RATIS_SERVER=9856, STANDALONE=9859],
networkLocation: /default, certSerialId: null, persistedOpState: IN_SERVICE,
persistedOpStateExpiryEpochSec: 0}
2022-09-22 20:17:42,723 INFO
org.apache.hadoop.hdds.scm.container.replication.ReplicationManager:
Replication Monitor Thread took 5 milliseconds for processing 238 containers.
2022-09-22 20:17:54,645 INFO
org.apache.hadoop.hdds.scm.pipeline.PipelineReportHandler: Reported pipeline
PipelineID=50b59b4b-4e14-4e00-9051-eb4689467a11 is not found
2022-09-22 20:17:54,661 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth
successful for recon/[email protected]
(auth:KERBEROS)
2022-09-22 20:17:54,666 INFO
SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager:
Authorization successful for
recon/[email protected] (auth:KERBEROS)
for protocol=interface
org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocol
2022-09-22 20:17:54,667 INFO org.apache.hadoop.ipc.Server: IPC Server handler
35 on 9860, call Call#89 Retry#0
org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocol.submitRequest
from 172.27.136.69:45687
org.apache.hadoop.hdds.scm.pipeline.PipelineNotFoundException:
PipelineID=50b59b4b-4e14-4e00-9051-eb4689467a11 not found
at
org.apache.hadoop.hdds.scm.pipeline.PipelineStateMap.getPipeline(PipelineStateMap.java:157)
at
org.apache.hadoop.hdds.scm.pipeline.PipelineStateManagerImpl.getPipeline(PipelineStateManagerImpl.java:137)
at jdk.internal.reflect.GeneratedMethodAccessor31.invoke(Unknown Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.invokeLocal(SCMHAInvocationHandler.java:87)
at
org.apache.hadoop.hdds.scm.ha.SCMHAInvocationHandler.invoke(SCMHAInvocationHandler.java:72)
at com.sun.proxy.$Proxy17.getPipeline(Unknown Source)
at
org.apache.hadoop.hdds.scm.pipeline.PipelineManagerImpl.getPipeline(PipelineManagerImpl.java:272)
at
org.apache.hadoop.hdds.scm.server.SCMClientProtocolServer.getPipeline(SCMClientProtocolServer.java:693)
at
org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.getPipeline(StorageContainerLocationProtocolServerSideTranslatorPB.java:860)
at
org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.processRequest(StorageContainerLocationProtocolServerSideTranslatorPB.java:581)
at
org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
at
org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.submitRequest(StorageContainerLocationProtocolServerSideTranslatorPB.java:213)
at
org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos$StorageContainerLocationProtocolService$2.callBlockingMethod(StorageContainerLocationProtocolProtos.java:61195)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:533)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:989)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:917)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2894)
2022-09-22 20:17:57,141 INFO
org.apache.hadoop.hdds.scm.container.replication.OverReplicatedProcessor:
Processed 0 over replicated containers, failed processing 0
2022-09-22 20:17:57,164 INFO
org.apache.hadoop.hdds.scm.container.replication.UnderReplicatedProcessor:
Processed 0 under replicated containers, failed processing 0
2022-09-22 20:17:57,724 INFO
org.apache.hadoop.hdds.scm.container.replication.ReplicationManager: Sending
close container command for container #4012 to datanode
323115a5-980c-42a6-be10-43f30ddec3f5{ip: 172.27.10.10, host:
quasar-nkmwlh-4.quasar-nkmwlh.root.hwx.site, ports: [REPLICATION=9886,
RATIS=9858, RATIS_ADMIN=9857, RATIS_SERVER=9856, STANDALONE=9859],
networkLocation: /default, certSerialId: null, persistedOpState: IN_SERVICE,
persistedOpStateExpiryEpochSec: 0}.
2022-09-22 20:17:57,729 ERROR
org.apache.hadoop.hdds.scm.container.replication.ReplicationManager: Exception
in Replication Monitor Thread.
java.lang.IllegalArgumentException: Replica Index in replicaSet for containerID
4012must be between 1 and 5. But the given index is: 0
at
org.apache.hadoop.hdds.scm.container.replication.ECContainerReplicaCount.ensureIndexWithinBounds(ECContainerReplicaCount.java:426)
at
org.apache.hadoop.hdds.scm.container.replication.ECContainerReplicaCount.<init>(ECContainerReplicaCount.java:99)
at
org.apache.hadoop.hdds.scm.container.replication.ECContainerHealthCheck.checkHealth(ECContainerHealthCheck.java:50)
at
org.apache.hadoop.hdds.scm.container.replication.ReplicationManager.processContainer(ReplicationManager.java:423)
at
org.apache.hadoop.hdds.scm.container.replication.ReplicationManager.processAll(ReplicationManager.java:278)
at
org.apache.hadoop.hdds.scm.container.replication.ReplicationManager.run(ReplicationManager.java:538)
at java.base/java.lang.Thread.run(Thread.java:834)
2022-09-22 20:17:57,731 INFO org.apache.hadoop.util.ExitUtil: Exiting with
status 1: java.lang.IllegalArgumentException: Replica Index in replicaSet for
containerID 4012must be between 1 and 5. But the given index is: 0
2022-09-22 20:17:57,736 INFO
org.apache.hadoop.hdds.scm.server.StorageContainerManager: Container Balancer
is not running.
2022-09-22 20:17:57,736 INFO
org.apache.hadoop.hdds.scm.server.StorageContainerManager: Stopping Replication
Manager Service.
2022-09-22 20:17:57,736 INFO
org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down StorageContainerManager at
quasar-nkmwlh-2.quasar-nkmwlh.root.hwx.site/172.27.136.69
************************************************************/
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
