[
https://issues.apache.org/jira/browse/HDDS-10430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Andika updated HDDS-10430:
-------------------------------
Target Version/s: 2.0.0, 1.4.2
> Race condition around Pipeline#nodesInOrder
> -------------------------------------------
>
> Key: HDDS-10430
> URL: https://issues.apache.org/jira/browse/HDDS-10430
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Attila Doroszlai
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.0.0
>
> Attachments: 6285_review.patch
>
>
> Found this potential bug in a HDDS-7593 derived branch (but looks more
> general) on [~weichiu]'s cluster.
> {code}
> 24/02/26 20:39:13 INFO retry.RetryInvocationHandler:
> com.google.protobuf.ServiceException:
> org.apache.hadoop.ipc.RemoteException(java.lang.IndexOut0fBoundsException):
> Index: 2, Size: 3
> at java.util.ArrayList.rangeCheck(ArrayList.java:657)
> at java.util.ArrayList.get(ArrayList.java:433)
> at
> org.apache.hadoop.hdds.scm.pipeline.Pipeline.getProtobufMessage(Pipeline.java:414)
> at
> org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfo.getProtobuf(OmKeyLocationInfo.java:123)
> at
> org.apache.hadoop.ozone.om.helpers.OmKeyLocationInfoGroup.getProtobuf(OmKeyLocationInfoGroup.java:135)
> at
> org.apache.hadoop.ozone.om.helpers.OmKeyInfo.getProtobuf(OmKeyInfo.java:652)
> at
> org.apache.hadoop.ozone.om.helpers.OmKeyInfo.getProtobuf(OmKeyInfo.java:634)
> at
> org.apache.hadoop.ozone.om.helpers.OmKeyInfo.getProtobuf(OmKeyInfo.java:602)
> at
> org.apache.hadoop.ozone.om.helpers.KeyInfoWithVolumeContext.toProtobuf(KeyInfoWithVolumeContext.java:63)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.getKeyInfo(OzoneManagerRequestHandler.java:653)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleReadRequest(OzoneManagerRequestHandler.java:337)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitReadRequestToOM(OzoneManagerProtocolServerSideTranslatorPB.java:250)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.internalProcessRequest(OzoneManagerProtocolServerSideTranslatorPB.java:196)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:156)
> at
> org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:89)
> at
> org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:147)
> at
> org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:533)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:994)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:922)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2899)
> {code}
> thrown from {{nodes.get(i)}} in snippet:
> https://github.com/apache/ozone/blob/e1d123149ecd7d6817f09a43707570513a71fbda/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/pipeline/Pipeline.java#L414
> {code:java|title=Pipeline#getProtobufMessage}
> // To save the message size on wire, only transfer the node order based on
> // network topology
> List<DatanodeDetails> nodes = nodesInOrder;
> if (!nodes.isEmpty()) {
> for (int i = 0; i < nodes.size(); i++) {
> Iterator<DatanodeDetails> it = nodeStatus.keySet().iterator();
> for (int j = 0; j < nodeStatus.keySet().size(); j++) {
> if (it.next().equals(nodes.get(i))) {
> builder.addMemberOrders(j);
> break;
> }
> }
> }
> {code}
> *Suspected cause:*
> {{nodes}} is changed during {{nodes.get(i)}} due to lack of synchronization.
> This is most likely caused by (at least) one other thread modifying {{nodes}}
> array at the same time using {{setNodesInOrder()}}: cc [~sumitagrawl]
> {code}
> public void setNodesInOrder(List<DatanodeDetails> nodes) {
> nodesInOrder.clear();
> if (null == nodes) {
> return;
> }
> nodesInOrder.addAll(nodes);
> }
> {code}
> {{setNodesInOrder()}} is at least called from
> [{{sortDatanodes()}}|https://github.com/apache/ozone/blob/c1d7b433d80baf21f32f48474db49406814dd6d2/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/KeyManagerImpl.java#L1867]
> or
> [{{allocateBlock()}}|https://github.com/apache/ozone/blob/838cc2691b0652ade8de13c0c6156d4ca1b64751/hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMBlockProtocolServer.java#L213].
> And the former is further widely called from {{lookupKey()}},
> {{getOzoneFileStatus()}}, {{listStatus()}}, {{getKeyInfo()}}, etc.
> [~weichiu] pls help add more relevant log details here.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
