[
https://issues.apache.org/jira/browse/HDDS-9292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai reassigned HDDS-9292:
--------------------------------------
Assignee: Rishabh Patel (was: Ashish Kumar)
> [disabled] Intermittent failure in S3 multipart copy with HA proxy
> ------------------------------------------------------------------
>
> Key: HDDS-9292
> URL: https://issues.apache.org/jira/browse/HDDS-9292
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Datanode, S3
> Reporter: Attila Doroszlai
> Assignee: Rishabh Patel
> Priority: Major
>
> Download of object created via multipart upload in S3 HA environment is
> failing intermittently.
> {code}
> Executing test ozone/test-s3-haproxy.sh
> ...
> Test Multipart Upload with the simplified aws s3 cp API | FAIL |
> ...
> ERROR: Test execution of ozone/test-s3-haproxy.sh is FAILED!!!!
> {code}
> S3 Gateway log:
> {code}
> 2023-09-15 05:06:57,143 [qtp2112233878-18] ERROR scm.XceiverClientGrpc:
> Failed to execute command GetBlock on the pipeline Pipeline[ Id:
> 3e70c06d-5798-4025-8791-78b8b2500f11, Nodes:
> 9416081d-f44a-4263-bc28-1838037e3ca1(ozone_datanode_1.ozone_default/172.19.0.7)82a30eb8-2baa-482f-a23a-ad010ae2684d(ozone_datanode_3.ozone_default/172.19.0.10)c229b2ba-c364-4c38-879e-952c8da62282(ozone_datanode_2.ozone_default/172.19.0.3),
> ReplicationConfig: STANDALONE/THREE, State:OPEN,
> leaderId:c229b2ba-c364-4c38-879e-952c8da62282,
> CreationTimestamp2023-09-15T05:05:12.106Z[UTC]].
> 2023-09-15 05:06:57,144 [qtp2112233878-18] WARN
> storage.ContainerProtocolCalls: Failed to get block #111677748019200096 in
> container #1 from
> 9416081d-f44a-4263-bc28-1838037e3ca1(ozone_datanode_1.ozone_default/172.19.0.7);
> will try another datanode.
> org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
> bcsId 388 mismatches with existing block Id 387 for block conID: 1 locID:
> 111677748019200096 bcsId: 388.
> at
> org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.validateContainerResponse(ContainerProtocolCalls.java:674)
> at
> org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.lambda$createValidators$4(ContainerProtocolCalls.java:685)
> at
> org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithRetry(XceiverClientGrpc.java:407)
> at
> org.apache.hadoop.hdds.scm.XceiverClientGrpc.lambda$sendCommandWithTraceIDAndRetry$0(XceiverClientGrpc.java:347)
> at
> org.apache.hadoop.hdds.tracing.TracingUtil.executeInSpan(TracingUtil.java:169)
> at
> org.apache.hadoop.hdds.tracing.TracingUtil.executeInNewSpan(TracingUtil.java:149)
> at
> org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommandWithTraceIDAndRetry(XceiverClientGrpc.java:342)
> at
> org.apache.hadoop.hdds.scm.XceiverClientGrpc.sendCommand(XceiverClientGrpc.java:323)
> at
> org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.getBlock(ContainerProtocolCalls.java:208)
> at
> org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.lambda$getBlock$0(ContainerProtocolCalls.java:186)
> at
> org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.tryEachDatanode(ContainerProtocolCalls.java:146)
> at
> org.apache.hadoop.hdds.scm.storage.ContainerProtocolCalls.getBlock(ContainerProtocolCalls.java:185)
> at
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.getChunkInfos(BlockInputStream.java:255)
> at
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.initialize(BlockInputStream.java:146)
> at
> org.apache.hadoop.hdds.scm.storage.BlockInputStream.readWithStrategy(BlockInputStream.java:308)
> at
> org.apache.hadoop.hdds.scm.storage.ExtendedInputStream.read(ExtendedInputStream.java:56)
> at
> org.apache.hadoop.hdds.scm.storage.ByteArrayReader.readFromBlock(ByteArrayReader.java:57)
> at
> org.apache.hadoop.hdds.scm.storage.MultipartInputStream.readWithStrategy(MultipartInputStream.java:96)
> at
> org.apache.hadoop.hdds.scm.storage.ExtendedInputStream.read(ExtendedInputStream.java:56)
> at
> org.apache.hadoop.ozone.client.io.OzoneInputStream.read(OzoneInputStream.java:56)
> at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1384)
> at
> org.apache.hadoop.ozone.s3.endpoint.ObjectEndpoint.lambda$get$1(ObjectEndpoint.java:407)
> {code}
> Message about bcdId mismatch appears in all 3 datanode logs, too.
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/08/30/25061/acceptance-misc
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/09/12/25292/acceptance-misc
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/09/13/25337/acceptance-misc
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/09/13/25345/acceptance-misc
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/09/14/25355/acceptance-misc
> *
> https://github.com/adoroszlai/ozone-build-results/blob/master/2023/09/15/25394/acceptance-misc
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
