[
https://issues.apache.org/jira/browse/HDDS-11308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17873020#comment-17873020
]
Ethan Rose commented on HDDS-11308:
-----------------------------------
Rather than add synchronization on the EC path, a better fix might be removing
the requirement that putBlocks of different blocks in the same container must
be serialized. This would mean keeping block count and used bytes of an open
container in memory and flushing it on container close, instead of incrementing
and flushing with each putBlock. On startup, the DN would have to check and
load these values for open containers only. This would be about
{{ozone.scm.pipeline.owner.container.count}} *
{{ozone.scm.datanode.pipeline.limit}} containers, but pipeline limit is usually
coupled to volume count and volumes are loaded concurrently.
Concurrent putBlocks updating different block entries within the same container
in the DB should not cause a problem with the existing code as is.
> ClassCastException during simultaneous putBlock operations on an EC container
> -----------------------------------------------------------------------------
>
> Key: HDDS-11308
> URL: https://issues.apache.org/jira/browse/HDDS-11308
> Project: Apache Ozone
> Issue Type: Bug
> Components: EC, Ozone Datanode
> Reporter: Ethan Rose
> Priority: Major
>
> h2. Issue
> While writing EC data the following exception was observed:
> {code:java}
> 2024-07-30 10:42:22,547 WARN
> [ChunkReader-8]-org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler:
> Operation: PutBlock , Trace ID: , Message: java.lang.ClassCastException:
> java.util.HashMap$Node cannot be cast to java.util.HashMap$TreeNode , Result:
> CONTAINER_INTERNAL_ERROR , StorageContainerException Occurred.
> org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
> java.lang.ClassCastException: java.util.HashMap$Node cannot be cast to
> java.util.HashMap$TreeNode
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handle(KeyValueHandler.java:228)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:328)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.lambda$dispatch$0(HddsDispatcher.java:176)
> at
> org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:175)
> at
> org.apache.hadoop.ozone.container.common.transport.server.GrpcXceiverService$1.onNext(GrpcXceiverService.java:57)
> at
> org.apache.hadoop.ozone.container.common.transport.server.GrpcXceiverService$1.onNext(GrpcXceiverService.java:50)
> at
> org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:262)
> at
> org.apache.ratis.thirdparty.io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33)
> at
> org.apache.hadoop.hdds.tracing.GrpcServerInterceptor$1.onMessage(GrpcServerInterceptor.java:49)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:333)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:316)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:835)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> Caused by: java.lang.ClassCastException: java.util.HashMap$Node cannot be
> cast to java.util.HashMap$TreeNode
> at java.util.HashMap$TreeNode.moveRootToFront(HashMap.java:1859)
> at java.util.HashMap$TreeNode.putTreeVal(HashMap.java:2038)
> at java.util.HashMap.putVal(HashMap.java:639)
> at java.util.HashMap.put(HashMap.java:613)
> at java.util.HashSet.add(HashSet.java:220)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.addToPendingPutBlockCache(KeyValueContainer.java:831)
> at
> org.apache.hadoop.ozone.container.keyvalue.impl.BlockManagerImpl.persistPutBlock(BlockManagerImpl.java:209)
> at
> org.apache.hadoop.ozone.container.keyvalue.impl.BlockManagerImpl.putBlock(BlockManagerImpl.java:103)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handlePutBlock(KeyValueHandler.java:551)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.dispatchRequest(KeyValueHandler.java:254)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handle(KeyValueHandler.java:225)
> ... 17 more
> 2024-07-30 10:42:22,550 WARN
> [ChunkReader-8]-org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer:
> Moving container /data/...current/containerDir189/227982 to state UNHEALTHY
> from state:OPEN
> {code}
> Container 227982 is an EC container, and Ozone was later able to restore all
> healthy replicas using offline reconstruction.
> h2. Cause
> There is a putBlock cache for tracking updates to the block count metadata of
> an open container. The cache is not thread safe so when two threads access it
> at the same time we may get undefined behavior like the exception shown above.
> h3. Explanation
> The cache is implemented as a non-concurrent {{{}HashSet{}}}. The [code
> comments acknowledge
> this|https://github.com/apache/ozone/blob/98369a8343f12479af99db4c83696945d40c1e3d/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainer.java#L112]
> and assume only one instance of putBlock per container will run at a time.
> This assumption that only one putBlock per container will run at a time is
> also assumed by the metadata updates done by putBlock, which increment
> counters in RocksDB and can have incorrect values if two updates happen at
> the same time (time of check time of use, see HDDS-8129 causing issues like
> HDDS-5359)
> These assumptions were probably in place before EC was added. For Ratis, the
> {{ContainerStateMachine}} enforces that only one putBlock per container will
> run at a time. For EC, however, there is no such enforcement. EC write
> requests go through {{XceiverServerGrpc}} to {{GrpcXceiverService}} where
> they are handled by a [thread
> pool|https://github.com/apache/ozone/blob/9ba4a73f46156cae31c6c074290549fdb95e6f2e/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/XceiverServerGrpc.java#L149]
> that assumes it is only handling reads. If two EC clients issue putBlock for
> different blocks in the same container, we may see:
> # Incorrect metadata counters.
> # Exceptions like the one above that mark the container unhealthy and fail
> the write.
> Based on inspection of the {{HashSet}} code, I believe this specific
> exception will only occur if the two concurrent putBlock IDs hash to the same
> bin in the set. This explains why the issue has existed for about 2 years but
> have just seen it for the first time now.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
