Ethan Rose created HDDS-11308:
---------------------------------
Summary: ClassCastException during simultaneous putBlock
operations on an EC container
Key: HDDS-11308
URL: https://issues.apache.org/jira/browse/HDDS-11308
Project: Apache Ozone
Issue Type: Bug
Components: EC, Ozone Datanode
Reporter: Ethan Rose
h2. Issue
While writing EC data the following exception was observed:
{code:java}
2024-07-30 10:42:22,547 WARN
[ChunkReader-8]-org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler:
Operation: PutBlock , Trace ID: , Message: java.lang.ClassCastException:
java.util.HashMap$Node cannot be cast to java.util.HashMap$TreeNode , Result:
CONTAINER_INTERNAL_ERROR , StorageContainerException Occurred.
org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
java.lang.ClassCastException: java.util.HashMap$Node cannot be cast to
java.util.HashMap$TreeNode
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handle(KeyValueHandler.java:228)
at
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:328)
at
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.lambda$dispatch$0(HddsDispatcher.java:176)
at
org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
at
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:175)
at
org.apache.hadoop.ozone.container.common.transport.server.GrpcXceiverService$1.onNext(GrpcXceiverService.java:57)
at
org.apache.hadoop.ozone.container.common.transport.server.GrpcXceiverService$1.onNext(GrpcXceiverService.java:50)
at
org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:262)
at
org.apache.ratis.thirdparty.io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33)
at
org.apache.hadoop.hdds.tracing.GrpcServerInterceptor$1.onMessage(GrpcServerInterceptor.java:49)
at
org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:333)
at
org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:316)
at
org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:835)
at
org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.ClassCastException: java.util.HashMap$Node cannot be cast
to java.util.HashMap$TreeNode
at java.util.HashMap$TreeNode.moveRootToFront(HashMap.java:1859)
at java.util.HashMap$TreeNode.putTreeVal(HashMap.java:2038)
at java.util.HashMap.putVal(HashMap.java:639)
at java.util.HashMap.put(HashMap.java:613)
at java.util.HashSet.add(HashSet.java:220)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.addToPendingPutBlockCache(KeyValueContainer.java:831)
at
org.apache.hadoop.ozone.container.keyvalue.impl.BlockManagerImpl.persistPutBlock(BlockManagerImpl.java:209)
at
org.apache.hadoop.ozone.container.keyvalue.impl.BlockManagerImpl.putBlock(BlockManagerImpl.java:103)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handlePutBlock(KeyValueHandler.java:551)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.dispatchRequest(KeyValueHandler.java:254)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handle(KeyValueHandler.java:225)
... 17 more
2024-07-30 10:42:22,550 WARN
[ChunkReader-8]-org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer:
Moving container /data/...current/containerDir189/227982 to state UNHEALTHY
from state:OPEN
{code}
Container 227982 is an EC container, and Ozone was later able to restore all
healthy replicas using offline reconstruction.
h2. Cause
There is a putBlock cache for tracking updates to the block count metadata of
an open container. The cache is not thread safe so when two threads access it
at the same time we may get undefined behavior like the exception shown above.
h3. Explanation
The cache is implemented as a non-concurrent {{{}HashSet{}}}. The [code
comments acknowledge
this|https://github.com/apache/ozone/blob/98369a8343f12479af99db4c83696945d40c1e3d/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainer.java#L112]
and assume only one instance of putBlock per container will run at a time.
This assumption that only one putBlock per container will run at a time is also
assumed by the metadata updates done by putBlock, which increment counters in
RocksDB and can have incorrect values if two updates happen at the same time
(time of check time of use, see HDDS-8129 causing issues like HDDS-5359)
This assumptions were probably in place before EC was added. For Ratis, the
{{ContainerStateMachine}} enforces that only one putBlock per container will
run at a time. For EC, however, there is no such enforcement. EC write requests
go through {{XceiverServerGrpc}} to {{GrpcXceiverService}} where they are
handled by a [thread
pool|https://github.com/apache/ozone/blob/9ba4a73f46156cae31c6c074290549fdb95e6f2e/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/XceiverServerGrpc.java#L149]
that assumes it is only handling reads. If two EC clients issue putBlock for
different blocks in the same container, we may see:
# Incorrect metadata counters.
# Exceptions like the one above that mark the container unhealthy and fail the
write.
Based on inspection of the {{HashSet}} code, I believe this specific exception
will only occur if the two concurrent putBlock IDs hash to the same bin in the
set. This explains why the issue has existed for about 2 years but have just
seen it for the first time now.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
