[
https://issues.apache.org/jira/browse/HDDS-11308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ethan Rose updated HDDS-11308:
------------------------------
Description:
h2. Issue
While writing EC data the following exception was observed:
{code:java}
2024-07-30 10:42:22,547 WARN
[ChunkReader-8]-org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler:
Operation: PutBlock , Trace ID: , Message: java.lang.ClassCastException:
java.util.HashMap$Node cannot be cast to java.util.HashMap$TreeNode , Result:
CONTAINER_INTERNAL_ERROR , StorageContainerException Occurred.
org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
java.lang.ClassCastException: java.util.HashMap$Node cannot be cast to
java.util.HashMap$TreeNode
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handle(KeyValueHandler.java:228)
at
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:328)
at
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.lambda$dispatch$0(HddsDispatcher.java:176)
at
org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
at
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:175)
at
org.apache.hadoop.ozone.container.common.transport.server.GrpcXceiverService$1.onNext(GrpcXceiverService.java:57)
at
org.apache.hadoop.ozone.container.common.transport.server.GrpcXceiverService$1.onNext(GrpcXceiverService.java:50)
at
org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:262)
at
org.apache.ratis.thirdparty.io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33)
at
org.apache.hadoop.hdds.tracing.GrpcServerInterceptor$1.onMessage(GrpcServerInterceptor.java:49)
at
org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:333)
at
org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:316)
at
org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:835)
at
org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.ClassCastException: java.util.HashMap$Node cannot be cast
to java.util.HashMap$TreeNode
at java.util.HashMap$TreeNode.moveRootToFront(HashMap.java:1859)
at java.util.HashMap$TreeNode.putTreeVal(HashMap.java:2038)
at java.util.HashMap.putVal(HashMap.java:639)
at java.util.HashMap.put(HashMap.java:613)
at java.util.HashSet.add(HashSet.java:220)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.addToPendingPutBlockCache(KeyValueContainer.java:831)
at
org.apache.hadoop.ozone.container.keyvalue.impl.BlockManagerImpl.persistPutBlock(BlockManagerImpl.java:209)
at
org.apache.hadoop.ozone.container.keyvalue.impl.BlockManagerImpl.putBlock(BlockManagerImpl.java:103)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handlePutBlock(KeyValueHandler.java:551)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.dispatchRequest(KeyValueHandler.java:254)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handle(KeyValueHandler.java:225)
... 17 more
2024-07-30 10:42:22,550 WARN
[ChunkReader-8]-org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer:
Moving container /data/...current/containerDir189/227982 to state UNHEALTHY
from state:OPEN
{code}
Container 227982 is an EC container, and Ozone was later able to restore all
healthy replicas using offline reconstruction.
h2. Cause
There is a putBlock cache for tracking updates to the block count metadata of
an open container. The cache is not thread safe so when two threads access it
at the same time we may get undefined behavior like the exception shown above.
h3. Explanation
The cache is implemented as a non-concurrent {{{}HashSet{}}}. The [code
comments acknowledge
this|https://github.com/apache/ozone/blob/98369a8343f12479af99db4c83696945d40c1e3d/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainer.java#L112]
and assume only one instance of putBlock per container will run at a time.
This assumption that only one putBlock per container will run at a time is also
assumed by the metadata updates done by putBlock, which increment counters in
RocksDB and can have incorrect values if two updates happen at the same time
(time of check time of use, see HDDS-8129 causing issues like HDDS-5359)
These assumptions were probably in place before EC was added. For Ratis, the
{{ContainerStateMachine}} enforces that only one putBlock per container will
run at a time. For EC, however, there is no such enforcement. EC write requests
go through {{XceiverServerGrpc}} to {{GrpcXceiverService}} where they are
handled by a [thread
pool|https://github.com/apache/ozone/blob/9ba4a73f46156cae31c6c074290549fdb95e6f2e/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/XceiverServerGrpc.java#L149]
that assumes it is only handling reads. If two EC clients issue putBlock for
different blocks in the same container, we may see:
# Incorrect metadata counters.
# Exceptions like the one above that mark the container unhealthy and fail the
write.
Based on inspection of the {{HashSet}} code, I believe this specific exception
will only occur if the two concurrent putBlock IDs hash to the same bin in the
set. This explains why the issue has existed for about 2 years but have just
seen it for the first time now.
was:
h2. Issue
While writing EC data the following exception was observed:
{code:java}
2024-07-30 10:42:22,547 WARN
[ChunkReader-8]-org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler:
Operation: PutBlock , Trace ID: , Message: java.lang.ClassCastException:
java.util.HashMap$Node cannot be cast to java.util.HashMap$TreeNode , Result:
CONTAINER_INTERNAL_ERROR , StorageContainerException Occurred.
org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
java.lang.ClassCastException: java.util.HashMap$Node cannot be cast to
java.util.HashMap$TreeNode
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handle(KeyValueHandler.java:228)
at
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:328)
at
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.lambda$dispatch$0(HddsDispatcher.java:176)
at
org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
at
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:175)
at
org.apache.hadoop.ozone.container.common.transport.server.GrpcXceiverService$1.onNext(GrpcXceiverService.java:57)
at
org.apache.hadoop.ozone.container.common.transport.server.GrpcXceiverService$1.onNext(GrpcXceiverService.java:50)
at
org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:262)
at
org.apache.ratis.thirdparty.io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33)
at
org.apache.hadoop.hdds.tracing.GrpcServerInterceptor$1.onMessage(GrpcServerInterceptor.java:49)
at
org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:333)
at
org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:316)
at
org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:835)
at
org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.ClassCastException: java.util.HashMap$Node cannot be cast
to java.util.HashMap$TreeNode
at java.util.HashMap$TreeNode.moveRootToFront(HashMap.java:1859)
at java.util.HashMap$TreeNode.putTreeVal(HashMap.java:2038)
at java.util.HashMap.putVal(HashMap.java:639)
at java.util.HashMap.put(HashMap.java:613)
at java.util.HashSet.add(HashSet.java:220)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.addToPendingPutBlockCache(KeyValueContainer.java:831)
at
org.apache.hadoop.ozone.container.keyvalue.impl.BlockManagerImpl.persistPutBlock(BlockManagerImpl.java:209)
at
org.apache.hadoop.ozone.container.keyvalue.impl.BlockManagerImpl.putBlock(BlockManagerImpl.java:103)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handlePutBlock(KeyValueHandler.java:551)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.dispatchRequest(KeyValueHandler.java:254)
at
org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handle(KeyValueHandler.java:225)
... 17 more
2024-07-30 10:42:22,550 WARN
[ChunkReader-8]-org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer:
Moving container /data/...current/containerDir189/227982 to state UNHEALTHY
from state:OPEN
{code}
Container 227982 is an EC container, and Ozone was later able to restore all
healthy replicas using offline reconstruction.
h2. Cause
There is a putBlock cache for tracking updates to the block count metadata of
an open container. The cache is not thread safe so when two threads access it
at the same time we may get undefined behavior like the exception shown above.
h3. Explanation
The cache is implemented as a non-concurrent {{{}HashSet{}}}. The [code
comments acknowledge
this|https://github.com/apache/ozone/blob/98369a8343f12479af99db4c83696945d40c1e3d/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainer.java#L112]
and assume only one instance of putBlock per container will run at a time.
This assumption that only one putBlock per container will run at a time is also
assumed by the metadata updates done by putBlock, which increment counters in
RocksDB and can have incorrect values if two updates happen at the same time
(time of check time of use, see HDDS-8129 causing issues like HDDS-5359)
This assumptions were probably in place before EC was added. For Ratis, the
{{ContainerStateMachine}} enforces that only one putBlock per container will
run at a time. For EC, however, there is no such enforcement. EC write requests
go through {{XceiverServerGrpc}} to {{GrpcXceiverService}} where they are
handled by a [thread
pool|https://github.com/apache/ozone/blob/9ba4a73f46156cae31c6c074290549fdb95e6f2e/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/XceiverServerGrpc.java#L149]
that assumes it is only handling reads. If two EC clients issue putBlock for
different blocks in the same container, we may see:
# Incorrect metadata counters.
# Exceptions like the one above that mark the container unhealthy and fail the
write.
Based on inspection of the {{HashSet}} code, I believe this specific exception
will only occur if the two concurrent putBlock IDs hash to the same bin in the
set. This explains why the issue has existed for about 2 years but have just
seen it for the first time now.
> ClassCastException during simultaneous putBlock operations on an EC container
> -----------------------------------------------------------------------------
>
> Key: HDDS-11308
> URL: https://issues.apache.org/jira/browse/HDDS-11308
> Project: Apache Ozone
> Issue Type: Bug
> Components: EC, Ozone Datanode
> Reporter: Ethan Rose
> Priority: Major
>
> h2. Issue
> While writing EC data the following exception was observed:
> {code:java}
> 2024-07-30 10:42:22,547 WARN
> [ChunkReader-8]-org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler:
> Operation: PutBlock , Trace ID: , Message: java.lang.ClassCastException:
> java.util.HashMap$Node cannot be cast to java.util.HashMap$TreeNode , Result:
> CONTAINER_INTERNAL_ERROR , StorageContainerException Occurred.
> org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException:
> java.lang.ClassCastException: java.util.HashMap$Node cannot be cast to
> java.util.HashMap$TreeNode
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handle(KeyValueHandler.java:228)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatchRequest(HddsDispatcher.java:328)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.lambda$dispatch$0(HddsDispatcher.java:176)
> at
> org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
> at
> org.apache.hadoop.ozone.container.common.impl.HddsDispatcher.dispatch(HddsDispatcher.java:175)
> at
> org.apache.hadoop.ozone.container.common.transport.server.GrpcXceiverService$1.onNext(GrpcXceiverService.java:57)
> at
> org.apache.hadoop.ozone.container.common.transport.server.GrpcXceiverService$1.onNext(GrpcXceiverService.java:50)
> at
> org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:262)
> at
> org.apache.ratis.thirdparty.io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33)
> at
> org.apache.hadoop.hdds.tracing.GrpcServerInterceptor$1.onMessage(GrpcServerInterceptor.java:49)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:333)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:316)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:835)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> Caused by: java.lang.ClassCastException: java.util.HashMap$Node cannot be
> cast to java.util.HashMap$TreeNode
> at java.util.HashMap$TreeNode.moveRootToFront(HashMap.java:1859)
> at java.util.HashMap$TreeNode.putTreeVal(HashMap.java:2038)
> at java.util.HashMap.putVal(HashMap.java:639)
> at java.util.HashMap.put(HashMap.java:613)
> at java.util.HashSet.add(HashSet.java:220)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer.addToPendingPutBlockCache(KeyValueContainer.java:831)
> at
> org.apache.hadoop.ozone.container.keyvalue.impl.BlockManagerImpl.persistPutBlock(BlockManagerImpl.java:209)
> at
> org.apache.hadoop.ozone.container.keyvalue.impl.BlockManagerImpl.putBlock(BlockManagerImpl.java:103)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handlePutBlock(KeyValueHandler.java:551)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.dispatchRequest(KeyValueHandler.java:254)
> at
> org.apache.hadoop.ozone.container.keyvalue.KeyValueHandler.handle(KeyValueHandler.java:225)
> ... 17 more
> 2024-07-30 10:42:22,550 WARN
> [ChunkReader-8]-org.apache.hadoop.ozone.container.keyvalue.KeyValueContainer:
> Moving container /data/...current/containerDir189/227982 to state UNHEALTHY
> from state:OPEN
> {code}
> Container 227982 is an EC container, and Ozone was later able to restore all
> healthy replicas using offline reconstruction.
> h2. Cause
> There is a putBlock cache for tracking updates to the block count metadata of
> an open container. The cache is not thread safe so when two threads access it
> at the same time we may get undefined behavior like the exception shown above.
> h3. Explanation
> The cache is implemented as a non-concurrent {{{}HashSet{}}}. The [code
> comments acknowledge
> this|https://github.com/apache/ozone/blob/98369a8343f12479af99db4c83696945d40c1e3d/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainer.java#L112]
> and assume only one instance of putBlock per container will run at a time.
> This assumption that only one putBlock per container will run at a time is
> also assumed by the metadata updates done by putBlock, which increment
> counters in RocksDB and can have incorrect values if two updates happen at
> the same time (time of check time of use, see HDDS-8129 causing issues like
> HDDS-5359)
> These assumptions were probably in place before EC was added. For Ratis, the
> {{ContainerStateMachine}} enforces that only one putBlock per container will
> run at a time. For EC, however, there is no such enforcement. EC write
> requests go through {{XceiverServerGrpc}} to {{GrpcXceiverService}} where
> they are handled by a [thread
> pool|https://github.com/apache/ozone/blob/9ba4a73f46156cae31c6c074290549fdb95e6f2e/hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/XceiverServerGrpc.java#L149]
> that assumes it is only handling reads. If two EC clients issue putBlock for
> different blocks in the same container, we may see:
> # Incorrect metadata counters.
> # Exceptions like the one above that mark the container unhealthy and fail
> the write.
> Based on inspection of the {{HashSet}} code, I believe this specific
> exception will only occur if the two concurrent putBlock IDs hash to the same
> bin in the set. This explains why the issue has existed for about 2 years but
> have just seen it for the first time now.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
