Ronald Macmaster created HDFS-18006:
---------------------------------------
Summary: DFSClient: Shared Memory Leak when
REQUEST_SHORT_CIRCUIT_FDS fails
Key: HDFS-18006
URL: https://issues.apache.org/jira/browse/HDFS-18006
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.5.0, 3.4.1, 3.6.0
Environment: client: hadoop 3.4.1 (HBase 2.5.11 Regionserver)
server: hadoop 3.4.2
Reporter: Ronald Macmaster
When the DFSClient {{BlockReaderFactory#createShortCircuitReplicaInfo}}
initializes shared memory slots for Short Circuit Reads, it doesn't properly
clean it up it if an unexpected error is encountered. This leaks shared memory
on the DataNode host over a prolonged period of time.
The problem is below in {{{}BlockReaderFactory#requestFileDescriptors{}}}. The
shmem slot is only free'd on {{ERROR_ACCESS_TOKEN}} (HDFS-16198) or when it
throws an {{{}IOException{}}}, ({{{}createShortCircuitReplicaInfo{}}} catches
it above).
Three other {{requestFileDescriptors}} cases can return {{null}} without
freeing the slot:
* {{SUCCESS}}
* {{ERROR_UNSUPPORTED}}
* {{{}default{}}}, which handles general {{ERRORresponses from the DataNode.}}
Even though the DataNode unregisters its side of the slot in
{{DataXceiver#requestShortCircuitFds}} on failure, it cannot fully release the
memory until the client closes its end as well!
h2. Context
We discovered this issue while experimenting with machine decom/recom on our
HBase clusters. If a DataNode rejoins the cluster without previously decomming
properly, it can trigger a mass block deletions and ReplicaNotFoundExceptions
on the HBase Regionserver's DFSClient (which caches the old block locations).
The {{DataNode}} handles {{ReplicaNotFoundException}} by returning an ERROR
response to the client and this causes it to temporarily disable short circuit
reads. We curiously tuned down dfs.domain.socket.disable.interval.seconds to 0
to see if we could bypass this otherwise harmless error (the read is retried
over TCP), but encountered a shared memory leak at scale.
{code:java}
WARN impl.BlockReaderFactory: BlockReaderFactory(fileName=..., block=...):
unknown response code ERROR while attempting to set up short-circuit access. No
data exists for block BP-...:blk_... . Short-circuit read for DataNode
DatanodeInfoWithStorage[...] is disabled temporarily for 600 seconds based on
dfs.domain.socket.disable.interval.seconds.
{code}
{code:java}
WARN [RpcServer.default.FPRWQ.Fifo.read.handler=157,queue=16,port=16022]
impl.BlockReaderFactory: I/O error constructing remote block reader for block
BP-..:blk_..
java.io.IOException: Got error, status=ERROR, status message opReadBlock
BP-..:blk_.. received exception
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not
found for BP-..:blk_... The block may have been removed recently by the
balancer or by intentionally reducing the replication factor. This condition is
usually harmless. To be certain, please check the preceding datanode log
messages for signs of a more serious issue., for OP_READ_BLOCK,
self=/<ip>:<port>, remote=/<ip>:<port>, for file
/hbase/data/<ns>/<table>/<region>/<col>/<hfile>, for pool BP-.. block ..
at
org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:128)
~[hadoop-hdfs-client-3.4.1.jar:?]
at
org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:104)
~[hadoop-hdfs-client-3.4.1.jar:?]
at
org.apache.hadoop.hdfs.client.impl.BlockReaderRemote.checkSuccess(BlockReaderRemote.java:452)
~[hadoop-hdfs-client-3.4.1.jar:?]
at
org.apache.hadoop.hdfs.client.impl.BlockReaderRemote.newBlockReader(BlockReaderRemote.java:420)
~[hadoop-hdfs-client-3.4.1.jar:?]
at
org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:861)
~[hadoop-hdfs-client-3.4.1.jar:?]
at
org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:757)
~[hadoop-hdfs-client-3.4.1.jar:?]
at
org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.build(BlockReaderFactory.java:381)
~[hadoop-hdfs-client-3.4.1.jar:?]
at
org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:715)
~[hadoop-hdfs-client-3.4.1.jar:?]
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]