Ronald Macmaster created HDFS-18006:
---------------------------------------

             Summary: DFSClient: Shared Memory Leak when 
REQUEST_SHORT_CIRCUIT_FDS fails
                 Key: HDFS-18006
                 URL: https://issues.apache.org/jira/browse/HDFS-18006
             Project: Hadoop HDFS
          Issue Type: Bug
    Affects Versions: 3.5.0, 3.4.1, 3.6.0
         Environment: client: hadoop 3.4.1 (HBase 2.5.11 Regionserver)
server: hadoop 3.4.2
            Reporter: Ronald Macmaster


When the DFSClient {{BlockReaderFactory#createShortCircuitReplicaInfo}} 
initializes shared memory slots for Short Circuit Reads, it doesn't properly 
clean it up it if an unexpected error is encountered. This leaks shared memory 
on the DataNode host over a prolonged period of time.

The problem is below in {{{}BlockReaderFactory#requestFileDescriptors{}}}. The 
shmem slot is only free'd on {{ERROR_ACCESS_TOKEN}} (HDFS-16198) or when it 
throws an {{{}IOException{}}}, ({{{}createShortCircuitReplicaInfo{}}} catches 
it above).

Three other {{requestFileDescriptors}} cases can return {{null}} without 
freeing the slot:
 * {{SUCCESS}}
 * {{ERROR_UNSUPPORTED}}
 * {{{}default{}}}, which handles general {{ERRORresponses from the DataNode.}}

Even though the DataNode unregisters its side of the slot in 
{{DataXceiver#requestShortCircuitFds}} on failure, it cannot fully release the 
memory until the client closes its end as well! 
h2. Context

We discovered this issue while experimenting with machine decom/recom on our 
HBase clusters. If a DataNode rejoins the cluster without previously decomming 
properly, it can trigger a mass block deletions and ReplicaNotFoundExceptions 
on the HBase Regionserver's DFSClient (which caches the old block locations).  

The {{DataNode}} handles {{ReplicaNotFoundException}} by returning an ERROR 
response to the client and this causes it to temporarily disable short circuit 
reads. We curiously tuned down dfs.domain.socket.disable.interval.seconds to 0 
to see if we could bypass this otherwise harmless error (the read is retried 
over TCP), but encountered a shared memory leak at scale.
 
{code:java}
WARN impl.BlockReaderFactory: BlockReaderFactory(fileName=..., block=...): 
unknown response code ERROR while attempting to set up short-circuit access. No 
data exists for block BP-...:blk_... . Short-circuit read for DataNode 
DatanodeInfoWithStorage[...] is disabled temporarily for 600 seconds based on 
dfs.domain.socket.disable.interval.seconds.
{code}

{code:java}
WARN  [RpcServer.default.FPRWQ.Fifo.read.handler=157,queue=16,port=16022] 
impl.BlockReaderFactory: I/O error constructing remote block reader for block 
BP-..:blk_..
java.io.IOException: Got error, status=ERROR, status message opReadBlock 
BP-..:blk_.. received exception 
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not 
found for BP-..:blk_... The block may have been removed recently by the 
balancer or by intentionally reducing the replication factor. This condition is 
usually harmless. To be certain, please check the preceding datanode log 
messages for signs of a more serious issue., for OP_READ_BLOCK, 
self=/<ip>:<port>, remote=/<ip>:<port>, for file 
/hbase/data/<ns>/<table>/<region>/<col>/<hfile>, for pool BP-.. block ..
        at 
org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:128)
 ~[hadoop-hdfs-client-3.4.1.jar:?]
        at 
org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:104)
 ~[hadoop-hdfs-client-3.4.1.jar:?]
        at 
org.apache.hadoop.hdfs.client.impl.BlockReaderRemote.checkSuccess(BlockReaderRemote.java:452)
 ~[hadoop-hdfs-client-3.4.1.jar:?]
        at 
org.apache.hadoop.hdfs.client.impl.BlockReaderRemote.newBlockReader(BlockReaderRemote.java:420)
 ~[hadoop-hdfs-client-3.4.1.jar:?]
        at 
org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:861)
 ~[hadoop-hdfs-client-3.4.1.jar:?]
        at 
org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:757)
 ~[hadoop-hdfs-client-3.4.1.jar:?]
        at 
org.apache.hadoop.hdfs.client.impl.BlockReaderFactory.build(BlockReaderFactory.java:381)
 ~[hadoop-hdfs-client-3.4.1.jar:?]
        at 
org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:715) 
~[hadoop-hdfs-client-3.4.1.jar:?]
{code}

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to