Longping Jie created HDFS-17676:
-----------------------------------
Summary: The frequency of submitting DFSInputStream refresh block
locations in the LocatedBlocksRefresher thread is too fast.
Key: HDFS-17676
URL: https://issues.apache.org/jira/browse/HDFS-17676
Project: Hadoop HDFS
Issue Type: Improvement
Components: dfsclient
Affects Versions: 3.3.6
Reporter: Longping Jie
Attachments: image-2024-11-29-10-12-45-483.png
LocatedBlocksRefresher thread is responsible for submitting the task of
DFSInputStream refresh block locations.
When a region server opens tens of thousands of DFSInputstreams, these
DFSInputstream refresh block locations will be executed in a short period of
time, which will put great pressure on the datanode, create a large number of
DataXceiver threads in a short period of time, and cause GC fluctuations,
resulting in time-consuming hbase queries. Burr appears
!image-2024-11-29-10-12-45-483.png!
Even if you set the size of refreshThreadPool to 1, you can't limit the rate
well, consider adding a rate limiting mechanism, which can be enabled by
parameter configuration to limit the refresh frequency.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
