[
https://issues.apache.org/jira/browse/HDFS-16519?focusedWorklogId=751301&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-751301
]
ASF GitHub Bot logged work on HDFS-16519:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 31/Mar/22 23:58
Start Date: 31/Mar/22 23:58
Worklog Time Spent: 10m
Work Description: tomscut commented on pull request #4101:
URL: https://github.com/apache/hadoop/pull/4101#issuecomment-1085245717
> @tomscut Thanks for your review. Yes, background reconstruction is
necessary. With missing block(s) of block group, client has to read `dataUnits`
blocks in block group to decode the target data. This causes `dataUnits` times
of network traffic and corresponding cpu costs. In our product cluster, we
encountered backgroud reconstruction exhausts the network traffic of DN node.
This leads to the failure of client read/write. The speed of read/write and
reconstruction is in conflict, we can not have both of them actually. And in
practice, we set throttler of EC reconstruction to half of available network
traffic, for example, 500 MB/s to 1GB/s NIC. In this PR, we set the default
value to 0, which indicates there is no limit to reconstruction at all, to be
sure there is no side effect to upgrade.
Thanks @cndaimin for your explanation.
Hi @tasanuma @Hexiaoqiao, could you please give me some advice? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 751301)
Time Spent: 50m (was: 40m)
> Add throttler to EC reconstruction
> ----------------------------------
>
> Key: HDFS-16519
> URL: https://issues.apache.org/jira/browse/HDFS-16519
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, ec
> Affects Versions: 3.3.1, 3.3.2
> Reporter: daimin
> Assignee: daimin
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> HDFS already have throttlers for data transfer(replication) and balancer, the
> throttlers reduce the impact of these background procedures to user
> read/write.
> We should add a throttler to EC background reconstruction too.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
