[
https://issues.apache.org/jira/browse/HDFS-17052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17737148#comment-17737148
]
ASF GitHub Bot commented on HDFS-17052:
---------------------------------------
zhtttylz commented on PR #5759:
URL: https://github.com/apache/hadoop/pull/5759#issuecomment-1607490022
> Great catch here! It's make sense to me. I have some thoughts to discuss
with you. The solution here may involve multiple calls to `chooseOnce`, some of
which may be unnecessary and waste some time. The root cause of this problem is
that `BlockPlacementPolicyDefault#getMaxNodesPerRack` cannot return an accurate
value for `maxNodesPerRack`. How about compute a right value before the loop in
`chooseEvenlyFromRemainingRacks`? Will this be more efficient?
>
> ```
> private void chooseEvenlyFromRemainingRacks(Node writer,
> Set<Node> excludedNodes, long blocksize, int maxNodesPerRack,
> List<DatanodeStorageInfo> results, boolean avoidStaleNodes,
> EnumMap<StorageType, Integer> storageTypes, int totalReplicaExpected,
> NotEnoughReplicasException e) throws NotEnoughReplicasException {
> int numResultsOflastChoose = 0;
> NotEnoughReplicasException lastException = e;
> int bestEffortMaxNodesPerRack = maxNodesPerRack;
> Map<String, Integer> nodesPerRack = new HashMap<>();
> for (DatanodeStorageInfo dsInfo : results) {
> String rackName =
dsInfo.getDatanodeDescriptor().getNetworkLocation();
> nodesPerRack.merge(rackName, 1, Integer::sum);
> }
> for (int numNodes : nodesPerRack.values()) {
> if (numNodes > bestEffortMaxNodesPerRack) {
> bestEffortMaxNodesPerRack = numNodes;
> }
> }
> while (results.size() != totalReplicaExpected &&
> numResultsOflastChoose != results.size()) {
> ```
Thank you for your review. We appreciate your suggestion. However, in
situations where the number of available racks is insufficient to meet the
requirements of the Erasure Coding storage type, each write operation would
trigger the invocation of the
`BlockPlacementPolicyRackFaultTolerant#chooseEvenlyFromRemainingRacks` method.
It's important to note that each invocation of this method involves the
calculation of `chooseEvenlyFromRemainingRacks`.We are uncertain about the
potential efficiency implications of adopting this approach.
> Erasure coding reconstruction failed when num of storageType rack NOT enough
> ----------------------------------------------------------------------------
>
> Key: HDFS-17052
> URL: https://issues.apache.org/jira/browse/HDFS-17052
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ec
> Affects Versions: 3.4.0
> Reporter: Hualong Zhang
> Assignee: Hualong Zhang
> Priority: Major
> Labels: pull-request-available
> Attachments: failed reconstruction ec in same rack-1.png, write ec in
> same rack.png
>
>
> When writing EC data, if the number of racks matching the storageType is
> insufficient, more than one block are allowed to be written to the same rack
> !write ec in same rack.png|width=962,height=604!
>
>
>
> However, during EC block recovery, it is not possible to recover on the same
> rack, which deviates from the expected behavior.
> !failed reconstruction ec in same rack-1.png|width=946,height=413!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
