[jira] [Updated] (HDFS-10966) Enhance Dispatcher logic on deciding when to give up a source DataNode
[
https://issues.apache.org/jira/browse/HDFS-10966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kihwal Lee updated HDFS-10966:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
2.7.4
Status: Resolved (was: Patch Available)
> Enhance Dispatcher logic on deciding when to give up a source DataNode
> --
>
> Key: HDFS-10966
> URL: https://issues.apache.org/jira/browse/HDFS-10966
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Reporter: Zhe Zhang
>Assignee: Mark Wagner
> Fix For: 2.7.4, 3.0.0-alpha2
>
> Attachments: HDFS-10966-branch-2.7.00.patch, HDFS-10966.00.patch,
> HDFS-10966.01.patch
>
>
> When a {{Dispatcher}} thread works on a source DataNode, in each iteration it
> tries to execute a {{PendingMove}}. If no block is moved after 5 iterations,
> this source (over-utlized) DataNode is given up for this Balancer iteration
> (20 mins). This is problematic if the source DataNode was heavily loaded in
> the beginning of the iteration. It will quickly encounter 5 unsuccessful
> moves and be abandoned.
> We should enhance this logic by e.g. using elapsed time instead of number of
> iterations.
> {code}
> // Check if the previous move was successful
> } else {
> // source node cannot find a pending block to move, iteration +1
> noPendingMoveIteration++;
> // in case no blocks can be moved for source node's task,
> // jump out of while-loop after 5 iterations.
> if (noPendingMoveIteration >= MAX_NO_PENDING_MOVE_ITERATIONS) {
> LOG.info("Failed to find a pending move " +
> noPendingMoveIteration
> + " times. Skipping " + this);
> resetScheduledSize();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10966) Enhance Dispatcher logic on deciding when to give up a source DataNode
[
https://issues.apache.org/jira/browse/HDFS-10966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-10966:
-
Attachment: HDFS-10966-branch-2.7.00.patch
Thanks [~kihwal] for the final review! Attaching branch-2.7 patch.
> Enhance Dispatcher logic on deciding when to give up a source DataNode
> --
>
> Key: HDFS-10966
> URL: https://issues.apache.org/jira/browse/HDFS-10966
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Reporter: Zhe Zhang
>Assignee: Mark Wagner
> Attachments: HDFS-10966-branch-2.7.00.patch, HDFS-10966.00.patch,
> HDFS-10966.01.patch
>
>
> When a {{Dispatcher}} thread works on a source DataNode, in each iteration it
> tries to execute a {{PendingMove}}. If no block is moved after 5 iterations,
> this source (over-utlized) DataNode is given up for this Balancer iteration
> (20 mins). This is problematic if the source DataNode was heavily loaded in
> the beginning of the iteration. It will quickly encounter 5 unsuccessful
> moves and be abandoned.
> We should enhance this logic by e.g. using elapsed time instead of number of
> iterations.
> {code}
> // Check if the previous move was successful
> } else {
> // source node cannot find a pending block to move, iteration +1
> noPendingMoveIteration++;
> // in case no blocks can be moved for source node's task,
> // jump out of while-loop after 5 iterations.
> if (noPendingMoveIteration >= MAX_NO_PENDING_MOVE_ITERATIONS) {
> LOG.info("Failed to find a pending move " +
> noPendingMoveIteration
> + " times. Skipping " + this);
> resetScheduledSize();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10966) Enhance Dispatcher logic on deciding when to give up a source DataNode
[
https://issues.apache.org/jira/browse/HDFS-10966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-10966:
-
Attachment: HDFS-10966.01.patch
Thanks Kihwal for the review! Uploading a new patch:
# This patch does change the Balancer behavior introduced by HDFS-4261, around
the timeout logic. But I don't think there's a negative effect. By staying in
the {{dispatchBlocks}} while loop longer, the only overhead is to check
{{chooseNextMove}}, which only checks local states, without issuing NameNode
workload. Even if we jump out of the while loop, the thread for that Source
cannot be reused at another Source anyway.
# Added to {{hdfs-default.xml}}, thx for the catch.
# I think it is a good idea, added.
> Enhance Dispatcher logic on deciding when to give up a source DataNode
> --
>
> Key: HDFS-10966
> URL: https://issues.apache.org/jira/browse/HDFS-10966
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Reporter: Zhe Zhang
>Assignee: Mark Wagner
> Attachments: HDFS-10966.00.patch, HDFS-10966.01.patch
>
>
> When a {{Dispatcher}} thread works on a source DataNode, in each iteration it
> tries to execute a {{PendingMove}}. If no block is moved after 5 iterations,
> this source (over-utlized) DataNode is given up for this Balancer iteration
> (20 mins). This is problematic if the source DataNode was heavily loaded in
> the beginning of the iteration. It will quickly encounter 5 unsuccessful
> moves and be abandoned.
> We should enhance this logic by e.g. using elapsed time instead of number of
> iterations.
> {code}
> // Check if the previous move was successful
> } else {
> // source node cannot find a pending block to move, iteration +1
> noPendingMoveIteration++;
> // in case no blocks can be moved for source node's task,
> // jump out of while-loop after 5 iterations.
> if (noPendingMoveIteration >= MAX_NO_PENDING_MOVE_ITERATIONS) {
> LOG.info("Failed to find a pending move " +
> noPendingMoveIteration
> + " times. Skipping " + this);
> resetScheduledSize();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10966) Enhance Dispatcher logic on deciding when to give up a source DataNode
[
https://issues.apache.org/jira/browse/HDFS-10966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-10966:
-
Attachment: HDFS-10966.00.patch
> Enhance Dispatcher logic on deciding when to give up a source DataNode
> --
>
> Key: HDFS-10966
> URL: https://issues.apache.org/jira/browse/HDFS-10966
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Reporter: Zhe Zhang
>Assignee: Mark Wagner
> Attachments: HDFS-10966.00.patch
>
>
> When a {{Dispatcher}} thread works on a source DataNode, in each iteration it
> tries to execute a {{PendingMove}}. If no block is moved after 5 iterations,
> this source (over-utlized) DataNode is given up for this Balancer iteration
> (20 mins). This is problematic if the source DataNode was heavily loaded in
> the beginning of the iteration. It will quickly encounter 5 unsuccessful
> moves and be abandoned.
> We should enhance this logic by e.g. using elapsed time instead of number of
> iterations.
> {code}
> // Check if the previous move was successful
> } else {
> // source node cannot find a pending block to move, iteration +1
> noPendingMoveIteration++;
> // in case no blocks can be moved for source node's task,
> // jump out of while-loop after 5 iterations.
> if (noPendingMoveIteration >= MAX_NO_PENDING_MOVE_ITERATIONS) {
> LOG.info("Failed to find a pending move " +
> noPendingMoveIteration
> + " times. Skipping " + this);
> resetScheduledSize();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10966) Enhance Dispatcher logic on deciding when to give up a source DataNode
[
https://issues.apache.org/jira/browse/HDFS-10966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-10966:
-
Status: Patch Available (was: Open)
[~mwagner] did an internal patch (based on 2.6). I'm uploading on his behalf.
[~kihwal] Could you take a look? Thanks!
> Enhance Dispatcher logic on deciding when to give up a source DataNode
> --
>
> Key: HDFS-10966
> URL: https://issues.apache.org/jira/browse/HDFS-10966
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer & mover
>Reporter: Zhe Zhang
>Assignee: Mark Wagner
>
> When a {{Dispatcher}} thread works on a source DataNode, in each iteration it
> tries to execute a {{PendingMove}}. If no block is moved after 5 iterations,
> this source (over-utlized) DataNode is given up for this Balancer iteration
> (20 mins). This is problematic if the source DataNode was heavily loaded in
> the beginning of the iteration. It will quickly encounter 5 unsuccessful
> moves and be abandoned.
> We should enhance this logic by e.g. using elapsed time instead of number of
> iterations.
> {code}
> // Check if the previous move was successful
> } else {
> // source node cannot find a pending block to move, iteration +1
> noPendingMoveIteration++;
> // in case no blocks can be moved for source node's task,
> // jump out of while-loop after 5 iterations.
> if (noPendingMoveIteration >= MAX_NO_PENDING_MOVE_ITERATIONS) {
> LOG.info("Failed to find a pending move " +
> noPendingMoveIteration
> + " times. Skipping " + this);
> resetScheduledSize();
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
