[jira] [Updated] (HDFS-10598) DiskBalancer does not execute multi-steps plan.
[
https://issues.apache.org/jira/browse/HDFS-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-10598:
---
Fix Version/s: (was: 3.0.0-alpha2)
3.0.0-alpha1
> DiskBalancer does not execute multi-steps plan.
> ---
>
> Key: HDFS-10598
> URL: https://issues.apache.org/jira/browse/HDFS-10598
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: diskbalancer
>Affects Versions: 3.0.0-beta1
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Critical
> Fix For: 3.0.0-alpha1
>
> Attachments: HDFS-10598.00.patch
>
>
> I set up a 3 DN node cluster, each one with 2 small disks. After creating
> some files to fill HDFS, I added two more small disks to one DN. And run the
> diskbalancer on this DataNode.
> The disk usage before running diskbalancer:
> {code}
> /dev/loop0 3.9G 2.1G 1.6G 58% /mnt/data1
> /dev/loop1 3.9G 2.6G 1.1G 71% /mnt/data2
> /dev/loop2 3.9G 17M 3.6G 1% /mnt/data3
> /dev/loop3 3.9G 17M 3.6G 1% /mnt/data4
> {code}
> However, after running diskbalancer (i.e., {{-query}} shows {{PLAN_DONE}})
> {code}
> /dev/loop0 3.9G 1.2G 2.5G 32% /mnt/data1
> /dev/loop1 3.9G 2.6G 1.1G 71% /mnt/data2
> /dev/loop2 3.9G 953M 2.7G 26% /mnt/data3
> /dev/loop3 3.9G 17M 3.6G 1% /mnt/data4
> {code}
> It is suspicious that in {{DiskBalancerMover#copyBlocks}}, every return does
> {{this.setExitFlag}} which prevents {{copyBlocks()}} be called multiple times
> from {{DiskBalancer#executePlan}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10598) DiskBalancer does not execute multi-steps plan.
[
https://issues.apache.org/jira/browse/HDFS-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-10598:
-
Assignee: Lei (Eddy) Xu (was: Anu Engineer)
> DiskBalancer does not execute multi-steps plan.
> ---
>
> Key: HDFS-10598
> URL: https://issues.apache.org/jira/browse/HDFS-10598
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: diskbalancer
>Affects Versions: 3.0.0-beta1
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Critical
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10598.00.patch
>
>
> I set up a 3 DN node cluster, each one with 2 small disks. After creating
> some files to fill HDFS, I added two more small disks to one DN. And run the
> diskbalancer on this DataNode.
> The disk usage before running diskbalancer:
> {code}
> /dev/loop0 3.9G 2.1G 1.6G 58% /mnt/data1
> /dev/loop1 3.9G 2.6G 1.1G 71% /mnt/data2
> /dev/loop2 3.9G 17M 3.6G 1% /mnt/data3
> /dev/loop3 3.9G 17M 3.6G 1% /mnt/data4
> {code}
> However, after running diskbalancer (i.e., {{-query}} shows {{PLAN_DONE}})
> {code}
> /dev/loop0 3.9G 1.2G 2.5G 32% /mnt/data1
> /dev/loop1 3.9G 2.6G 1.1G 71% /mnt/data2
> /dev/loop2 3.9G 953M 2.7G 26% /mnt/data3
> /dev/loop3 3.9G 17M 3.6G 1% /mnt/data4
> {code}
> It is suspicious that in {{DiskBalancerMover#copyBlocks}}, every return does
> {{this.setExitFlag}} which prevents {{copyBlocks()}} be called multiple times
> from {{DiskBalancer#executePlan}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10598) DiskBalancer does not execute multi-steps plan.
[
https://issues.apache.org/jira/browse/HDFS-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-10598:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.0.0-alpha2
Status: Resolved (was: Patch Available)
Thanks again [~eddyxu] for the contribution. I committed this patch to trunk.
> DiskBalancer does not execute multi-steps plan.
> ---
>
> Key: HDFS-10598
> URL: https://issues.apache.org/jira/browse/HDFS-10598
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: diskbalancer
>Affects Versions: 3.0.0-beta1
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Critical
> Fix For: 3.0.0-alpha2
>
> Attachments: HDFS-10598.00.patch
>
>
> I set up a 3 DN node cluster, each one with 2 small disks. After creating
> some files to fill HDFS, I added two more small disks to one DN. And run the
> diskbalancer on this DataNode.
> The disk usage before running diskbalancer:
> {code}
> /dev/loop0 3.9G 2.1G 1.6G 58% /mnt/data1
> /dev/loop1 3.9G 2.6G 1.1G 71% /mnt/data2
> /dev/loop2 3.9G 17M 3.6G 1% /mnt/data3
> /dev/loop3 3.9G 17M 3.6G 1% /mnt/data4
> {code}
> However, after running diskbalancer (i.e., {{-query}} shows {{PLAN_DONE}})
> {code}
> /dev/loop0 3.9G 1.2G 2.5G 32% /mnt/data1
> /dev/loop1 3.9G 2.6G 1.1G 71% /mnt/data2
> /dev/loop2 3.9G 953M 2.7G 26% /mnt/data3
> /dev/loop3 3.9G 17M 3.6G 1% /mnt/data4
> {code}
> It is suspicious that in {{DiskBalancerMover#copyBlocks}}, every return does
> {{this.setExitFlag}} which prevents {{copyBlocks()}} be called multiple times
> from {{DiskBalancer#executePlan}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10598) DiskBalancer does not execute multi-steps plan.
[
https://issues.apache.org/jira/browse/HDFS-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-10598:
-
Affects Version/s: (was: 2.8.0)
> DiskBalancer does not execute multi-steps plan.
> ---
>
> Key: HDFS-10598
> URL: https://issues.apache.org/jira/browse/HDFS-10598
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: diskbalancer
>Affects Versions: 3.0.0-beta1
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Critical
> Attachments: HDFS-10598.00.patch
>
>
> I set up a 3 DN node cluster, each one with 2 small disks. After creating
> some files to fill HDFS, I added two more small disks to one DN. And run the
> diskbalancer on this DataNode.
> The disk usage before running diskbalancer:
> {code}
> /dev/loop0 3.9G 2.1G 1.6G 58% /mnt/data1
> /dev/loop1 3.9G 2.6G 1.1G 71% /mnt/data2
> /dev/loop2 3.9G 17M 3.6G 1% /mnt/data3
> /dev/loop3 3.9G 17M 3.6G 1% /mnt/data4
> {code}
> However, after running diskbalancer (i.e., {{-query}} shows {{PLAN_DONE}})
> {code}
> /dev/loop0 3.9G 1.2G 2.5G 32% /mnt/data1
> /dev/loop1 3.9G 2.6G 1.1G 71% /mnt/data2
> /dev/loop2 3.9G 953M 2.7G 26% /mnt/data3
> /dev/loop3 3.9G 17M 3.6G 1% /mnt/data4
> {code}
> It is suspicious that in {{DiskBalancerMover#copyBlocks}}, every return does
> {{this.setExitFlag}} which prevents {{copyBlocks()}} be called multiple times
> from {{DiskBalancer#executePlan}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10598) DiskBalancer does not execute multi-steps plan.
[
https://issues.apache.org/jira/browse/HDFS-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-10598:
-
Target Version/s: 3.0.0-beta1 (was: 2.9.0, 3.0.0-beta1)
> DiskBalancer does not execute multi-steps plan.
> ---
>
> Key: HDFS-10598
> URL: https://issues.apache.org/jira/browse/HDFS-10598
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: diskbalancer
>Affects Versions: 3.0.0-beta1
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Critical
> Attachments: HDFS-10598.00.patch
>
>
> I set up a 3 DN node cluster, each one with 2 small disks. After creating
> some files to fill HDFS, I added two more small disks to one DN. And run the
> diskbalancer on this DataNode.
> The disk usage before running diskbalancer:
> {code}
> /dev/loop0 3.9G 2.1G 1.6G 58% /mnt/data1
> /dev/loop1 3.9G 2.6G 1.1G 71% /mnt/data2
> /dev/loop2 3.9G 17M 3.6G 1% /mnt/data3
> /dev/loop3 3.9G 17M 3.6G 1% /mnt/data4
> {code}
> However, after running diskbalancer (i.e., {{-query}} shows {{PLAN_DONE}})
> {code}
> /dev/loop0 3.9G 1.2G 2.5G 32% /mnt/data1
> /dev/loop1 3.9G 2.6G 1.1G 71% /mnt/data2
> /dev/loop2 3.9G 953M 2.7G 26% /mnt/data3
> /dev/loop3 3.9G 17M 3.6G 1% /mnt/data4
> {code}
> It is suspicious that in {{DiskBalancerMover#copyBlocks}}, every return does
> {{this.setExitFlag}} which prevents {{copyBlocks()}} be called multiple times
> from {{DiskBalancer#executePlan}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10598) DiskBalancer does not execute multi-steps plan.
[
https://issues.apache.org/jira/browse/HDFS-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HDFS-10598:
-
Assignee: Lei (Eddy) Xu (was: Anu Engineer)
Fix Version/s: (was: 2.9.0)
Status: Patch Available (was: Open)
> DiskBalancer does not execute multi-steps plan.
> ---
>
> Key: HDFS-10598
> URL: https://issues.apache.org/jira/browse/HDFS-10598
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: diskbalancer
>Affects Versions: 2.8.0, 3.0.0-beta1
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Critical
> Attachments: HDFS-10598.00.patch
>
>
> I set up a 3 DN node cluster, each one with 2 small disks. After creating
> some files to fill HDFS, I added two more small disks to one DN. And run the
> diskbalancer on this DataNode.
> The disk usage before running diskbalancer:
> {code}
> /dev/loop0 3.9G 2.1G 1.6G 58% /mnt/data1
> /dev/loop1 3.9G 2.6G 1.1G 71% /mnt/data2
> /dev/loop2 3.9G 17M 3.6G 1% /mnt/data3
> /dev/loop3 3.9G 17M 3.6G 1% /mnt/data4
> {code}
> However, after running diskbalancer (i.e., {{-query}} shows {{PLAN_DONE}})
> {code}
> /dev/loop0 3.9G 1.2G 2.5G 32% /mnt/data1
> /dev/loop1 3.9G 2.6G 1.1G 71% /mnt/data2
> /dev/loop2 3.9G 953M 2.7G 26% /mnt/data3
> /dev/loop3 3.9G 17M 3.6G 1% /mnt/data4
> {code}
> It is suspicious that in {{DiskBalancerMover#copyBlocks}}, every return does
> {{this.setExitFlag}} which prevents {{copyBlocks()}} be called multiple times
> from {{DiskBalancer#executePlan}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-10598) DiskBalancer does not execute multi-steps plan.
[
https://issues.apache.org/jira/browse/HDFS-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HDFS-10598:
-
Attachment: HDFS-10598.00.patch
Upload the patch that changes {{DiskBalancerMover#copyBlocks}} to not
{{setExitFlag}} for normal exit case. And it {{setExitFlag}} from
{{executePlan()}}.
However, whether it needs to {{setExitFlag()}} in {{executePlan()}} is unclear
to me. [~anu] could you give some inputs of the cases it were designed for?
Thanks.
> DiskBalancer does not execute multi-steps plan.
> ---
>
> Key: HDFS-10598
> URL: https://issues.apache.org/jira/browse/HDFS-10598
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: diskbalancer
>Affects Versions: 2.8.0, 3.0.0-beta1
>Reporter: Lei (Eddy) Xu
>Assignee: Anu Engineer
>Priority: Critical
> Fix For: 2.9.0
>
> Attachments: HDFS-10598.00.patch
>
>
> I set up a 3 DN node cluster, each one with 2 small disks. After creating
> some files to fill HDFS, I added two more small disks to one DN. And run the
> diskbalancer on this DataNode.
> The disk usage before running diskbalancer:
> {code}
> /dev/loop0 3.9G 2.1G 1.6G 58% /mnt/data1
> /dev/loop1 3.9G 2.6G 1.1G 71% /mnt/data2
> /dev/loop2 3.9G 17M 3.6G 1% /mnt/data3
> /dev/loop3 3.9G 17M 3.6G 1% /mnt/data4
> {code}
> However, after running diskbalancer (i.e., {{-query}} shows {{PLAN_DONE}})
> {code}
> /dev/loop0 3.9G 1.2G 2.5G 32% /mnt/data1
> /dev/loop1 3.9G 2.6G 1.1G 71% /mnt/data2
> /dev/loop2 3.9G 953M 2.7G 26% /mnt/data3
> /dev/loop3 3.9G 17M 3.6G 1% /mnt/data4
> {code}
> It is suspicious that in {{DiskBalancerMover#copyBlocks}}, every return does
> {{this.setExitFlag}} which prevents {{copyBlocks()}} be called multiple times
> from {{DiskBalancer#executePlan}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
