[jira] [Updated] (HDFS-9922) Upgrade Domain placement policy status marks a good block in violation when there are decommissioned nodes
[
https://issues.apache.org/jira/browse/HDFS-9922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-9922:
--
Fix Version/s: 2.9.0
> Upgrade Domain placement policy status marks a good block in violation when
> there are decommissioned nodes
> --
>
> Key: HDFS-9922
> URL: https://issues.apache.org/jira/browse/HDFS-9922
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Minor
> Fix For: 2.9.0, 3.0.0-alpha1, 2.8.2
>
> Attachments: HDFS-9922.branch-2.8.001.patch,
> HDFS-9922-trunk-v1.patch, HDFS-9922-trunk-v2.patch, HDFS-9922-trunk-v3.patch,
> HDFS-9922-trunk-v4.patch
>
>
> When there are replicas of a block on a decommissioned node,
> BlockPlacementStatusWithUpgradeDomain#isUpgradeDomainPolicySatisfied returns
> false when it should return true. This is because numberOfReplicas is the
> number of in-service replicas for the block and upgradeDomains.size() is the
> number of upgrade domains across all replicas of the block. Specifically, we
> hit this scenario when numberOfReplicas is equal to upgradeDomainFactor and
> upgradeDomains.size() is greater than numberOfReplicas.
> {code}
> private boolean isUpgradeDomainPolicySatisfied() {
> if (numberOfReplicas <= upgradeDomainFactor) {
> return (numberOfReplicas == upgradeDomains.size());
> } else {
> return upgradeDomains.size() >= upgradeDomainFactor;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9922) Upgrade Domain placement policy status marks a good block in violation when there are decommissioned nodes
[
https://issues.apache.org/jira/browse/HDFS-9922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ming Ma updated HDFS-9922:
--
Resolution: Fixed
Fix Version/s: (was: 2.9.0)
2.8.2
Status: Resolved (was: Patch Available)
Backported to branch-2.8 per the discussion in the umbrella jira. The failed
tests pass locally.
> Upgrade Domain placement policy status marks a good block in violation when
> there are decommissioned nodes
> --
>
> Key: HDFS-9922
> URL: https://issues.apache.org/jira/browse/HDFS-9922
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Minor
> Fix For: 2.8.2, 3.0.0-alpha1
>
> Attachments: HDFS-9922.branch-2.8.001.patch,
> HDFS-9922-trunk-v1.patch, HDFS-9922-trunk-v2.patch, HDFS-9922-trunk-v3.patch,
> HDFS-9922-trunk-v4.patch
>
>
> When there are replicas of a block on a decommissioned node,
> BlockPlacementStatusWithUpgradeDomain#isUpgradeDomainPolicySatisfied returns
> false when it should return true. This is because numberOfReplicas is the
> number of in-service replicas for the block and upgradeDomains.size() is the
> number of upgrade domains across all replicas of the block. Specifically, we
> hit this scenario when numberOfReplicas is equal to upgradeDomainFactor and
> upgradeDomains.size() is greater than numberOfReplicas.
> {code}
> private boolean isUpgradeDomainPolicySatisfied() {
> if (numberOfReplicas <= upgradeDomainFactor) {
> return (numberOfReplicas == upgradeDomains.size());
> } else {
> return upgradeDomains.size() >= upgradeDomainFactor;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9922) Upgrade Domain placement policy status marks a good block in violation when there are decommissioned nodes
[
https://issues.apache.org/jira/browse/HDFS-9922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ming Ma updated HDFS-9922:
--
Attachment: HDFS-9922.branch-2.8.001.patch
> Upgrade Domain placement policy status marks a good block in violation when
> there are decommissioned nodes
> --
>
> Key: HDFS-9922
> URL: https://issues.apache.org/jira/browse/HDFS-9922
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Minor
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: HDFS-9922.branch-2.8.001.patch,
> HDFS-9922-trunk-v1.patch, HDFS-9922-trunk-v2.patch, HDFS-9922-trunk-v3.patch,
> HDFS-9922-trunk-v4.patch
>
>
> When there are replicas of a block on a decommissioned node,
> BlockPlacementStatusWithUpgradeDomain#isUpgradeDomainPolicySatisfied returns
> false when it should return true. This is because numberOfReplicas is the
> number of in-service replicas for the block and upgradeDomains.size() is the
> number of upgrade domains across all replicas of the block. Specifically, we
> hit this scenario when numberOfReplicas is equal to upgradeDomainFactor and
> upgradeDomains.size() is greater than numberOfReplicas.
> {code}
> private boolean isUpgradeDomainPolicySatisfied() {
> if (numberOfReplicas <= upgradeDomainFactor) {
> return (numberOfReplicas == upgradeDomains.size());
> } else {
> return upgradeDomains.size() >= upgradeDomainFactor;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9922) Upgrade Domain placement policy status marks a good block in violation when there are decommissioned nodes
[
https://issues.apache.org/jira/browse/HDFS-9922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ming Ma updated HDFS-9922:
--
Status: Patch Available (was: Reopened)
> Upgrade Domain placement policy status marks a good block in violation when
> there are decommissioned nodes
> --
>
> Key: HDFS-9922
> URL: https://issues.apache.org/jira/browse/HDFS-9922
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Minor
> Fix For: 2.9.0, 3.0.0-alpha1
>
> Attachments: HDFS-9922.branch-2.8.001.patch,
> HDFS-9922-trunk-v1.patch, HDFS-9922-trunk-v2.patch, HDFS-9922-trunk-v3.patch,
> HDFS-9922-trunk-v4.patch
>
>
> When there are replicas of a block on a decommissioned node,
> BlockPlacementStatusWithUpgradeDomain#isUpgradeDomainPolicySatisfied returns
> false when it should return true. This is because numberOfReplicas is the
> number of in-service replicas for the block and upgradeDomains.size() is the
> number of upgrade domains across all replicas of the block. Specifically, we
> hit this scenario when numberOfReplicas is equal to upgradeDomainFactor and
> upgradeDomains.size() is greater than numberOfReplicas.
> {code}
> private boolean isUpgradeDomainPolicySatisfied() {
> if (numberOfReplicas <= upgradeDomainFactor) {
> return (numberOfReplicas == upgradeDomains.size());
> } else {
> return upgradeDomains.size() >= upgradeDomainFactor;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9922) Upgrade Domain placement policy status marks a good block in violation when there are decommissioned nodes
[
https://issues.apache.org/jira/browse/HDFS-9922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ming Ma updated HDFS-9922:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.9.0
Status: Resolved (was: Patch Available)
I have committed the patch to trunk and branch-2. Thanks [~ctrezzo] for the
contribution and [~eddyxu] for the review.
> Upgrade Domain placement policy status marks a good block in violation when
> there are decommissioned nodes
> --
>
> Key: HDFS-9922
> URL: https://issues.apache.org/jira/browse/HDFS-9922
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Minor
> Fix For: 2.9.0
>
> Attachments: HDFS-9922-trunk-v1.patch, HDFS-9922-trunk-v2.patch,
> HDFS-9922-trunk-v3.patch, HDFS-9922-trunk-v4.patch
>
>
> When there are replicas of a block on a decommissioned node,
> BlockPlacementStatusWithUpgradeDomain#isUpgradeDomainPolicySatisfied returns
> false when it should return true. This is because numberOfReplicas is the
> number of in-service replicas for the block and upgradeDomains.size() is the
> number of upgrade domains across all replicas of the block. Specifically, we
> hit this scenario when numberOfReplicas is equal to upgradeDomainFactor and
> upgradeDomains.size() is greater than numberOfReplicas.
> {code}
> private boolean isUpgradeDomainPolicySatisfied() {
> if (numberOfReplicas <= upgradeDomainFactor) {
> return (numberOfReplicas == upgradeDomains.size());
> } else {
> return upgradeDomains.size() >= upgradeDomainFactor;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9922) Upgrade Domain placement policy status marks a good block in violation when there are decommissioned nodes
[
https://issues.apache.org/jira/browse/HDFS-9922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ming Ma updated HDFS-9922:
--
Attachment: HDFS-9922-trunk-v4.patch
Given Chris is on vacation and the patch is almost ready, here is the updated
patch that fixes the checkstyle issue. [~eddyxu], do you have time to review
it? Thanks.
> Upgrade Domain placement policy status marks a good block in violation when
> there are decommissioned nodes
> --
>
> Key: HDFS-9922
> URL: https://issues.apache.org/jira/browse/HDFS-9922
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chris Trezzo
>Assignee: Ming Ma
>Priority: Minor
> Attachments: HDFS-9922-trunk-v1.patch, HDFS-9922-trunk-v2.patch,
> HDFS-9922-trunk-v3.patch, HDFS-9922-trunk-v4.patch
>
>
> When there are replicas of a block on a decommissioned node,
> BlockPlacementStatusWithUpgradeDomain#isUpgradeDomainPolicySatisfied returns
> false when it should return true. This is because numberOfReplicas is the
> number of in-service replicas for the block and upgradeDomains.size() is the
> number of upgrade domains across all replicas of the block. Specifically, we
> hit this scenario when numberOfReplicas is equal to upgradeDomainFactor and
> upgradeDomains.size() is greater than numberOfReplicas.
> {code}
> private boolean isUpgradeDomainPolicySatisfied() {
> if (numberOfReplicas <= upgradeDomainFactor) {
> return (numberOfReplicas == upgradeDomains.size());
> } else {
> return upgradeDomains.size() >= upgradeDomainFactor;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9922) Upgrade Domain placement policy status marks a good block in violation when there are decommissioned nodes
[
https://issues.apache.org/jira/browse/HDFS-9922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ming Ma updated HDFS-9922:
--
Assignee: Chris Trezzo (was: Ming Ma)
> Upgrade Domain placement policy status marks a good block in violation when
> there are decommissioned nodes
> --
>
> Key: HDFS-9922
> URL: https://issues.apache.org/jira/browse/HDFS-9922
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Minor
> Attachments: HDFS-9922-trunk-v1.patch, HDFS-9922-trunk-v2.patch,
> HDFS-9922-trunk-v3.patch, HDFS-9922-trunk-v4.patch
>
>
> When there are replicas of a block on a decommissioned node,
> BlockPlacementStatusWithUpgradeDomain#isUpgradeDomainPolicySatisfied returns
> false when it should return true. This is because numberOfReplicas is the
> number of in-service replicas for the block and upgradeDomains.size() is the
> number of upgrade domains across all replicas of the block. Specifically, we
> hit this scenario when numberOfReplicas is equal to upgradeDomainFactor and
> upgradeDomains.size() is greater than numberOfReplicas.
> {code}
> private boolean isUpgradeDomainPolicySatisfied() {
> if (numberOfReplicas <= upgradeDomainFactor) {
> return (numberOfReplicas == upgradeDomains.size());
> } else {
> return upgradeDomains.size() >= upgradeDomainFactor;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9922) Upgrade Domain placement policy status marks a good block in violation when there are decommissioned nodes
[
https://issues.apache.org/jira/browse/HDFS-9922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Trezzo updated HDFS-9922:
---
Attachment: HDFS-9922-trunk-v3.patch
V3 attached.
# Fixed import issues in TestUpgradeDomainBlockPlacementPolicy.
# Removed extra live node filter in BlockPlacementPolicyWithUpgradeDomain.
> Upgrade Domain placement policy status marks a good block in violation when
> there are decommissioned nodes
> --
>
> Key: HDFS-9922
> URL: https://issues.apache.org/jira/browse/HDFS-9922
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Minor
> Attachments: HDFS-9922-trunk-v1.patch, HDFS-9922-trunk-v2.patch,
> HDFS-9922-trunk-v3.patch
>
>
> When there are replicas of a block on a decommissioned node,
> BlockPlacementStatusWithUpgradeDomain#isUpgradeDomainPolicySatisfied returns
> false when it should return true. This is because numberOfReplicas is the
> number of in-service replicas for the block and upgradeDomains.size() is the
> number of upgrade domains across all replicas of the block. Specifically, we
> hit this scenario when numberOfReplicas is equal to upgradeDomainFactor and
> upgradeDomains.size() is greater than numberOfReplicas.
> {code}
> private boolean isUpgradeDomainPolicySatisfied() {
> if (numberOfReplicas <= upgradeDomainFactor) {
> return (numberOfReplicas == upgradeDomains.size());
> } else {
> return upgradeDomains.size() >= upgradeDomainFactor;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9922) Upgrade Domain placement policy status marks a good block in violation when there are decommissioned nodes
[
https://issues.apache.org/jira/browse/HDFS-9922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Trezzo updated HDFS-9922:
---
Attachment: HDFS-9922-trunk-v2.patch
V2 attached.
# Spoke with [~mingma] offline and will not modify the name of
{{numOfReplicas}} since this might not be the replication factor when erasure
encoding is in use.
# Modified the condition in BlockPlacementStatusWithUpgradeDomain.
# Added more tests in TestUpgradeDomainBlockPlacementPolicy.
> Upgrade Domain placement policy status marks a good block in violation when
> there are decommissioned nodes
> --
>
> Key: HDFS-9922
> URL: https://issues.apache.org/jira/browse/HDFS-9922
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Minor
> Attachments: HDFS-9922-trunk-v1.patch, HDFS-9922-trunk-v2.patch
>
>
> When there are replicas of a block on a decommissioned node,
> BlockPlacementStatusWithUpgradeDomain#isUpgradeDomainPolicySatisfied returns
> false when it should return true. This is because numberOfReplicas is the
> number of in-service replicas for the block and upgradeDomains.size() is the
> number of upgrade domains across all replicas of the block. Specifically, we
> hit this scenario when numberOfReplicas is equal to upgradeDomainFactor and
> upgradeDomains.size() is greater than numberOfReplicas.
> {code}
> private boolean isUpgradeDomainPolicySatisfied() {
> if (numberOfReplicas <= upgradeDomainFactor) {
> return (numberOfReplicas == upgradeDomains.size());
> } else {
> return upgradeDomains.size() >= upgradeDomainFactor;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-9922) Upgrade Domain placement policy status marks a good block in violation when there are decommissioned nodes
[
https://issues.apache.org/jira/browse/HDFS-9922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Trezzo updated HDFS-9922:
---
Status: Patch Available (was: In Progress)
> Upgrade Domain placement policy status marks a good block in violation when
> there are decommissioned nodes
> --
>
> Key: HDFS-9922
> URL: https://issues.apache.org/jira/browse/HDFS-9922
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Minor
> Attachments: HDFS-9922-trunk-v1.patch
>
>
> When there are replicas of a block on a decommissioned node,
> BlockPlacementStatusWithUpgradeDomain#isUpgradeDomainPolicySatisfied returns
> false when it should return true. This is because numberOfReplicas is the
> number of in-service replicas for the block and upgradeDomains.size() is the
> number of upgrade domains across all replicas of the block. Specifically, we
> hit this scenario when numberOfReplicas is equal to upgradeDomainFactor and
> upgradeDomains.size() is greater than numberOfReplicas.
> {code}
> private boolean isUpgradeDomainPolicySatisfied() {
> if (numberOfReplicas <= upgradeDomainFactor) {
> return (numberOfReplicas == upgradeDomains.size());
> } else {
> return upgradeDomains.size() >= upgradeDomainFactor;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9922) Upgrade Domain placement policy status marks a good block in violation when there are decommissioned nodes
[
https://issues.apache.org/jira/browse/HDFS-9922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Trezzo updated HDFS-9922:
---
Attachment: HDFS-9922-trunk-v1.patch
[~mingma] v1 patch attached.
> Upgrade Domain placement policy status marks a good block in violation when
> there are decommissioned nodes
> --
>
> Key: HDFS-9922
> URL: https://issues.apache.org/jira/browse/HDFS-9922
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Minor
> Attachments: HDFS-9922-trunk-v1.patch
>
>
> When there are replicas of a block on a decommissioned node,
> BlockPlacementStatusWithUpgradeDomain#isUpgradeDomainPolicySatisfied returns
> false when it should return true. This is because numberOfReplicas is the
> number of in-service replicas for the block and upgradeDomains.size() is the
> number of upgrade domains across all replicas of the block. Specifically, we
> hit this scenario when numberOfReplicas is equal to upgradeDomainFactor and
> upgradeDomains.size() is greater than numberOfReplicas.
> {code}
> private boolean isUpgradeDomainPolicySatisfied() {
> if (numberOfReplicas <= upgradeDomainFactor) {
> return (numberOfReplicas == upgradeDomains.size());
> } else {
> return upgradeDomains.size() >= upgradeDomainFactor;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9922) Upgrade Domain placement policy status marks a good block in violation when there are decommissioned nodes
[
https://issues.apache.org/jira/browse/HDFS-9922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Trezzo updated HDFS-9922:
---
Issue Type: Sub-task (was: Bug)
Parent: HDFS-7541
> Upgrade Domain placement policy status marks a good block in violation when
> there are decommissioned nodes
> --
>
> Key: HDFS-9922
> URL: https://issues.apache.org/jira/browse/HDFS-9922
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Chris Trezzo
>Assignee: Chris Trezzo
>Priority: Minor
>
> When there are replicas of a block on a decommissioned node,
> BlockPlacementStatusWithUpgradeDomain#isUpgradeDomainPolicySatisfied returns
> false when it should return true. This is because numberOfReplicas is the
> number of in-service replicas for the block and upgradeDomains.size() is the
> number of upgrade domains across all replicas of the block. Specifically, we
> hit this scenario when numberOfReplicas is equal to upgradeDomainFactor and
> upgradeDomains.size() is greater than numberOfReplicas.
> {code}
> private boolean isUpgradeDomainPolicySatisfied() {
> if (numberOfReplicas <= upgradeDomainFactor) {
> return (numberOfReplicas == upgradeDomains.size());
> } else {
> return upgradeDomains.size() >= upgradeDomainFactor;
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
