[jira] [Commented] (HDFS-9083) Replication violates block placement policy.
[
https://issues.apache.org/jira/browse/HDFS-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15802687#comment-15802687
]
Junping Du commented on HDFS-9083:
--
Got it. Thanks for the confirmation!
> Replication violates block placement policy.
>
>
> Key: HDFS-9083
> URL: https://issues.apache.org/jira/browse/HDFS-9083
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Fix For: 2.7.2, 2.6.3
>
> Attachments: HDFS-9083-Test fix-branch-2.7.patch,
> HDFS-9083-branch-2.6.patch, HDFS-9083-branch-2.7.patch
>
>
> Recently we are noticing many cases in which all the replica of the block are
> residing on the same rack.
> During the block creation, the block placement policy was honored.
> But after node failure event in some specific manner, the block ends up in
> such state.
> On investigating more I found out that BlockManager#blockHasEnoughRacks is
> dependent on the config (net.topology.script.file.name)
> {noformat}
> if (!this.shouldCheckForEnoughRacks) {
> return true;
> }
> {noformat}
> We specify DNSToSwitchMapping implementation (our own custom implementation)
> via net.topology.node.switch.mapping.impl and no longer use
> net.topology.script.file.name config.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9083) Replication violates block placement policy.
[
https://issues.apache.org/jira/browse/HDFS-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15802662#comment-15802662
]
Kihwal Lee commented on HDFS-9083:
--
[~djp], that's correct.
> Replication violates block placement policy.
>
>
> Key: HDFS-9083
> URL: https://issues.apache.org/jira/browse/HDFS-9083
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Fix For: 2.7.2, 2.6.3
>
> Attachments: HDFS-9083-Test fix-branch-2.7.patch,
> HDFS-9083-branch-2.6.patch, HDFS-9083-branch-2.7.patch
>
>
> Recently we are noticing many cases in which all the replica of the block are
> residing on the same rack.
> During the block creation, the block placement policy was honored.
> But after node failure event in some specific manner, the block ends up in
> such state.
> On investigating more I found out that BlockManager#blockHasEnoughRacks is
> dependent on the config (net.topology.script.file.name)
> {noformat}
> if (!this.shouldCheckForEnoughRacks) {
> return true;
> }
> {noformat}
> We specify DNSToSwitchMapping implementation (our own custom implementation)
> via net.topology.node.switch.mapping.impl and no longer use
> net.topology.script.file.name config.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9083) Replication violates block placement policy.
[
https://issues.apache.org/jira/browse/HDFS-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15802521#comment-15802521
]
Junping Du commented on HDFS-9083:
--
Hi [~shahrs87], [~kihwal] and [~sjlee0], I assume the issue fixed here is only
applied to 2.7/2.6 but not affect 2.8.0 and 3.0.0. Can you confirm it? Thanks!
> Replication violates block placement policy.
>
>
> Key: HDFS-9083
> URL: https://issues.apache.org/jira/browse/HDFS-9083
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Fix For: 2.7.2, 2.6.3
>
> Attachments: HDFS-9083-Test fix-branch-2.7.patch,
> HDFS-9083-branch-2.6.patch, HDFS-9083-branch-2.7.patch
>
>
> Recently we are noticing many cases in which all the replica of the block are
> residing on the same rack.
> During the block creation, the block placement policy was honored.
> But after node failure event in some specific manner, the block ends up in
> such state.
> On investigating more I found out that BlockManager#blockHasEnoughRacks is
> dependent on the config (net.topology.script.file.name)
> {noformat}
> if (!this.shouldCheckForEnoughRacks) {
> return true;
> }
> {noformat}
> We specify DNSToSwitchMapping implementation (our own custom implementation)
> via net.topology.node.switch.mapping.impl and no longer use
> net.topology.script.file.name config.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9083) Replication violates block placement policy.
[
https://issues.apache.org/jira/browse/HDFS-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037727#comment-15037727
]
Brahma Reddy Battula commented on HDFS-9083:
[~xyao] Uploaded addendum patch to fix testcase failure.. It's ok for me,If you
want it done in separate jira,
> Replication violates block placement policy.
>
>
> Key: HDFS-9083
> URL: https://issues.apache.org/jira/browse/HDFS-9083
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Fix For: 2.7.2, 2.6.3
>
> Attachments: HDFS-9083-Test fix-branch-2.7.patch,
> HDFS-9083-branch-2.6.patch, HDFS-9083-branch-2.7.patch
>
>
> Recently we are noticing many cases in which all the replica of the block are
> residing on the same rack.
> During the block creation, the block placement policy was honored.
> But after node failure event in some specific manner, the block ends up in
> such state.
> On investigating more I found out that BlockManager#blockHasEnoughRacks is
> dependent on the config (net.topology.script.file.name)
> {noformat}
> if (!this.shouldCheckForEnoughRacks) {
> return true;
> }
> {noformat}
> We specify DNSToSwitchMapping implementation (our own custom implementation)
> via net.topology.node.switch.mapping.impl and no longer use
> net.topology.script.file.name config.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9083) Replication violates block placement policy.
[
https://issues.apache.org/jira/browse/HDFS-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037554#comment-15037554
]
Brahma Reddy Battula commented on HDFS-9083:
[~xyao] Yes, It's not handled in branch-2.7..
> Replication violates block placement policy.
>
>
> Key: HDFS-9083
> URL: https://issues.apache.org/jira/browse/HDFS-9083
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Fix For: 2.7.2, 2.6.3
>
> Attachments: HDFS-9083-branch-2.6.patch, HDFS-9083-branch-2.7.patch
>
>
> Recently we are noticing many cases in which all the replica of the block are
> residing on the same rack.
> During the block creation, the block placement policy was honored.
> But after node failure event in some specific manner, the block ends up in
> such state.
> On investigating more I found out that BlockManager#blockHasEnoughRacks is
> dependent on the config (net.topology.script.file.name)
> {noformat}
> if (!this.shouldCheckForEnoughRacks) {
> return true;
> }
> {noformat}
> We specify DNSToSwitchMapping implementation (our own custom implementation)
> via net.topology.node.switch.mapping.impl and no longer use
> net.topology.script.file.name config.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9083) Replication violates block placement policy.
[
https://issues.apache.org/jira/browse/HDFS-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037760#comment-15037760
]
Brahma Reddy Battula commented on HDFS-9083:
HDFS-9501 Raised for this testcase fix.
> Replication violates block placement policy.
>
>
> Key: HDFS-9083
> URL: https://issues.apache.org/jira/browse/HDFS-9083
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Fix For: 2.7.2, 2.6.3
>
> Attachments: HDFS-9083-Test fix-branch-2.7.patch,
> HDFS-9083-branch-2.6.patch, HDFS-9083-branch-2.7.patch
>
>
> Recently we are noticing many cases in which all the replica of the block are
> residing on the same rack.
> During the block creation, the block placement policy was honored.
> But after node failure event in some specific manner, the block ends up in
> such state.
> On investigating more I found out that BlockManager#blockHasEnoughRacks is
> dependent on the config (net.topology.script.file.name)
> {noformat}
> if (!this.shouldCheckForEnoughRacks) {
> return true;
> }
> {noformat}
> We specify DNSToSwitchMapping implementation (our own custom implementation)
> via net.topology.node.switch.mapping.impl and no longer use
> net.topology.script.file.name config.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9083) Replication violates block placement policy.
[
https://issues.apache.org/jira/browse/HDFS-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037361#comment-15037361
]
Brahma Reddy Battula commented on HDFS-9083:
[~xyao] thanks for pointing same..
{code}
cluster = new MiniDFSCluster.Builder(conf).numDataNodes(capacities.length)
.hosts(new String[]{"localhost", "localhost"})
.racks(new String[]{"rack0", "rack1"}).simulatedCapacities(capacities).build()
{code}
2 DNs are started with "rack1". Ideally we should not create 2 DNs with the
same hostname.And Pinning depends on favoredNodes.DFSClient#create(..) only
uses host:port, if favoredNodes is created by new InetSocketAddress(ip, port)
DFSClient will attempt a reverse lookup locally to get host:port, instead of
sending ip:port directly to NameNode.
.
MiniDFSCluster use fake hostname "host1.foo.com" to start DataNodes.DFSClient
doesn't use StaticMapping. So if DFSClient do reverse lookup, "127.0.0.1:8020"
becomes "localhost:8020".
Fix can be like following which I did same in branch-2 and Trunk.
{code}
+String[] hosts = {"host0", "host1"};
String[] racks = { RACK0, RACK1 };
int numOfDatanodes = capacities.length;
cluster = new MiniDFSCluster.Builder(conf).numDataNodes(capacities.length)
- .hosts(new String[]{"localhost", "localhost"})
- .racks(racks).simulatedCapacities(capacities).build();
+.hosts(hosts).racks(racks).simulatedCapacities(capacities).build();
try {
cluster.waitActive();
@@ -377,7 +377,10 @@ public void testBalancerWithPinnedBlocks() throws
Exception {
long totalUsedSpace = totalCapacity * 8 / 10;
InetSocketAddress[] favoredNodes = new InetSocketAddress[numOfDatanodes];
for (int i = 0; i < favoredNodes.length; i++) {
-favoredNodes[i] = cluster.getDataNodes().get(i).getXferAddress();
+// DFSClient will attempt reverse lookup. In case it resolves
+// "127.0.0.1" to "localhost", we manually specify the hostname.
+int port = cluster.getDataNodes().get(i).getXferAddress().getPort();
+favoredNodes[i] = new InetSocketAddress(hosts[i], port);
{code}
> Replication violates block placement policy.
>
>
> Key: HDFS-9083
> URL: https://issues.apache.org/jira/browse/HDFS-9083
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Fix For: 2.7.2, 2.6.3
>
> Attachments: HDFS-9083-branch-2.6.patch, HDFS-9083-branch-2.7.patch
>
>
> Recently we are noticing many cases in which all the replica of the block are
> residing on the same rack.
> During the block creation, the block placement policy was honored.
> But after node failure event in some specific manner, the block ends up in
> such state.
> On investigating more I found out that BlockManager#blockHasEnoughRacks is
> dependent on the config (net.topology.script.file.name)
> {noformat}
> if (!this.shouldCheckForEnoughRacks) {
> return true;
> }
> {noformat}
> We specify DNSToSwitchMapping implementation (our own custom implementation)
> via net.topology.node.switch.mapping.impl and no longer use
> net.topology.script.file.name config.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9083) Replication violates block placement policy.
[
https://issues.apache.org/jira/browse/HDFS-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037293#comment-15037293
]
Xiaoyu Yao commented on HDFS-9083:
--
The 2.7 patch caused failure of TestBalancer#testBalancerWithPinnedBlocks. The
test was passing without this patch.
[~shahrs87], can you take a look?
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m;
support was removed in 8.0
Running org.apache.hadoop.hdfs.server.balancer.TestBalancer
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 12.888 sec <<<
FAILURE! - in org.apache.hadoop.hdfs.server.balancer.TestBalancer
testBalancerWithPinnedBlocks(org.apache.hadoop.hdfs.server.balancer.TestBalancer)
Time elapsed: 12.748 sec <<< FAILURE!
java.lang.AssertionError: expected:<-3> but was:<0>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.hdfs.server.balancer.TestBalancer.testBalancerWithPinnedBlocks(TestBalancer.java:362)
Results :
Failed tests:
TestBalancer.testBalancerWithPinnedBlocks:362 expected:<-3> but was:<0>
> Replication violates block placement policy.
>
>
> Key: HDFS-9083
> URL: https://issues.apache.org/jira/browse/HDFS-9083
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Fix For: 2.7.2, 2.6.3
>
> Attachments: HDFS-9083-branch-2.6.patch, HDFS-9083-branch-2.7.patch
>
>
> Recently we are noticing many cases in which all the replica of the block are
> residing on the same rack.
> During the block creation, the block placement policy was honored.
> But after node failure event in some specific manner, the block ends up in
> such state.
> On investigating more I found out that BlockManager#blockHasEnoughRacks is
> dependent on the config (net.topology.script.file.name)
> {noformat}
> if (!this.shouldCheckForEnoughRacks) {
> return true;
> }
> {noformat}
> We specify DNSToSwitchMapping implementation (our own custom implementation)
> via net.topology.node.switch.mapping.impl and no longer use
> net.topology.script.file.name config.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9083) Replication violates block placement policy.
[
https://issues.apache.org/jira/browse/HDFS-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037436#comment-15037436
]
Xiaoyu Yao commented on HDFS-9083:
--
Thanks [~brahmareddy] for the explanation. That helps to understand the issue.
Is this fixed in 2.7.x branches such as branch-2.7.1 or branch-2.7.2? If not,
we need a separate ticket for the unit test fix.
> Replication violates block placement policy.
>
>
> Key: HDFS-9083
> URL: https://issues.apache.org/jira/browse/HDFS-9083
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Fix For: 2.7.2, 2.6.3
>
> Attachments: HDFS-9083-branch-2.6.patch, HDFS-9083-branch-2.7.patch
>
>
> Recently we are noticing many cases in which all the replica of the block are
> residing on the same rack.
> During the block creation, the block placement policy was honored.
> But after node failure event in some specific manner, the block ends up in
> such state.
> On investigating more I found out that BlockManager#blockHasEnoughRacks is
> dependent on the config (net.topology.script.file.name)
> {noformat}
> if (!this.shouldCheckForEnoughRacks) {
> return true;
> }
> {noformat}
> We specify DNSToSwitchMapping implementation (our own custom implementation)
> via net.topology.node.switch.mapping.impl and no longer use
> net.topology.script.file.name config.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9083) Replication violates block placement policy.
[
https://issues.apache.org/jira/browse/HDFS-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15023284#comment-15023284
]
Sangjin Lee commented on HDFS-9083:
---
Hi [~shahrs87], could you please create a patch for branch-2.6? I don't think
the changes in 2.7.x apply cleanly to 2.6.x.
> Replication violates block placement policy.
>
>
> Key: HDFS-9083
> URL: https://issues.apache.org/jira/browse/HDFS-9083
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS, namenode
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Fix For: 2.7.2
>
> Attachments: HDFS-9083-branch-2.7.patch
>
>
> Recently we are noticing many cases in which all the replica of the block are
> residing on the same rack.
> During the block creation, the block placement policy was honored.
> But after node failure event in some specific manner, the block ends up in
> such state.
> On investigating more I found out that BlockManager#blockHasEnoughRacks is
> dependent on the config (net.topology.script.file.name)
> {noformat}
> if (!this.shouldCheckForEnoughRacks) {
> return true;
> }
> {noformat}
> We specify DNSToSwitchMapping implementation (our own custom implementation)
> via net.topology.node.switch.mapping.impl and no longer use
> net.topology.script.file.name config.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9083) Replication violates block placement policy.
[
https://issues.apache.org/jira/browse/HDFS-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15023171#comment-15023171
]
Rushabh S Shah commented on HDFS-9083:
--
Yes. This bug is there in 2.6 also.
> Replication violates block placement policy.
>
>
> Key: HDFS-9083
> URL: https://issues.apache.org/jira/browse/HDFS-9083
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS, namenode
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Fix For: 2.7.2
>
> Attachments: HDFS-9083-branch-2.7.patch
>
>
> Recently we are noticing many cases in which all the replica of the block are
> residing on the same rack.
> During the block creation, the block placement policy was honored.
> But after node failure event in some specific manner, the block ends up in
> such state.
> On investigating more I found out that BlockManager#blockHasEnoughRacks is
> dependent on the config (net.topology.script.file.name)
> {noformat}
> if (!this.shouldCheckForEnoughRacks) {
> return true;
> }
> {noformat}
> We specify DNSToSwitchMapping implementation (our own custom implementation)
> via net.topology.node.switch.mapping.impl and no longer use
> net.topology.script.file.name config.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9083) Replication violates block placement policy.
[
https://issues.apache.org/jira/browse/HDFS-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15019105#comment-15019105
]
Sangjin Lee commented on HDFS-9083:
---
Should this be backported to branch-2.6?
> Replication violates block placement policy.
>
>
> Key: HDFS-9083
> URL: https://issues.apache.org/jira/browse/HDFS-9083
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS, namenode
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Fix For: 2.7.2
>
> Attachments: HDFS-9083-branch-2.7.patch
>
>
> Recently we are noticing many cases in which all the replica of the block are
> residing on the same rack.
> During the block creation, the block placement policy was honored.
> But after node failure event in some specific manner, the block ends up in
> such state.
> On investigating more I found out that BlockManager#blockHasEnoughRacks is
> dependent on the config (net.topology.script.file.name)
> {noformat}
> if (!this.shouldCheckForEnoughRacks) {
> return true;
> }
> {noformat}
> We specify DNSToSwitchMapping implementation (our own custom implementation)
> via net.topology.node.switch.mapping.impl and no longer use
> net.topology.script.file.name config.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9083) Replication violates block placement policy.
[
https://issues.apache.org/jira/browse/HDFS-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978452#comment-14978452
]
Rushabh S Shah commented on HDFS-9083:
--
[~jingzhao] [~mingma] [~brahmareddy]: Thanks for the reviews.
I ran all the hdfs tests since jenkins failed to run the tests.
The following tests failed:
{noformat}
TestSecureNNWithQJM#testSecureMode
TestSecureNNWithQJM#testSecondaryNameNodeHttpAddressNotNeeded
TestAppendSnapshotTruncate#testAST
TestBalancer#testTwoReplicaShouldNotInSameDN
TestBalancer#testBalancerWithPinnedBlocks
TestBalancer#testBalancerWithZeroThreadsForMove
TestBalancerWithSaslDataTransfer#testBalancer0Integrity
TestBalancerWithSaslDataTransfer#testBalancer0Authentication
TestBalancerWithSaslDataTransfer#testBalancer0Privacy
TestBalancerWithNodeGroup#testBalancerWithNodeGroup
TestBalancerWithNodeGroup#testBalancerEndInNoMoveProgress
TestSaslDataTransfer#testServerSaslNoClientSasl
TestSaslDataTransfer#testClientAndServerDoNotHaveCommonQop
TestSaslDataTransfer#testAuthentication
TestSaslDataTransfer#testPrivacy
TestSaslDataTransfer#testNoSaslAndSecurePortsIgnored
TestSaslDataTransfer#testIntegrity
{noformat}
I ran all these tests multiple times.
All these tests failed always except TestAppendSnapshotTruncate#testAST, which
failed intermittently.
I ran all the failed tests without my patch also and they failed.
So none of the test failures are related to my patch.
I will start the test-patch.sh on my machine and upload the results shortly.
> Replication violates block placement policy.
>
>
> Key: HDFS-9083
> URL: https://issues.apache.org/jira/browse/HDFS-9083
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS, namenode
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Attachments: HDFS-9083-branch-2.7.patch
>
>
> Recently we are noticing many cases in which all the replica of the block are
> residing on the same rack.
> During the block creation, the block placement policy was honored.
> But after node failure event in some specific manner, the block ends up in
> such state.
> On investigating more I found out that BlockManager#blockHasEnoughRacks is
> dependent on the config (net.topology.script.file.name)
> {noformat}
> if (!this.shouldCheckForEnoughRacks) {
> return true;
> }
> {noformat}
> We specify DNSToSwitchMapping implementation (our own custom implementation)
> via net.topology.node.switch.mapping.impl and no longer use
> net.topology.script.file.name config.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9083) Replication violates block placement policy.
[
https://issues.apache.org/jira/browse/HDFS-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14979094#comment-14979094
]
Rushabh S Shah commented on HDFS-9083:
--
Ran the findbugs on hadoop-hdfs-project and there is one findbugs warning in
PBImageTextWriter.java.
I haven't made any changes in that file.
> Replication violates block placement policy.
>
>
> Key: HDFS-9083
> URL: https://issues.apache.org/jira/browse/HDFS-9083
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS, namenode
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Attachments: HDFS-9083-branch-2.7.patch
>
>
> Recently we are noticing many cases in which all the replica of the block are
> residing on the same rack.
> During the block creation, the block placement policy was honored.
> But after node failure event in some specific manner, the block ends up in
> such state.
> On investigating more I found out that BlockManager#blockHasEnoughRacks is
> dependent on the config (net.topology.script.file.name)
> {noformat}
> if (!this.shouldCheckForEnoughRacks) {
> return true;
> }
> {noformat}
> We specify DNSToSwitchMapping implementation (our own custom implementation)
> via net.topology.node.switch.mapping.impl and no longer use
> net.topology.script.file.name config.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9083) Replication violates block placement policy.
[
https://issues.apache.org/jira/browse/HDFS-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14979126#comment-14979126
]
Rushabh S Shah commented on HDFS-9083:
--
[~kihwal]: Thanks for committing.
> Replication violates block placement policy.
>
>
> Key: HDFS-9083
> URL: https://issues.apache.org/jira/browse/HDFS-9083
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS, namenode
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Fix For: 2.7.2
>
> Attachments: HDFS-9083-branch-2.7.patch
>
>
> Recently we are noticing many cases in which all the replica of the block are
> residing on the same rack.
> During the block creation, the block placement policy was honored.
> But after node failure event in some specific manner, the block ends up in
> such state.
> On investigating more I found out that BlockManager#blockHasEnoughRacks is
> dependent on the config (net.topology.script.file.name)
> {noformat}
> if (!this.shouldCheckForEnoughRacks) {
> return true;
> }
> {noformat}
> We specify DNSToSwitchMapping implementation (our own custom implementation)
> via net.topology.node.switch.mapping.impl and no longer use
> net.topology.script.file.name config.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9083) Replication violates block placement policy.
[
https://issues.apache.org/jira/browse/HDFS-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14979115#comment-14979115
]
Kihwal Lee commented on HDFS-9083:
--
+1
> Replication violates block placement policy.
>
>
> Key: HDFS-9083
> URL: https://issues.apache.org/jira/browse/HDFS-9083
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS, namenode
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Attachments: HDFS-9083-branch-2.7.patch
>
>
> Recently we are noticing many cases in which all the replica of the block are
> residing on the same rack.
> During the block creation, the block placement policy was honored.
> But after node failure event in some specific manner, the block ends up in
> such state.
> On investigating more I found out that BlockManager#blockHasEnoughRacks is
> dependent on the config (net.topology.script.file.name)
> {noformat}
> if (!this.shouldCheckForEnoughRacks) {
> return true;
> }
> {noformat}
> We specify DNSToSwitchMapping implementation (our own custom implementation)
> via net.topology.node.switch.mapping.impl and no longer use
> net.topology.script.file.name config.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9083) Replication violates block placement policy.
[
https://issues.apache.org/jira/browse/HDFS-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977073#comment-14977073
]
Jing Zhao commented on HDFS-9083:
-
The patch looks good to me. [~mingma] and [~brahmareddy], do you also want to
take a look at the patch since you guys worked on HDFS-8647.
> Replication violates block placement policy.
>
>
> Key: HDFS-9083
> URL: https://issues.apache.org/jira/browse/HDFS-9083
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS, namenode
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Attachments: HDFS-9083-branch-2.7.patch
>
>
> Recently we are noticing many cases in which all the replica of the block are
> residing on the same rack.
> During the block creation, the block placement policy was honored.
> But after node failure event in some specific manner, the block ends up in
> such state.
> On investigating more I found out that BlockManager#blockHasEnoughRacks is
> dependent on the config (net.topology.script.file.name)
> {noformat}
> if (!this.shouldCheckForEnoughRacks) {
> return true;
> }
> {noformat}
> We specify DNSToSwitchMapping implementation (our own custom implementation)
> via net.topology.node.switch.mapping.impl and no longer use
> net.topology.script.file.name config.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9083) Replication violates block placement policy.
[
https://issues.apache.org/jira/browse/HDFS-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977301#comment-14977301
]
Ming Ma commented on HDFS-9083:
---
LGTM. [~shahrs87] any test failures in branch 2.7(if any, it should be just
test code update)?
> Replication violates block placement policy.
>
>
> Key: HDFS-9083
> URL: https://issues.apache.org/jira/browse/HDFS-9083
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS, namenode
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Attachments: HDFS-9083-branch-2.7.patch
>
>
> Recently we are noticing many cases in which all the replica of the block are
> residing on the same rack.
> During the block creation, the block placement policy was honored.
> But after node failure event in some specific manner, the block ends up in
> such state.
> On investigating more I found out that BlockManager#blockHasEnoughRacks is
> dependent on the config (net.topology.script.file.name)
> {noformat}
> if (!this.shouldCheckForEnoughRacks) {
> return true;
> }
> {noformat}
> We specify DNSToSwitchMapping implementation (our own custom implementation)
> via net.topology.node.switch.mapping.impl and no longer use
> net.topology.script.file.name config.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9083) Replication violates block placement policy.
[
https://issues.apache.org/jira/browse/HDFS-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977646#comment-14977646
]
Brahma Reddy Battula commented on HDFS-9083:
[~jingzhao] thanks for pinging. Patch LGTM.[~shahrs87] As jenkins did not
run,can you confirm about testcase failures in branc-2.7..?
> Replication violates block placement policy.
>
>
> Key: HDFS-9083
> URL: https://issues.apache.org/jira/browse/HDFS-9083
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS, namenode
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Attachments: HDFS-9083-branch-2.7.patch
>
>
> Recently we are noticing many cases in which all the replica of the block are
> residing on the same rack.
> During the block creation, the block placement policy was honored.
> But after node failure event in some specific manner, the block ends up in
> such state.
> On investigating more I found out that BlockManager#blockHasEnoughRacks is
> dependent on the config (net.topology.script.file.name)
> {noformat}
> if (!this.shouldCheckForEnoughRacks) {
> return true;
> }
> {noformat}
> We specify DNSToSwitchMapping implementation (our own custom implementation)
> via net.topology.node.switch.mapping.impl and no longer use
> net.topology.script.file.name config.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9083) Replication violates block placement policy.
[
https://issues.apache.org/jira/browse/HDFS-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14974953#comment-14974953
]
Hadoop QA commented on HDFS-9083:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12768797/HDFS-9083-branch-2.7.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | branch-2 / baa2998 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/13198/console |
This message was automatically generated.
> Replication violates block placement policy.
>
>
> Key: HDFS-9083
> URL: https://issues.apache.org/jira/browse/HDFS-9083
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS, namenode
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
> Attachments: HDFS-9083-branch-2.7.patch
>
>
> Recently we are noticing many cases in which all the replica of the block are
> residing on the same rack.
> During the block creation, the block placement policy was honored.
> But after node failure event in some specific manner, the block ends up in
> such state.
> On investigating more I found out that BlockManager#blockHasEnoughRacks is
> dependent on the config (net.topology.script.file.name)
> {noformat}
> if (!this.shouldCheckForEnoughRacks) {
> return true;
> }
> {noformat}
> We specify DNSToSwitchMapping implementation (our own custom implementation)
> via net.topology.node.switch.mapping.impl and no longer use
> net.topology.script.file.name config.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9083) Replication violates block placement policy.
[
https://issues.apache.org/jira/browse/HDFS-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14974829#comment-14974829
]
Jing Zhao commented on HDFS-9083:
-
Thanks for working on this, [~rushabh.shah]. Yes, we need to fix this in
branch-2.7 and currently this is a blocker for 2.7.2.
> Replication violates block placement policy.
>
>
> Key: HDFS-9083
> URL: https://issues.apache.org/jira/browse/HDFS-9083
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS, namenode
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>Priority: Blocker
>
> Recently we are noticing many cases in which all the replica of the block are
> residing on the same rack.
> During the block creation, the block placement policy was honored.
> But after node failure event in some specific manner, the block ends up in
> such state.
> On investigating more I found out that BlockManager#blockHasEnoughRacks is
> dependent on the config (net.topology.script.file.name)
> {noformat}
> if (!this.shouldCheckForEnoughRacks) {
> return true;
> }
> {noformat}
> We specify DNSToSwitchMapping implementation (our own custom implementation)
> via net.topology.node.switch.mapping.impl and no longer use
> net.topology.script.file.name config.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9083) Replication violates block placement policy.
[
https://issues.apache.org/jira/browse/HDFS-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956929#comment-14956929
]
Rushabh S Shah commented on HDFS-9083:
--
bq. Most of the change looks good. It seems this will also fix HDFS-9083. cc:
Rushabh S Shah.
[~mingma]: Thanks for letting me know. Do we need to fix it in branch-2.7 ?
> Replication violates block placement policy.
>
>
> Key: HDFS-9083
> URL: https://issues.apache.org/jira/browse/HDFS-9083
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS, namenode
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>
> Recently we are noticing many cases in which all the replica of the block are
> residing on the same rack.
> During the block creation, the block placement policy was honored.
> But after node failure event in some specific manner, the block ends up in
> such state.
> On investigating more I found out that BlockManager#blockHasEnoughRacks is
> dependent on the config (net.topology.script.file.name)
> {noformat}
> if (!this.shouldCheckForEnoughRacks) {
> return true;
> }
> {noformat}
> We specify DNSToSwitchMapping implementation (our own custom implementation)
> via net.topology.node.switch.mapping.impl and no longer use
> net.topology.script.file.name config.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9083) Replication violates block placement policy.
[
https://issues.apache.org/jira/browse/HDFS-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956932#comment-14956932
]
Rushabh S Shah commented on HDFS-9083:
--
This is a conversation from
https://issues.apache.org/jira/browse/HDFS-8647?focusedCommentId=14956537=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14956537
> Replication violates block placement policy.
>
>
> Key: HDFS-9083
> URL: https://issues.apache.org/jira/browse/HDFS-9083
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS, namenode
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>
> Recently we are noticing many cases in which all the replica of the block are
> residing on the same rack.
> During the block creation, the block placement policy was honored.
> But after node failure event in some specific manner, the block ends up in
> such state.
> On investigating more I found out that BlockManager#blockHasEnoughRacks is
> dependent on the config (net.topology.script.file.name)
> {noformat}
> if (!this.shouldCheckForEnoughRacks) {
> return true;
> }
> {noformat}
> We specify DNSToSwitchMapping implementation (our own custom implementation)
> via net.topology.node.switch.mapping.impl and no longer use
> net.topology.script.file.name config.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9083) Replication violates block placement policy.
[
https://issues.apache.org/jira/browse/HDFS-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14769020#comment-14769020
]
Jagadesh Kiran N commented on HDFS-9083:
[~shahrs87] no problem , please assign
> Replication violates block placement policy.
>
>
> Key: HDFS-9083
> URL: https://issues.apache.org/jira/browse/HDFS-9083
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS, namenode
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Jagadesh Kiran N
>
> Recently we are noticing many cases in which all the replica of the block are
> residing on the same rack.
> During the block creation, the block placement policy was honored.
> But after node failure event in some specific manner, the block ends up in
> such state.
> On investigating more I found out that BlockManager#blockHasEnoughRacks is
> dependent on the config (net.topology.script.file.name)
> {noformat}
> if (!this.shouldCheckForEnoughRacks) {
> return true;
> }
> {noformat}
> We specify DNSToSwitchMapping implementation (our own custom implementation)
> via net.topology.node.switch.mapping.impl and no longer use
> net.topology.script.file.name config.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9083) Replication violates block placement policy.
[
https://issues.apache.org/jira/browse/HDFS-9083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14769009#comment-14769009
]
Rushabh S Shah commented on HDFS-9083:
--
[~jagadesh.kiran]: Do you mind if I assign the jira to myself.
I have started working on the patch.
I missed to assign to myself when I created the jira.
> Replication violates block placement policy.
>
>
> Key: HDFS-9083
> URL: https://issues.apache.org/jira/browse/HDFS-9083
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: HDFS, namenode
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Jagadesh Kiran N
>
> Recently we are noticing many cases in which all the replica of the block are
> residing on the same rack.
> During the block creation, the block placement policy was honored.
> But after node failure event in some specific manner, the block ends up in
> such state.
> On investigating more I found out that BlockManager#blockHasEnoughRacks is
> dependent on the config (net.topology.script.file.name)
> {noformat}
> if (!this.shouldCheckForEnoughRacks) {
> return true;
> }
> {noformat}
> We specify DNSToSwitchMapping implementation (our own custom implementation)
> via net.topology.node.switch.mapping.impl and no longer use
> net.topology.script.file.name config.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
