[
https://issues.apache.org/jira/browse/HDFS-10810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491119#comment-15491119
]
Mingliang Liu commented on HDFS-10810:
--------------------------------------
{quote}
Brahma Reddy Battula: we can remove v3 patch from jira..?
{quote}
I posted the v3 patch to help me understand the root cause of the problem.
Sorry for the confusion. I deleted it now. Let's focus on the original v2 patch.
{quote}
Tsz Wo Nicholas Sze: Why v3 does not work?
{quote}
One change I made in deleted v3 patch was the number of DNs vs. replication
factor. In the v2 test there are 3 DNs while the replication factor is set 10
from 3. I updated the v3 patch for this, and it works as what v2 expects. That
is, the test fails without the patch, while it passes with the patch. Just for
the record, I pasted as following instead of uploading a new patch.
{code}
@Test(timeout = 60000)
public void testSetReplicationWhenBatchIBR() throws Exception {
final Configuration conf = new HdfsConfiguration();
conf.setLong(DFSConfigKeys.DFS_BLOCKREPORT_INCREMENTAL_INTERVAL_MSEC_KEY,
30000);
conf.setLong(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, 1024);
conf.setInt(DFSConfigKeys.DFS_NAMENODE_FILE_CLOSE_NUM_COMMITTED_ALLOWED_KEY,
1);
try (MiniDFSCluster cluster = new MiniDFSCluster.Builder(conf)
.numDataNodes(3).build()) {
cluster.waitActive();
final DistributedFileSystem dfs = cluster.getFileSystem();
DFSTestUtil.createFile(dfs, new Path("/testSetReplicationWhenBatchIBR-1"),
1024L, (short) 3, 0L);
//sending the FBR to Delay next IBR
cluster.triggerBlockReports();
Thread.sleep(3000);
final Path filePath = new Path("/testSetReplicationWhenBatchIBR-2");
DFSTestUtil.createFile(dfs, filePath, 1024L, (short) 3, 0L);
dfs.setReplication(filePath, (short) 10);
Thread.sleep(3000);
assertEquals(0,
cluster.getNamesystem().getBlockManager().getMissingBlocksCount());
assertEquals(1, cluster.getNamesystem().getBlockManager()
.getUnderReplicatedBlocksCount());
}
}
{code}
{quote}
Tsz Wo Nicholas Sze: I believe you have found a real bug. I just want to
understand more the details.
{quote}
Me too. Again, the basic idea of taking pending replicas into account when
updating the {{neededReconstructions}} makes sense to me.
> Setreplication removing block from underconstrcution temporarily when batch
> IBR is enabled.
> --------------------------------------------------------------------------------------------
>
> Key: HDFS-10810
> URL: https://issues.apache.org/jira/browse/HDFS-10810
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Brahma Reddy Battula
> Assignee: Brahma Reddy Battula
> Attachments: HDFS-10810-002.patch, HDFS-10810.patch
>
>
> 1)Batch IBR is enabled with number of committed blocks allowed=1
> 2) Written one block and closed the file without waiting for IBR
> 3)Setreplication called immediately on the file.
> So till the finalized IBR Received, this block will be marked as corrupt.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
