[
https://issues.apache.org/jira/browse/HDFS-8978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Walter Su updated HDFS-8978:
----------------------------
Attachment: HDFS-8978-HDFS-7285.01.patch
1. fix {{TestDFSStripedOutputStreamWithFailure}}. Right now StripedOutputStream
doesn't handle remote exception. So I just throw it out.
2. fix a race condition. After file closed, a race between block reader and IBR
of some parity blocks. The test case verify parity block before parity block
was reported. So it failed.
Add {{StripedFileTestUtil.waitBlockGroupsReported(..)}}, similar to
{{DFSTestUtil.waitReplication(..)}}
{noformat}
2015-08-27 17:01:14,158 [Thread-1] INFO
hdfs.TestDFSStripedOutputStreamWithFailure
(TestDFSStripedOutputStreamWithFailure.java:checkData(444)) - i,j=0, 0,
numCellInBlock=1, blockSize=65536,
lb=LocatedBlock{BP-1919703598-9.96.1.34-1440666072467:blk_-9223372036854775792_1001;
getBlockSize()=65536; corrupt=false; offset=0;
locs=[DatanodeInfoWithStorage[127.0.0.1:45887,DS-0c4ce2aa-40a8-4ed8-9941-489205a59299,DISK]]}
2015-08-27 17:01:14,161 [Thread-1] INFO
hdfs.TestDFSStripedOutputStreamWithFailure
(TestDFSStripedOutputStreamWithFailure.java:checkData(444)) - i,j=1, 1,
numCellInBlock=1, blockSize=65536,
lb=LocatedBlock{BP-1919703598-9.96.1.34-1440666072467:blk_-9223372036854775791_1001;
getBlockSize()=65536; corrupt=false; offset=0;
locs=[DatanodeInfoWithStorage[127.0.0.1:39496,DS-e4002b94-cc0a-4020-bb6a-d995a45e31fa,DISK]]}
2015-08-27 17:01:14,164 [Thread-1] INFO
hdfs.TestDFSStripedOutputStreamWithFailure
(TestDFSStripedOutputStreamWithFailure.java:checkData(444)) - i,j=2, 2,
numCellInBlock=1, blockSize=65535,
lb=LocatedBlock{BP-1919703598-9.96.1.34-1440666072467:blk_-9223372036854775790_1001;
getBlockSize()=65535; corrupt=false; offset=0;
locs=[DatanodeInfoWithStorage[127.0.0.1:39584,DS-1b7892ee-f465-4755-87a2-f87c8f323c08,DISK]]}
2015-08-27 17:01:14,166 [Thread-1] INFO
hdfs.TestDFSStripedOutputStreamWithFailure
(TestDFSStripedOutputStreamWithFailure.java:checkData(444)) - i,j=3, 3,
numCellInBlock=0, blockSize=0, lb=null
2015-08-27 17:01:14,166 [Thread-1] INFO
hdfs.TestDFSStripedOutputStreamWithFailure
(TestDFSStripedOutputStreamWithFailure.java:checkData(444)) - i,j=4, 4,
numCellInBlock=0, blockSize=0, lb=null
2015-08-27 17:01:14,166 [Thread-1] INFO
hdfs.TestDFSStripedOutputStreamWithFailure
(TestDFSStripedOutputStreamWithFailure.java:checkData(444)) - i,j=5, 5,
numCellInBlock=0, blockSize=0, lb=null
2015-08-27 17:01:14,166 [Thread-1] INFO
hdfs.TestDFSStripedOutputStreamWithFailure
(TestDFSStripedOutputStreamWithFailure.java:checkData(444)) - i,j=6, 0,
numCellInBlock=1, blockSize=65536,
lb=LocatedBlock{BP-1919703598-9.96.1.34-1440666072467:blk_-9223372036854775786_1001;
getBlockSize()=65536; corrupt=false; offset=0;
locs=[DatanodeInfoWithStorage[127.0.0.1:38728,DS-aa540f42-bb6f-4923-a951-5196d4c2a810,DISK]]}
2015-08-27 17:01:14,169 [Thread-1] INFO
hdfs.TestDFSStripedOutputStreamWithFailure
(TestDFSStripedOutputStreamWithFailure.java:checkData(444)) - i,j=7, 0,
numCellInBlock=1, blockSize=65536, lb=null
2015-08-27 17:01:14,169 [Thread-1] INFO
hdfs.TestDFSStripedOutputStreamWithFailure
(TestDFSStripedOutputStreamWithFailure.java:checkData(444)) - i,j=8, 0,
numCellInBlock=1, blockSize=65536, lb=null
....
i=2, killedDnIndex=1: arrays first differed at element [1];
Expected :127
Actual :0
<Click to see difference>
at
org.junit.internal.ComparisonCriteria.arrayEquals(ComparisonCriteria.java:50)
at org.junit.Assert.internalArrayEquals(Assert.java:473)
at org.junit.Assert.assertArrayEquals(Assert.java:294)
at
org.apache.hadoop.hdfs.TestDFSStripedOutputStream.verifyParity(TestDFSStripedOutputStream.java:288)
at
org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure.checkData(TestDFSStripedOutputStreamWithFailure.java:487)
at
org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure.runTest(TestDFSStripedOutputStreamWithFailure.java:340)
at
org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure.runTest(TestDFSStripedOutputStreamWithFailure.java:268)
at
org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure.runTest(TestDFSStripedOutputStreamWithFailure.java:146)
at
org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure$TestBase.run(TestDFSStripedOutputStreamWithFailure.java:520)
at
org.apache.hadoop.hdfs.TestDFSStripedOutputStreamWithFailure$TestBase.test0(TestDFSStripedOutputStreamWithFailure.java:523)
{noformat}
3. disable block recovery for some test cases. Just for correctness.
> Erasure coding: fix 2 failed tests of DFSStripedOutputStream
> ------------------------------------------------------------
>
> Key: HDFS-8978
> URL: https://issues.apache.org/jira/browse/HDFS-8978
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: test
> Reporter: Walter Su
> Assignee: Walter Su
> Priority: Minor
> Attachments: HDFS-8978-HDFS-7285.01.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
