[jira] [Commented] (HDFS-8202) Improve end to end stirpping file test to add erasure recovering test
[
https://issues.apache.org/jira/browse/HDFS-8202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14645270#comment-14645270
]
Zhe Zhang commented on HDFS-8202:
-
Thanks Xinwei for updating the patch. The patch LGTM except for the following
minor issues:
# {{TestReadStripedFileWithDecoding}} has an unused import and {{import
org.apache.hadoop.hdfs.protocol.*}}
# We should probably convert {{Assert.fail}} to {{assertTrue}}.
{code}
int recoverBlkNum = dataBlkDelNum + parityBlkDelNum;
if (dataBlkDelNum 0 || parityBlkDelNum 0) {
Assert.fail(dataBlkDelNum and parityBlkDelNum should be positive);
}
if (recoverBlkNum parityBlocks) {
Assert.fail(The sum of +
dataBlkDelNum and parityBlkDelNum should be between 1 ~
+ parityBlocks);
}
{code}
# We can add some randomness to the following code, maybe as a follow-on:
{code}
for (int i = 0; i indices.length; i++) {
if (j dataBlkDelNum) {
if (indices[i] dataBlocks) {
delDataBlkIndices[j++] = i;
}
}
if (k parityBlkDelNum) {
if (indices[i] = dataBlocks) {
delParityBlkIndices[k++] = i;
}
}
}
{code}
# Calling {{TestDFSStripedOutputStreamWithFailure#killDatanode}} from its peer
class ({{TestWriteStripedFileWithFailure}}) doesn't look very neat. As a
follow-on we can move it to a utility class.
# {{TestWriteStripedFileWithFailure}} actually fails, could you debug the
issue? Thanks.
Improve end to end stirpping file test to add erasure recovering test
-
Key: HDFS-8202
URL: https://issues.apache.org/jira/browse/HDFS-8202
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Xinwei Qin
Attachments: HDFS-8202-HDFS-7285.003.patch,
HDFS-8202-HDFS-7285.004.patch, HDFS-8202-HDFS-7285.005.patch,
HDFS-8202.001.patch, HDFS-8202.002.patch
This to follow on HDFS-8201 to add erasure recovering test in the end to end
stripping file test:
* After writing certain blocks to the test file, delete some block file;
* Read the file content and compare, see if any recovering issue, or verify
the erasure recovering works or not.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8202) Improve end to end stirpping file test to add erasure recovering test
[
https://issues.apache.org/jira/browse/HDFS-8202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14639658#comment-14639658
]
Zhe Zhang commented on HDFS-8202:
-
Thanks for updating the patch Xinwei.
bq. but we can not do this for testReadWithDNFailure, since each test case of
it needs a new cluster with no dead Datanode.
I simply meant something like:
{code}
for (int fileLength : fileLengths) {
for (int dnFailureNum : dnFailureNums) {
testReadWithDNFailure(...);
{code}
Improve end to end stirpping file test to add erasure recovering test
-
Key: HDFS-8202
URL: https://issues.apache.org/jira/browse/HDFS-8202
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Xinwei Qin
Attachments: HDFS-8202-HDFS-7285.003.patch,
HDFS-8202-HDFS-7285.004.patch, HDFS-8202.001.patch, HDFS-8202.002.patch
This to follow on HDFS-8201 to add erasure recovering test in the end to end
stripping file test:
* After writing certain blocks to the test file, delete some block file;
* Read the file content and compare, see if any recovering issue, or verify
the erasure recovering works or not.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8202) Improve end to end stirpping file test to add erasure recovering test
[
https://issues.apache.org/jira/browse/HDFS-8202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14639710#comment-14639710
]
Walter Su commented on HDFS-8202:
-
bq. each test case of it needs a new cluster with no dead Datanode.
The concern is valid.
bq. I simply meant something like:
Yes, the code is more clean.
Hi, [~xinwei]. You can move the setup() and teardown() to the impl function
{{testReadWithDNFailure()}}. So you'll have a new cluster for each test case.
Improve end to end stirpping file test to add erasure recovering test
-
Key: HDFS-8202
URL: https://issues.apache.org/jira/browse/HDFS-8202
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Xinwei Qin
Attachments: HDFS-8202-HDFS-7285.003.patch,
HDFS-8202-HDFS-7285.004.patch, HDFS-8202.001.patch, HDFS-8202.002.patch
This to follow on HDFS-8201 to add erasure recovering test in the end to end
stripping file test:
* After writing certain blocks to the test file, delete some block file;
* Read the file content and compare, see if any recovering issue, or verify
the erasure recovering works or not.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8202) Improve end to end stirpping file test to add erasure recovering test
[
https://issues.apache.org/jira/browse/HDFS-8202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14636340#comment-14636340
]
Zhe Zhang commented on HDFS-8202:
-
Thanks for the work Xinwei!
The patch looks good overall. A few comments:
# Can we write a loop instead of manually add all possibilities for
{{testReadWithDNFailure}} and {{testReadCorruptedData}}?
# {{testReadWithBlockCorrupted}} could use a simple Javadoc
# Similar comments apply to {{TestWriteStripedFileWithFailure}} tests.
# We can also eliminate the {{srcPath}} parameter of {{writeFileWithDNFailure}}
if the same value is always used.
Improve end to end stirpping file test to add erasure recovering test
-
Key: HDFS-8202
URL: https://issues.apache.org/jira/browse/HDFS-8202
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Xinwei Qin
Attachments: HDFS-8202-HDFS-7285.003.patch, HDFS-8202.001.patch,
HDFS-8202.002.patch
This to follow on HDFS-8201 to add erasure recovering test in the end to end
stripping file test:
* After writing certain blocks to the test file, delete some block file;
* Read the file content and compare, see if any recovering issue, or verify
the erasure recovering works or not.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8202) Improve end to end stirpping file test to add erasure recovering test
[ https://issues.apache.org/jira/browse/HDFS-8202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629094#comment-14629094 ] Xinwei Qin commented on HDFS-8202: --- Hi, [~zhz], thanks for your clarify. I will move HDFS-8259 and HDFS-8260 patch here. Improve end to end stirpping file test to add erasure recovering test - Key: HDFS-8202 URL: https://issues.apache.org/jira/browse/HDFS-8202 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng Assignee: Xinwei Qin Attachments: HDFS-8202.001.patch, HDFS-8202.002.patch This to follow on HDFS-8201 to add erasure recovering test in the end to end stripping file test: * After writing certain blocks to the test file, delete some block file; * Read the file content and compare, see if any recovering issue, or verify the erasure recovering works or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8202) Improve end to end stirpping file test to add erasure recovering test
[ https://issues.apache.org/jira/browse/HDFS-8202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14628774#comment-14628774 ] Zhe Zhang commented on HDFS-8202: - Thanks [~xinwei] for creating HDFS-8259 patch. I think that HDFS-8259 is for real-system testing while this JIRA aims to improve unit test codes. So the patch should perhaps be moved here. Improve end to end stirpping file test to add erasure recovering test - Key: HDFS-8202 URL: https://issues.apache.org/jira/browse/HDFS-8202 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng Assignee: Xinwei Qin Attachments: HDFS-8202.001.patch, HDFS-8202.002.patch This to follow on HDFS-8201 to add erasure recovering test in the end to end stripping file test: * After writing certain blocks to the test file, delete some block file; * Read the file content and compare, see if any recovering issue, or verify the erasure recovering works or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8202) Improve end to end stirpping file test to add erasure recovering test
[ https://issues.apache.org/jira/browse/HDFS-8202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14618097#comment-14618097 ] Xinwei Qin commented on HDFS-8202: --- Maybe this jira is duplicated to HDFS-8259, I have attached a patch there, we can review and discuss in that jira. So, I feel this jira can be closed now. Improve end to end stirpping file test to add erasure recovering test - Key: HDFS-8202 URL: https://issues.apache.org/jira/browse/HDFS-8202 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng Assignee: Xinwei Qin Attachments: HDFS-8202.001.patch, HDFS-8202.002.patch This to follow on HDFS-8201 to add erasure recovering test in the end to end stripping file test: * After writing certain blocks to the test file, delete some block file; * Read the file content and compare, see if any recovering issue, or verify the erasure recovering works or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8202) Improve end to end stirpping file test to add erasure recovering test
[ https://issues.apache.org/jira/browse/HDFS-8202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534236#comment-14534236 ] Xinwei Qin commented on HDFS-8202: --- Initial patch for review. This patch is based on HDFS-8201. Improve end to end stirpping file test to add erasure recovering test - Key: HDFS-8202 URL: https://issues.apache.org/jira/browse/HDFS-8202 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng Assignee: Xinwei Qin Attachments: HDFS-8202.001.patch This to follow on HDFS-8201 to add erasure recovering test in the end to end stripping file test: * After writing certain blocks to the test file, delete some block file; * Read the file content and compare, see if any recovering issue, or verify the erasure recovering works or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
