[jira] [Commented] (HDFS-8201) Refactor the end to end test for stripping file writing and reading
[
https://issues.apache.org/jira/browse/HDFS-8201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15933916#comment-15933916
]
Kai Zheng commented on HDFS-8201:
-
Just did a check and found below tests that all can serve the end to end test
as desired here. So this is obsolete and we don't need it any more.
{noformat}
TestReadStripedFileWithDecoding
TestReadStripedFileWithMissingBlocks
TestDFSStripedInputStream
TestDFSStripedOutputStream
{noformat}
> Refactor the end to end test for stripping file writing and reading
> ---
>
> Key: HDFS-8201
> URL: https://issues.apache.org/jira/browse/HDFS-8201
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Kai Zheng
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-8201.001.patch, HDFS-8201.002.patch,
> HDFS-8201-HDFS-7285.003.patch, HDFS-8201-HDFS-7285.004.patch,
> HDFS-8201-HDFS-7285.005.patch
>
>
> According to off-line discussion with [~zhz] and [~xinwei], we need to
> implement an end to end test for stripping file support:
> * Create an EC zone;
> * Create a file in the zone;
> * Write various typical sizes of content to the file, each size maybe a test
> method;
> * Read the written content back;
> * Compare the written content and read content to ensure it's good;
> This jira aims to refactor the end to end test
> class(TestWriteReadStripedFile) in order to reuse them conveniently in the
> next test step for erasure encoding and recovering. Will open separate issue
> for it.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8201) Refactor the end to end test for stripping file writing and reading
[
https://issues.apache.org/jira/browse/HDFS-8201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816745#comment-15816745
]
Hadoop QA commented on HDFS-8201:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HDFS-8201 does not apply to HDFS-7285. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-8201 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12734445/HDFS-8201-HDFS-7285.005.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/18138/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Refactor the end to end test for stripping file writing and reading
> ---
>
> Key: HDFS-8201
> URL: https://issues.apache.org/jira/browse/HDFS-8201
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Xinwei Qin
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-8201-HDFS-7285.003.patch,
> HDFS-8201-HDFS-7285.004.patch, HDFS-8201-HDFS-7285.005.patch,
> HDFS-8201.001.patch, HDFS-8201.002.patch
>
>
> According to off-line discussion with [~zhz] and [~xinwei], we need to
> implement an end to end test for stripping file support:
> * Create an EC zone;
> * Create a file in the zone;
> * Write various typical sizes of content to the file, each size maybe a test
> method;
> * Read the written content back;
> * Compare the written content and read content to ensure it's good;
> This jira aims to refactor the end to end test
> class(TestWriteReadStripedFile) in order to reuse them conveniently in the
> next test step for erasure encoding and recovering. Will open separate issue
> for it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8201) Refactor the end to end test for stripping file writing and reading
[ https://issues.apache.org/jira/browse/HDFS-8201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14575173#comment-14575173 ] Zhe Zhang commented on HDFS-8201: - Moving as a follow-on since those tests are still being modified. Maybe we can refactor after they are stabilized. Refactor the end to end test for stripping file writing and reading --- Key: HDFS-8201 URL: https://issues.apache.org/jira/browse/HDFS-8201 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng Assignee: Xinwei Qin Attachments: HDFS-8201-HDFS-7285.003.patch, HDFS-8201-HDFS-7285.004.patch, HDFS-8201-HDFS-7285.005.patch, HDFS-8201.001.patch, HDFS-8201.002.patch According to off-line discussion with [~zhz] and [~xinwei], we need to implement an end to end test for stripping file support: * Create an EC zone; * Create a file in the zone; * Write various typical sizes of content to the file, each size maybe a test method; * Read the written content back; * Compare the written content and read content to ensure it's good; This jira aims to refactor the end to end test class(TestWriteReadStripedFile) in order to reuse them conveniently in the next test step for erasure encoding and recovering. Will open separate issue for it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8201) Refactor the end to end test for stripping file writing and reading
[
https://issues.apache.org/jira/browse/HDFS-8201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14554239#comment-14554239
]
Xinwei Qin commented on HDFS-8201:
---
Thanks [~zhz] for the review.
1. The JIRA description and summary has been updated.
2. I don't think it is necessary to eventually subclass {{TestWriteRead}}, as
the non-EC files can be tested in the {{TestWriteRead}}, and the
{{TestWriteReadStripedFile}} class can only aim to the striped files. What do
you think?
Refactor the end to end test for stripping file writing and reading
---
Key: HDFS-8201
URL: https://issues.apache.org/jira/browse/HDFS-8201
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Xinwei Qin
Attachments: HDFS-8201-HDFS-7285.003.patch,
HDFS-8201-HDFS-7285.004.patch, HDFS-8201.001.patch, HDFS-8201.002.patch
According to off-line discussion with [~zhz] and [~xinwei], we need to
implement an end to end test for stripping file support:
* Create an EC zone;
* Create a file in the zone;
* Write various typical sizes of content to the file, each size maybe a test
method;
* Read the written content back;
* Compare the written content and read content to ensure it's good;
This jira aims to refactor the end to end test
class(TestWriteReadStripedFile) in order to reuse them conveniently in the
next test step for erasure encoding and recovering. Will open separate issue
for it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8201) Refactor the end to end test for stripping file writing and reading
[
https://issues.apache.org/jira/browse/HDFS-8201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14554668#comment-14554668
]
Hadoop QA commented on HDFS-8201:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 5m 13s | Pre-patch HDFS-7285 compilation
is healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 38s | There were no new javac warning
messages. |
| {color:red}-1{color} | release audit | 0m 13s | The applied patch generated
1 release audit warnings. |
| {color:green}+1{color} | checkstyle | 2m 17s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 31s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 3m 13s | The patch appears to introduce 6
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 1m 18s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 172m 43s | Tests failed in hadoop-hdfs. |
| | | 194m 44s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| | Inconsistent synchronization of
org.apache.hadoop.hdfs.DFSOutputStream.streamer; locked 89% of time
Unsynchronized access at DFSOutputStream.java:89% of time Unsynchronized
access at DFSOutputStream.java:[line 146] |
| | Possible null pointer dereference of arr$ in
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoStripedUnderConstruction.initializeBlockRecovery(long)
Dereferenced at BlockInfoStripedUnderConstruction.java:arr$ in
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoStripedUnderConstruction.initializeBlockRecovery(long)
Dereferenced at BlockInfoStripedUnderConstruction.java:[line 194] |
| | Unread field:field be static? At ErasureCodingWorker.java:[line 254] |
| | Should
org.apache.hadoop.hdfs.server.datanode.erasurecode.ErasureCodingWorker$StripedReader
be a _static_ inner class? At ErasureCodingWorker.java:inner class? At
ErasureCodingWorker.java:[lines 907-914] |
| | Result of integer multiplication cast to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.constructInternalBlock(LocatedStripedBlock,
int, int, int, int) At StripedBlockUtil.java:to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.constructInternalBlock(LocatedStripedBlock,
int, int, int, int) At StripedBlockUtil.java:[line 108] |
| | Result of integer multiplication cast to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.getStartOffsetsForInternalBlocks(ECSchema,
int, LocatedStripedBlock, long) At StripedBlockUtil.java:to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.getStartOffsetsForInternalBlocks(ECSchema,
int, LocatedStripedBlock, long) At StripedBlockUtil.java:[line 409] |
| Failed unit tests | hadoop.hdfs.server.blockmanagement.TestBlockInfo |
| | hadoop.hdfs.server.blockmanagement.TestReplicationPolicy |
| | hadoop.hdfs.server.namenode.TestAuditLogs |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
| | hadoop.hdfs.TestEncryptedTransfer |
| | hadoop.hdfs.server.namenode.TestFileTruncate |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12734445/HDFS-8201-HDFS-7285.005.patch
|
| Optional Tests | javac unit findbugs checkstyle |
| git revision | HDFS-7285 / 9fdb5be |
| Release Audit |
https://builds.apache.org/job/PreCommit-HDFS-Build/11086/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/11086/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11086/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11086/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11086/console |
This message was automatically generated.
Refactor the end to end test for stripping file writing and reading
---
Key: HDFS-8201
URL: https://issues.apache.org/jira/browse/HDFS-8201
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Xinwei Qin
