[jira] [Commented] (HDFS-8201) Add an end to end test for stripping file writing and reading
[
https://issues.apache.org/jira/browse/HDFS-8201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14541449#comment-14541449
]
Zhe Zhang commented on HDFS-8201:
-
Thanks Xinwei for the work! The patch looks good to me overall. Just a few
minors:
# We should update JIRA description and summary. Right now the path mainly
refactors the {{TestWriteReadStripedFile}} class
# It'd be ideal to eventually subclass {{TestWriteRead}}, in order to cover all
test cases developed for non-EC files. This is a future we can think about.
So I'm +1 pending the JIRA update.
Add an end to end test for stripping file writing and reading
-
Key: HDFS-8201
URL: https://issues.apache.org/jira/browse/HDFS-8201
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Xinwei Qin
Attachments: HDFS-8201-HDFS-7285.003.patch,
HDFS-8201-HDFS-7285.004.patch, HDFS-8201.001.patch, HDFS-8201.002.patch
According to off-line discussion with [~zhz] and [~xinwei], we need to
implement an end to end test for stripping file support:
* Create an EC zone;
* Create a file in the zone;
* Write various typical sizes of content to the file, each size maybe a test
method;
* Read the written content back;
* Compare the written content and read content to ensure it's good;
The test facility is subject to add more steps for erasure encoding and
recovering. Will open separate issue for it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8201) Add an end to end test for stripping file writing and reading
[
https://issues.apache.org/jira/browse/HDFS-8201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14539648#comment-14539648
]
Hadoop QA commented on HDFS-8201:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 5m 15s | Pre-patch HDFS-7285 compilation
is healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 27s | There were no new javac warning
messages. |
| {color:red}-1{color} | release audit | 0m 13s | The applied patch generated
1 release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 38s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 36s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 31s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 3m 10s | The patch appears to introduce 9
new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | native | 1m 20s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 200m 26s | Tests failed in hadoop-hdfs. |
| | | 220m 41s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| | Inconsistent synchronization of
org.apache.hadoop.hdfs.DFSOutputStream.streamer; locked 89% of time
Unsynchronized access at DFSOutputStream.java:89% of time Unsynchronized
access at DFSOutputStream.java:[line 146] |
| | Possible null pointer dereference of arr$ in
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoStripedUnderConstruction.initializeBlockRecovery(long)
Dereferenced at BlockInfoStripedUnderConstruction.java:arr$ in
org.apache.hadoop.hdfs.server.blockmanagement.BlockInfoStripedUnderConstruction.initializeBlockRecovery(long)
Dereferenced at BlockInfoStripedUnderConstruction.java:[line 194] |
| | Unread field:field be static? At ErasureCodingWorker.java:[line 252] |
| | Should
org.apache.hadoop.hdfs.server.datanode.erasurecode.ErasureCodingWorker$StripedReader
be a _static_ inner class? At ErasureCodingWorker.java:inner class? At
ErasureCodingWorker.java:[lines 911-913] |
| | Found reliance on default encoding in
org.apache.hadoop.hdfs.server.namenode.ErasureCodingZoneManager.createErasureCodingZone(String,
ECSchema):in
org.apache.hadoop.hdfs.server.namenode.ErasureCodingZoneManager.createErasureCodingZone(String,
ECSchema): String.getBytes() At ErasureCodingZoneManager.java:[line 117] |
| | Found reliance on default encoding in
org.apache.hadoop.hdfs.server.namenode.ErasureCodingZoneManager.getECZoneInfo(INodesInPath):in
org.apache.hadoop.hdfs.server.namenode.ErasureCodingZoneManager.getECZoneInfo(INodesInPath):
new String(byte[]) At ErasureCodingZoneManager.java:[line 81] |
| | Dead store to counts in
org.apache.hadoop.hdfs.server.namenode.INodeFile.storagespaceConsumed(BlockStoragePolicy)
At
INodeFile.java:org.apache.hadoop.hdfs.server.namenode.INodeFile.storagespaceConsumed(BlockStoragePolicy)
At INodeFile.java:[line 833] |
| | Dead store to dataBlkNum in
org.apache.hadoop.hdfs.util.StripedBlockUtil.calcualteChunkPositionsInBuf(ECSchema,
LocatedStripedBlock, byte[], int, int, int, int, long, int,
StripedBlockUtil$AlignedStripe[]) At
StripedBlockUtil.java:org.apache.hadoop.hdfs.util.StripedBlockUtil.calcualteChunkPositionsInBuf(ECSchema,
LocatedStripedBlock, byte[], int, int, int, int, long, int,
StripedBlockUtil$AlignedStripe[]) At StripedBlockUtil.java:[line 467] |
| | Result of integer multiplication cast to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.planReadPortions(int, int, long,
int, int) At StripedBlockUtil.java:to long in
org.apache.hadoop.hdfs.util.StripedBlockUtil.planReadPortions(int, int, long,
int, int) At StripedBlockUtil.java:[line 206] |
| Failed unit tests | hadoop.tracing.TestTraceAdmin |
| | hadoop.hdfs.util.TestStripedBlockUtil |
| | hadoop.hdfs.server.blockmanagement.TestBlockInfo |
| | hadoop.hdfs.TestDistributedFileSystem |
| | hadoop.hdfs.TestWriteReadStripedFile |
| | hadoop.fs.TestSymlinkHdfsFileSystem |
| | hadoop.hdfs.server.namenode.TestFileTruncate |
| | hadoop.hdfs.TestDFSStripedInputStream |
| | hadoop.fs.TestSymlinkHdfsFileContext |
| | hadoop.hdfs.TestRecoverStripedFile |
| | hadoop.hdfs.server.datanode.TestIncrementalBlockReports |
| |
hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerWithStripedBlocks |
| | hadoop.hdfs.server.namenode.TestFSImage |
| | hadoop.hdfs.server.namenode.TestFSEditLogLoader |
| | hadoop.hdfs.TestDFSPermission |
| | hadoop.hdfs.TestErasureCodingZones |
| |
[jira] [Commented] (HDFS-8201) Add an end to end test for stripping file writing and reading
[
https://issues.apache.org/jira/browse/HDFS-8201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14537478#comment-14537478
]
Xinwei Qin commented on HDFS-8201:
---
Attached the patch with a name {{HDFS-8201-HDFS-7285.003.patch}}. [~rakeshr],
thanks for your adivce.
Add an end to end test for stripping file writing and reading
-
Key: HDFS-8201
URL: https://issues.apache.org/jira/browse/HDFS-8201
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Xinwei Qin
Attachments: HDFS-8201-HDFS-7285.003.patch, HDFS-8201.001.patch,
HDFS-8201.002.patch
According to off-line discussion with [~zhz] and [~xinwei], we need to
implement an end to end test for stripping file support:
* Create an EC zone;
* Create a file in the zone;
* Write various typical sizes of content to the file, each size maybe a test
method;
* Read the written content back;
* Compare the written content and read content to ensure it's good;
The test facility is subject to add more steps for erasure encoding and
recovering. Will open separate issue for it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8201) Add an end to end test for stripping file writing and reading
[
https://issues.apache.org/jira/browse/HDFS-8201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534230#comment-14534230
]
Xinwei Qin commented on HDFS-8201:
---
The {{TestWriteReadStripedFile}} has implemented a comprehensive end to end
test for writing and reading, the 002 patch mainly separates the write and read
test method in order to reused them in next test such as HDFS-8202.
Add an end to end test for stripping file writing and reading
-
Key: HDFS-8201
URL: https://issues.apache.org/jira/browse/HDFS-8201
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Xinwei Qin
Attachments: HDFS-8201.001.patch, HDFS-8201.002.patch
According to off-line discussion with [~zhz] and [~xinwei], we need to
implement an end to end test for stripping file support:
* Create an EC zone;
* Create a file in the zone;
* Write various typical sizes of content to the file, each size maybe a test
method;
* Read the written content back;
* Compare the written content and read content to ensure it's good;
The test facility is subject to add more steps for erasure encoding and
recovering. Will open separate issue for it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8201) Add an end to end test for stripping file writing and reading
[
https://issues.apache.org/jira/browse/HDFS-8201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534401#comment-14534401
]
Rakesh R commented on HDFS-8201:
Thanks [~xinwei] for the patch. Just one quick observation - could you please
attach the patch with a name like {{HDFS-8201-HDFS-7285.003.patch}}. This way
will help to trigger the jenkins in HDFS-7285 branch and get the report.
Add an end to end test for stripping file writing and reading
-
Key: HDFS-8201
URL: https://issues.apache.org/jira/browse/HDFS-8201
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Xinwei Qin
Attachments: HDFS-8201.001.patch, HDFS-8201.002.patch
According to off-line discussion with [~zhz] and [~xinwei], we need to
implement an end to end test for stripping file support:
* Create an EC zone;
* Create a file in the zone;
* Write various typical sizes of content to the file, each size maybe a test
method;
* Read the written content back;
* Compare the written content and read content to ensure it's good;
The test facility is subject to add more steps for erasure encoding and
recovering. Will open separate issue for it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8201) Add an end to end test for stripping file writing and reading
[ https://issues.apache.org/jira/browse/HDFS-8201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14518545#comment-14518545 ] Xinwei Qin commented on HDFS-8201: --- [~zhz] I have noticed that JIRA. I think your suggestion is very good. Add an end to end test for stripping file writing and reading - Key: HDFS-8201 URL: https://issues.apache.org/jira/browse/HDFS-8201 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng Assignee: Xinwei Qin Attachments: HDFS-8201.001.patch According to off-line discussion with [~zhz] and [~xinwei], we need to implement an end to end test for stripping file support: * Create an EC zone; * Create a file in the zone; * Write various typical sizes of content to the file, each size maybe a test method; * Read the written content back; * Compare the written content and read content to ensure it's good; The test facility is subject to add more steps for erasure encoding and recovering. Will open separate issue for it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8201) Add an end to end test for stripping file writing and reading
[
https://issues.apache.org/jira/browse/HDFS-8201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14516333#comment-14516333
]
Zhe Zhang commented on HDFS-8201:
-
Thanks for working on this Xinwei!
Not sure if you have noticed, {{TestDFSStripedInputStream}} was added as part
of HDFS-8136. It looks quite similar to this patch. Maybe we should cleanup and
optimize that test under this JIRA?
Add an end to end test for stripping file writing and reading
-
Key: HDFS-8201
URL: https://issues.apache.org/jira/browse/HDFS-8201
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Kai Zheng
Assignee: Xinwei Qin
Attachments: HDFS-8201.001.patch
According to off-line discussion with [~zhz] and [~xinwei], we need to
implement an end to end test for stripping file support:
* Create an EC zone;
* Create a file in the zone;
* Write various typical sizes of content to the file, each size maybe a test
method;
* Read the written content back;
* Compare the written content and read content to ensure it's good;
The test facility is subject to add more steps for erasure encoding and
recovering. Will open separate issue for it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8201) Add an end to end test for stripping file writing and reading
[ https://issues.apache.org/jira/browse/HDFS-8201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504468#comment-14504468 ] Kai Zheng commented on HDFS-8201: - I'm not sure. This work would rather end with a unit test, focusing stripping writing and reading. I thought HDFS-8197 is good for system integration tests. Add an end to end test for stripping file writing and reading - Key: HDFS-8201 URL: https://issues.apache.org/jira/browse/HDFS-8201 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng Assignee: Xinwei Qin According to off-line discussion with [~zhz] and [~xinwei], we need to implement an end to end test for stripping file support: * Create an EC zone; * Create a file in the zone; * Write various typical sizes of content to the file, each size maybe a test method; * Read the written content back; * Compare the written content and read content to ensure it's good; The test facility is subject to add more steps for erasure encoding and recovering. Will open separate issue for it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8201) Add an end to end test for stripping file writing and reading
[ https://issues.apache.org/jira/browse/HDFS-8201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504476#comment-14504476 ] Kai Sasaki commented on HDFS-8201: -- [~drankye] I see. If the purpose of this JIRA is like what you mentioned, please keep it. Thank you for clarifying! Add an end to end test for stripping file writing and reading - Key: HDFS-8201 URL: https://issues.apache.org/jira/browse/HDFS-8201 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng Assignee: Xinwei Qin According to off-line discussion with [~zhz] and [~xinwei], we need to implement an end to end test for stripping file support: * Create an EC zone; * Create a file in the zone; * Write various typical sizes of content to the file, each size maybe a test method; * Read the written content back; * Compare the written content and read content to ensure it's good; The test facility is subject to add more steps for erasure encoding and recovering. Will open separate issue for it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8201) Add an end to end test for stripping file writing and reading
[ https://issues.apache.org/jira/browse/HDFS-8201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504451#comment-14504451 ] Kai Sasaki commented on HDFS-8201: -- [~drankye] I wonder this JIRA might be duplicate to [HDFS-8197|https://issues.apache.org/jira/browse/HDFS-8197]. Can I file this JIRA under HDFS-8197? Add an end to end test for stripping file writing and reading - Key: HDFS-8201 URL: https://issues.apache.org/jira/browse/HDFS-8201 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng Assignee: Xinwei Qin According to off-line discussion with [~zhz] and [~xinwei], we need to implement an end to end test for stripping file support: * Create an EC zone; * Create a file in the zone; * Write various typical sizes of content to the file, each size maybe a test method; * Read the written content back; * Compare the written content and read content to ensure it's good; The test facility is subject to add more steps for erasure encoding and recovering. Will open separate issue for it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8201) Add an end to end test for stripping file writing and reading
[ https://issues.apache.org/jira/browse/HDFS-8201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504273#comment-14504273 ] Kai Zheng commented on HDFS-8201: - HDFS-8202 was also opened to improve the test for erasure recovering. Add an end to end test for stripping file writing and reading - Key: HDFS-8201 URL: https://issues.apache.org/jira/browse/HDFS-8201 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Kai Zheng Assignee: Xinwei Qin According to off-line discussion with [~zhz] and [~xinwei], we need to implement an end to end test for stripping file support: * Create an EC zone; * Create a file in the zone; * Write various typical sizes of content to the file, each size maybe a test method; * Read the written content back; * Compare the written content and read content to ensure it's good; The test facility is subject to add more steps for erasure encoding and recovering. Will open separate issue for it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
