[jira] [Commented] (HDFS-12872) EC Checksum broken when BlockAccessToken is enabled
[
https://issues.apache.org/jira/browse/HDFS-12872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16279669#comment-16279669
]
Hudson commented on HDFS-12872:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13334 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13334/])
HDFS-12872. EC Checksum broken when BlockAccessToken is enabled. (xiao: rev
56b1ff80dd9fbcde8d21a604eff0babb3a16418f)
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileChecksum.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/LocatedBlocks.java
> EC Checksum broken when BlockAccessToken is enabled
> ---
>
> Key: HDFS-12872
> URL: https://issues.apache.org/jira/browse/HDFS-12872
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Labels: hdfs-ec-3.0-must-do
> Fix For: 3.0.0
>
> Attachments: HDFS-12872.01.patch, HDFS-12872.02.patch,
> HDFS-12872.repro.patch
>
>
> It appears {{hdfs ec -checksum}} doesn't work when block access token is
> enabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12872) EC Checksum broken when BlockAccessToken is enabled
[
https://issues.apache.org/jira/browse/HDFS-12872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16279663#comment-16279663
]
Xiao Chen commented on HDFS-12872:
--
Thanks Uma, committing this.
> EC Checksum broken when BlockAccessToken is enabled
> ---
>
> Key: HDFS-12872
> URL: https://issues.apache.org/jira/browse/HDFS-12872
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-12872.01.patch, HDFS-12872.02.patch,
> HDFS-12872.repro.patch
>
>
> It appears {{hdfs ec -checksum}} doesn't work when block access token is
> enabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12872) EC Checksum broken when BlockAccessToken is enabled
[
https://issues.apache.org/jira/browse/HDFS-12872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16279406#comment-16279406
]
Uma Maheswara Rao G commented on HDFS-12872:
[~xiaochen] Thanks for the update on the patch. Latest patch looks good to me.
+1
> EC Checksum broken when BlockAccessToken is enabled
> ---
>
> Key: HDFS-12872
> URL: https://issues.apache.org/jira/browse/HDFS-12872
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-12872.01.patch, HDFS-12872.02.patch,
> HDFS-12872.repro.patch
>
>
> It appears {{hdfs ec -checksum}} doesn't work when block access token is
> enabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12872) EC Checksum broken when BlockAccessToken is enabled
[
https://issues.apache.org/jira/browse/HDFS-12872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16279400#comment-16279400
]
Xiao Chen commented on HDFS-12872:
--
Failed tests are not related and passed locally.
> EC Checksum broken when BlockAccessToken is enabled
> ---
>
> Key: HDFS-12872
> URL: https://issues.apache.org/jira/browse/HDFS-12872
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-12872.01.patch, HDFS-12872.02.patch,
> HDFS-12872.repro.patch
>
>
> It appears {{hdfs ec -checksum}} doesn't work when block access token is
> enabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12872) EC Checksum broken when BlockAccessToken is enabled
[
https://issues.apache.org/jira/browse/HDFS-12872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16279162#comment-16279162
]
genericqa commented on HDFS-12872:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 32s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 2s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
24s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}122m 53s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
29s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}184m 55s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.mover.TestMover |
| | hadoop.hdfs.server.mover.TestStorageMover |
| | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure030 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HDFS-12872 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12900601/HDFS-12872.02.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 6e5be83fb102 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personalit
[jira] [Commented] (HDFS-12872) EC Checksum broken when BlockAccessToken is enabled
[
https://issues.apache.org/jira/browse/HDFS-12872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16279105#comment-16279105
]
Xiao Chen commented on HDFS-12872:
--
Test failures are all OOM and doesn't look related.
[~umamaheswararao] mind giving another look? Thanks a lot!
> EC Checksum broken when BlockAccessToken is enabled
> ---
>
> Key: HDFS-12872
> URL: https://issues.apache.org/jira/browse/HDFS-12872
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-12872.01.patch, HDFS-12872.02.patch,
> HDFS-12872.repro.patch
>
>
> It appears {{hdfs ec -checksum}} doesn't work when block access token is
> enabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12872) EC Checksum broken when BlockAccessToken is enabled
[
https://issues.apache.org/jira/browse/HDFS-12872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16278169#comment-16278169
]
genericqa commented on HDFS-12872:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 33s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
39s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 18s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
5s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}139m 38s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
22s{color} | {color:red} The patch generated 19 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}211m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeUUID |
| | hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate |
| | hadoop.hdfs.TestDFSClientFailover |
| | hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality |
| | hadoop.hdfs.server.namenode.TestNameNodeRecovery |
| | hadoop.hdfs.server.namenode.TestFSDirectory |
| | hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| | hadoop.hdfs.TestDFSRollback |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration |
| | hadoop.hdfs.TestFileCreationEmpty |
| | hadoop.hdfs.server.datanode.checker.TestStorageLocationChecker |
| | hadoop.hdfs.server.namenode.TestQuotaByStorageType |
| | hadoop.hdfs.server.datanode.TestDiskError |
| | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
| | hadoop.hdfs.server.namenode.TestNameNodeRpcServer |
[jira] [Commented] (HDFS-12872) EC Checksum broken when BlockAccessToken is enabled
[
https://issues.apache.org/jira/browse/HDFS-12872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16277463#comment-16277463
]
Uma Maheswara Rao G commented on HDFS-12872:
HI [~xiaochen], Thanks for working on it. I think it make sense to set
blocktocken at group level to use for calculating DN.
{code}
sb.setBlockToken(blockTokenSecretManager.generateToken(
+NameNode.getRemoteUser().getShortUserName(),
+internalBlock, EnumSet.of(mode), b.getStorageTypes(),
+b.getStorageIDs()));
{code}
Isn't this code common to else part now? should we remove else part code and
make this set for block irrespective of isStriped ?
> EC Checksum broken when BlockAccessToken is enabled
> ---
>
> Key: HDFS-12872
> URL: https://issues.apache.org/jira/browse/HDFS-12872
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-12872.01.patch, HDFS-12872.repro.patch
>
>
> It appears {{hdfs ec -checksum}} doesn't work when block access token is
> enabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12872) EC Checksum broken when BlockAccessToken is enabled
[
https://issues.apache.org/jira/browse/HDFS-12872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16275399#comment-16275399
]
Xiao Chen commented on HDFS-12872:
--
Great, thanks Uma!
> EC Checksum broken when BlockAccessToken is enabled
> ---
>
> Key: HDFS-12872
> URL: https://issues.apache.org/jira/browse/HDFS-12872
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-12872.01.patch, HDFS-12872.repro.patch
>
>
> It appears {{hdfs ec -checksum}} doesn't work when block access token is
> enabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12872) EC Checksum broken when BlockAccessToken is enabled
[
https://issues.apache.org/jira/browse/HDFS-12872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16275332#comment-16275332
]
Uma Maheswara Rao G commented on HDFS-12872:
[~xiaochen] Thanks for pinging me. Let me get some time for this. I will
provide my review late today. Thanks
> EC Checksum broken when BlockAccessToken is enabled
> ---
>
> Key: HDFS-12872
> URL: https://issues.apache.org/jira/browse/HDFS-12872
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-12872.01.patch, HDFS-12872.repro.patch
>
>
> It appears {{hdfs ec -checksum}} doesn't work when block access token is
> enabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12872) EC Checksum broken when BlockAccessToken is enabled
[
https://issues.apache.org/jira/browse/HDFS-12872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16275159#comment-16275159
]
genericqa commented on HDFS-12872:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 36s{color} | {color:orange} hadoop-hdfs-project: The patch generated 3 new +
117 unchanged - 0 fixed = 120 total (was 117) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 27s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
23s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}139m 29s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}193m 22s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure000 |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.TestDFSUpgradeFromImage |
| | hadoop.hdfs.TestHdfsAdmin |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
| | hadoop.fs.TestUnbuffer |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.TestErasureCodingPolicies |
| | hadoop.hdfs.TestFetchImage |
| | hadoop.hdfs.TestSafeModeWithStripedFile |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoop.hdfs.TestReadStripedFileWithMissingBlocks |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 |
| | hadoop.hdfs.
[jira] [Commented] (HDFS-12872) EC Checksum broken when BlockAccessToken is enabled
[
https://issues.apache.org/jira/browse/HDFS-12872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16274861#comment-16274861
]
Xiao Chen commented on HDFS-12872:
--
Regarding the related question, Eddy pointed out that my test was writing all 0
bytes - stupid me. But even writing random bytes I still see exactly the same
physical blocks on disks Will look into that in a bit, please comment if
you have any thoughts. Thanks!
> EC Checksum broken when BlockAccessToken is enabled
> ---
>
> Key: HDFS-12872
> URL: https://issues.apache.org/jira/browse/HDFS-12872
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-12872.01.patch, HDFS-12872.repro.patch
>
>
> It appears {{hdfs ec -checksum}} doesn't work when block access token is
> enabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12872) EC Checksum broken when BlockAccessToken is enabled
[
https://issues.apache.org/jira/browse/HDFS-12872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16273066#comment-16273066
]
Andrew Wang commented on HDFS-12872:
If we can get it in before the other blockers are resolved, I'm fine with
including this.
> EC Checksum broken when BlockAccessToken is enabled
> ---
>
> Key: HDFS-12872
> URL: https://issues.apache.org/jira/browse/HDFS-12872
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-12872.repro.patch
>
>
> It appears {{hdfs ec -checksum}} doesn't work when block access token is
> enabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12872) EC Checksum broken when BlockAccessToken is enabled
[
https://issues.apache.org/jira/browse/HDFS-12872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16273029#comment-16273029
]
Xiao Chen commented on HDFS-12872:
--
Ping [~andrew.wang], I'm hoping to include this in 3.0.0 too. Perhaps not a
blocker but still fundamental.
> EC Checksum broken when BlockAccessToken is enabled
> ---
>
> Key: HDFS-12872
> URL: https://issues.apache.org/jira/browse/HDFS-12872
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding
>Reporter: Xiao Chen
>Assignee: Xiao Chen
>Priority: Critical
> Labels: hdfs-ec-3.0-must-do
> Attachments: HDFS-12872.repro.patch
>
>
> It appears {{hdfs ec -checksum}} doesn't work when block access token is
> enabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12872) EC Checksum broken when BlockAccessToken is enabled
[
https://issues.apache.org/jira/browse/HDFS-12872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16272173#comment-16272173
]
genericqa commented on HDFS-12872:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 39s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 35s{color} | {color:orange} hadoop-hdfs-project: The patch generated 2 new +
3 unchanged - 0 fixed = 5 total (was 3) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 9s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
17s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}153m 16s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}206m 33s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.tools.TestWebHDFSStoragePolicyCommands |
| | hadoop.hdfs.TestDFSRollback |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 |
| | hadoop.hdfs.server.balancer.TestBalancerWithSaslDataTransfer |
| |
hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerForContentSummary |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure170 |
| | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
| | hadoop.hdfs.TestEncryptionZones |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
| | hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer |
| | hadoop.hdfs.tools.TestViewFSStoragePolicyCommands |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure |
| | hadoo
[jira] [Commented] (HDFS-12872) EC Checksum broken when BlockAccessToken is enabled
[
https://issues.apache.org/jira/browse/HDFS-12872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271922#comment-16271922
]
Xiao Chen commented on HDFS-12872:
--
Hi [~eddyxu] [~drankye] [~umamaheswararao],
Have some questions on this one. Appreciate your inputs.
I'm attaching a patch for discussion. The test fails without my attempted fix,
passes with.
My understanding is the checksum has to be done at the blockgroup level, by
calculating each internal block in the group. From the following code, it
appears this is done on a single datanode instead of by the client. (DN later
will calculate in {{DataXceiver#blockGroupChecksum}} by running the computers).
So it would be {{done}} as long as it can get result from any 1 DN.
{code}
boolean done = false;
for (int j = 0; !done && j < datanodes.length; j++) {
try {
tryDatanode(blockGroup, stripedBlockInfo, datanodes[j],
requestedNumBytes);
done = true;
} catch (InvalidBlockTokenException ibte) {
if (bgIdx > getLastRetriedIndex()) {
setLastRetriedIndex(bgIdx);
done = true; // actually it's not done; but we'll retry
bgIdx--; // repeat at bgIdx-th block
setRefetchBlocks(true);
}
} catch (IOException ie) {
LOG.warn("src={}" + ", datanodes[{}]={}",
getSrc(), j, datanodes[j], ie);
}
}
{code}
So it appears this bug is just that we'd want to always use the first
blocktoken (index==0) to authenticate with the datanodes, since the
blockgroup's id will be equal to the id of the first internal block.
Is my understanding above correct?
I think for testing purpose we may want to make sure this works for all DNs,
not 1 DN, in unit tests. (Verified the posted patch works by removing {{!done
&&}}, and look for {{got reply from }} messages.
A related question:
I was trying to look at the actual storage of these blocks. This was done by
first run the unit test to clean up local dir, then set a break point in the
end of the test. It seems to me, listing the local dir I always get the same
blocks on different datanodes (e.g. 9223372036854775792, 9223372036854775791
and 9223372036854775790 are the same). Is this by design? Why are they the same
if they're storing different cells?
{noformat}
xiao-MBP:hadoop xiao$ for f in $(find
/Users/xiao/Desktop/repo/hdfs/xiao/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/
-name *blk*);do ls -lh $f ;done
-rw-r--r-- 1 xiao staff 128M Nov 29 15:54
/Users/xiao/Desktop/repo/hdfs/xiao/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data//data1/current/BP-50698545-10.0.0.51-1511999667872/current/finalized/subdir0/subdir0/blk_-9223372036854775792
-rw-r--r-- 1 xiao staff 1.0M Nov 29 15:54
/Users/xiao/Desktop/repo/hdfs/xiao/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data//data1/current/BP-50698545-10.0.0.51-1511999667872/current/finalized/subdir0/subdir0/blk_-9223372036854775792_1001.meta
-rw-r--r-- 1 xiao staff72M Nov 29 15:54
/Users/xiao/Desktop/repo/hdfs/xiao/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data//data2/current/BP-50698545-10.0.0.51-1511999667872/current/finalized/subdir0/subdir0/blk_-9223372036854775774
-rw-r--r-- 1 xiao staff 576K Nov 29 15:54
/Users/xiao/Desktop/repo/hdfs/xiao/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data//data2/current/BP-50698545-10.0.0.51-1511999667872/current/finalized/subdir0/subdir0/blk_-9223372036854775774_1002.meta
-rw-r--r-- 1 xiao staff 128M Nov 29 15:54
/Users/xiao/Desktop/repo/hdfs/xiao/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data//data3/current/BP-50698545-10.0.0.51-1511999667872/current/finalized/subdir0/subdir0/blk_-9223372036854775791
-rw-r--r-- 1 xiao staff 1.0M Nov 29 15:54
/Users/xiao/Desktop/repo/hdfs/xiao/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data//data3/current/BP-50698545-10.0.0.51-1511999667872/current/finalized/subdir0/subdir0/blk_-9223372036854775791_1001.meta
-rw-r--r-- 1 xiao staff72M Nov 29 15:54
/Users/xiao/Desktop/repo/hdfs/xiao/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data//data4/current/BP-50698545-10.0.0.51-1511999667872/current/finalized/subdir0/subdir0/blk_-9223372036854775776
-rw-r--r-- 1 xiao staff 576K Nov 29 15:54
/Users/xiao/Desktop/repo/hdfs/xiao/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data//data4/current/BP-50698545-10.0.0.51-1511999667872/current/finalized/subdir0/subdir0/blk_-9223372036854775776_1002.meta
-rw-r--r-- 1 xiao staff 128M Nov 29 15:54
/Users/xiao/Desktop/repo/hdfs/xiao/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data//data5/current/BP-50698545-10.0.0.51-1511999667872/current/finalized/subdir0/subdir0/blk_-9223372036854775790
-rw-r--r-- 1 xiao staff 1.0M Nov 29 15:5
