[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15097181#comment-15097181 ] Vinod Kumar Vavilapalli commented on HDFS-9289: --- Pulled this into 2.7.2 to keep the release up-to-date with 2.6.3. Changing fix-versions to reflect the same. > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Fix For: 2.7.2, 2.6.3 > > Attachments: HDFS-9289-branch-2.6.patch, HDFS-9289.1.patch, > HDFS-9289.2.patch, HDFS-9289.3.patch, HDFS-9289.4.patch, HDFS-9289.5.patch, > HDFS-9289.6.patch, HDFS-9289.7.patch, HDFS-9289.branch-2.7.patch, > HDFS-9289.branch-2.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15024945#comment-15024945 ] Zhe Zhang commented on HDFS-9289: - Thanks Sangjin. Yes the conflicts were mostly trivial. I was just not sure if I did the right update to CHANGES.TXT. I just committed the change to branch-2.6. > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Fix For: 3.0.0, 2.7.3, 2.6.3 > > Attachments: HDFS-9289-branch-2.6.patch, HDFS-9289.1.patch, > HDFS-9289.2.patch, HDFS-9289.3.patch, HDFS-9289.4.patch, HDFS-9289.5.patch, > HDFS-9289.6.patch, HDFS-9289.7.patch, HDFS-9289.branch-2.7.patch, > HDFS-9289.branch-2.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[
https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15023221#comment-15023221
]
Sangjin Lee commented on HDFS-9289:
---
Hi [~zhz], the only differences I see between the 2.7 commit and the 2.6 patch
are a couple of type changes (e.g. BlockInfoContiguous), right? If so, I think
it might be good to follow a cherry-pick route, rather than creating a separate
patch and commit.
If you're not familiar with it already, you can do
{noformat}
git cherry-pick -x
{noformat}
Naturally CHANGES.txt will come up as a conflict, and you can just make it like
what's proposed in the proposed patch. You'll also want to touch up the other
files just like what you did in the patch. You then stage those files that were
further modified. Finally you can finalize the cherry-pick via
{noformat}
git cherry-pick --continue
{noformat}
If you're not comfortable with that, I'd be OK with a new separate commit too
this time. Let me know if you have a question either way.
> Make DataStreamer#block thread safe and verify genStamp in commitBlock
> --
>
> Key: HDFS-9289
> URL: https://issues.apache.org/jira/browse/HDFS-9289
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Chang Li
>Assignee: Chang Li
>Priority: Critical
> Fix For: 3.0.0, 2.7.3
>
> Attachments: HDFS-9289-branch-2.6.patch, HDFS-9289.1.patch,
> HDFS-9289.2.patch, HDFS-9289.3.patch, HDFS-9289.4.patch, HDFS-9289.5.patch,
> HDFS-9289.6.patch, HDFS-9289.7.patch, HDFS-9289.branch-2.7.patch,
> HDFS-9289.branch-2.patch
>
>
> we have seen a case of corrupt block which is caused by file complete after a
> pipelineUpdate, but the file complete with the old block genStamp. This
> caused the replicas of two datanodes in updated pipeline to be viewed as
> corrupte. Propose to check genstamp when commit block
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15019084#comment-15019084 ] Sangjin Lee commented on HDFS-9289: --- Does this issue exist in 2.6.x? Should this be backported to branch-2.6? > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Fix For: 3.0.0, 2.7.3 > > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch, HDFS-9289.7.patch, > HDFS-9289.branch-2.7.patch, HDFS-9289.branch-2.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15019220#comment-15019220 ] Zhe Zhang commented on HDFS-9289: - I think this is a good addition to branch-2.6. > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Fix For: 3.0.0, 2.7.3 > > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch, HDFS-9289.7.patch, > HDFS-9289.branch-2.7.patch, HDFS-9289.branch-2.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15019222#comment-15019222 ] Zhe Zhang commented on HDFS-9289: - [~sjlee0] Yes this is a valid bug even for 2.6.x. > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Fix For: 3.0.0, 2.7.3 > > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch, HDFS-9289.7.patch, > HDFS-9289.branch-2.7.patch, HDFS-9289.branch-2.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15019228#comment-15019228 ] Sangjin Lee commented on HDFS-9289: --- Thanks [~zhz]! Could you please cherry-pick the commit to branch-2.6? > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Fix For: 3.0.0, 2.7.3 > > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch, HDFS-9289.7.patch, > HDFS-9289.branch-2.7.patch, HDFS-9289.branch-2.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14989831#comment-14989831 ] Rushabh S Shah commented on HDFS-9289: -- [~zhz]: Given that we are going to release 2.7.2 shortly , do we need to cherry pick it to 2.7.2 ? > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Fix For: 2.8.0 > > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch, HDFS-9289.7.patch, > HDFS-9289.branch-2.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14989919#comment-14989919 ] Kihwal Lee commented on HDFS-9289: -- bq. I will target it to 2.7.3 and cherry-pick to branch-2.7. We can't cherry-pick since the files/components have been split and moved around. [~lichangleo], can you post 2.7 version of the patch? > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Fix For: 2.8.0 > > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch, HDFS-9289.7.patch, > HDFS-9289.branch-2.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14989903#comment-14989903 ] Kihwal Lee commented on HDFS-9289: -- 2.7.2 is already cut. I will target it to 2.7.3 and cherry-pick to branch-2.7. > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Fix For: 2.8.0 > > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch, HDFS-9289.7.patch, > HDFS-9289.branch-2.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14989994#comment-14989994 ] Zhe Zhang commented on HDFS-9289: - Thanks for the discussions above, 2.7.3 sounds a good target version for this change. > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Fix For: 2.8.0 > > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch, HDFS-9289.7.patch, > HDFS-9289.branch-2.7.patch, HDFS-9289.branch-2.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14990065#comment-14990065 ] Kihwal Lee commented on HDFS-9289: -- Looks like a correct port of the patch. The new test succeeds. Committed to branch-2.7. > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Fix For: 2.7.3 > > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch, HDFS-9289.7.patch, > HDFS-9289.branch-2.7.patch, HDFS-9289.branch-2.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[
https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987944#comment-14987944
]
Hadoop QA commented on HDFS-9289:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 8s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 5s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk cannot run
convertXmlToText from findbugs {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 39s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 19s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 13s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 13s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 24s
{color} | {color:red} Patch generated 1 new checkstyle issues in
hadoop-hdfs-project (total was 247, now 247). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 33s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 22s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 29s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 2s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 38s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 1s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 22s
{color} | {color:red} Patch generated 56 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 178m 46s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits |
| | hadoop.hdfs.server.blockmanagement.TestNodeCount |
| JDK v1.7.0_79 Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock |
| | hadoop.hdfs.server.namenode.ha.TestDNFencing |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.TestFileCreationDelete |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
\\
\\
|| Subsystem
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[
https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987673#comment-14987673
]
Zhe Zhang commented on HDFS-9289:
-
Thanks Chang. +1 on the latest patch pending Jenkins.
I'll file a follow-on JIRA to cleanup {{addBlockToFile}} a bit. Maybe we can
consolidate {{numStripes}} and {{len}}.
> Make DataStreamer#block thread safe and verify genStamp in commitBlock
> --
>
> Key: HDFS-9289
> URL: https://issues.apache.org/jira/browse/HDFS-9289
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Chang Li
>Assignee: Chang Li
>Priority: Critical
> Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch,
> HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch, HDFS-9289.7.patch
>
>
> we have seen a case of corrupt block which is caused by file complete after a
> pipelineUpdate, but the file complete with the old block genStamp. This
> caused the replicas of two datanodes in updated pipeline to be viewed as
> corrupte. Propose to check genstamp when commit block
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988292#comment-14988292 ] Zhe Zhang commented on HDFS-9289: - Thanks Chang. I just committed to branch-2 as well. Changed fix version to 2.8.0. > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Fix For: 2.8.0 > > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch, HDFS-9289.7.patch, > HDFS-9289.branch-2.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988304#comment-14988304 ] Hudson commented on HDFS-9289: -- FAILURE: Integrated in Hadoop-trunk-Commit #8750 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8750/]) HDFS-9289. Make DataStreamer#block thread safe and verify genStamp in (zhz: rev dac0463a4e20dfb3a802355919fc22b8e017a4e1) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCommitBlockWithInvalidGenStamp.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestQuotaWithStripedBlocks.java > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Fix For: 2.8.0 > > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch, HDFS-9289.7.patch, > HDFS-9289.branch-2.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988185#comment-14988185 ] Zhe Zhang commented on HDFS-9289: - Thanks Change. I just committed the patch to trunk. Do you mind creating a patch for branch-2? The main conflict is on the test. > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch, HDFS-9289.7.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988093#comment-14988093 ] Chang Li commented on HDFS-9289: Thanks [~zhz] for further review! I have test those above failed unit tests above on my own machine with my patch on, and those test passed. They are not related to the changes of my patch. > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch, HDFS-9289.7.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[
https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14987991#comment-14987991
]
Hadoop QA commented on HDFS-9289:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 5s
{color} | {color:blue} docker + precommit patch detected. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 51s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk cannot run
convertXmlToText from findbugs {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 25s
{color} | {color:green} trunk passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 12s
{color} | {color:green} trunk passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
6s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 19s
{color} | {color:red} Patch generated 1 new checkstyle issues in
hadoop-hdfs-project (total was 247, now 247). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 28s
{color} | {color:green} the patch passed with JDK v1.8.0_60 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 11s
{color} | {color:green} the patch passed with JDK v1.7.0_79 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 48s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_60. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 48s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_60. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 57s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_79. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 55s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.7.0_79. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 19s
{color} | {color:red} Patch generated 58 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 155m 57s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_60 Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestNodeCount |
| | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| JDK v1.7.0_79 Failed junit tests | hadoop.fs.shell.TestHdfsTextCommand |
| | hadoop.hdfs.TestRollingUpgrade |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.7.1 Server=1.7.1
Image:test-patch-base-hadoop-date2015-11-03 |
| JIRA Patch URL |
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988639#comment-14988639 ] Hudson commented on HDFS-9289: -- SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #634 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/634/]) HDFS-9289. Make DataStreamer#block thread safe and verify genStamp in (zhz: rev dac0463a4e20dfb3a802355919fc22b8e017a4e1) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestQuotaWithStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCommitBlockWithInvalidGenStamp.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Fix For: 2.8.0 > > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch, HDFS-9289.7.patch, > HDFS-9289.branch-2.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988600#comment-14988600 ] Hudson commented on HDFS-9289: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2564 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2564/]) HDFS-9289. Make DataStreamer#block thread safe and verify genStamp in (zhz: rev dac0463a4e20dfb3a802355919fc22b8e017a4e1) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestQuotaWithStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCommitBlockWithInvalidGenStamp.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Fix For: 2.8.0 > > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch, HDFS-9289.7.patch, > HDFS-9289.branch-2.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988619#comment-14988619 ] Hudson commented on HDFS-9289: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #1357 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1357/]) HDFS-9289. Make DataStreamer#block thread safe and verify genStamp in (zhz: rev dac0463a4e20dfb3a802355919fc22b8e017a4e1) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCommitBlockWithInvalidGenStamp.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestQuotaWithStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Fix For: 2.8.0 > > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch, HDFS-9289.7.patch, > HDFS-9289.branch-2.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1498#comment-1498 ] Hudson commented on HDFS-9289: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #568 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/568/]) HDFS-9289. Make DataStreamer#block thread safe and verify genStamp in (zhz: rev dac0463a4e20dfb3a802355919fc22b8e017a4e1) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCommitBlockWithInvalidGenStamp.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestQuotaWithStripedBlocks.java > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Fix For: 2.8.0 > > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch, HDFS-9289.7.patch, > HDFS-9289.branch-2.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988801#comment-14988801 ] Hudson commented on HDFS-9289: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #623 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/623/]) HDFS-9289. Make DataStreamer#block thread safe and verify genStamp in (zhz: rev dac0463a4e20dfb3a802355919fc22b8e017a4e1) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCommitBlockWithInvalidGenStamp.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestQuotaWithStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Fix For: 2.8.0 > > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch, HDFS-9289.7.patch, > HDFS-9289.branch-2.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14988900#comment-14988900 ] Hudson commented on HDFS-9289: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2505 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2505/]) HDFS-9289. Make DataStreamer#block thread safe and verify genStamp in (zhz: rev dac0463a4e20dfb3a802355919fc22b8e017a4e1) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestQuotaWithStripedBlocks.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCommitBlockWithInvalidGenStamp.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DataStreamer.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.7.1 >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Fix For: 2.8.0 > > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch, HDFS-9289.7.patch, > HDFS-9289.branch-2.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14985415#comment-14985415 ] Daryn Sharp commented on HDFS-9289: --- I'd rather see the InvalidGenStampException instead of a generic IOE. Else it's hard for client code to intelligently deal with exceptions and for tests to verify that the correct/expected IOE. > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[
https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14986303#comment-14986303
]
Zhe Zhang commented on HDFS-9289:
-
The test looks reasonable. I can see 2 minor issues:
# {{DFSTestUtil#addBlockToFile}} is very similar as {{addStripedBlockToFile}}.
At the minimum we should change the comment (not adding a "block group").
Ideally we should consolidate the two methods with a switch for striped or not.
# The {{TestCommitBlock}} class only has 1 test and should be named more
specifically, something like {{TestCommitBlockWithInvalidGenStamp}}
If we create a new {{InvalidGenStampException}} class, I guess we should apply
it to all GS-related NN exceptions? E.g. {{BlockManager#checkReplicaCorrupt}}.
How about we do it under a separate JIRA?
> Make DataStreamer#block thread safe and verify genStamp in commitBlock
> --
>
> Key: HDFS-9289
> URL: https://issues.apache.org/jira/browse/HDFS-9289
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Chang Li
>Assignee: Chang Li
>Priority: Critical
> Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch,
> HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch
>
>
> we have seen a case of corrupt block which is caused by file complete after a
> pipelineUpdate, but the file complete with the old block genStamp. This
> caused the replicas of two datanodes in updated pipeline to be viewed as
> corrupte. Propose to check genstamp when commit block
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14981971#comment-14981971 ] Zhe Zhang commented on HDFS-9289: - Thanks Chang for the update. The main code change looks good. I'll take a closer look at the test tomorrow. I just updated the JIRA title. Please see if it looks OK. > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9289) Make DataStreamer#block thread safe and verify genStamp in commitBlock
[ https://issues.apache.org/jira/browse/HDFS-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14982617#comment-14982617 ] Chang Li commented on HDFS-9289: Thanks [~zhz], the jira title looks good to me. Looking forward to your further review. > Make DataStreamer#block thread safe and verify genStamp in commitBlock > -- > > Key: HDFS-9289 > URL: https://issues.apache.org/jira/browse/HDFS-9289 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Chang Li >Assignee: Chang Li >Priority: Critical > Attachments: HDFS-9289.1.patch, HDFS-9289.2.patch, HDFS-9289.3.patch, > HDFS-9289.4.patch, HDFS-9289.5.patch, HDFS-9289.6.patch > > > we have seen a case of corrupt block which is caused by file complete after a > pipelineUpdate, but the file complete with the old block genStamp. This > caused the replicas of two datanodes in updated pipeline to be viewed as > corrupte. Propose to check genstamp when commit block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
