[jira] [Commented] (HDFS-15293) Relax the condition for accepting a fsimage when receiving a checkpoint
[ https://issues.apache.org/jira/browse/HDFS-15293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110734#comment-17110734 ] Akira Ajisaka commented on HDFS-15293: -- Hi [~gabor.bota], if you create 3.1.4-RC1, would you include this? > Relax the condition for accepting a fsimage when receiving a checkpoint > > > Key: HDFS-15293 > URL: https://issues.apache.org/jira/browse/HDFS-15293 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Critical > Labels: multi-sbnn, release-blocker > Fix For: 3.3.0, 3.2.2, 2.10.1, 3.1.5 > > Attachments: HDFS-15293.001.patch, HDFS-15293.002.patch > > > HDFS-12979 introduced the logic that, if ANN sees consecutive fs image upload > from Standby with a small delta comparing to previous fsImage. ANN would > reject this image. This is to avoid overly frequent fsImage in case of when > there are multiple Standby node. However this check could be too stringent. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15293) Relax the condition for accepting a fsimage when receiving a checkpoint
[ https://issues.apache.org/jira/browse/HDFS-15293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110733#comment-17110733 ] Akira Ajisaka commented on HDFS-15293: -- Chery-picked to branch-3.3 and branch-3.3.0. > Relax the condition for accepting a fsimage when receiving a checkpoint > > > Key: HDFS-15293 > URL: https://issues.apache.org/jira/browse/HDFS-15293 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Critical > Labels: multi-sbnn, release-blocker > Fix For: 3.2.2, 2.10.1, 3.3.1, 3.1.5 > > Attachments: HDFS-15293.001.patch, HDFS-15293.002.patch > > > HDFS-12979 introduced the logic that, if ANN sees consecutive fs image upload > from Standby with a small delta comparing to previous fsImage. ANN would > reject this image. This is to avoid overly frequent fsImage in case of when > there are multiple Standby node. However this check could be too stringent. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15293) Relax the condition for accepting a fsimage when receiving a checkpoint
[ https://issues.apache.org/jira/browse/HDFS-15293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110561#comment-17110561 ] Chen Liang commented on HDFS-15293: --- I have committed to trunk, branch-3.2, branch-3.1 and branch-2.10. Thanks the reviewers! > Relax the condition for accepting a fsimage when receiving a checkpoint > > > Key: HDFS-15293 > URL: https://issues.apache.org/jira/browse/HDFS-15293 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Critical > Labels: multi-sbnn, release-blocker > Attachments: HDFS-15293.001.patch, HDFS-15293.002.patch > > > HDFS-12979 introduced the logic that, if ANN sees consecutive fs image upload > from Standby with a small delta comparing to previous fsImage. ANN would > reject this image. This is to avoid overly frequent fsImage in case of when > there are multiple Standby node. However this check could be too stringent. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15293) Relax the condition for accepting a fsimage when receiving a checkpoint
[ https://issues.apache.org/jira/browse/HDFS-15293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110550#comment-17110550 ] Hudson commented on HDFS-15293: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18272 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18272/]) HDFS-15293. Relax the condition for accepting a fsimage when receiving a (vagarychen: rev 7bb902bc0d0c62d63a8960db444de3abb0a6ad22) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ImageServlet.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCheckpoint.java > Relax the condition for accepting a fsimage when receiving a checkpoint > > > Key: HDFS-15293 > URL: https://issues.apache.org/jira/browse/HDFS-15293 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Critical > Labels: multi-sbnn, release-blocker > Attachments: HDFS-15293.001.patch, HDFS-15293.002.patch > > > HDFS-12979 introduced the logic that, if ANN sees consecutive fs image upload > from Standby with a small delta comparing to previous fsImage. ANN would > reject this image. This is to avoid overly frequent fsImage in case of when > there are multiple Standby node. However this check could be too stringent. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15293) Relax the condition for accepting a fsimage when receiving a checkpoint
[ https://issues.apache.org/jira/browse/HDFS-15293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17109727#comment-17109727 ] Akira Ajisaka commented on HDFS-15293: -- +1, thanks [~vagarychen] for the update. > Relax the condition for accepting a fsimage when receiving a checkpoint > > > Key: HDFS-15293 > URL: https://issues.apache.org/jira/browse/HDFS-15293 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Critical > Labels: multi-sbnn, release-blocker > Attachments: HDFS-15293.001.patch, HDFS-15293.002.patch > > > HDFS-12979 introduced the logic that, if ANN sees consecutive fs image upload > from Standby with a small delta comparing to previous fsImage. ANN would > reject this image. This is to avoid overly frequent fsImage in case of when > there are multiple Standby node. However this check could be too stringent. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15293) Relax the condition for accepting a fsimage when receiving a checkpoint
[ https://issues.apache.org/jira/browse/HDFS-15293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108710#comment-17108710 ] Konstantin Shvachko commented on HDFS-15293: Looks like you have one longish line #644. Other than that it looks great. +1 as well. > Relax the condition for accepting a fsimage when receiving a checkpoint > > > Key: HDFS-15293 > URL: https://issues.apache.org/jira/browse/HDFS-15293 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Critical > Labels: multi-sbnn, release-blocker > Attachments: HDFS-15293.001.patch, HDFS-15293.002.patch > > > HDFS-12979 introduced the logic that, if ANN sees consecutive fs image upload > from Standby with a small delta comparing to previous fsImage. ANN would > reject this image. This is to avoid overly frequent fsImage in case of when > there are multiple Standby node. However this check could be too stringent. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15293) Relax the condition for accepting a fsimage when receiving a checkpoint
[
https://issues.apache.org/jira/browse/HDFS-15293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108640#comment-17108640
]
Hadoop QA commented on HDFS-15293:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
55s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 24m
35s{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 19m
49s{color} | {color:red} branch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m
22s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 51s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 131 unchanged - 0 fixed = 132 total (was 131) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 14m
2s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 96m
36s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}173m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://builds.apache.org/job/PreCommit-HDFS-Build/29288/artifact/out/Dockerfile
|
| JIRA Issue | HDFS-15293 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13003060/HDFS-15293.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 835db8aa1034 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / ac4a2e11d98 |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDFS-Build/29288/artifact/out/branch-mvninstall-root.txt
|
| checkstyle |
[jira] [Commented] (HDFS-15293) Relax the condition for accepting a fsimage when receiving a checkpoint
[ https://issues.apache.org/jira/browse/HDFS-15293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108491#comment-17108491 ] Chen Liang commented on HDFS-15293: --- Updated v002 patch to address Akira's comments. > Relax the condition for accepting a fsimage when receiving a checkpoint > > > Key: HDFS-15293 > URL: https://issues.apache.org/jira/browse/HDFS-15293 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Critical > Labels: multi-sbnn, release-blocker > Attachments: HDFS-15293.001.patch, HDFS-15293.002.patch > > > HDFS-12979 introduced the logic that, if ANN sees consecutive fs image upload > from Standby with a small delta comparing to previous fsImage. ANN would > reject this image. This is to avoid overly frequent fsImage in case of when > there are multiple Standby node. However this check could be too stringent. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15293) Relax the condition for accepting a fsimage when receiving a checkpoint
[ https://issues.apache.org/jira/browse/HDFS-15293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108470#comment-17108470 ] Chen Liang commented on HDFS-15293: --- Hi [~aajisaka] sorry I have been busy dealing with some internal work. Will update the patch later today. Also [~shv] would like to get your thoughts on this as you have been looking into our internal version of this fix. > Relax the condition for accepting a fsimage when receiving a checkpoint > > > Key: HDFS-15293 > URL: https://issues.apache.org/jira/browse/HDFS-15293 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Critical > Labels: multi-sbnn, release-blocker > Attachments: HDFS-15293.001.patch > > > HDFS-12979 introduced the logic that, if ANN sees consecutive fs image upload > from Standby with a small delta comparing to previous fsImage. ANN would > reject this image. This is to avoid overly frequent fsImage in case of when > there are multiple Standby node. However this check could be too stringent. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15293) Relax the condition for accepting a fsimage when receiving a checkpoint
[ https://issues.apache.org/jira/browse/HDFS-15293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17107966#comment-17107966 ] Akira Ajisaka commented on HDFS-15293: -- Hi [~vagarychen], would you update the patch to address my above comment? This issue blocks some releases, so I'd like to resolve this earlier. > Relax the condition for accepting a fsimage when receiving a checkpoint > > > Key: HDFS-15293 > URL: https://issues.apache.org/jira/browse/HDFS-15293 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Labels: multi-sbnn, release-blocker > Attachments: HDFS-15293.001.patch > > > HDFS-12979 introduced the logic that, if ANN sees consecutive fs image upload > from Standby with a small delta comparing to previous fsImage. ANN would > reject this image. This is to avoid overly frequent fsImage in case of when > there are multiple Standby node. However this check could be too stringent. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15293) Relax the condition for accepting a fsimage when receiving a checkpoint
[
https://issues.apache.org/jira/browse/HDFS-15293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17104510#comment-17104510
]
Kihwal Lee commented on HDFS-15293:
---
{quote}Because the checks only skip an image if BOTH conditions are met: 1.
time delta too small AND 2. txnid delta too small. It's an AND not OR.
{quote}
You are right. It works as is since the condition for the rejection will not be
met if a checkpointing was created due to the configured txns limit.
+1
> Relax the condition for accepting a fsimage when receiving a checkpoint
>
>
> Key: HDFS-15293
> URL: https://issues.apache.org/jira/browse/HDFS-15293
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Chen Liang
>Assignee: Chen Liang
>Priority: Major
> Labels: multi-sbnn, release-blocker
> Attachments: HDFS-15293.001.patch
>
>
> HDFS-12979 introduced the logic that, if ANN sees consecutive fs image upload
> from Standby with a small delta comparing to previous fsImage. ANN would
> reject this image. This is to avoid overly frequent fsImage in case of when
> there are multiple Standby node. However this check could be too stringent.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15293) Relax the condition for accepting a fsimage when receiving a checkpoint
[ https://issues.apache.org/jira/browse/HDFS-15293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101307#comment-17101307 ] Akira Ajisaka commented on HDFS-15293: -- Thank you [~shv] and [~vagarychen]. The change looks straightforward to me. Minor nits: * ImageServlet#setRecentImageCheckTimePrecision can be package-private. * MiniDFSCluster is AutoCloseable, so you can use try-with-resources in the test. I'm +1 if these are addressed. The test failures are not related to the patch. All the tests except TestBPOfferService passed on my local. TestBPOfferService: HDFS-15296 > Relax the condition for accepting a fsimage when receiving a checkpoint > > > Key: HDFS-15293 > URL: https://issues.apache.org/jira/browse/HDFS-15293 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Labels: multi-sbnn > Attachments: HDFS-15293.001.patch > > > HDFS-12979 introduced the logic that, if ANN sees consecutive fs image upload > from Standby with a small delta comparing to previous fsImage. ANN would > reject this image. This is to avoid overly frequent fsImage in case of when > there are multiple Standby node. However this check could be too stringent. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15293) Relax the condition for accepting a fsimage when receiving a checkpoint
[
https://issues.apache.org/jira/browse/HDFS-15293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101256#comment-17101256
]
Hadoop QA commented on HDFS-15293:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 19s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m
35s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
31s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 46s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 131 unchanged - 0 fixed = 132 total (was 131) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 29s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}106m 37s{color}
| {color:red} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}185m 26s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestFileCreation |
| | hadoop.hdfs.server.datanode.TestBPOfferService |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
|
| | hadoop.hdfs.TestSafeModeWithStripedFile |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://builds.apache.org/job/PreCommit-HDFS-Build/29241/artifact/out/Dockerfile
|
| JIRA Issue | HDFS-15293 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13002214/HDFS-15293.001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 26e52af6471a 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 92e3ebb4019 |
| Default
[jira] [Commented] (HDFS-15293) Relax the condition for accepting a fsimage when receiving a checkpoint
[ https://issues.apache.org/jira/browse/HDFS-15293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101136#comment-17101136 ] Chen Liang commented on HDFS-15293: --- Had some offline discussion with [~shv], the txnid check does not seem be relevant here actually. Post v001 patch. This is based on our internal version of this fix, with some additional logging added to capture this behavior. > Relax the condition for accepting a fsimage when receiving a checkpoint > > > Key: HDFS-15293 > URL: https://issues.apache.org/jira/browse/HDFS-15293 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Labels: multi-sbnn > Attachments: HDFS-15293.001.patch > > > HDFS-12979 introduced the logic that, if ANN sees consecutive fs image upload > from Standby with a small delta comparing to previous fsImage. ANN would > reject this image. This is to avoid overly frequent fsImage in case of when > there are multiple Standby node. However this check could be too stringent. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15293) Relax the condition for accepting a fsimage when receiving a checkpoint
[ https://issues.apache.org/jira/browse/HDFS-15293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094792#comment-17094792 ] Chen Liang commented on HDFS-15293: --- [~shv] I don't think the the issue you mentioned will actually happen currently. Because the checks only skip an image if BOTH conditions are met: 1. time delta too small AND 2. txnid delta too small. It's an AND not OR. So in the case you mentioned, it is true that time delta will always be considered too small due to the ridiculously large interval, but if configured with a small txnid, it is easy to get enough txnid, so txnid delta won't be considered too small. It is not that time delta being small alone leads to rejecting an image. But indeed, it is possible that in a cluster with ridiculously large interval, plus a extremely light load (so txnid barely make progress), both conditions will always be true. In this case the checkpoint will all be rejected. Although realistically I don't think there is much value doing checkpoint in such situation any way, it is probably not a good idea to change behavior of the system by effectively rejecting all images from happening. Because of this, I'm thinking of removing the txnid condition all together, so the check only looks at time delta and allow any txnid delta. It seems more tricky to justify preventing all the use cases with slow txnid increase. (Time always proceed, but not necessarily txnid.) I think we were targeting mainly time condition originally. > Relax the condition for accepting a fsimage when receiving a checkpoint > > > Key: HDFS-15293 > URL: https://issues.apache.org/jira/browse/HDFS-15293 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Chen Liang >Assignee: Chen Liang >Priority: Major > Labels: multi-sbnn > > HDFS-12979 introduced the logic that, if ANN sees consecutive fs image upload > from Standby with a small delta comparing to previous fsImage. ANN would > reject this image. This is to avoid overly frequent fsImage in case of when > there are multiple Standby node. However this check could be too stringent. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
