[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[ https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537407#comment-15537407 ] Arpit Agarwal commented on HDFS-10713: -- Thanks for the backport [~zhz]. > Throttle FsNameSystem lock warnings > --- > > Key: HDFS-10713 > URL: https://issues.apache.org/jira/browse/HDFS-10713 > Project: Hadoop HDFS > Issue Type: Bug > Components: logging, namenode >Reporter: Arpit Agarwal >Assignee: Hanisha Koneru > Fix For: 2.8.0, 2.7.4, 3.0.0-alpha2 > > Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch, > HDFS-10713.002.patch, HDFS-10713.003.patch, HDFS-10713.004.patch, > HDFS-10713.005.patch, HDFS-10713.006.patch, HDFS-10713.007.patch, > HDFS-10713.008.patch, HDFS-10713.009.patch, HDFS-10713.010.patch, > HDFS-10713.011.patch, HDFS-10713.012.patch, HDFS-10713.branch-2.000.patch > > > The NameNode logs a message if the FSNamesystem write lock is held by a > thread for over 1 second. These messages can be throttled to at one most one > per x minutes to avoid potentially filling up NN logs. We can also log the > number of suppressed notices since the last log message. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[ https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15537395#comment-15537395 ] Zhe Zhang commented on HDFS-10713: -- I backported this change to branch-2.7 since the original read/write lock logging patches are there. > Throttle FsNameSystem lock warnings > --- > > Key: HDFS-10713 > URL: https://issues.apache.org/jira/browse/HDFS-10713 > Project: Hadoop HDFS > Issue Type: Bug > Components: logging, namenode >Reporter: Arpit Agarwal >Assignee: Hanisha Koneru > Fix For: 2.8.0, 2.7.4, 3.0.0-alpha2 > > Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch, > HDFS-10713.002.patch, HDFS-10713.003.patch, HDFS-10713.004.patch, > HDFS-10713.005.patch, HDFS-10713.006.patch, HDFS-10713.007.patch, > HDFS-10713.008.patch, HDFS-10713.009.patch, HDFS-10713.010.patch, > HDFS-10713.011.patch, HDFS-10713.012.patch, HDFS-10713.branch-2.000.patch > > > The NameNode logs a message if the FSNamesystem write lock is held by a > thread for over 1 second. These messages can be throttled to at one most one > per x minutes to avoid potentially filling up NN logs. We can also log the > number of suppressed notices since the last log message. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[ https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526786#comment-15526786 ] Erik Krogen commented on HDFS-10713: [~arpitagarwal], I'm confident that my HDFS-10817 patch would not have added latency cost since it is just manipulating a ThreadLocal but I agree that there is potential for both this patch and HDFS-10475, which I am working on, to add overheads. I was hoping to do a little bit of microbenchmarking as part of HDFS-10475; I could include this patch in the comparison. > Throttle FsNameSystem lock warnings > --- > > Key: HDFS-10713 > URL: https://issues.apache.org/jira/browse/HDFS-10713 > Project: Hadoop HDFS > Issue Type: Bug > Components: logging, namenode >Reporter: Arpit Agarwal >Assignee: Hanisha Koneru > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch, > HDFS-10713.002.patch, HDFS-10713.003.patch, HDFS-10713.004.patch, > HDFS-10713.005.patch, HDFS-10713.006.patch, HDFS-10713.007.patch, > HDFS-10713.008.patch, HDFS-10713.009.patch, HDFS-10713.010.patch, > HDFS-10713.011.patch, HDFS-10713.012.patch, HDFS-10713.branch-2.000.patch > > > The NameNode logs a message if the FSNamesystem write lock is held by a > thread for over 1 second. These messages can be throttled to at one most one > per x minutes to avoid potentially filling up NN logs. We can also log the > number of suppressed notices since the last log message. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[ https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15526626#comment-15526626 ] Arpit Agarwal commented on HDFS-10713: -- bq. Since HDFS-10817 and related JIRAs are adding some overhead to a critical section of the NN (literally), has anybody measured throughput/latency differences when compared to branch-2? Hi [~xkrogen], do you have any thoughts on this question by Chris? > Throttle FsNameSystem lock warnings > --- > > Key: HDFS-10713 > URL: https://issues.apache.org/jira/browse/HDFS-10713 > Project: Hadoop HDFS > Issue Type: Bug > Components: logging, namenode >Reporter: Arpit Agarwal >Assignee: Hanisha Koneru > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch, > HDFS-10713.002.patch, HDFS-10713.003.patch, HDFS-10713.004.patch, > HDFS-10713.005.patch, HDFS-10713.006.patch, HDFS-10713.007.patch, > HDFS-10713.008.patch, HDFS-10713.009.patch, HDFS-10713.010.patch, > HDFS-10713.011.patch, HDFS-10713.012.patch, HDFS-10713.branch-2.000.patch > > > The NameNode logs a message if the FSNamesystem write lock is held by a > thread for over 1 second. These messages can be throttled to at one most one > per x minutes to avoid potentially filling up NN logs. We can also log the > number of suppressed notices since the last log message. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[
https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525118#comment-15525118
]
Hadoop QA commented on HDFS-10713:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
49s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
7s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} branch-2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
43s{color} | {color:green} branch-2 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 29s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 203 unchanged - 3 fixed = 204 total (was 206) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
44s{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 49m
31s{color} | {color:green} hadoop-hdfs in the patch passed with JDK v1.7.0_111.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}146m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:b59b8b7 |
| JIRA Issue | HDFS-10713 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830430/HDFS-10713.branch-2.000.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 31dc7abb6ef4 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | branch-2 / 809a45a |
| Default Java | 1.7.0_111 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_101
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[ https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524943#comment-15524943 ] Arpit Agarwal commented on HDFS-10713: -- +1 for the branch-2 patch, pending Jenkins. > Throttle FsNameSystem lock warnings > --- > > Key: HDFS-10713 > URL: https://issues.apache.org/jira/browse/HDFS-10713 > Project: Hadoop HDFS > Issue Type: Bug > Components: logging, namenode >Reporter: Arpit Agarwal >Assignee: Hanisha Koneru > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch, > HDFS-10713.002.patch, HDFS-10713.003.patch, HDFS-10713.004.patch, > HDFS-10713.005.patch, HDFS-10713.006.patch, HDFS-10713.007.patch, > HDFS-10713.008.patch, HDFS-10713.009.patch, HDFS-10713.010.patch, > HDFS-10713.011.patch, HDFS-10713.012.patch, HDFS-10713.branch-2.000.patch > > > The NameNode logs a message if the FSNamesystem write lock is held by a > thread for over 1 second. These messages can be throttled to at one most one > per x minutes to avoid potentially filling up NN logs. We can also log the > number of suppressed notices since the last log message. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[ https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15524693#comment-15524693 ] Hudson commented on HDFS-10713: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10491 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10491/]) HDFS-10713. Throttle FsNameSystem lock warnings. Contributed by Hanisha (arp: rev 059058f9614613667d5385f76022294e07e140aa) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java > Throttle FsNameSystem lock warnings > --- > > Key: HDFS-10713 > URL: https://issues.apache.org/jira/browse/HDFS-10713 > Project: Hadoop HDFS > Issue Type: Bug > Components: logging, namenode >Reporter: Arpit Agarwal >Assignee: Hanisha Koneru > Fix For: 3.0.0-alpha2 > > Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch, > HDFS-10713.002.patch, HDFS-10713.003.patch, HDFS-10713.004.patch, > HDFS-10713.005.patch, HDFS-10713.006.patch, HDFS-10713.007.patch, > HDFS-10713.008.patch, HDFS-10713.009.patch, HDFS-10713.010.patch, > HDFS-10713.011.patch, HDFS-10713.012.patch, HDFS-10713.branch-2.000.patch > > > The NameNode logs a message if the FSNamesystem write lock is held by a > thread for over 1 second. These messages can be throttled to at one most one > per x minutes to avoid potentially filling up NN logs. We can also log the > number of suppressed notices since the last log message. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[ https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15514601#comment-15514601 ] Hanisha Koneru commented on HDFS-10713: --- The junit tests pass locally. - hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock -- Intermittent test failure. A bug has already been raised for this: HDFS-10426 - hadoop.hdfs.server.datanode.TestDataNodeLifeline -- Intermittent test failure. A bug has already been raised for this: HDFS-9911 > Throttle FsNameSystem lock warnings > --- > > Key: HDFS-10713 > URL: https://issues.apache.org/jira/browse/HDFS-10713 > Project: Hadoop HDFS > Issue Type: Bug > Components: logging, namenode >Reporter: Arpit Agarwal >Assignee: Hanisha Koneru > Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch, > HDFS-10713.002.patch, HDFS-10713.003.patch, HDFS-10713.004.patch, > HDFS-10713.005.patch, HDFS-10713.006.patch, HDFS-10713.007.patch, > HDFS-10713.008.patch, HDFS-10713.009.patch, HDFS-10713.010.patch, > HDFS-10713.011.patch, HDFS-10713.012.patch > > > The NameNode logs a message if the FSNamesystem write lock is held by a > thread for over 1 second. These messages can be throttled to at one most one > per x minutes to avoid potentially filling up NN logs. We can also log the > number of suppressed notices since the last log message. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[
https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15505320#comment-15505320
]
Hadoop QA commented on HDFS-10713:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 25s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 194 unchanged - 3 fixed = 196 total (was 197) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 8s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 84m 34s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock |
| | hadoop.hdfs.server.datanode.TestDataNodeLifeline |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10713 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829300/HDFS-10713.012.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux cf02940b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 98bdb51 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16801/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16801/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16801/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16801/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Throttle FsNameSystem lock warnings
> ---
>
>
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[
https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504738#comment-15504738
]
Chris Douglas commented on HDFS-10713:
--
+0 on v11. Since HDFS-10817 and related JIRAs are adding some overhead to a
critical section of the NN (literally), has anybody measured throughput/latency
differences when compared to branch-2?
* Minor:
{noformat}
+ do {
+localLongestReadLock = longestReadLockHeldInterval.get();
+if (readLockInterval < localLongestReadLock) {
+ break;
+}
+ } while (!longestReadLockHeldInterval.compareAndSet(
+ localLongestReadLock, readLockInterval));
{noformat}
Equivalently
{code}
do {
localLongestReadLock = longestReadLockHeldInterval.get();
} while (localLongestReadLock - readLockInterval < 0
&& !longestReadLockHeldInterval.compareAndSet(localLongestReadLock,
readLockInterval));
{code}
Following
[guidance|https://docs.oracle.com/javase/8/docs/api/java/lang/System.html#nanoTime--]
on {{System#nanoTime}} and avoiding redundant assignment when
{{readLockInterval == localLongestReadLock}}.
* Are the {{\@VisibleForTesting}} methods {{setTimeStampOfLastReadLockReport}}
and {{setTimeStampOfLastWriteLockReport}} necessary for the unit test?
* Missing replacement call to {{FakeTimer#advance}}:
{noformat}
fsn.writeLock();
-Thread.sleep(writeLockReportingThreshold / 2);
fsn.writeUnlock();
{noformat}
> Throttle FsNameSystem lock warnings
> ---
>
> Key: HDFS-10713
> URL: https://issues.apache.org/jira/browse/HDFS-10713
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: logging, namenode
>Reporter: Arpit Agarwal
>Assignee: Hanisha Koneru
> Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch,
> HDFS-10713.002.patch, HDFS-10713.003.patch, HDFS-10713.004.patch,
> HDFS-10713.005.patch, HDFS-10713.006.patch, HDFS-10713.007.patch,
> HDFS-10713.008.patch, HDFS-10713.009.patch, HDFS-10713.010.patch,
> HDFS-10713.011.patch
>
>
> The NameNode logs a message if the FSNamesystem write lock is held by a
> thread for over 1 second. These messages can be throttled to at one most one
> per x minutes to avoid potentially filling up NN logs. We can also log the
> number of suppressed notices since the last log message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[ https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15501335#comment-15501335 ] Arpit Agarwal commented on HDFS-10713: -- +1 for the v11 patch. I'll hold off committing for a couple of days. > Throttle FsNameSystem lock warnings > --- > > Key: HDFS-10713 > URL: https://issues.apache.org/jira/browse/HDFS-10713 > Project: Hadoop HDFS > Issue Type: Bug > Components: logging, namenode >Reporter: Arpit Agarwal >Assignee: Hanisha Koneru > Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch, > HDFS-10713.002.patch, HDFS-10713.003.patch, HDFS-10713.004.patch, > HDFS-10713.005.patch, HDFS-10713.006.patch, HDFS-10713.007.patch, > HDFS-10713.008.patch, HDFS-10713.009.patch, HDFS-10713.010.patch, > HDFS-10713.011.patch > > > The NameNode logs a message if the FSNamesystem write lock is held by a > thread for over 1 second. These messages can be throttled to at one most one > per x minutes to avoid potentially filling up NN logs. We can also log the > number of suppressed notices since the last log message. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[
https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15500103#comment-15500103
]
Hadoop QA commented on HDFS-10713:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 25s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 194 unchanged - 3 fixed = 195 total (was 197) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 56m 11s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 75m 46s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.security.TestDelegationToken |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10713 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829073/HDFS-10713.011.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 29fb44fc5ac0 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / f67237c |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16786/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/16786/artifact/patchprocess/whitespace-eol.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16786/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16786/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16786/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[
https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498130#comment-15498130
]
Arpit Agarwal commented on HDFS-10713:
--
v10 patch lgtm. I think this block was left in by mistake in the second
do-while loop in readUnlock, it can be removed.
{code}
localLongestReadLock = longestReadLockHeldInterval.get();
if (readLockInterval > localLongestReadLock) {
longestReadLockHeldInterval.compareAndSet(
localLongestReadLock, readLockInterval);
}
{code}
Will hold off committing until next week in case Chris, Erik or others have
additional comments.
> Throttle FsNameSystem lock warnings
> ---
>
> Key: HDFS-10713
> URL: https://issues.apache.org/jira/browse/HDFS-10713
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: logging, namenode
>Reporter: Arpit Agarwal
>Assignee: Hanisha Koneru
> Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch,
> HDFS-10713.002.patch, HDFS-10713.003.patch, HDFS-10713.004.patch,
> HDFS-10713.005.patch, HDFS-10713.006.patch, HDFS-10713.007.patch,
> HDFS-10713.008.patch, HDFS-10713.009.patch, HDFS-10713.010.patch
>
>
> The NameNode logs a message if the FSNamesystem write lock is held by a
> thread for over 1 second. These messages can be throttled to at one most one
> per x minutes to avoid potentially filling up NN logs. We can also log the
> number of suppressed notices since the last log message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[
https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497756#comment-15497756
]
Hadoop QA commented on HDFS-10713:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 25s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 194 unchanged - 3 fixed = 195 total (was 197) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 62m 5s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 81m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10713 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12828918/HDFS-10713.010.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 709212a55191 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / f6f3a44 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16781/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/16781/artifact/patchprocess/whitespace-eol.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16781/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16781/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16781/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[
https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497753#comment-15497753
]
Hadoop QA commented on HDFS-10713:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
37s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 28s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 194 unchanged - 3 fixed = 196 total (was 197) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 78m
23s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 99m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10713 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12828915/HDFS-10713.009.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 071d92e9c5a1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / f6f3a44 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16780/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/16780/artifact/patchprocess/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16780/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16780/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Throttle FsNameSystem lock warnings
> ---
>
> Key: HDFS-10713
> URL: https://issues.apache.org/jira/browse/HDFS-10713
>
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[
https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497173#comment-15497173
]
Hadoop QA commented on HDFS-10713:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 23s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 194 unchanged - 3 fixed = 195 total (was 197) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 56m 47s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 75m 18s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDFSShell |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10713 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12828876/HDFS-10713.008.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux c0e04a852f20 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / b09a03c |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16774/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/16774/artifact/patchprocess/whitespace-eol.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16774/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16774/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16774/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[ https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497148#comment-15497148 ] Arpit Agarwal commented on HDFS-10713: -- Thanks for the updated patch Hanisha. longestReadLockHeldInterval needs its own do-while loop for atomic update (can be before the existing do-while loop). Also the three test functions in FsNameSystem (setTimer, setTimeStampOfLastReadLockReport and setTimeStampOfLastWriteLockReport) can be package private. LGTM otherwise. > Throttle FsNameSystem lock warnings > --- > > Key: HDFS-10713 > URL: https://issues.apache.org/jira/browse/HDFS-10713 > Project: Hadoop HDFS > Issue Type: Bug > Components: logging, namenode >Reporter: Arpit Agarwal >Assignee: Hanisha Koneru > Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch, > HDFS-10713.002.patch, HDFS-10713.003.patch, HDFS-10713.004.patch, > HDFS-10713.005.patch, HDFS-10713.006.patch, HDFS-10713.007.patch, > HDFS-10713.008.patch > > > The NameNode logs a message if the FSNamesystem write lock is held by a > thread for over 1 second. These messages can be throttled to at one most one > per x minutes to avoid potentially filling up NN logs. We can also log the > number of suppressed notices since the last log message. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[
https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15495093#comment-15495093
]
Hadoop QA commented on HDFS-10713:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 24s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 7 new + 194 unchanged - 3 fixed = 201 total (was 197) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 56m 32s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 75m 6s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.TestSecureNameNode |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10713 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12828775/HDFS-10713.007.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux e6f2cebfb395 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / ec3ea18 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16764/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16764/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16764/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16764/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Throttle FsNameSystem lock warnings
> ---
>
> Key: HDFS-10713
> URL:
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[ https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15488784#comment-15488784 ] Andrew Wang commented on HDFS-10713: Thanks for working on this Hanisha. I had the same review comment as Erik above about mocking to avoid sleeping in tests. Would prefer if we addressed it here so this patch is standalone. > Throttle FsNameSystem lock warnings > --- > > Key: HDFS-10713 > URL: https://issues.apache.org/jira/browse/HDFS-10713 > Project: Hadoop HDFS > Issue Type: Bug > Components: logging, namenode >Reporter: Arpit Agarwal >Assignee: Hanisha Koneru > Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch, > HDFS-10713.002.patch, HDFS-10713.003.patch, HDFS-10713.004.patch, > HDFS-10713.005.patch, HDFS-10713.006.patch > > > The NameNode logs a message if the FSNamesystem write lock is held by a > thread for over 1 second. These messages can be throttled to at one most one > per x minutes to avoid potentially filling up NN logs. We can also log the > number of suppressed notices since the last log message. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[
https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15488530#comment-15488530
]
Chris Douglas commented on HDFS-10713:
--
Using a {{synchronized}} block after releasing the read lock will limit read
concurrency. Can this use the pattern from HDFS-10742 that doesn't use locks?
> Throttle FsNameSystem lock warnings
> ---
>
> Key: HDFS-10713
> URL: https://issues.apache.org/jira/browse/HDFS-10713
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: logging, namenode
>Reporter: Arpit Agarwal
>Assignee: Hanisha Koneru
> Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch,
> HDFS-10713.002.patch, HDFS-10713.003.patch, HDFS-10713.004.patch,
> HDFS-10713.005.patch, HDFS-10713.006.patch
>
>
> The NameNode logs a message if the FSNamesystem write lock is held by a
> thread for over 1 second. These messages can be throttled to at one most one
> per x minutes to avoid potentially filling up NN logs. We can also log the
> number of suppressed notices since the last log message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[
https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15488521#comment-15488521
]
Erik Krogen commented on HDFS-10713:
One comment on this is that it would be ideal to change it to use
{{o.a.h.util.Timer.monotonicNow()}} instead of {{Time.monotonicNow()}} to allow
for the unit tests to complete quickly without real sleeps, especially now that
we have a longer sleep for the reporting threshold. However I plan to do this
as part of the work in progress in HDFS-10475 so I wouldn't worry about it
here.
Good work! Thanks for incorporating my read lock reporting changes into this
throttling.
> Throttle FsNameSystem lock warnings
> ---
>
> Key: HDFS-10713
> URL: https://issues.apache.org/jira/browse/HDFS-10713
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: logging, namenode
>Reporter: Arpit Agarwal
>Assignee: Hanisha Koneru
> Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch,
> HDFS-10713.002.patch, HDFS-10713.003.patch, HDFS-10713.004.patch,
> HDFS-10713.005.patch, HDFS-10713.006.patch
>
>
> The NameNode logs a message if the FSNamesystem write lock is held by a
> thread for over 1 second. These messages can be throttled to at one most one
> per x minutes to avoid potentially filling up NN logs. We can also log the
> number of suppressed notices since the last log message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[ https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15488362#comment-15488362 ] Hanisha Koneru commented on HDFS-10713: --- The unit test TestDFSUpgradeWithHA fails intermittently. Local rerun is successful. A bug has already been raised for this: HDFS-10644. > Throttle FsNameSystem lock warnings > --- > > Key: HDFS-10713 > URL: https://issues.apache.org/jira/browse/HDFS-10713 > Project: Hadoop HDFS > Issue Type: Bug > Components: logging, namenode >Reporter: Arpit Agarwal >Assignee: Hanisha Koneru > Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch, > HDFS-10713.002.patch, HDFS-10713.003.patch, HDFS-10713.004.patch, > HDFS-10713.005.patch, HDFS-10713.006.patch > > > The NameNode logs a message if the FSNamesystem write lock is held by a > thread for over 1 second. These messages can be throttled to at one most one > per x minutes to avoid potentially filling up NN logs. We can also log the > number of suppressed notices since the last log message. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[
https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15488329#comment-15488329
]
Hadoop QA commented on HDFS-10713:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 32s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 194 unchanged - 3 fixed = 195 total (was 197) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 69m 0s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 92m 34s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10713 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12828309/HDFS-10713.006.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 514ff61187c2 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 08d8e0b |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16734/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16734/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16734/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16734/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Throttle FsNameSystem lock warnings
> ---
>
> Key: HDFS-10713
> URL:
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[ https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15488093#comment-15488093 ] Arpit Agarwal commented on HDFS-10713: -- +1 pending Jenkins. > Throttle FsNameSystem lock warnings > --- > > Key: HDFS-10713 > URL: https://issues.apache.org/jira/browse/HDFS-10713 > Project: Hadoop HDFS > Issue Type: Bug > Components: logging, namenode >Reporter: Arpit Agarwal >Assignee: Hanisha Koneru > Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch, > HDFS-10713.002.patch, HDFS-10713.003.patch, HDFS-10713.004.patch, > HDFS-10713.005.patch, HDFS-10713.006.patch > > > The NameNode logs a message if the FSNamesystem write lock is held by a > thread for over 1 second. These messages can be throttled to at one most one > per x minutes to avoid potentially filling up NN logs. We can also log the > number of suppressed notices since the last log message. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[
https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15485124#comment-15485124
]
Arpit Agarwal commented on HDFS-10713:
--
Also patch will have to be rebased since HDFS-10742 just introduced
{{dfs.lock.suppress.warning.interval}}, we can reuse the same setting. Instead
of using conf.getLong you will have to use conf.getTimeDuration like.
{code}
conf.getTimeDuration(
DFSConfigKeys.DFS_LOCK_SUPPRESS_WARNING_INTERVAL_KEY,
DFSConfigKeys.DFS_LOCK_SUPPRESS_WARNING_INTERVAL_DEFAULT,
TimeUnit.MILLISECONDS),
{code}
> Throttle FsNameSystem lock warnings
> ---
>
> Key: HDFS-10713
> URL: https://issues.apache.org/jira/browse/HDFS-10713
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: logging, namenode
>Reporter: Arpit Agarwal
>Assignee: Hanisha Koneru
> Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch,
> HDFS-10713.002.patch, HDFS-10713.003.patch, HDFS-10713.004.patch,
> HDFS-10713.005.patch
>
>
> The NameNode logs a message if the FSNamesystem write lock is held by a
> thread for over 1 second. These messages can be throttled to at one most one
> per x minutes to avoid potentially filling up NN logs. We can also log the
> number of suppressed notices since the last log message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[
https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15485104#comment-15485104
]
Arpit Agarwal commented on HDFS-10713:
--
Thanks [~hanishakoneru]. Few more comments:
# Log statement in {{writeUnlock}} says _Longest read-lock held interval_. It
should be _Longest write-lock held interval_.
# Move the synchronized block inside the {{if (needReport && readLockInterval
>= this.readLockReportingThreshold)}} so every lock release does not have to
get the {{readUnlockSyncLock}} mutex.
# Nitpick: {{readUnlock}} and {{writeUnlock}} both call {{monotonicNow()}}
twice. We can cache the value returned by the first call to avoid the second
call.
> Throttle FsNameSystem lock warnings
> ---
>
> Key: HDFS-10713
> URL: https://issues.apache.org/jira/browse/HDFS-10713
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: logging, namenode
>Reporter: Arpit Agarwal
>Assignee: Hanisha Koneru
> Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch,
> HDFS-10713.002.patch, HDFS-10713.003.patch, HDFS-10713.004.patch,
> HDFS-10713.005.patch
>
>
> The NameNode logs a message if the FSNamesystem write lock is held by a
> thread for over 1 second. These messages can be throttled to at one most one
> per x minutes to avoid potentially filling up NN logs. We can also log the
> number of suppressed notices since the last log message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[
https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15475110#comment-15475110
]
Hadoop QA commented on HDFS-10713:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 38s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 590 unchanged - 1 fixed = 591 total (was 591) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 88m 52s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}113m 6s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.server.namenode.ha.TestHAAppend |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery
|
| Timed out junit tests | org.apache.hadoop.hdfs.TestLeaseRecovery2 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10713 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12827636/HDFS-10713.005.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 8fbdeed16d65 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / cae33118 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16676/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16676/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16676/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
|
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[
https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15474750#comment-15474750
]
Hadoop QA commented on HDFS-10713:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 30s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 591 unchanged - 1 fixed = 592 total (was 592) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
48s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 3 new + 0
unchanged - 0 fixed = 3 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 58m 26s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 77m 34s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Synchronization on Integer in
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.readUnlock() At
FSNamesystem.java: At FSNamesystem.java:[line 1569] |
| | Synchronization on FSNamesystem.numReadLockWarningsSuppressed in futile
attempt to guard it At FSNamesystem.java:attempt to guard it At
FSNamesystem.java:[line 1578] |
| | Synchronization on FSNamesystem.numReadLockWarningsSuppressed in futile
attempt to guard it At FSNamesystem.java:attempt to guard it At
FSNamesystem.java:[line 1583] |
| Failed junit tests | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.TestCrcCorruption |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10713 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12827621/HDFS-10713.004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 1f3bae201d94 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 20a20c2 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[
https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15468265#comment-15468265
]
Arpit Agarwal commented on HDFS-10713:
--
Thanks for the updated patch [~hanishakoneru]. The read lock is not exclusive
so the following sequence of operations in readUnlock won't see a consistent
state. This block of code probably needs to synchronize on a separate mutex
after releasing the read lock.
{code}
if (needReport && readLockInterval >= this.readLockReportingThreshold) {
if (readLockInterval > longestReadLockHeldInterval) {
longestReadLockHeldInterval = readLockInterval;
}
if (currentTime - timeStampOfLastReadLockReport > this
.lockSuppressWarningInterval) {
logReport = true;
numSuppressedWarnings = numReadLockWarningsSuppressed;
numReadLockWarningsSuppressed = 0;
longestLockHeldInterval = longestReadLockHeldInterval;
longestReadLockHeldInterval = 0;
timeStampOfLastReadLockReport = currentTime;
} else {
numReadLockWarningsSuppressed++;
}
}
this.fsLock.readLock().unlock();
{code}
I also had some questions about the _needReport_ logic I posted on HDFS-10817.
> Throttle FsNameSystem lock warnings
> ---
>
> Key: HDFS-10713
> URL: https://issues.apache.org/jira/browse/HDFS-10713
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: logging, namenode
>Reporter: Arpit Agarwal
>Assignee: Hanisha Koneru
> Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch,
> HDFS-10713.002.patch, HDFS-10713.003.patch
>
>
> The NameNode logs a message if the FSNamesystem write lock is held by a
> thread for over 1 second. These messages can be throttled to at one most one
> per x minutes to avoid potentially filling up NN logs. We can also log the
> number of suppressed notices since the last log message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[
https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15460191#comment-15460191
]
Hadoop QA commented on HDFS-10713:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 32s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 590 unchanged - 1 fixed = 591 total (was 591) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 77m
36s{color} | {color:green} hadoop-hdfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 99m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10713 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12826950/HDFS-10713.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 9db6c4e88dde 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 07650bc |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16627/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16627/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16627/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Throttle FsNameSystem lock warnings
> ---
>
> Key: HDFS-10713
> URL: https://issues.apache.org/jira/browse/HDFS-10713
> Project: Hadoop HDFS
> Issue
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[ https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459771#comment-15459771 ] Erik Krogen commented on HDFS-10713: In HDFS-10475 we hope to add more detailed metrics for which calls are causing long held locks. We are actively working on this so it should hopefully subsume any useful information that would be gathered by keeping all of the stack traces. > Throttle FsNameSystem lock warnings > --- > > Key: HDFS-10713 > URL: https://issues.apache.org/jira/browse/HDFS-10713 > Project: Hadoop HDFS > Issue Type: Bug > Components: logging, namenode >Reporter: Arpit Agarwal >Assignee: Hanisha Koneru > Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch, > HDFS-10713.002.patch > > > The NameNode logs a message if the FSNamesystem write lock is held by a > thread for over 1 second. These messages can be throttled to at one most one > per x minutes to avoid potentially filling up NN logs. We can also log the > number of suppressed notices since the last log message. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[
https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459759#comment-15459759
]
Mingliang Liu commented on HDFS-10713:
--
{quote}
If we just capture the longest lock held interval but lose the stack trace
could that be an acceptable solution. Mingliang Liu, what do you think?
{quote}
That makes sense in order to avoid overhead of too frequent capturing the stack
trace. If we find in the future we need the longest lock holding stack trace
heavily, we can add it back. Before that, operators can reduce the config
{{dfs.lock.suppress.warning.interval.ms}} for finer-granularity information.
Thanks.
> Throttle FsNameSystem lock warnings
> ---
>
> Key: HDFS-10713
> URL: https://issues.apache.org/jira/browse/HDFS-10713
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: logging, namenode
>Reporter: Arpit Agarwal
>Assignee: Hanisha Koneru
> Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch,
> HDFS-10713.002.patch
>
>
> The NameNode logs a message if the FSNamesystem write lock is held by a
> thread for over 1 second. These messages can be throttled to at one most one
> per x minutes to avoid potentially filling up NN logs. We can also log the
> number of suppressed notices since the last log message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[ https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15456206#comment-15456206 ] Arpit Agarwal commented on HDFS-10713: -- Re capturing the thread stack every time we see a higher lock held interval - worst case we may need to capture ~120 stack traces in 2 minutes. That will probably not add too much overhead but I am not sure. If we just capture the longest lock held interval but lose the stack trace could that be an acceptable solution. [~liuml07], what do you think? > Throttle FsNameSystem lock warnings > --- > > Key: HDFS-10713 > URL: https://issues.apache.org/jira/browse/HDFS-10713 > Project: Hadoop HDFS > Issue Type: Bug > Components: logging, namenode >Reporter: Arpit Agarwal >Assignee: Hanisha Koneru > Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch, > HDFS-10713.002.patch > > > The NameNode logs a message if the FSNamesystem write lock is held by a > thread for over 1 second. These messages can be throttled to at one most one > per x minutes to avoid potentially filling up NN logs. We can also log the > number of suppressed notices since the last log message. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[ https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15453832#comment-15453832 ] Hanisha Koneru commented on HDFS-10713: --- We can cache the thread's stackTrace and lock time and once a new thread lock warning is encountered, we can log the report for the previous thread. This way all thread's information will be preserved. > Throttle FsNameSystem lock warnings > --- > > Key: HDFS-10713 > URL: https://issues.apache.org/jira/browse/HDFS-10713 > Project: Hadoop HDFS > Issue Type: Bug > Components: logging, namenode >Reporter: Arpit Agarwal >Assignee: Hanisha Koneru > Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch, > HDFS-10713.002.patch > > > The NameNode logs a message if the FSNamesystem write lock is held by a > thread for over 1 second. These messages can be throttled to at one most one > per x minutes to avoid potentially filling up NN logs. We can also log the > number of suppressed notices since the last log message. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[ https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15453607#comment-15453607 ] Jing Zhao commented on HDFS-10713: -- bq. we have a look at the feasibility to expose this information to nntop metrics. No... not exposing this information to nntop, but to publish op -> lock-holding-time mapping information to metrics as nntop does today. > Throttle FsNameSystem lock warnings > --- > > Key: HDFS-10713 > URL: https://issues.apache.org/jira/browse/HDFS-10713 > Project: Hadoop HDFS > Issue Type: Bug > Components: logging, namenode >Reporter: Arpit Agarwal >Assignee: Hanisha Koneru > Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch, > HDFS-10713.002.patch > > > The NameNode logs a message if the FSNamesystem write lock is held by a > thread for over 1 second. These messages can be throttled to at one most one > per x minutes to avoid potentially filling up NN logs. We can also log the > number of suppressed notices since the last log message. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[
https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15453359#comment-15453359
]
Mingliang Liu commented on HDFS-10713:
--
{code:title=DFSConfigKeys.java}
414 // Threshold for how long the write lock warnings must be suppressed
415 public static final String DFS_LOCK_SUPPRESS_WARNING_INTERVAL_MS_KEY =
416 "dfs.lock.suppress.warning.interval.ms";
417 public static final long
DFS_LOCK_SUPPRESS_WARNING_INTERVAL_MS_DEFAULT =
418 12L;
{code}
And
{code:title=hdfs-default.xml}
2626
2627 dfs.lock.suppress.warning.interval.ms
2628 1000
2629 The interval between reporting lock warnings.
2630
2631
2632
{code}
I believe the default value of config key
{{dfs.lock.suppress.warning.interval.ms}} is 2 mins not 1 second?
Minor comments:
In line 1571, message {{"Number of suppressed write-lock reports: " +
numSuppressedWarnings);}} should have a "\n" or "\t" before it.
> Throttle FsNameSystem lock warnings
> ---
>
> Key: HDFS-10713
> URL: https://issues.apache.org/jira/browse/HDFS-10713
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: logging, namenode
>Reporter: Arpit Agarwal
>Assignee: Hanisha Koneru
> Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch,
> HDFS-10713.002.patch
>
>
> The NameNode logs a message if the FSNamesystem write lock is held by a
> thread for over 1 second. These messages can be throttled to at one most one
> per x minutes to avoid potentially filling up NN logs. We can also log the
> number of suppressed notices since the last log message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[
https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15452881#comment-15452881
]
Arpit Agarwal commented on HDFS-10713:
--
Also we can rename the config key to {{dfs.lock.suppress.warning.interval.ms}}.
Then it can also be used by the DataNode for HDFS-10742.
> Throttle FsNameSystem lock warnings
> ---
>
> Key: HDFS-10713
> URL: https://issues.apache.org/jira/browse/HDFS-10713
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: logging, namenode
>Reporter: Arpit Agarwal
>Assignee: Hanisha Koneru
> Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch
>
>
> The NameNode logs a message if the FSNamesystem write lock is held by a
> thread for over 1 second. These messages can be throttled to at one most one
> per x minutes to avoid potentially filling up NN logs. We can also log the
> number of suppressed notices since the last log message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[ https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15451986#comment-15451986 ] Arpit Agarwal commented on HDFS-10713: -- Thanks [~hanishakoneru]. The patch lgtm. The test case needs fixing. In TestFsNameSystem.java: # Line 319 - the assertFalse should be assertTrue. Also the comment on line 313 should be fixed. # Line 308 - the sleep can be removed. We can also change the writeLockReportingThreshold to 1 millisecond to cut down the test run time. > Throttle FsNameSystem lock warnings > --- > > Key: HDFS-10713 > URL: https://issues.apache.org/jira/browse/HDFS-10713 > Project: Hadoop HDFS > Issue Type: Bug > Components: logging, namenode >Reporter: Arpit Agarwal >Assignee: Hanisha Koneru > Attachments: HDFS-10713.000.patch, HDFS-10713.001.patch > > > The NameNode logs a message if the FSNamesystem write lock is held by a > thread for over 1 second. These messages can be throttled to at one most one > per x minutes to avoid potentially filling up NN logs. We can also log the > number of suppressed notices since the last log message. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[
https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15450964#comment-15450964
]
Hadoop QA commented on HDFS-10713:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 31s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 2 new + 586 unchanged - 1 fixed = 588 total (was 587) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 58m 58s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 80m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.tools.TestDFSZKFailoverController |
| | hadoop.hdfs.server.namenode.TestFSNamesystem |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10713 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12826297/HDFS-10713.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 1aab5711ce1f 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 20ae1fa |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16586/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/16586/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/16586/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/16586/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[
https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15449776#comment-15449776
]
Arpit Agarwal commented on HDFS-10713:
--
Thanks for the patch [~hanishakoneru].
# We can do the timestamp checks just before the
{{this.fsLock.writeLock().unlock()}} call, so the new state is protected by the
write lock. The actual logging can be done outside the write lock though. Then
you don't need to make the members volatile anymore.
# We can cache the value returned by {{monotonicNow()}} instead of calling it
twice.
> Throttle FsNameSystem lock warnings
> ---
>
> Key: HDFS-10713
> URL: https://issues.apache.org/jira/browse/HDFS-10713
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: logging, namenode
>Reporter: Arpit Agarwal
>Assignee: Hanisha Koneru
> Attachments: HDFS-10713.000.patch
>
>
> The NameNode logs a message if the FSNamesystem write lock is held by a
> thread for over 1 second. These messages can be throttled to at one most one
> per x minutes to avoid potentially filling up NN logs. We can also log the
> number of suppressed notices since the last log message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10713) Throttle FsNameSystem lock warnings
[
https://issues.apache.org/jira/browse/HDFS-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15447515#comment-15447515
]
Hadoop QA commented on HDFS-10713:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
6s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 31s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 9 new + 586 unchanged - 1 fixed = 595 total (was 587) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
6s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 6s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 83m 35s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | Increment of volatile field
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.numWriteLockWarningsSuppressed
in org.apache.hadoop.hdfs.server.namenode.FSNamesystem.writeUnlock() At
FSNamesystem.java:in
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.writeUnlock() At
FSNamesystem.java:[line 1560] |
| Failed junit tests | hadoop.tools.TestHdfsConfigFields |
| | hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Issue | HDFS-10713 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12826090/HDFS-10713.000.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 60b3080209c6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 8b57be1 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/16570/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| findbugs |
https://builds.apache.org/job/PreCommit-HDFS-Build/16570/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
|
| unit |
