[jira] [Commented] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager
[
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16370240#comment-16370240
]
Ted Yu commented on HBASE-19954:
Added warning in javadoc of the test classes.
2 test classes are not enough since clearing shutdown hooks is done when
MiniDFSCluster is shutdown in hadoop 3.
> ShutdownHook should check whether shutdown hook is tracked by
> ShutdownHookManager
> -
>
> Key: HBASE-19954
> URL: https://issues.apache.org/jira/browse/HBASE-19954
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: 19954.v1.txt, 19954.v2.txt, 19954.v3.txt, 19954.v4.txt
>
>
> Currently ShutdownHook#suppressHdfsShutdownHook() does the following:
> {code}
> synchronized (fsShutdownHooks) {
> boolean isFSCacheDisabled =
> fs.getConf().getBoolean("fs.hdfs.impl.disable.cache", false);
> if (!isFSCacheDisabled &&
> !fsShutdownHooks.containsKey(hdfsClientFinalizer)
> && !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> {code}
> There is no check that ShutdownHookManager still tracks the shutdown hook,
> leading to potential RuntimeException (as can be observed in hadoop3 Jenkins
> job).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager
[
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369765#comment-16369765
]
Mike Drob commented on HBASE-19954:
---
Right, so is it the same thing happening? We start and stop a DFS cluster with
no region server start in between? I assume using a local file system is not an
option for the intent if the test.
Splitting into multiple classes is fine. Don't need 3 though, just 2 is enough.
Please add a comment to the one that doesn't use minihbase with a warning to
developers to not add tests that try to stay region server there later.
> ShutdownHook should check whether shutdown hook is tracked by
> ShutdownHookManager
> -
>
> Key: HBASE-19954
> URL: https://issues.apache.org/jira/browse/HBASE-19954
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: 19954.v1.txt, 19954.v2.txt, 19954.v3.txt
>
>
> Currently ShutdownHook#suppressHdfsShutdownHook() does the following:
> {code}
> synchronized (fsShutdownHooks) {
> boolean isFSCacheDisabled =
> fs.getConf().getBoolean("fs.hdfs.impl.disable.cache", false);
> if (!isFSCacheDisabled &&
> !fsShutdownHooks.containsKey(hdfsClientFinalizer)
> && !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> {code}
> There is no check that ShutdownHookManager still tracks the shutdown hook,
> leading to potential RuntimeException (as can be observed in hadoop3 Jenkins
> job).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager
[
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369688#comment-16369688
]
Hadoop QA commented on HBASE-19954:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m
2s{color} | {color:blue} The patch file was not named according to hbase's
naming conventions. Please see
https://yetus.apache.org/documentation/0.7.0/precommit-patchnames for
instructions. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
54s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
4s{color} | {color:red} hbase-server: The patch generated 1 new + 4 unchanged -
1 fixed = 5 total (was 5) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
42s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
18m 52s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}103m
55s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}141m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19954 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12911176/19954.v3.txt |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux d716038ecad3 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 391790ddb0 |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/11574/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/11574/testReport/ |
| Max.
[jira] [Commented] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager
[
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369672#comment-16369672
]
Hadoop QA commented on HBASE-19954:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m
2s{color} | {color:blue} The patch file was not named according to hbase's
naming conventions. Please see
https://yetus.apache.org/documentation/0.7.0/precommit-patchnames for
instructions. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
41s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
3s{color} | {color:red} hbase-server: The patch generated 1 new + 7 unchanged -
1 fixed = 8 total (was 8) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
34s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
18m 38s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}135m
46s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}175m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19954 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12911174/19954.v2.txt |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 16e31f7bd63a 3.13.0-133-generic #182-Ubuntu SMP Tue Sep 19
15:49:21 UTC 2017 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / a9a6eed372 |
| maven | version: Apache Maven 3.5.2
(138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z) |
| Default Java | 1.8.0_151 |
| checkstyle |
[jira] [Commented] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager
[
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369649#comment-16369649
]
Ted Yu commented on HBASE-19954:
Mike:
I re-read HBASE-17922.
See my comment above on which patch added the shutdown hook clearing.
> ShutdownHook should check whether shutdown hook is tracked by
> ShutdownHookManager
> -
>
> Key: HBASE-19954
> URL: https://issues.apache.org/jira/browse/HBASE-19954
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: 19954.v1.txt, 19954.v2.txt, 19954.v3.txt
>
>
> Currently ShutdownHook#suppressHdfsShutdownHook() does the following:
> {code}
> synchronized (fsShutdownHooks) {
> boolean isFSCacheDisabled =
> fs.getConf().getBoolean("fs.hdfs.impl.disable.cache", false);
> if (!isFSCacheDisabled &&
> !fsShutdownHooks.containsKey(hdfsClientFinalizer)
> && !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> {code}
> There is no check that ShutdownHookManager still tracks the shutdown hook,
> leading to potential RuntimeException (as can be observed in hadoop3 Jenkins
> job).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager
[
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369643#comment-16369643
]
Mike Drob commented on HBASE-19954:
---
[~tedyu] - see the discussion on HBASE-17922 please. I suspect it is relevant.
> ShutdownHook should check whether shutdown hook is tracked by
> ShutdownHookManager
> -
>
> Key: HBASE-19954
> URL: https://issues.apache.org/jira/browse/HBASE-19954
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: 19954.v1.txt, 19954.v2.txt, 19954.v3.txt
>
>
> Currently ShutdownHook#suppressHdfsShutdownHook() does the following:
> {code}
> synchronized (fsShutdownHooks) {
> boolean isFSCacheDisabled =
> fs.getConf().getBoolean("fs.hdfs.impl.disable.cache", false);
> if (!isFSCacheDisabled &&
> !fsShutdownHooks.containsKey(hdfsClientFinalizer)
> && !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> {code}
> There is no check that ShutdownHookManager still tracks the shutdown hook,
> leading to potential RuntimeException (as can be observed in hadoop3 Jenkins
> job).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager
[
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369635#comment-16369635
]
Ted Yu commented on HBASE-19954:
In the @After method of TestBlockReorder, shutdownMiniCluster() is called.
Clearing shutdown hooks came from this patch:
https://issues.apache.org/jira/secure/attachment/12751805/HDFS-8838-HDFS-7285-20150821.patch#file-6
whose parent JIRA, HDFS-7285, was incorporated into hadoop3.
In patch v3, I separate the 3 subtests into 3 tests since they use different
replication count.
I ran the tests locally, including the new files, which passed.
> ShutdownHook should check whether shutdown hook is tracked by
> ShutdownHookManager
> -
>
> Key: HBASE-19954
> URL: https://issues.apache.org/jira/browse/HBASE-19954
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: 19954.v1.txt, 19954.v2.txt, 19954.v3.txt
>
>
> Currently ShutdownHook#suppressHdfsShutdownHook() does the following:
> {code}
> synchronized (fsShutdownHooks) {
> boolean isFSCacheDisabled =
> fs.getConf().getBoolean("fs.hdfs.impl.disable.cache", false);
> if (!isFSCacheDisabled &&
> !fsShutdownHooks.containsKey(hdfsClientFinalizer)
> && !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> {code}
> There is no check that ShutdownHookManager still tracks the shutdown hook,
> leading to potential RuntimeException (as can be observed in hadoop3 Jenkins
> job).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager
[
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369617#comment-16369617
]
stack commented on HBASE-19954:
---
What about this question: "Why is the exception seen?/Why in hadoop3 do we get
this exception and not in hadoop2?"
> ShutdownHook should check whether shutdown hook is tracked by
> ShutdownHookManager
> -
>
> Key: HBASE-19954
> URL: https://issues.apache.org/jira/browse/HBASE-19954
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: 19954.v1.txt, 19954.v2.txt
>
>
> Currently ShutdownHook#suppressHdfsShutdownHook() does the following:
> {code}
> synchronized (fsShutdownHooks) {
> boolean isFSCacheDisabled =
> fs.getConf().getBoolean("fs.hdfs.impl.disable.cache", false);
> if (!isFSCacheDisabled &&
> !fsShutdownHooks.containsKey(hdfsClientFinalizer)
> && !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> {code}
> There is no check that ShutdownHookManager still tracks the shutdown hook,
> leading to potential RuntimeException (as can be observed in hadoop3 Jenkins
> job).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager
[
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369613#comment-16369613
]
stack commented on HBASE-19954:
---
bq. Testing block reorder is orthogonal to shutdown hook installation.
No it is not. Both are meddling with HDFS in unorthodox ways.
FS should be same for all three tests, not change dependent on random order in
which tests run.
> ShutdownHook should check whether shutdown hook is tracked by
> ShutdownHookManager
> -
>
> Key: HBASE-19954
> URL: https://issues.apache.org/jira/browse/HBASE-19954
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: 19954.v1.txt, 19954.v2.txt
>
>
> Currently ShutdownHook#suppressHdfsShutdownHook() does the following:
> {code}
> synchronized (fsShutdownHooks) {
> boolean isFSCacheDisabled =
> fs.getConf().getBoolean("fs.hdfs.impl.disable.cache", false);
> if (!isFSCacheDisabled &&
> !fsShutdownHooks.containsKey(hdfsClientFinalizer)
> && !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> {code}
> There is no check that ShutdownHookManager still tracks the shutdown hook,
> leading to potential RuntimeException (as can be observed in hadoop3 Jenkins
> job).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager
[
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369611#comment-16369611
]
Ted Yu commented on HBASE-19954:
Patch v2 adds audience annotation for the ShutdownHookManager class.
hasShutdownHook() is exercised by the TestBlockReorder against hadoop3.
If specific scenario is desired to test hasShutdownHook(), let me know.
> ShutdownHook should check whether shutdown hook is tracked by
> ShutdownHookManager
> -
>
> Key: HBASE-19954
> URL: https://issues.apache.org/jira/browse/HBASE-19954
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: 19954.v1.txt, 19954.v2.txt
>
>
> Currently ShutdownHook#suppressHdfsShutdownHook() does the following:
> {code}
> synchronized (fsShutdownHooks) {
> boolean isFSCacheDisabled =
> fs.getConf().getBoolean("fs.hdfs.impl.disable.cache", false);
> if (!isFSCacheDisabled &&
> !fsShutdownHooks.containsKey(hdfsClientFinalizer)
> && !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> {code}
> There is no check that ShutdownHookManager still tracks the shutdown hook,
> leading to potential RuntimeException (as can be observed in hadoop3 Jenkins
> job).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager
[
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16369609#comment-16369609
]
Ted Yu commented on HBASE-19954:
Testing block reorder is orthogonal to shutdown hook installation.
When code for subtests (not just TestBlockReorder) is put into one subtest, it
deviates from the original intent of the tests.
ShutdownHook.install() is static. So more refactoring is needed for unit tests
to override.
> ShutdownHook should check whether shutdown hook is tracked by
> ShutdownHookManager
> -
>
> Key: HBASE-19954
> URL: https://issues.apache.org/jira/browse/HBASE-19954
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: 19954.v1.txt
>
>
> Currently ShutdownHook#suppressHdfsShutdownHook() does the following:
> {code}
> synchronized (fsShutdownHooks) {
> boolean isFSCacheDisabled =
> fs.getConf().getBoolean("fs.hdfs.impl.disable.cache", false);
> if (!isFSCacheDisabled &&
> !fsShutdownHooks.containsKey(hdfsClientFinalizer)
> && !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> {code}
> There is no check that ShutdownHookManager still tracks the shutdown hook,
> leading to potential RuntimeException (as can be observed in hadoop3 Jenkins
> job).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager
[
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368715#comment-16368715
]
stack commented on HBASE-19954:
---
Good.
So, we need to reinstall the hook for each test? Or break up this class into
multiple classes, one per test, so fs is reinitialized for each? Or collapse
the tests in this class all together into one test w a comment why the test is
so large?
> ShutdownHook should check whether shutdown hook is tracked by
> ShutdownHookManager
> -
>
> Key: HBASE-19954
> URL: https://issues.apache.org/jira/browse/HBASE-19954
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: 19954.v1.txt
>
>
> Currently ShutdownHook#suppressHdfsShutdownHook() does the following:
> {code}
> synchronized (fsShutdownHooks) {
> boolean isFSCacheDisabled =
> fs.getConf().getBoolean("fs.hdfs.impl.disable.cache", false);
> if (!isFSCacheDisabled &&
> !fsShutdownHooks.containsKey(hdfsClientFinalizer)
> && !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> {code}
> There is no check that ShutdownHookManager still tracks the shutdown hook,
> leading to potential RuntimeException (as can be observed in hadoop3 Jenkins
> job).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager
[
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368420#comment-16368420
]
Ted Yu commented on HBASE-19954:
Did some debugging by installing hadoop-common with additional logging into
local maven repo.
{code}
2018-02-17 16:14:14,573 INFO [Time-limited test]
util.ShutdownHookManager(286): clearing hooks
2018-02-17 16:14:14,588 INFO [Time-limited test]
hbase.HBaseTestingUtility(1114): Minicluster is down
2018-02-17 16:14:14,627 INFO [Time-limited test] hbase.ResourceChecker(172):
after: fs.TestBlockReorder#testBlockLocationReorder Thread=110 (was 8)
{code}
Note the above was the first test in TestBlockReorder where the {{hooks}} Set
of hadoop ShutdownHookManager was cleared (first line).
The 'Failed suppression' exception happened in the second subtest where the
FileSystem$Cache$ClientFinalizer instance was no longer in the Set.
I dumped the contents of the {{hooks}} Set at time of the exception and saw
fsdataset.impl.BlockPoolSlice instances but no ClientFinalizer instance.
After poking around hadoop ShutdownHookManager, I don't see bug.
> ShutdownHook should check whether shutdown hook is tracked by
> ShutdownHookManager
> -
>
> Key: HBASE-19954
> URL: https://issues.apache.org/jira/browse/HBASE-19954
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: 19954.v1.txt
>
>
> Currently ShutdownHook#suppressHdfsShutdownHook() does the following:
> {code}
> synchronized (fsShutdownHooks) {
> boolean isFSCacheDisabled =
> fs.getConf().getBoolean("fs.hdfs.impl.disable.cache", false);
> if (!isFSCacheDisabled &&
> !fsShutdownHooks.containsKey(hdfsClientFinalizer)
> && !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> {code}
> There is no check that ShutdownHookManager still tracks the shutdown hook,
> leading to potential RuntimeException (as can be observed in hadoop3 Jenkins
> job).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager
[
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368134#comment-16368134
]
stack commented on HBASE-19954:
---
You don't say why it happens. Why in hadoop3 do we get this exception and not
in hadoop2? Why is hook removed earlier or not installed?
Looking at the patch, there is refactoring and no test. Seems easy enough to
add. Compound checks like that done in ShutdownHook.java are easy to get wrong.
Needs Audience.
I tried it and seems to address the below.
I see this exception when TestBlockReorder fails against hadoop3. The exception
is:
{code}
java.lang.RuntimeException: Failed suppression of fs shutdown hook:
org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@771d03a8
at
org.apache.hadoop.hbase.regionserver.ShutdownHook.suppressHdfsShutdownHook(ShutdownHook.java:207)
at
org.apache.hadoop.hbase.regionserver.ShutdownHook.install(ShutdownHook.java:85)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:927)
at
org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:187)
at
org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:133)
at
org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:171)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:360)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1942)
at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:307)
at
org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:168)
at java.lang.Thread.run(Thread.java:745)
{code}
> ShutdownHook should check whether shutdown hook is tracked by
> ShutdownHookManager
> -
>
> Key: HBASE-19954
> URL: https://issues.apache.org/jira/browse/HBASE-19954
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Fix For: 2.0.0-beta-2
>
> Attachments: 19954.v1.txt
>
>
> Currently ShutdownHook#suppressHdfsShutdownHook() does the following:
> {code}
> synchronized (fsShutdownHooks) {
> boolean isFSCacheDisabled =
> fs.getConf().getBoolean("fs.hdfs.impl.disable.cache", false);
> if (!isFSCacheDisabled &&
> !fsShutdownHooks.containsKey(hdfsClientFinalizer)
> && !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> {code}
> There is no check that ShutdownHookManager still tracks the shutdown hook,
> leading to potential RuntimeException (as can be observed in hadoop3 Jenkins
> job).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager
[
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357267#comment-16357267
]
Ted Yu commented on HBASE-19954:
Here is related test output with exception (the first line was added for
debugging):
{code}
2018-02-08 17:22:23,382 DEBUG [RS:0;cn012:38253]
util.ShutdownHookManager$ShutdownHookManagerV2(94): removing hook
org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@6a0c707e
2018-02-08 17:22:23,382 DEBUG [M:0;cn012:41573] executor.ExecutorService(92):
Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/cn012:0,
corePoolSize=5, maxPoolSize=5
2018-02-08 17:22:23,382 DEBUG [M:0;cn012:41573] executor.ExecutorService(92):
Starting executor service name=M_LOG_REPLAY_OPS-master/cn012:0,
corePoolSize=10, maxPoolSize=10
2018-02-08 17:22:23,382 DEBUG [M:0;cn012:41573] executor.ExecutorService(92):
Starting executor service name=MASTER_TABLE_OPERATIONS-master/cn012:0,
corePoolSize=1, maxPoolSize=1
2018-02-08 17:22:23,389 ERROR [RS:0;cn012:38253]
helpers.MarkerIgnoringBase(159): * ABORTING region server
cn012.l42scl.hortonworks.com,38253,1518110542271: Unhandled: Failed suppression
of fs shutdown hook:
org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@6a0c707e *
java.lang.RuntimeException: Failed suppression of fs shutdown hook:
org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@6a0c707e
at
org.apache.hadoop.hbase.regionserver.ShutdownHook.suppressHdfsShutdownHook(ShutdownHook.java:207)
at
org.apache.hadoop.hbase.regionserver.ShutdownHook.install(ShutdownHook.java:85)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:927)
at
org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:187)
at
org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:133)
at
org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:171)
at java.security.AccessController.doPrivileged(Native Method)
{code}
I also added a debug log in
ShutdownHookManager#ShutdownHookManagerV2#addShutdownHook() which didn't show
up in the test output.
Since there was no check that hadoop tracks the ClientFinalizer before calling
#removeShutdownHook, the false return from hadoop resulted in the
RuntimeException to be thrown.
> ShutdownHook should check whether shutdown hook is tracked by
> ShutdownHookManager
> -
>
> Key: HBASE-19954
> URL: https://issues.apache.org/jira/browse/HBASE-19954
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 19954.v1.txt
>
>
> Currently ShutdownHook#suppressHdfsShutdownHook() does the following:
> {code}
> synchronized (fsShutdownHooks) {
> boolean isFSCacheDisabled =
> fs.getConf().getBoolean("fs.hdfs.impl.disable.cache", false);
> if (!isFSCacheDisabled &&
> !fsShutdownHooks.containsKey(hdfsClientFinalizer)
> && !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> {code}
> There is no check that ShutdownHookManager still tracks the shutdown hook,
> leading to potential RuntimeException (as can be observed in hadoop3 Jenkins
> job).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager
[
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357212#comment-16357212
]
stack commented on HBASE-19954:
---
What is the exception seen? Why does it happen? Why would the manager not be
tracking the shutdown hook still? Has it been removed? If so, by whom? Thanks.
> ShutdownHook should check whether shutdown hook is tracked by
> ShutdownHookManager
> -
>
> Key: HBASE-19954
> URL: https://issues.apache.org/jira/browse/HBASE-19954
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 19954.v1.txt
>
>
> Currently ShutdownHook#suppressHdfsShutdownHook() does the following:
> {code}
> synchronized (fsShutdownHooks) {
> boolean isFSCacheDisabled =
> fs.getConf().getBoolean("fs.hdfs.impl.disable.cache", false);
> if (!isFSCacheDisabled &&
> !fsShutdownHooks.containsKey(hdfsClientFinalizer)
> && !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> {code}
> There is no check that ShutdownHookManager still tracks the shutdown hook,
> leading to potential RuntimeException (as can be observed in hadoop3 Jenkins
> job).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager
[
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356879#comment-16356879
]
Ted Yu commented on HBASE-19954:
[~chia7712]:
Can you take a look ?
> ShutdownHook should check whether shutdown hook is tracked by
> ShutdownHookManager
> -
>
> Key: HBASE-19954
> URL: https://issues.apache.org/jira/browse/HBASE-19954
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 19954.v1.txt
>
>
> Currently ShutdownHook#suppressHdfsShutdownHook() does the following:
> {code}
> synchronized (fsShutdownHooks) {
> boolean isFSCacheDisabled =
> fs.getConf().getBoolean("fs.hdfs.impl.disable.cache", false);
> if (!isFSCacheDisabled &&
> !fsShutdownHooks.containsKey(hdfsClientFinalizer)
> && !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> {code}
> There is no check that ShutdownHookManager still tracks the shutdown hook,
> leading to potential RuntimeException (as can be observed in hadoop3 Jenkins
> job).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager
[
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356344#comment-16356344
]
Ted Yu commented on HBASE-19954:
Test failure was not related.
I ran the test locally with patch and it passed.
> ShutdownHook should check whether shutdown hook is tracked by
> ShutdownHookManager
> -
>
> Key: HBASE-19954
> URL: https://issues.apache.org/jira/browse/HBASE-19954
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 19954.v1.txt
>
>
> Currently ShutdownHook#suppressHdfsShutdownHook() does the following:
> {code}
> synchronized (fsShutdownHooks) {
> boolean isFSCacheDisabled =
> fs.getConf().getBoolean("fs.hdfs.impl.disable.cache", false);
> if (!isFSCacheDisabled &&
> !fsShutdownHooks.containsKey(hdfsClientFinalizer)
> && !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> {code}
> There is no check that ShutdownHookManager still tracks the shutdown hook,
> leading to potential RuntimeException (as can be observed in hadoop3 Jenkins
> job).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager
[
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356310#comment-16356310
]
Hadoop QA commented on HBASE-19954:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m
2s{color} | {color:blue} The patch file was not named according to hbase's
naming conventions. Please see
https://yetus.apache.org/documentation/0.7.0/precommit-patchnames for
instructions. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
54s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
6s{color} | {color:red} hbase-server: The patch generated 1 new + 7 unchanged -
1 fixed = 8 total (was 8) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 3 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
47s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
18m 21s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 92m 0s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}129m 45s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.procedure.TestFailedProcCleanup |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:eee3b01 |
| JIRA Issue | HBASE-19954 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12909675/19954.v1.txt |
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 64218491fde2 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / d95940776e |
| maven | version: Apache Maven 3.5.2
[jira] [Commented] (HBASE-19954) ShutdownHook should check whether shutdown hook is tracked by ShutdownHookManager
[
https://issues.apache.org/jira/browse/HBASE-19954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356187#comment-16356187
]
Ted Yu commented on HBASE-19954:
Patch v1 simplifies ShutdownHookManager by removing ShutdownHookManagerV1 which
was developed for Hadoop 20.x/1.x.
TestBlockReorder passes against hadoop3 - without the patch it fails.
> ShutdownHook should check whether shutdown hook is tracked by
> ShutdownHookManager
> -
>
> Key: HBASE-19954
> URL: https://issues.apache.org/jira/browse/HBASE-19954
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Major
> Attachments: 19954.v1.txt
>
>
> Currently ShutdownHook#suppressHdfsShutdownHook() does the following:
> {code}
> synchronized (fsShutdownHooks) {
> boolean isFSCacheDisabled =
> fs.getConf().getBoolean("fs.hdfs.impl.disable.cache", false);
> if (!isFSCacheDisabled &&
> !fsShutdownHooks.containsKey(hdfsClientFinalizer)
> && !ShutdownHookManager.deleteShutdownHook(hdfsClientFinalizer)) {
> {code}
> There is no check that ShutdownHookManager still tracks the shutdown hook,
> leading to potential RuntimeException (as can be observed in hadoop3 Jenkins
> job).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
