[
https://issues.apache.org/jira/browse/HBASE-19803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16338521#comment-16338521
]
Appy edited comment on HBASE-19803 at 1/25/18 1:23 AM:
-------------------------------------------------------
So to iterate faster in trying to understand what/when/how this happens, i
switched to hbase-http module and made following two changes:
- Added System.exit(0) at the end of all tests of TestConfServlet (since it
runs first)
- Added 5 sec sleep to TestHttpServer so i can {{kill -9 }}it when it's running
Some important mvn props:
reuseForks=false (that's our default in root pom)
surefire.firstPartForkCount = 1, so next jvm is not started until previous one
ends.
It's interesting that the error is thrown only after all tests finish running,
and not immediately when some test calls System.exit(0) or some test jvm
crashes.
{noformat}
$ mvn test -PrunAllTests -pl hbase-http
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.hadoop.hbase.http.conf.TestConfServlet
Running org.apache.hadoop.hbase.http.jmx.TestJMXJsonServlet
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.63 s - in
org.apache.hadoop.hbase.http.jmx.TestJMXJsonServlet
Running org.apache.hadoop.hbase.http.lib.TestStaticUserWebFilter
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.338 s - in
org.apache.hadoop.hbase.http.lib.TestStaticUserWebFilter
Running org.apache.hadoop.hbase.http.log.TestLogLevel
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.73 s - in
org.apache.hadoop.hbase.http.log.TestLogLevel
Running org.apache.hadoop.hbase.http.TestGlobalFilter
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.724 s - in
org.apache.hadoop.hbase.http.TestGlobalFilter
Running org.apache.hadoop.hbase.http.TestHtmlQuoting
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.143 s - in
org.apache.hadoop.hbase.http.TestHtmlQuoting
Running org.apache.hadoop.hbase.http.TestHttpRequestLog
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.376 s - in
org.apache.hadoop.hbase.http.TestHttpRequestLog
Running org.apache.hadoop.hbase.http.TestHttpRequestLogAppender
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.223 s - in
org.apache.hadoop.hbase.http.TestHttpRequestLogAppender
Running org.apache.hadoop.hbase.http.TestHttpServer
Running org.apache.hadoop.hbase.http.TestHttpServerLifecycle
Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 0.002 s - in
org.apache.hadoop.hbase.http.TestHttpServerLifecycle
Running org.apache.hadoop.hbase.http.TestHttpServerWebapps
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.546 s - in
org.apache.hadoop.hbase.http.TestHttpServerWebapps
Running org.apache.hadoop.hbase.http.TestPathFilter
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.766 s - in
org.apache.hadoop.hbase.http.TestPathFilter
Running org.apache.hadoop.hbase.http.TestServletFilter
Tests run: 3, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.757 s - in
org.apache.hadoop.hbase.http.TestServletFilter
Running org.apache.hadoop.hbase.http.TestSpnegoHttpServer
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.651 s - in
org.apache.hadoop.hbase.http.TestSpnegoHttpServer
Running org.apache.hadoop.hbase.http.TestSSLHttpServer
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.389 s - in
org.apache.hadoop.hbase.http.TestSSLHttpServer
Results:
Tests run: 31, Failures: 0, Errors: 0, Skipped: 7
------------------------------------------------------------------------
BUILD FAILURE
------------------------------------------------------------------------
Total time: 41.684 s
Finished at: 2018-01-24T17:17:36-08:00
Final Memory: 77M/867M
------------------------------------------------------------------------
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:2.20.1:test (default-test) on
project hbase-http: There are test failures.
[ERROR]
[ERROR] Please refer to
/Users/appy/apache/hbase/hbase-http/target/surefire-reports for the individual
test results.
[ERROR] Please refer to dump files (if any exist) [date]-jvmRun[N].dump,
[date].dumpstream and [date]-jvmRun[N].dumpstream.
[ERROR] ExecutionException The forked VM terminated without properly saying
goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /Users/appy/apache/hbase/hbase-http &&
/Library/Java/JavaVirtualMachines/jdk1.8.0_152.jdk/Contents/Home/jre/bin/java
-enableassertions -Dhbase.build.id=2018-01-25T01:16:55Z -Xmx2800m
-Djava.security.egd=file:/dev/./urandom -Djava.net.preferIPv4Stack=true
-Djava.awt.headless=true
-XX:ErrorFile=/Users/appy/apache/hbase/hbase-http/target/classes/hs_err_pid%p.log
-jar
/Users/appy/apache/hbase/hbase-http/target/surefire/surefirebooter1670234724455998120.jar
/Users/appy/apache/hbase/hbase-http/target/surefire
2018-01-24T17-17-04_960-jvmRun1 surefire826046890227282714tmp
surefire_08917843261632866625tmp
[ERROR] Process Exit Code: 0
[ERROR] Crashed tests:
[ERROR] org.apache.hadoop.hbase.http.conf.TestConfServlet
[ERROR] ExecutionException The forked VM terminated without properly saying
goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /Users/appy/apache/hbase/hbase-http &&
/Library/Java/JavaVirtualMachines/jdk1.8.0_152.jdk/Contents/Home/jre/bin/java
-enableassertions -Dhbase.build.id=2018-01-25T01:16:55Z -Xmx2800m
-Djava.security.egd=file:/dev/./urandom -Djava.net.preferIPv4Stack=true
-Djava.awt.headless=true
-XX:ErrorFile=/Users/appy/apache/hbase/hbase-http/target/classes/hs_err_pid%p.log
-jar
/Users/appy/apache/hbase/hbase-http/target/surefire/surefirebooter8079462470974596876.jar
/Users/appy/apache/hbase/hbase-http/target/surefire
2018-01-24T17-17-04_960-jvmRun1 surefire799216347605645008tmp
surefire_84750862494178897145tmp
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 137
[ERROR] Crashed tests:
[ERROR] org.apache.hadoop.hbase.http.TestHttpServer
[ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException:
ExecutionException The forked VM terminated without properly saying goodbye. VM
crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /Users/appy/apache/hbase/hbase-http &&
/Library/Java/JavaVirtualMachines/jdk1.8.0_152.jdk/Contents/Home/jre/bin/java
-enableassertions -Dhbase.build.id=2018-01-25T01:16:55Z -Xmx2800m
-Djava.security.egd=file:/dev/./urandom -Djava.net.preferIPv4Stack=true
-Djava.awt.headless=true
-XX:ErrorFile=/Users/appy/apache/hbase/hbase-http/target/classes/hs_err_pid%p.log
-jar
/Users/appy/apache/hbase/hbase-http/target/surefire/surefirebooter1670234724455998120.jar
/Users/appy/apache/hbase/hbase-http/target/surefire
2018-01-24T17-17-04_960-jvmRun1 surefire826046890227282714tmp
surefire_08917843261632866625tmp
[ERROR] Process Exit Code: 0
[ERROR] Crashed tests:
[ERROR] org.apache.hadoop.hbase.http.conf.TestConfServlet
[ERROR] ExecutionException The forked VM terminated without properly saying
goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /Users/appy/apache/hbase/hbase-http &&
/Library/Java/JavaVirtualMachines/jdk1.8.0_152.jdk/Contents/Home/jre/bin/java
-enableassertions -Dhbase.build.id=2018-01-25T01:16:55Z -Xmx2800m
-Djava.security.egd=file:/dev/./urandom -Djava.net.preferIPv4Stack=true
-Djava.awt.headless=true
-XX:ErrorFile=/Users/appy/apache/hbase/hbase-http/target/classes/hs_err_pid%p.log
-jar
/Users/appy/apache/hbase/hbase-http/target/surefire/surefirebooter8079462470974596876.jar
/Users/appy/apache/hbase/hbase-http/target/surefire
2018-01-24T17-17-04_960-jvmRun1 surefire799216347605645008tmp
surefire_84750862494178897145tmp
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 137
[ERROR] Crashed tests:
[ERROR] org.apache.hadoop.hbase.http.TestHttpServer
[ERROR] at
org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:496)
[ERROR] at
org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:443)
[ERROR] at
org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:295)
[ERROR] at
org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:246)
[ERROR] at
org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1124)
[ERROR] at
org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:954)
[ERROR] at
org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:832)
[ERROR] at
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
[ERROR] at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
[ERROR] at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
[ERROR] at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
[ERROR] at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
[ERROR] at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
[ERROR] at
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
[ERROR] at
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
[ERROR] at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
[ERROR] at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
[ERROR] at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
[ERROR] at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
[ERROR] at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
[ERROR] at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
[ERROR] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR] at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[ERROR] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR] at java.lang.reflect.Method.invoke(Method.java:498)
[ERROR] at
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
[ERROR] at
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
[ERROR] at
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
[ERROR] at
org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
[ERROR] Caused by:
org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM
terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /Users/appy/apache/hbase/hbase-http &&
/Library/Java/JavaVirtualMachines/jdk1.8.0_152.jdk/Contents/Home/jre/bin/java
-enableassertions -Dhbase.build.id=2018-01-25T01:16:55Z -Xmx2800m
-Djava.security.egd=file:/dev/./urandom -Djava.net.preferIPv4Stack=true
-Djava.awt.headless=true
-XX:ErrorFile=/Users/appy/apache/hbase/hbase-http/target/classes/hs_err_pid%p.log
-jar
/Users/appy/apache/hbase/hbase-http/target/surefire/surefirebooter8079462470974596876.jar
/Users/appy/apache/hbase/hbase-http/target/surefire
2018-01-24T17-17-04_960-jvmRun1 surefire799216347605645008tmp
surefire_84750862494178897145tmp
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 137
[ERROR] Crashed tests:
[ERROR] org.apache.hadoop.hbase.http.TestHttpServer
[ERROR] at
org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:686)
[ERROR] at
org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:535)
[ERROR] at
org.apache.maven.plugin.surefire.booterclient.ForkStarter.access$700(ForkStarter.java:116)
[ERROR] at
org.apache.maven.plugin.surefire.booterclient.ForkStarter$2.call(ForkStarter.java:431)
[ERROR] at
org.apache.maven.plugin.surefire.booterclient.ForkStarter$2.call(ForkStarter.java:408)
[ERROR] at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[ERROR] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[ERROR] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[ERROR] at java.lang.Thread.run(Thread.java:748)
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
{noformat}
was (Author: appy):
So to iterate faster in trying to understand what/when/how this happens, i
switched to hbase-http module, added System.exit(0) at the end of
--TestGlobalFilter#testServletFilter.-- all tests of TestConfServlet (since it
runs first)
forkedProcessTimeoutInSeconds=10sec (should be fine since no http test takes
more than 4 sec on my machine, see list below)
reuseForks=false (that's our default in root pom)
{noformat}
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.hadoop.hbase.http.conf.TestConfServlet
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.756 s - in
org.apache.hadoop.hbase.http.conf.TestConfServlet
Running org.apache.hadoop.hbase.http.jmx.TestJMXJsonServlet
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.806 s - in
org.apache.hadoop.hbase.http.jmx.TestJMXJsonServlet
Running org.apache.hadoop.hbase.http.lib.TestStaticUserWebFilter
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.274 s - in
org.apache.hadoop.hbase.http.lib.TestStaticUserWebFilter
Running org.apache.hadoop.hbase.http.log.TestLogLevel
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.786 s - in
org.apache.hadoop.hbase.http.log.TestLogLevel
Running org.apache.hadoop.hbase.http.TestGlobalFilter
Running org.apache.hadoop.hbase.http.TestHtmlQuoting
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.291 s - in
org.apache.hadoop.hbase.http.TestHtmlQuoting
Running org.apache.hadoop.hbase.http.TestHttpRequestLog
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.415 s - in
org.apache.hadoop.hbase.http.TestHttpRequestLog
Running org.apache.hadoop.hbase.http.TestHttpRequestLogAppender
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.253 s - in
org.apache.hadoop.hbase.http.TestHttpRequestLogAppender
Running org.apache.hadoop.hbase.http.TestHttpServer
Tests run: 15, Failures: 0, Errors: 0, Skipped: 2, Time elapsed: 3.275 s - in
org.apache.hadoop.hbase.http.TestHttpServer
Running org.apache.hadoop.hbase.http.TestHttpServerLifecycle
Tests run: 6, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 0.002 s - in
org.apache.hadoop.hbase.http.TestHttpServerLifecycle
Running org.apache.hadoop.hbase.http.TestHttpServerWebapps
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.557 s - in
org.apache.hadoop.hbase.http.TestHttpServerWebapps
Running org.apache.hadoop.hbase.http.TestPathFilter
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.71 s - in
org.apache.hadoop.hbase.http.TestPathFilter
Running org.apache.hadoop.hbase.http.TestServletFilter
Tests run: 3, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.82 s - in
org.apache.hadoop.hbase.http.TestServletFilter
Running org.apache.hadoop.hbase.http.TestSpnegoHttpServer
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.881 s - in
org.apache.hadoop.hbase.http.TestSpnegoHttpServer
Running org.apache.hadoop.hbase.http.TestSSLHttpServer
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.33 s - in
org.apache.hadoop.hbase.http.TestSSLHttpServer
Results:
Tests run: 48, Failures: 0, Errors: 0, Skipped: 9
{noformat}
> False positive for the HBASE-Find-Flaky-Tests job
> -------------------------------------------------
>
> Key: HBASE-19803
> URL: https://issues.apache.org/jira/browse/HBASE-19803
> Project: HBase
> Issue Type: Bug
> Reporter: Duo Zhang
> Priority: Major
>
> It reports two hangs for TestAsyncTableGetMultiThreaded, but I checked the
> surefire output
> https://builds.apache.org/job/HBASE-Flaky-Tests/24830/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestAsyncTableGetMultiThreaded-output.txt
> This one was likely to be killed in the middle of the run within 20 seconds.
> https://builds.apache.org/job/HBASE-Flaky-Tests/24852/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.client.TestAsyncTableGetMultiThreaded-output.txt
> This one was also killed within about 1 minutes.
> The test is declared as LargeTests so the time limit should be 10 minutes. It
> seems that the jvm may crash during the mvn test run and then we will kill
> all the running tests and then we may mark some of them as hang which leads
> to the false positive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
