[
https://issues.apache.org/jira/browse/PHOENIX-6288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17258242#comment-17258242
]
Istvan Toth commented on PHOENIX-6288:
--------------------------------------
Looking at the 2.2 run in #178, I noticed something very strange:
in org.apache.phoenix.end2end.UngroupedIT.txt with 08:53 mtime has the
{noformat}
-------------------------------------------------------------------------------
Test set: org.apache.phoenix.end2end.UngroupedIT
-------------------------------------------------------------------------------
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.036 s <<<
FAILURE! - in org.apache.phoenix.end2end.UngroupedIT
org.apache.phoenix.end2end.UngroupedIT Time elapsed: 0.036 s <<< ERROR!
java.lang.RuntimeException: java.io.IOException: Shutting down
at org.apache.phoenix.query.BaseTest.initMiniCluster(BaseTest.java:557)
at org.apache.phoenix.query.BaseTest.setUpTestCluster(BaseTest.java:453)
at
org.apache.phoenix.query.BaseTest.checkClusterInitialized(BaseTest.java:439)
at org.apache.phoenix.query.BaseTest.setUpTestDriver(BaseTest.java:525)
at org.apache.phoenix.query.BaseTest.setUpTestDriver(BaseTest.java:520)
at
org.apache.phoenix.end2end.ParallelStatsDisabledIT.doSetup(ParallelStatsDisabledIT.java:62)
at sun.reflect.GeneratedMethodAccessor144.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.apache.phoenix.SystemExitRule$1.evaluate(SystemExitRule.java:40)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:27)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
at
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
at
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeLazy(JUnitCoreWrapper.java:119)
at
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:87)
at
org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
at
org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:158)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:383)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:344)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:417)
Caused by: java.io.IOException: Shutting down
at
org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:266)
at
org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:116)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniHBaseCluster(HBaseTestingUtility.java:1169)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1134)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1088)
at org.apache.phoenix.query.BaseTest.initMiniCluster(BaseTest.java:552)
... 38 more
Caused by: java.lang.RuntimeException: Master not active after 200000ms
at
org.apache.hadoop.hbase.util.JVMClusterUtil.waitForEvent(JVMClusterUtil.java:229)
at
org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:185)
at
org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:413)
at
org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:259)
... 43 more
{noformat}
error.
At the same time, in org.apache.phoenix.end2end.UngroupedIT-output.txt we have
the following lines:
{noformat}
2021-01-04 08:53:48,120 INFO [main]
org.apache.hadoop.hbase.HBaseTestingUtility(467):
System.getProperty("hadoop.log.dir") already set to:
/home/jenkins/jenkins-agent/workspace/oenix_Phoenix-mulitbranch_master/HBASE_2.2/phoenix-core/target/test-data/adda432a-b81f-4dae-8d4a-f07757076886/hadoop.log.dir
so I do NOT create it in target/test-data/fde08e09-7415-a424-2509-9b3cf3b34de1
2021-01-04 08:53:48,120 INFO [main]
org.apache.hadoop.hbase.HBaseTestingUtility(467):
System.getProperty("hadoop.tmp.dir") already set to:
/home/jenkins/jenkins-agent/workspace/oenix_Phoenix-mulitbranch_master/HBASE_2.2/phoenix-core/target/test-data/adda432a-b81f-4dae-8d4a-f07757076886/hadoop.tmp.dir
so I do NOT create it in target/test-data/fde08e09-7415-a424-2509-9b3cf3b34de1
2021-01-04 08:53:48,120 WARN [main]
org.apache.hadoop.hbase.HBaseTestingUtility(471): hadoop.tmp.dir property value
differs in configuration and system: Configuration=/tmp/hadoop-jenkins while
System=/home/jenkins/jenkins-agent/workspace/oenix_Phoenix-mulitbranch_master/HBASE_2.2/phoenix-core/target/test-data/adda432a-b81f-4dae-8d4a-f07757076886/hadoop.tmp.dir
Erasing configuration value by system value.
2021-01-04 08:53:48,121 DEBUG [main]
org.apache.hadoop.hbase.HBaseTestingUtility(350): Setting hbase.rootdir to
/home/jenkins/jenkins-agent/workspace/oenix_Phoenix-mulitbranch_master/HBASE_2.2/phoenix-core/target/test-data/fde08e09-7415-a424-2509-9b3cf3b34de1
2021-01-04 08:53:48,121 INFO [main]
org.apache.hadoop.hbase.HBaseTestingUtility(1108): Starting up minicluster with
option: StartMiniClusterOption{numMasters=1, masterClass=null,
numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null,
numZkServers=1, createRootDir=false, createWALDir=false}
2021-01-04 08:53:48,121 INFO [main]
org.apache.hadoop.hbase.HBaseTestingUtility(1122): STARTING DFS
2021-01-04 08:53:48,122 INFO [main]
org.apache.hadoop.hbase.HBaseTestingUtility(797): Setting test.cache.data to
/home/jenkins/jenkins-agent/workspace/oenix_Phoenix-mulitbranch_master/HBASE_2.2/phoenix-core/target/test-data/fde08e09-7415-a424-2509-9b3cf3b34de1/test.cache.data
in system properties and HBase conf
2021-01-04 08:53:48,123 INFO [main]
org.apache.hadoop.hbase.HBaseTestingUtility(797): Setting hadoop.tmp.dir to
/home/jenkins/jenkins-agent/workspace/oenix_Phoenix-mulitbranch_master/HBASE_2.2/phoenix-core/target/test-data/fde08e09-7415-a424-2509-9b3cf3b34de1/hadoop.tmp.dir
in system properties and HBase conf
2021-01-04 08:53:48,123 INFO [main]
org.apache.hadoop.hbase.HBaseTestingUtility(797): Setting hadoop.log.dir to
/home/jenkins/jenkins-agent/workspace/oenix_Phoenix-mulitbranch_master/HBASE_2.2/phoenix-core/target/test-data/fde08e09-7415-a424-2509-9b3cf3b34de1/hadoop.log.dir
in system properties and HBase conf
2021-01-04 08:53:48,123 INFO [main]
org.apache.hadoop.hbase.HBaseTestingUtility(797): Setting
mapreduce.cluster.local.dir to
/home/jenkins/jenkins-agent/workspace/oenix_Phoenix-mulitbranch_master/HBASE_2.2/phoenix-core/target/test-data/fde08e09-7415-a424-2509-9b3cf3b34de1/mapreduce.cluster.local.dir
in system properties and HBase conf
2021-01-04 08:53:48,124 INFO [main]
org.apache.hadoop.hbase.HBaseTestingUtility(797): Setting
mapreduce.cluster.temp.dir to
/home/jenkins/jenkins-agent/workspace/oenix_Phoenix-mulitbranch_master/HBASE_2.2/phoenix-core/target/test-data/fde08e09-7415-a424-2509-9b3cf3b34de1/mapreduce.cluster.temp.dir
in system properties and HBase conf
2021-01-04 08:53:48,124 INFO [main]
org.apache.hadoop.hbase.HBaseTestingUtility(784): read short circuit is OFF
2021-01-04 08:53:48,125 INFO [main]
org.apache.hadoop.hbase.HBaseTestingUtility(797): Setting
yarn.node-labels.fs-store.root-dir to
/home/jenkins/jenkins-agent/workspace/oenix_Phoenix-mulitbranch_master/HBASE_2.2/phoenix-core/target/test-data/fde08e09-7415-a424-2509-9b3cf3b34de1/yarn.node-labels.fs-store.root-dir
in system properties and HBase conf
2021-01-04 08:53:48,126 INFO [main]
org.apache.hadoop.hbase.HBaseTestingUtility(797): Setting
yarn.nodemanager.log-dirs to
/home/jenkins/jenkins-agent/workspace/oenix_Phoenix-mulitbranch_master/HBASE_2.2/phoenix-core/target/test-data/fde08e09-7415-a424-2509-9b3cf3b34de1/yarn.nodemanager.log-dirs
in system properties and HBase conf
2021-01-04 08:53:48,126 INFO [main]
org.apache.hadoop.hbase.HBaseTestingUtility(797): Setting
yarn.nodemanager.remote-app-log-dir to
/home/jenkins/jenkins-agent/workspace/oenix_Phoenix-mulitbranch_master/HBASE_2.2/phoenix-core/target/test-data/fde08e09-7415-a424-2509-9b3cf3b34de1/yarn.nodemanager.remote-app-log-dir
in system properties and HBase conf
2021-01-04 08:53:48,127 INFO [main]
org.apache.hadoop.hbase.HBaseTestingUtility(797): Setting
yarn.timeline-service.entity-group-fs-store.active-dir to
/home/jenkins/jenkins-agent/workspace/oenix_Phoenix-mulitbranch_master/HBASE_2.2/phoenix-core/target/test-data/fde08e09-7415-a424-2509-9b3cf3b34de1/yarn.timeline-service.entity-group-fs-store.active-dir
in system properties and HBase conf
2021-01-04 08:53:48,127 INFO [main]
org.apache.hadoop.hbase.HBaseTestingUtility(797): Setting
yarn.timeline-service.entity-group-fs-store.done-dir to
/home/jenkins/jenkins-agent/workspace/oenix_Phoenix-mulitbranch_master/HBASE_2.2/phoenix-core/target/test-data/fde08e09-7415-a424-2509-9b3cf3b34de1/yarn.timeline-service.entity-group-fs-store.done-dir
in system properties and HBase conf
2021-01-04 08:53:48,127 INFO [main]
org.apache.hadoop.hbase.HBaseTestingUtility(797): Setting
yarn.nodemanager.remote-app-log-dir to
/home/jenkins/jenkins-agent/workspace/oenix_Phoenix-mulitbranch_master/HBASE_2.2/phoenix-core/target/test-data/fde08e09-7415-a424-2509-9b3cf3b34de1/yarn.nodemanager.remote-app-log-dir
in system properties and HBase conf
2021-01-04 08:53:48,128 INFO [main]
org.apache.hadoop.hbase.HBaseTestingUtility(797): Setting
dfs.journalnode.edits.dir to
/home/jenkins/jenkins-agent/workspace/oenix_Phoenix-mulitbranch_master/HBASE_2.2/phoenix-core/target/test-data/fde08e09-7415-a424-2509-9b3cf3b34de1/dfs.journalnode.edits.dir
in system properties and HBase conf
2021-01-04 08:53:48,128 INFO [main]
org.apache.hadoop.hbase.HBaseTestingUtility(797): Setting
dfs.datanode.shared.file.descriptor.paths to
/home/jenkins/jenkins-agent/workspace/oenix_Phoenix-mulitbranch_master/HBASE_2.2/phoenix-core/target/test-data/fde08e09-7415-a424-2509-9b3cf3b34de1/dfs.datanode.shared.file.descriptor.paths
in system properties and HBase conf
2021-01-04 08:53:48,129 INFO [main]
org.apache.hadoop.hbase.HBaseTestingUtility(797): Setting nfs.dump.dir to
/home/jenkins/jenkins-agent/workspace/oenix_Phoenix-mulitbranch_master/HBASE_2.2/phoenix-core/target/test-data/fde08e09-7415-a424-2509-9b3cf3b34de1/nfs.dump.dir
in system properties and HBase conf
2021-01-04 08:53:48,129 INFO [main]
org.apache.hadoop.hbase.HBaseTestingUtility(797): Setting java.io.tmpdir to
/home/jenkins/jenkins-agent/workspace/oenix_Phoenix-mulitbranch_master/HBASE_2.2/phoenix-core/target/test-data/fde08e09-7415-a424-2509-9b3cf3b34de1/java.io.tmpdir
in system properties and HBase conf
2021-01-04 08:53:48,130 INFO [main]
org.apache.hadoop.hbase.HBaseTestingUtility(797): Setting java.io.tmpdir to
/home/jenkins/jenkins-agent/workspace/oenix_Phoenix-mulitbranch_master/HBASE_2.2/phoenix-core/target/test-data/fde08e09-7415-a424-2509-9b3cf3b34de1/java.io.tmpdir
in system properties and HBase conf
2021-01-04 08:53:48,130 INFO [main]
org.apache.hadoop.hbase.HBaseTestingUtility(797): Setting
dfs.journalnode.edits.dir to
/home/jenkins/jenkins-agent/workspace/oenix_Phoenix-mulitbranch_master/HBASE_2.2/phoenix-core/target/test-data/fde08e09-7415-a424-2509-9b3cf3b34de1/dfs.journalnode.edits.dir
in system properties and HBase conf
2021-01-04 08:53:48,130 INFO [main]
org.apache.hadoop.hbase.HBaseTestingUtility(797): Setting
dfs.provided.aliasmap.inmemory.leveldb.dir to
/home/jenkins/jenkins-agent/workspace/oenix_Phoenix-mulitbranch_master/HBASE_2.2/phoenix-core/target/test-data/fde08e09-7415-a424-2509-9b3cf3b34de1/dfs.provided.aliasmap.inmemory.leveldb.dir
in system properties and HBase conf
2021-01-04 08:53:51,525 DEBUG [main]
org.apache.hadoop.hbase.HBaseTestingUtility(683): Setting hbase.rootdir to
/home/jenkins/jenkins-agent/workspace/oenix_Phoenix-mulitbranch_master/HBASE_2.2/phoenix-core/target/test-data/fde08e09-7415-a424-2509-9b3cf3b34de1
2021-01-04 08:53:52,130 INFO [main]
org.apache.hadoop.hbase.HBaseTestingUtility(1469): The hbase.fs.tmp.dir is set
to
hdfs://localhost:42205/user/jenkins/test-data/71c34ba2-fc1c-687c-b297-c57b3007e49b/hbase-staging
2021-01-04 08:53:58,806 INFO [main]
org.apache.hadoop.hbase.HBaseTestingUtility(1180): Minicluster is up;
activeMaster=10f5e95dc76e,38237,1609750432134
2021-01-04 08:53:58,807 INFO [main] org.apache.phoenix.query.BaseTest(554):
HBase minicluster startup complete in 10686 ms
{noformat}
showing that the minicluster in fact DID come up in 10 seconds.
This seems to suggests that either
* maven has tried to start two miniclusters, one thread succeeded
* maven has mis-attributed the setup failure from a previous test to this one
* Use of the new @BeforeParam annotation has introduced some strange new bug
* I completely misread the code or the logs.
> Minicluster startup problems on Jenkins
> ---------------------------------------
>
> Key: PHOENIX-6288
> URL: https://issues.apache.org/jira/browse/PHOENIX-6288
> Project: Phoenix
> Issue Type: Bug
> Reporter: Istvan Toth
> Assignee: Istvan Toth
> Priority: Critical
> Fix For: 5.1.0, 4.16.0
>
>
> We are sporadically getting Test failures on Jenkins that are caused by the
> miniCluster startup timeouts.
> {noformat}
> Caused by: java.lang.RuntimeException: Master not active after 30000ms at
> org.apache.hadoop.hbase.util.JVMClusterUtil.waitForEvent(JVMClusterUtil.java:232)
>
> at
> org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:188)
> at
> org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:430)
> at org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:259)
> ... 43 more
> {noformat}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
