I see these lines: 2015-07-20 21:27:21,791 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion (HRegion.java:createHRegion(5598)) - creating HRegion hbase:namespace HTD == 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', VERSIONS => '10', IN_MEMORY => 'true', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'true', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} RootDir = file:/Users/ssa/devel/myown/hadoop/mini-hdfs-cluster-maven-plugin/target/hbase-root/.tmp Table name == hbase:namespace
2015-07-20 21:27:21,802 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion (HRegion.java:doClose(1425)) - Closed hbase:namespace,,1437420441756.a69bbbcaf4a82786964da8d9cc62bea3. 2015-07-20 21:27:21,861 INFO [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:61781-0] hbase.MetaTableAccessor (MetaTableAccessor.java:addRegionsToMeta(1169)) - Added 2 2015-07-20 21:27:21,864 WARN [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:61781-0] zookeeper.ZKTableStateManager (ZKTableStateManager.java:setTableState(100)) - Moving table hbase:namespace state from ENABLING to ENABLED 2015-07-20 21:27:21,866 INFO [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:61781-0] handler.CreateTableHandler (CreateTableHandler.java:completed(219)) - Table, hbase:namespace, creation successful then it hungs and prints thread dump in few minutes. 2015-07-20 21:25 GMT+02:00 Esteban Gutierrez <este...@cloudera.com>: > > But do you see that thread printing anything in the logs? > > -- > Cloudera, Inc. > > > On Mon, Jul 20, 2015 at 12:07 PM, Serega Sheypak <serega.shey...@gmail.com > > wrote: > >> This is testing utiliy, it has few bytes of data to load. Running on >> oracle-jdk8 >> >> java.lang.Thread.run(Thread.java:745) >> Thread 178 (JvmPauseMonitor): >> State: TIMED_WAITING >> Blocked count: 3 >> Waited count: 398 >> Stack: >> java.lang.Thread.sleep(Native Method) >> >> org.apache.hadoop.hbase.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:159) >> java.lang.Thread.run(Thread.java:745) >> >> Thread 177 (JvmPauseMonitor): >> State: TIMED_WAITING >> Blocked count: 1 >> Waited count: 398 >> Stack: >> java.lang.Thread.sleep(Native Method) >> >> org.apache.hadoop.hbase.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:159) >> java.lang.Thread.run(Thread.java:745) >> >> 2015-07-20 19:30 GMT+02:00 Esteban Gutierrez <este...@cloudera.com>: >> >>> -user@hbase >>> >>> Hi Serega, >>> >>> The RunTimeException is pointing to a timeout of nearly 3min. Have tried >>> to find in the master log lines what is causing that 3min pause? do you see >>> any log line related to the JvmPauseMonitor? (perhaps some GC going on) >>> >>> thanks, >>> esteban. >>> >>> -- >>> Cloudera, Inc. >>> >>> >>> On Mon, Jul 20, 2015 at 12:28 AM, Serega Sheypak < >>> serega.shey...@gmail.com> wrote: >>> >>>> @Sean, thanks. I saw sometimes Cloudera guys help here. I also used >>>> Cloudera community forum. >>>> >>>> @Jean-Marc, nothing special, just maven-plugin wrapper around >>>> miniHbaseCluster >>>> Here is the code where failure happen: new >>>> MiniHBaseCluster(configuration, >>>> 1); >>>> >>>> There are nothing special from my side. I'm surprised, it always worked >>>> since CDH 4.4, just bump dependency versions, fix code to follow API >>>> changes and that' all. >>>> >>>> 2015-07-20 4:08 GMT+02:00 Jean-Marc Spaggiari <jean-m...@spaggiari.org >>>> >: >>>> >>>> > Hi Serega, >>>> > >>>> > What kind of tests are your trying to run? The HBase test suite? Or >>>> > something you developed yourself? >>>> > >>>> > JM >>>> > >>>> > 2015-07-19 17:47 GMT-04:00 Serega Sheypak <serega.shey...@gmail.com>: >>>> > >>>> > > Hi, bumped my testing stuff to CDH 5.4.4 and got failure while >>>> running >>>> > > tests. Here is a log >>>> > > >>>> > > 2015-07-19 23:40:18,607 INFO >>>> [SERGEYs-MBP:51977.activeMasterManager] >>>> > > master.TableNamespaceManager (TableNamespaceManager.java:start(85)) >>>> - >>>> > > Namespace table not found. Creating... >>>> > > >>>> > > 2015-07-19 23:40:18,620 INFO [ProcessThread(sid:0 cport:-1):] >>>> > > server.PrepRequestProcessor >>>> (PrepRequestProcessor.java:pRequest(645)) - >>>> > Got >>>> > > user-level KeeperException when processing >>>> sessionid:0x14ea84290170000 >>>> > > type:create cxid:0x1ac zxid:0x35 txntype:-1 reqpath:n/a Error >>>> > > Path:/hbase/table-lock/hbase:namespace Error:KeeperErrorCode = >>>> NoNode for >>>> > > /hbase/table-lock/hbase:namespace >>>> > > >>>> > > 2015-07-19 23:40:18,630 INFO >>>> > [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:51977-0] >>>> > > handler.CreateTableHandler (CreateTableHandler.java:process(189)) - >>>> > Create >>>> > > table hbase:namespace >>>> > > >>>> > > 2015-07-19 23:40:18,644 INFO >>>> [RegionOpenAndInitThread-hbase:namespace-1] >>>> > > regionserver.HRegion (HRegion.java:createHRegion(5598)) - creating >>>> > HRegion >>>> > > hbase:namespace HTD == 'hbase:namespace', {NAME => 'info', >>>> BLOOMFILTER => >>>> > > 'ROW', VERSIONS => '10', IN_MEMORY => 'true', KEEP_DELETED_CELLS => >>>> > > 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', >>>> COMPRESSION => >>>> > > 'NONE', CACHE_DATA_IN_L1 => 'true', MIN_VERSIONS => '0', BLOCKCACHE >>>> => >>>> > > 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} RootDir = >>>> > > >>>> > > >>>> > >>>> file:/Users/ssa/devel/myown/hadoop/mini-hdfs-cluster-maven-plugin/target/hbase-root/.tmp >>>> > > Table name == hbase:namespace >>>> > > >>>> > > 2015-07-19 23:40:18,655 INFO >>>> [RegionOpenAndInitThread-hbase:namespace-1] >>>> > > regionserver.HRegion (HRegion.java:doClose(1425)) - Closed >>>> > > hbase:namespace,,1437342018607.9c2e23572f970747f86ec499b89c281b. >>>> > > >>>> > > 2015-07-19 23:40:18,702 INFO >>>> > [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:51977-0] >>>> > > hbase.MetaTableAccessor >>>> (MetaTableAccessor.java:addRegionsToMeta(1169)) - >>>> > > Added 2 >>>> > > >>>> > > 2015-07-19 23:40:18,704 WARN >>>> > [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:51977-0] >>>> > > zookeeper.ZKTableStateManager >>>> > (ZKTableStateManager.java:setTableState(100)) >>>> > > - Moving table hbase:namespace state from ENABLING to ENABLED >>>> > > >>>> > > 2015-07-19 23:40:18,706 INFO >>>> > [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:51977-0] >>>> > > handler.CreateTableHandler (CreateTableHandler.java:completed(219)) >>>> - >>>> > > Table, hbase:namespace, creation successful >>>> > > >>>> > > Process Thread Dump: Thread dump because: Master not initialized >>>> after >>>> > > 200000ms seconds >>>> > > >>>> > > 186 active threads >>>> > > >>>> > > Thread 255 (MASTER_TABLE_OPERATIONS-SERGEYs-MBP:51977-0): >>>> > > >>>> > > State: WAITING >>>> > > >>>> > > Blocked count: 11 >>>> > > >>>> > > Waited count: 11 >>>> > > >>>> > > Waiting on >>>> > > >>>> > > >>>> > >>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@20b5f2ac >>>> > > >>>> > > Stack: >>>> > > >>>> > > sun.misc.Unsafe.park(Native Method) >>>> > > >>>> > > >>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) >>>> > > >>>> > > >>>> > > >>>> > > >>>> > >>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) >>>> > > >>>> > > >>>> > > >>>> > >>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) >>>> > > >>>> > > >>>> > > >>>> > > >>>> > >>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) >>>> > > >>>> > > >>>> > > >>>> > > >>>> > >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) >>>> > > >>>> > > >>>> > > >>>> > > >>>> > >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>> > > >>>> > > java.lang.Thread.run(Thread.java:745) >>>> > > >>>> > > Thread 249 (CatalogJanitor-SERGEYs-MBP:51977): >>>> > > >>>> > > >>>> > > ==== >>>> > > >>>> > > >>>> > > 2015-07-19 23:43:33,188 ERROR [main] hbase.MiniHBaseCluster >>>> > > (MiniHBaseCluster.java:init(229)) - Error starting cluster >>>> > > >>>> > > java.lang.RuntimeException: Master not initialized after 200000ms >>>> seconds >>>> > > >>>> > > at >>>> > > >>>> > > >>>> > >>>> org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:225) >>>> > > >>>> > > at >>>> > > >>>> > > >>>> > >>>> org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:436) >>>> > > >>>> > > at >>>> > >>>> org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:224) >>>> > > >>>> > > at >>>> > > >>>> org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:93) >>>> > > >>>> > > at >>>> > > >>>> org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:80) >>>> > > >>>> > > at >>>> > > >>>> org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:67) >>>> > > >>>> > > >>>> > > fails here: MINI_HBASE_CLUSTER = new >>>> MiniHBaseCluster(configuration, 1); >>>> > > >>>> > > >>>> > > What it could be? Have no idea. >>>> > > >>>> > >>>> >>> >>> >> >