I degrade project with hbase testing utility to these versions:
<hadoop.version>2.5.0-cdh5.2.0</hadoop.version>
<hadoop.mr.version>2.5.0-mr1-cdh5.2.0</hadoop.mr.version>
<hbase.version>0.98.6-cdh5.2.0</hbase.version>
It works.
I upgrade to these:
<hadoop.version>2.6.0-cdh5.4.4</hadoop.version>
<hadoop.mr.version>2.6.0-mr1-cdh5.4.4</hadoop.mr.version>
<hbase.version>1.0.0-cdh5.4.4</hbase.version>
it hangs...
2015-07-20 23:27 GMT+02:00 Serega Sheypak <serega.shey...@gmail.com>:
> I see these lines:
>
> 2015-07-20 21:27:21,791 INFO [RegionOpenAndInitThread-hbase:namespace-1]
> regionserver.HRegion (HRegion.java:createHRegion(5598)) - creating HRegion
> hbase:namespace HTD == 'hbase:namespace', {NAME => 'info', BLOOMFILTER =>
> 'ROW', VERSIONS => '10', IN_MEMORY => 'true', KEEP_DELETED_CELLS =>
> 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION =>
> 'NONE', CACHE_DATA_IN_L1 => 'true', MIN_VERSIONS => '0', BLOCKCACHE =>
> 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} RootDir =
> file:/Users/ssa/devel/myown/hadoop/mini-hdfs-cluster-maven-plugin/target/hbase-root/.tmp
> Table name == hbase:namespace
>
> 2015-07-20 21:27:21,802 INFO [RegionOpenAndInitThread-hbase:namespace-1]
> regionserver.HRegion (HRegion.java:doClose(1425)) - Closed
> hbase:namespace,,1437420441756.a69bbbcaf4a82786964da8d9cc62bea3.
>
> 2015-07-20 21:27:21,861 INFO
> [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:61781-0] hbase.MetaTableAccessor
> (MetaTableAccessor.java:addRegionsToMeta(1169)) - Added 2
>
> 2015-07-20 21:27:21,864 WARN
> [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:61781-0] zookeeper.ZKTableStateManager
> (ZKTableStateManager.java:setTableState(100)) - Moving table
> hbase:namespace state from ENABLING to ENABLED
>
> 2015-07-20 21:27:21,866 INFO
> [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:61781-0] handler.CreateTableHandler
> (CreateTableHandler.java:completed(219)) - Table, hbase:namespace, creation
> successful
>
>
> then it hungs and prints thread dump in few minutes.
>
>
> 2015-07-20 21:25 GMT+02:00 Esteban Gutierrez <este...@cloudera.com>:
>
>>
>> But do you see that thread printing anything in the logs?
>>
>> --
>> Cloudera, Inc.
>>
>>
>> On Mon, Jul 20, 2015 at 12:07 PM, Serega Sheypak <
>> serega.shey...@gmail.com> wrote:
>>
>>> This is testing utiliy, it has few bytes of data to load. Running on
>>> oracle-jdk8
>>>
>>> java.lang.Thread.run(Thread.java:745)
>>> Thread 178 (JvmPauseMonitor):
>>> State: TIMED_WAITING
>>> Blocked count: 3
>>> Waited count: 398
>>> Stack:
>>> java.lang.Thread.sleep(Native Method)
>>>
>>> org.apache.hadoop.hbase.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:159)
>>> java.lang.Thread.run(Thread.java:745)
>>>
>>> Thread 177 (JvmPauseMonitor):
>>> State: TIMED_WAITING
>>> Blocked count: 1
>>> Waited count: 398
>>> Stack:
>>> java.lang.Thread.sleep(Native Method)
>>>
>>> org.apache.hadoop.hbase.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:159)
>>> java.lang.Thread.run(Thread.java:745)
>>>
>>> 2015-07-20 19:30 GMT+02:00 Esteban Gutierrez <este...@cloudera.com>:
>>>
>>>> -user@hbase
>>>>
>>>> Hi Serega,
>>>>
>>>> The RunTimeException is pointing to a timeout of nearly 3min. Have
>>>> tried to find in the master log lines what is causing that 3min pause? do
>>>> you see any log line related to the JvmPauseMonitor? (perhaps some GC going
>>>> on)
>>>>
>>>> thanks,
>>>> esteban.
>>>>
>>>> --
>>>> Cloudera, Inc.
>>>>
>>>>
>>>> On Mon, Jul 20, 2015 at 12:28 AM, Serega Sheypak <
>>>> serega.shey...@gmail.com> wrote:
>>>>
>>>>> @Sean, thanks. I saw sometimes Cloudera guys help here. I also used
>>>>> Cloudera community forum.
>>>>>
>>>>> @Jean-Marc, nothing special, just maven-plugin wrapper around
>>>>> miniHbaseCluster
>>>>> Here is the code where failure happen: new
>>>>> MiniHBaseCluster(configuration,
>>>>> 1);
>>>>>
>>>>> There are nothing special from my side. I'm surprised, it always worked
>>>>> since CDH 4.4, just bump dependency versions, fix code to follow API
>>>>> changes and that' all.
>>>>>
>>>>> 2015-07-20 4:08 GMT+02:00 Jean-Marc Spaggiari <jean-m...@spaggiari.org
>>>>> >:
>>>>>
>>>>> > Hi Serega,
>>>>> >
>>>>> > What kind of tests are your trying to run? The HBase test suite? Or
>>>>> > something you developed yourself?
>>>>> >
>>>>> > JM
>>>>> >
>>>>> > 2015-07-19 17:47 GMT-04:00 Serega Sheypak <serega.shey...@gmail.com
>>>>> >:
>>>>> >
>>>>> > > Hi, bumped my testing stuff to CDH 5.4.4 and got failure while
>>>>> running
>>>>> > > tests. Here is a log
>>>>> > >
>>>>> > > 2015-07-19 23:40:18,607 INFO
>>>>> [SERGEYs-MBP:51977.activeMasterManager]
>>>>> > > master.TableNamespaceManager
>>>>> (TableNamespaceManager.java:start(85)) -
>>>>> > > Namespace table not found. Creating...
>>>>> > >
>>>>> > > 2015-07-19 23:40:18,620 INFO [ProcessThread(sid:0 cport:-1):]
>>>>> > > server.PrepRequestProcessor
>>>>> (PrepRequestProcessor.java:pRequest(645)) -
>>>>> > Got
>>>>> > > user-level KeeperException when processing
>>>>> sessionid:0x14ea84290170000
>>>>> > > type:create cxid:0x1ac zxid:0x35 txntype:-1 reqpath:n/a Error
>>>>> > > Path:/hbase/table-lock/hbase:namespace Error:KeeperErrorCode =
>>>>> NoNode for
>>>>> > > /hbase/table-lock/hbase:namespace
>>>>> > >
>>>>> > > 2015-07-19 23:40:18,630 INFO
>>>>> > [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:51977-0]
>>>>> > > handler.CreateTableHandler (CreateTableHandler.java:process(189)) -
>>>>> > Create
>>>>> > > table hbase:namespace
>>>>> > >
>>>>> > > 2015-07-19 23:40:18,644 INFO
>>>>> [RegionOpenAndInitThread-hbase:namespace-1]
>>>>> > > regionserver.HRegion (HRegion.java:createHRegion(5598)) - creating
>>>>> > HRegion
>>>>> > > hbase:namespace HTD == 'hbase:namespace', {NAME => 'info',
>>>>> BLOOMFILTER =>
>>>>> > > 'ROW', VERSIONS => '10', IN_MEMORY => 'true', KEEP_DELETED_CELLS =>
>>>>> > > 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER',
>>>>> COMPRESSION =>
>>>>> > > 'NONE', CACHE_DATA_IN_L1 => 'true', MIN_VERSIONS => '0',
>>>>> BLOCKCACHE =>
>>>>> > > 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} RootDir =
>>>>> > >
>>>>> > >
>>>>> >
>>>>> file:/Users/ssa/devel/myown/hadoop/mini-hdfs-cluster-maven-plugin/target/hbase-root/.tmp
>>>>> > > Table name == hbase:namespace
>>>>> > >
>>>>> > > 2015-07-19 23:40:18,655 INFO
>>>>> [RegionOpenAndInitThread-hbase:namespace-1]
>>>>> > > regionserver.HRegion (HRegion.java:doClose(1425)) - Closed
>>>>> > > hbase:namespace,,1437342018607.9c2e23572f970747f86ec499b89c281b.
>>>>> > >
>>>>> > > 2015-07-19 23:40:18,702 INFO
>>>>> > [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:51977-0]
>>>>> > > hbase.MetaTableAccessor
>>>>> (MetaTableAccessor.java:addRegionsToMeta(1169)) -
>>>>> > > Added 2
>>>>> > >
>>>>> > > 2015-07-19 23:40:18,704 WARN
>>>>> > [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:51977-0]
>>>>> > > zookeeper.ZKTableStateManager
>>>>> > (ZKTableStateManager.java:setTableState(100))
>>>>> > > - Moving table hbase:namespace state from ENABLING to ENABLED
>>>>> > >
>>>>> > > 2015-07-19 23:40:18,706 INFO
>>>>> > [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:51977-0]
>>>>> > > handler.CreateTableHandler
>>>>> (CreateTableHandler.java:completed(219)) -
>>>>> > > Table, hbase:namespace, creation successful
>>>>> > >
>>>>> > > Process Thread Dump: Thread dump because: Master not initialized
>>>>> after
>>>>> > > 200000ms seconds
>>>>> > >
>>>>> > > 186 active threads
>>>>> > >
>>>>> > > Thread 255 (MASTER_TABLE_OPERATIONS-SERGEYs-MBP:51977-0):
>>>>> > >
>>>>> > > State: WAITING
>>>>> > >
>>>>> > > Blocked count: 11
>>>>> > >
>>>>> > > Waited count: 11
>>>>> > >
>>>>> > > Waiting on
>>>>> > >
>>>>> > >
>>>>> >
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@20b5f2ac
>>>>> > >
>>>>> > > Stack:
>>>>> > >
>>>>> > > sun.misc.Unsafe.park(Native Method)
>>>>> > >
>>>>> > >
>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> >
>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> >
>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> >
>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> >
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> >
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>> > >
>>>>> > > java.lang.Thread.run(Thread.java:745)
>>>>> > >
>>>>> > > Thread 249 (CatalogJanitor-SERGEYs-MBP:51977):
>>>>> > >
>>>>> > >
>>>>> > > ====
>>>>> > >
>>>>> > >
>>>>> > > 2015-07-19 23:43:33,188 ERROR [main] hbase.MiniHBaseCluster
>>>>> > > (MiniHBaseCluster.java:init(229)) - Error starting cluster
>>>>> > >
>>>>> > > java.lang.RuntimeException: Master not initialized after 200000ms
>>>>> seconds
>>>>> > >
>>>>> > > at
>>>>> > >
>>>>> > >
>>>>> >
>>>>> org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:225)
>>>>> > >
>>>>> > > at
>>>>> > >
>>>>> > >
>>>>> >
>>>>> org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:436)
>>>>> > >
>>>>> > > at
>>>>> >
>>>>> org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:224)
>>>>> > >
>>>>> > > at
>>>>> > >
>>>>> org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:93)
>>>>> > >
>>>>> > > at
>>>>> > >
>>>>> org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:80)
>>>>> > >
>>>>> > > at
>>>>> > >
>>>>> org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:67)
>>>>> > >
>>>>> > >
>>>>> > > fails here: MINI_HBASE_CLUSTER = new
>>>>> MiniHBaseCluster(configuration, 1);
>>>>> > >
>>>>> > >
>>>>> > > What it could be? Have no idea.
>>>>> > >
>>>>> >
>>>>>
>>>>
>>>>
>>>
>>
>
