omg, looks like -Djava.net.preferIPv4Stack=true helped...
probably some weird macos update...
2015-07-20 23:56 GMT+02:00 Serega Sheypak <serega.shey...@gmail.com>:
> I degrade project with hbase testing utility to these versions:
> <hadoop.version>2.5.0-cdh5.2.0</hadoop.version>
> <hadoop.mr.version>2.5.0-mr1-cdh5.2.0</hadoop.mr.version>
> <hbase.version>0.98.6-cdh5.2.0</hbase.version>
>
> It works.
>
> I upgrade to these:
>
> <hadoop.version>2.6.0-cdh5.4.4</hadoop.version>
> <hadoop.mr.version>2.6.0-mr1-cdh5.4.4</hadoop.mr.version>
> <hbase.version>1.0.0-cdh5.4.4</hbase.version>
>
> it hangs...
>
> 2015-07-20 23:27 GMT+02:00 Serega Sheypak <serega.shey...@gmail.com>:
>
>> I see these lines:
>>
>> 2015-07-20 21:27:21,791 INFO [RegionOpenAndInitThread-hbase:namespace-1]
>> regionserver.HRegion (HRegion.java:createHRegion(5598)) - creating HRegion
>> hbase:namespace HTD == 'hbase:namespace', {NAME => 'info', BLOOMFILTER =>
>> 'ROW', VERSIONS => '10', IN_MEMORY => 'true', KEEP_DELETED_CELLS =>
>> 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION =>
>> 'NONE', CACHE_DATA_IN_L1 => 'true', MIN_VERSIONS => '0', BLOCKCACHE =>
>> 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} RootDir =
>> file:/Users/ssa/devel/myown/hadoop/mini-hdfs-cluster-maven-plugin/target/hbase-root/.tmp
>> Table name == hbase:namespace
>>
>> 2015-07-20 21:27:21,802 INFO [RegionOpenAndInitThread-hbase:namespace-1]
>> regionserver.HRegion (HRegion.java:doClose(1425)) - Closed
>> hbase:namespace,,1437420441756.a69bbbcaf4a82786964da8d9cc62bea3.
>>
>> 2015-07-20 21:27:21,861 INFO
>> [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:61781-0] hbase.MetaTableAccessor
>> (MetaTableAccessor.java:addRegionsToMeta(1169)) - Added 2
>>
>> 2015-07-20 21:27:21,864 WARN
>> [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:61781-0] zookeeper.ZKTableStateManager
>> (ZKTableStateManager.java:setTableState(100)) - Moving table
>> hbase:namespace state from ENABLING to ENABLED
>>
>> 2015-07-20 21:27:21,866 INFO
>> [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:61781-0] handler.CreateTableHandler
>> (CreateTableHandler.java:completed(219)) - Table, hbase:namespace, creation
>> successful
>>
>>
>> then it hungs and prints thread dump in few minutes.
>>
>>
>> 2015-07-20 21:25 GMT+02:00 Esteban Gutierrez <este...@cloudera.com>:
>>
>>>
>>> But do you see that thread printing anything in the logs?
>>>
>>> --
>>> Cloudera, Inc.
>>>
>>>
>>> On Mon, Jul 20, 2015 at 12:07 PM, Serega Sheypak <
>>> serega.shey...@gmail.com> wrote:
>>>
>>>> This is testing utiliy, it has few bytes of data to load. Running on
>>>> oracle-jdk8
>>>>
>>>> java.lang.Thread.run(Thread.java:745)
>>>> Thread 178 (JvmPauseMonitor):
>>>> State: TIMED_WAITING
>>>> Blocked count: 3
>>>> Waited count: 398
>>>> Stack:
>>>> java.lang.Thread.sleep(Native Method)
>>>>
>>>> org.apache.hadoop.hbase.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:159)
>>>> java.lang.Thread.run(Thread.java:745)
>>>>
>>>> Thread 177 (JvmPauseMonitor):
>>>> State: TIMED_WAITING
>>>> Blocked count: 1
>>>> Waited count: 398
>>>> Stack:
>>>> java.lang.Thread.sleep(Native Method)
>>>>
>>>> org.apache.hadoop.hbase.util.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:159)
>>>> java.lang.Thread.run(Thread.java:745)
>>>>
>>>> 2015-07-20 19:30 GMT+02:00 Esteban Gutierrez <este...@cloudera.com>:
>>>>
>>>>> -user@hbase
>>>>>
>>>>> Hi Serega,
>>>>>
>>>>> The RunTimeException is pointing to a timeout of nearly 3min. Have
>>>>> tried to find in the master log lines what is causing that 3min pause? do
>>>>> you see any log line related to the JvmPauseMonitor? (perhaps some GC
>>>>> going
>>>>> on)
>>>>>
>>>>> thanks,
>>>>> esteban.
>>>>>
>>>>> --
>>>>> Cloudera, Inc.
>>>>>
>>>>>
>>>>> On Mon, Jul 20, 2015 at 12:28 AM, Serega Sheypak <
>>>>> serega.shey...@gmail.com> wrote:
>>>>>
>>>>>> @Sean, thanks. I saw sometimes Cloudera guys help here. I also used
>>>>>> Cloudera community forum.
>>>>>>
>>>>>> @Jean-Marc, nothing special, just maven-plugin wrapper around
>>>>>> miniHbaseCluster
>>>>>> Here is the code where failure happen: new
>>>>>> MiniHBaseCluster(configuration,
>>>>>> 1);
>>>>>>
>>>>>> There are nothing special from my side. I'm surprised, it always
>>>>>> worked
>>>>>> since CDH 4.4, just bump dependency versions, fix code to follow API
>>>>>> changes and that' all.
>>>>>>
>>>>>> 2015-07-20 4:08 GMT+02:00 Jean-Marc Spaggiari <
>>>>>> jean-m...@spaggiari.org>:
>>>>>>
>>>>>> > Hi Serega,
>>>>>> >
>>>>>> > What kind of tests are your trying to run? The HBase test suite? Or
>>>>>> > something you developed yourself?
>>>>>> >
>>>>>> > JM
>>>>>> >
>>>>>> > 2015-07-19 17:47 GMT-04:00 Serega Sheypak <serega.shey...@gmail.com
>>>>>> >:
>>>>>> >
>>>>>> > > Hi, bumped my testing stuff to CDH 5.4.4 and got failure while
>>>>>> running
>>>>>> > > tests. Here is a log
>>>>>> > >
>>>>>> > > 2015-07-19 23:40:18,607 INFO
>>>>>> [SERGEYs-MBP:51977.activeMasterManager]
>>>>>> > > master.TableNamespaceManager
>>>>>> (TableNamespaceManager.java:start(85)) -
>>>>>> > > Namespace table not found. Creating...
>>>>>> > >
>>>>>> > > 2015-07-19 23:40:18,620 INFO [ProcessThread(sid:0 cport:-1):]
>>>>>> > > server.PrepRequestProcessor
>>>>>> (PrepRequestProcessor.java:pRequest(645)) -
>>>>>> > Got
>>>>>> > > user-level KeeperException when processing
>>>>>> sessionid:0x14ea84290170000
>>>>>> > > type:create cxid:0x1ac zxid:0x35 txntype:-1 reqpath:n/a Error
>>>>>> > > Path:/hbase/table-lock/hbase:namespace Error:KeeperErrorCode =
>>>>>> NoNode for
>>>>>> > > /hbase/table-lock/hbase:namespace
>>>>>> > >
>>>>>> > > 2015-07-19 23:40:18,630 INFO
>>>>>> > [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:51977-0]
>>>>>> > > handler.CreateTableHandler (CreateTableHandler.java:process(189))
>>>>>> -
>>>>>> > Create
>>>>>> > > table hbase:namespace
>>>>>> > >
>>>>>> > > 2015-07-19 23:40:18,644 INFO
>>>>>> [RegionOpenAndInitThread-hbase:namespace-1]
>>>>>> > > regionserver.HRegion (HRegion.java:createHRegion(5598)) - creating
>>>>>> > HRegion
>>>>>> > > hbase:namespace HTD == 'hbase:namespace', {NAME => 'info',
>>>>>> BLOOMFILTER =>
>>>>>> > > 'ROW', VERSIONS => '10', IN_MEMORY => 'true', KEEP_DELETED_CELLS
>>>>>> =>
>>>>>> > > 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER',
>>>>>> COMPRESSION =>
>>>>>> > > 'NONE', CACHE_DATA_IN_L1 => 'true', MIN_VERSIONS => '0',
>>>>>> BLOCKCACHE =>
>>>>>> > > 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} RootDir =
>>>>>> > >
>>>>>> > >
>>>>>> >
>>>>>> file:/Users/ssa/devel/myown/hadoop/mini-hdfs-cluster-maven-plugin/target/hbase-root/.tmp
>>>>>> > > Table name == hbase:namespace
>>>>>> > >
>>>>>> > > 2015-07-19 23:40:18,655 INFO
>>>>>> [RegionOpenAndInitThread-hbase:namespace-1]
>>>>>> > > regionserver.HRegion (HRegion.java:doClose(1425)) - Closed
>>>>>> > > hbase:namespace,,1437342018607.9c2e23572f970747f86ec499b89c281b.
>>>>>> > >
>>>>>> > > 2015-07-19 23:40:18,702 INFO
>>>>>> > [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:51977-0]
>>>>>> > > hbase.MetaTableAccessor
>>>>>> (MetaTableAccessor.java:addRegionsToMeta(1169)) -
>>>>>> > > Added 2
>>>>>> > >
>>>>>> > > 2015-07-19 23:40:18,704 WARN
>>>>>> > [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:51977-0]
>>>>>> > > zookeeper.ZKTableStateManager
>>>>>> > (ZKTableStateManager.java:setTableState(100))
>>>>>> > > - Moving table hbase:namespace state from ENABLING to ENABLED
>>>>>> > >
>>>>>> > > 2015-07-19 23:40:18,706 INFO
>>>>>> > [MASTER_TABLE_OPERATIONS-SERGEYs-MBP:51977-0]
>>>>>> > > handler.CreateTableHandler
>>>>>> (CreateTableHandler.java:completed(219)) -
>>>>>> > > Table, hbase:namespace, creation successful
>>>>>> > >
>>>>>> > > Process Thread Dump: Thread dump because: Master not initialized
>>>>>> after
>>>>>> > > 200000ms seconds
>>>>>> > >
>>>>>> > > 186 active threads
>>>>>> > >
>>>>>> > > Thread 255 (MASTER_TABLE_OPERATIONS-SERGEYs-MBP:51977-0):
>>>>>> > >
>>>>>> > > State: WAITING
>>>>>> > >
>>>>>> > > Blocked count: 11
>>>>>> > >
>>>>>> > > Waited count: 11
>>>>>> > >
>>>>>> > > Waiting on
>>>>>> > >
>>>>>> > >
>>>>>> >
>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@20b5f2ac
>>>>>> > >
>>>>>> > > Stack:
>>>>>> > >
>>>>>> > > sun.misc.Unsafe.park(Native Method)
>>>>>> > >
>>>>>> > >
>>>>>> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>>>>>> > >
>>>>>> > >
>>>>>> > >
>>>>>> > >
>>>>>> >
>>>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
>>>>>> > >
>>>>>> > >
>>>>>> > >
>>>>>> >
>>>>>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
>>>>>> > >
>>>>>> > >
>>>>>> > >
>>>>>> > >
>>>>>> >
>>>>>> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
>>>>>> > >
>>>>>> > >
>>>>>> > >
>>>>>> > >
>>>>>> >
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
>>>>>> > >
>>>>>> > >
>>>>>> > >
>>>>>> > >
>>>>>> >
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>>>> > >
>>>>>> > > java.lang.Thread.run(Thread.java:745)
>>>>>> > >
>>>>>> > > Thread 249 (CatalogJanitor-SERGEYs-MBP:51977):
>>>>>> > >
>>>>>> > >
>>>>>> > > ====
>>>>>> > >
>>>>>> > >
>>>>>> > > 2015-07-19 23:43:33,188 ERROR [main] hbase.MiniHBaseCluster
>>>>>> > > (MiniHBaseCluster.java:init(229)) - Error starting cluster
>>>>>> > >
>>>>>> > > java.lang.RuntimeException: Master not initialized after 200000ms
>>>>>> seconds
>>>>>> > >
>>>>>> > > at
>>>>>> > >
>>>>>> > >
>>>>>> >
>>>>>> org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:225)
>>>>>> > >
>>>>>> > > at
>>>>>> > >
>>>>>> > >
>>>>>> >
>>>>>> org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:436)
>>>>>> > >
>>>>>> > > at
>>>>>> >
>>>>>> org.apache.hadoop.hbase.MiniHBaseCluster.init(MiniHBaseCluster.java:224)
>>>>>> > >
>>>>>> > > at
>>>>>> > >
>>>>>> org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:93)
>>>>>> > >
>>>>>> > > at
>>>>>> > >
>>>>>> org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:80)
>>>>>> > >
>>>>>> > > at
>>>>>> > >
>>>>>> org.apache.hadoop.hbase.MiniHBaseCluster.<init>(MiniHBaseCluster.java:67)
>>>>>> > >
>>>>>> > >
>>>>>> > > fails here: MINI_HBASE_CLUSTER = new
>>>>>> MiniHBaseCluster(configuration, 1);
>>>>>> > >
>>>>>> > >
>>>>>> > > What it could be? Have no idea.
>>>>>> > >
>>>>>> >
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
