The hbase client hang issue happened between 0.20.1 client and 0.20.1
server.
If I switch client to 0.20.3 hbase jar, I see the following in region server
log:
2010-03-08 10:31:53,571 ERROR [IPC Server handler 25 on 60020]
regionserver.HRegionServer(844):
org.apache.hadoop.hbase.UnknownScannerException: Name: -1
at
org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1925)
at sun.reflect.GeneratedMethodAccessor29.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
Thanks
On Mon, Mar 8, 2010 at 10:40 AM, Jean-Daniel Cryans <jdcry...@apache.org>wrote:
> Ted,
>
> Are you using a mix of 0.20.3 and 0.20.1 like you said previously? If
> so, your client here is .3 and the server is .1?
>
> WRT the WREs. like Stack said in a previous email they happen when
> something went wrong like HDFS errors intermixed with region splits,
> etc. We are currently working in improving HBase's reliability in the
> face of such errors. To repair your table I suggest running the
> bin/add_table.rb script (see the comments at the top of that file for
> usage information). Else your clients will always get those errors and
> will just keep retrying.
>
> J-D
>
> On Mon, Mar 8, 2010 at 10:23 AM, Ted Yu <yuzhih...@gmail.com> wrote:
> > Hi,
> > We use the following code to retrieve data from hbase 0.20.1 which didn't
> > return after 30 minutes:
> >
> > ResultScanner _scanner = _data;
> > try {
> > Result[] _results = _scanner.next(defaultPageSize);
> > updateResults(pTable, _scanner, _results);
> >
> > There is no exception thrown.
> > Here is stack traces of the client:
> >
> > "IPC Client (47) connection to /10.10.31.135:60020 from an unknown user"
> > daemon prio=10 tid=0x00000000560de800 nid=0x34bc runnable
> > [0x000000004249b000]
> > java.lang.Thread.State: RUNNABLE
> > at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
> > at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
> > at
> sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
> > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> > - locked <0x00002aaaaffc9cb8> (a sun.nio.ch.Util$1)
> > - locked <0x00002aaaaffc9ca0> (a
> > java.util.Collections$UnmodifiableSet)
> > - locked <0x00002aaaaffc9678> (a sun.nio.ch.EPollSelectorImpl)
> > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
> > at
> >
> org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:332)
> > at
> >
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)
> > at
> > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
> > at
> > org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
> > at java.io.FilterInputStream.read(FilterInputStream.java:116)
> > at
> >
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBaseClient.java:279)
> > at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> > at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
> > - locked <0x00002aaaaffb2878> (a java.io.BufferedInputStream)
> > at java.io.DataInputStream.readInt(DataInputStream.java:370)
> > at
> >
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:504)
> > at
> >
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:448)
> >
> > "IPC Client (47) connection to /10.10.31.136:60020 from an unknown user"
> > daemon prio=10 tid=0x00000000565c1800 nid=0x34bb in Object.wait()
> > [0x000000004239a000]
> > java.lang.Thread.State: TIMED_WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > at
> >
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClient.java:404)
> > - locked <0x00002aaaaffad298> (a
> > org.apache.hadoop.hbase.ipc.HBaseClient$Connection)
> > at
> >
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:447)
> >
> > "AWT-EventQueue-0" prio=10 tid=0x0000000056456800 nid=0x34a8 in
> > Object.wait() [0x0000000041793000]
> > java.lang.Thread.State: WAITING (on object monitor)
> > at java.lang.Object.wait(Native Method)
> > at java.lang.Object.wait(Object.java:485)
> > at
> > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:717)
> > - locked <0x00002aaaae2d8d48> (a
> > org.apache.hadoop.hbase.ipc.HBaseClient$Call)
> > at
> > org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:328)
> > at $Proxy0.close(Unknown Source)
> > at
> >
> org.apache.hadoop.hbase.client.ScannerCallable.close(ScannerCallable.java:101)
> > at
> >
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:72)
> > at
> >
> org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:38)
> > at
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:988)
> > at
> >
> org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java:1887)
> > at
> >
> org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:2033)
> > at
> >
> org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:2051)
> > at
> >
> net.kindsight.webmap.gui.HBaseViewerPanel.updateUI(HBaseViewerPanel.java:390)
> > at
> >
> net.kindsight.webmap.gui.HBaseViewerPanel.access$900(HBaseViewerPanel.java:45)
> > at
> >
> net.kindsight.webmap.gui.HBaseViewerPanel$5.run(HBaseViewerPanel.java:362)
> > at
> java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:199)
> > at java.awt.EventQueue.dispatchEvent(EventQueue.java:597)
> > at
> >
> java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:269)
> > at
> >
> java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:184)
> > at
> >
> java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:174)
> > at
> > java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:169)
> > at
> > java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:161)
> > at java.awt.EventDispatchThread.run(EventDispatchThread.java:122)
> >
> > The regionserver on 10.10.31.136 ran fine.
> > For regionserver on 10.10.31.135, apart from some WREs, things were
> normal:
> > 2010-03-08 10:21:16,791 ERROR [IPC Server handler 17 on 60020]
> > regionserver.HRegionServer(844):
> > org.apache.hadoop.hbase.regionserver.WrongRegionException: Requested row
> out
> > of range for HRegion
> >
> ruletable,com.hoovers.www\x2Fcompanyindex\x2FMichigan\x2FBattle_Creek\x2FMarketing_and_Advertising_Services-1.html,1267860376861,
> >
> startKey='com.hoovers.www\x2Fcompanyindex\x2FMichigan\x2FBattle_Creek\x2FMarketing_and_Advertising_Services-1.html',
> >
> getEndKey()='com.hoovers.www\x2Fcompanyindex\x2FNew_Mexico\x2FVado\x2FBroadcasting_Industry-1.html',
> >
> row='com.xmradio.www\x2FpadData\x2Fpad_data_servlet.jsp\x3Fchannel\x3D66\x26rpc\x3DXMROUS\x26rnd\x3D8378'
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:1522)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1554)
> > at
> org.apache.hadoop.hbase.regionserver.HRegion.getLock(HRegion.java:1622)
> > at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:2278)
> > at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:1785)
> > at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:648)
> > at
> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> >
> > Has someone seen similar situation ?
> > Thanks
> >
>
