Ved:
The REST client failed because:
java.net.SocketTimeoutException: timed out waiting for rpc
The master is trying to talk to the regionserver that was hosting the
catalog .META. table but its getting:
java.io.IOException: Server not running
It looks like the regionserver that was hosting the .META. is not
running or crashed. Look in its logs to see if you can figure why it
went away.
St.Ack
Ved Prakash wrote:
hbase-root-rest-ved-desktop.log.2008-03-13
2008-03-13 14:30:46,067 INFO org.mortbay.util.Container: Started
[EMAIL PROTECTED]
2008-03-13 20:06:47,136 INFO org.mortbay.util.Container: Started
HttpContext[/,/]
2008-03-13 23:37:12,047 WARN rest: /api/CreativeViewDetails/row/4102047/:
java.lang.reflect.UndeclaredThrowableException
at $Proxy0.findRootRegion(Unknown Source)
at
org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRootRegion(
HConnectionManager.java:700)
at
org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegion(
HConnectionManager.java:329)
at
org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegion(
HConnectionManager.java:306)
at
org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegionInMeta(
HConnectionManager.java:402)
at
org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegion(
HConnectionManager.java:339)
at
org.apache.hadoop.hbase.HConnectionManager$TableServers.relocateRegion(
HConnectionManager.java:311)
at
org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegionInMeta(
HConnectionManager.java:476)
at
org.apache.hadoop.hbase.HConnectionManager$TableServers.locateRegion(
HConnectionManager.java:343)
at
org.apache.hadoop.hbase.HConnectionManager$TableServers.relocateRegion(
HConnectionManager.java:311)
at org.apache.hadoop.hbase.HTable.getRegionLocation(HTable.java:113)
at org.apache.hadoop.hbase.HTable.commit(HTable.java:977)
at org.apache.hadoop.hbase.HTable.commit(HTable.java:941)
at org.apache.hadoop.hbase.rest.TableHandler.putRowXml(
TableHandler.java:335)
at org.apache.hadoop.hbase.rest.TableHandler.putRow(
TableHandler.java:254)
at org.apache.hadoop.hbase.rest.TableHandler.doPost(
TableHandler.java:89)
at org.apache.hadoop.hbase.rest.Dispatcher.doPost(Dispatcher.java
:113)
at org.apache.hadoop.hbase.rest.Dispatcher.doPut(Dispatcher.java
:126)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:712)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java
:427)
at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(
WebApplicationHandler.java:475)
at org.mortbay.jetty.servlet.ServletHandler.handle(
ServletHandler.java:567)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
at org.mortbay.jetty.servlet.WebApplicationContext.handle(
WebApplicationContext.java:635)
at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
at org.mortbay.http.HttpServer.service(HttpServer.java:954)
at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java
:981)
at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
at org.mortbay.http.SocketListener.handleConnection(
SocketListener.java:244)
at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
Caused by: java.net.SocketTimeoutException: timed out waiting for rpc
response
at org.apache.hadoop.ipc.Client.call(Client.java:514)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Invoker.invoke(HbaseRPC.java
:210)
... 33 more
------------------------------------------------------
error listing from stdout where hbase master was running
(Partial list)
08/03/14 10:37:33 WARN ipc.Server: IPC Server Responder, call
next(-6267545063026670769) from 192.168.0.52:35791: output error
08/03/14 10:37:33 INFO ipc.Server: IPC Server handler 6 on 39677 caught:
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(
SocketChannelImpl.java:125)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:294)
at org.apache.hadoop.ipc.Server$Responder.processResponse(
Server.java:608)
at org.apache.hadoop.ipc.Server$Responder.doRespond(Server.java:666)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:930)
08/03/14 10:37:33 INFO ipc.Server: IPC Server handler 3 on 39677, call
openScanner(.META.,,1, [Lorg.apache.hadoop.io.Text;@e66f56, , 1205471252608,
null) from 192.168.0.52:35817: error: java.io.IOException: Server not
running
java.io.IOException: Server not running
at org.apache.hadoop.hbase.HRegionServer.checkOpen(
HRegionServer.java:1626)
at org.apache.hadoop.hbase.HRegionServer.openScanner(
HRegionServer.java:1448)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(
NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java
:413)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:910)
08/03/14 10:37:33 WARN hbase.HMaster: Scan one META region: {regionname:
.META.,,1, startKey: <>, server: 192.168.0.52:39677}
java.io.IOException: java.io.IOException: Server not running
at org.apache.hadoop.hbase.HRegionServer.checkOpen(
HRegionServer.java:1626)
at org.apache.hadoop.hbase.HRegionServer.openScanner(
HRegionServer.java:1448)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(
NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java
:413)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:910)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(
NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:494)
at
org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(
RemoteExceptionHandler.java:82)
at org.apache.hadoop.hbase.HMaster$BaseScanner.scanRegion(
HMaster.java:259)
at org.apache.hadoop.hbase.HMaster$MetaScanner.scanOneMetaRegion(
HMaster.java:687)
at org.apache.hadoop.hbase.HMaster$MetaScanner.maintenanceScan(
HMaster.java:743)
at org.apache.hadoop.hbase.HMaster$BaseScanner.chore(HMaster.java
:201)
at org.apache.hadoop.hbase.Chore.run(Chore.java:63)
08/03/14 10:37:33 INFO hbase.HMaster: all meta regions scanned
08/03/14 10:37:33 INFO ipc.Server: IPC Server handler 2 on 39677, call
getClosestRowBefore(.META.,,1, CreativeViewDetails,4111546,999999999999999,
9223372036854775807) from 192.168.0.52:35850: error: java.io.IOException:
Server not running
java.io.IOException: Server not running
at org.apache.hadoop.hbase.HRegionServer.checkOpen(
HRegionServer.java:1626)
at org.apache.hadoop.hbase.HRegionServer.getClosestRowBefore(
HRegionServer.java:1363)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(
NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java
:413)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:910)
08/03/14 10:37:36 INFO hbase.HRegionServer: RegionServer:0.cacheFlusherexiting
08/03/14 10:37:40 INFO hbase.HRegionServer: worker thread exiting
08/03/14 10:37:42 INFO hbase.Leases: RegionServer:0 2 lease(s) outstanding.
Waiting for them to expire.
08/03/14 10:37:47 ERROR hbase.HRegionServer: Error opening scanner (fsOk:
true)
java.io.IOException: HStoreScanner failed construction
at org.apache.hadoop.hbase.HStore$StoreFileScanner.<init>(
HStore.java:2172)
at org.apache.hadoop.hbase.HStore$HStoreScanner.<init>(HStore.java
:2293)
at org.apache.hadoop.hbase.HStore.getScanner(HStore.java:2083)
at org.apache.hadoop.hbase.HRegion$HScanner.<init>(HRegion.java
:1622)
at org.apache.hadoop.hbase.HRegion.getScanner(HRegion.java:1196)
at org.apache.hadoop.hbase.HRegionServer.openScanner(
HRegionServer.java:1453)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(
NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java
:413)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:910)
Caused by: java.net.SocketTimeoutException: timed out waiting for rpc
response
at org.apache.hadoop.ipc.Client.call(Client.java:514)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:198)
at org.apache.hadoop.dfs.$Proxy0.getFileInfo(Unknown Source)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(
DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(
RetryInvocationHandler.java:82)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(
RetryInvocationHandler.java:59)
at org.apache.hadoop.dfs.$Proxy0.getFileInfo(Unknown Source)
at org.apache.hadoop.dfs.DFSClient.getFileInfo(DFSClient.java:444)
at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(
DistributedFileSystem.java:334)
at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:538)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java
:1387)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java
:1382)
at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:263)
at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:242)
at org.apache.hadoop.hbase.HStoreFile$HbaseMapFile$HbaseReader
.<init>(HStoreFile.java:593)
at org.apache.hadoop.hbase.HStoreFile$BloomFilterMapFile$Reader
.<init>(HStoreFile.java:648)
at org.apache.hadoop.hbase.HStoreFile.getReader(HStoreFile.java:419)
at org.apache.hadoop.hbase.HStore$StoreFileScanner.<init>(
HStore.java:2147)
... 11 more
I hope this helps
