Himanshu told it clearly. To make to more clear I am adding :) When the range of rowkeys that you are looking for spread across 5 regions, at the client side there will be 5 exec requests created and submitted to a thread pool.[HBase client side thread pool associated with HTable] Now as per the availability of slots in this pool the operations will be requested to regions in RSs. So in different RS there can be parallel execution. Within one RS itself parallel execution on different regions also possible.
It is upto the client app to deal with the results what the Endpoint has returned. It will return a Map right? region name vs result from region. Client app need to take care about the ordering -Anoop- ________________________________________ From: [email protected] [[email protected]] on behalf of Himanshu Vashishtha [[email protected]] Sent: Thursday, March 14, 2013 11:15 PM To: [email protected] Subject: Re: Regionserver goes down while endpoint execution There is no ordering guarantee for the endpoint execution, other than that calls will be executed in parallel across all the regions. In case you have 5 regions, then there will be 5 separate calls to these regions. Then, you get 5 results from these regions at your client, where you use the Callback class to aggregate the results. You can define your ordering this aggregate class for sure. On Thu, Mar 14, 2013 at 10:15 AM, Ted Yu <[email protected]> wrote: > bq. provide the rowkey range as rowkey1 to rowkey100 in endpoint RPC client > > If I understand you correctly, you perform batching at the client as > described above. > The order would be as you expected. > > Cheers > > On Thu, Mar 14, 2013 at 10:09 AM, Kumar, Deepak8 > <[email protected]>wrote: > >> Hi,**** >> >> It seems due to huge data the RegionServer is getting down. Now I am >> trying to fetch the data in parts & is running fine. I need some more info >> about the Endpoint execution:**** >> >> ** ** >> >> My use case is to fetch the data from HBase as per some rowkey range & to >> render it at UI. Since endpoints are executed in parallel so I am looking >> to use it. **** >> >> ** ** >> >> **Ø **Suppose I provide the rowkey range as rowkey1 to rowkey100 in >> endpoint RPC client & these rowkeys are distributed at 5 regions across 4 >> region servers. If I fetch 10 records at a time, do we have any way to >> guarantee that it would come in serial order like first result would of >> rowkey1 to rowkey10, next time I set the start rowkey as rowkey11 & the >> fetch would be from rowkey11 to rowkey20, irrespective of the region & >> region servers?**** >> >> ** ** >> >> Regards,**** >> >> Deepak**** >> >> ** ** >> >> ** ** >> >> -----Original Message----- >> From: [email protected] [mailto:[email protected]] On Behalf Of >> Himanshu Vashishtha >> Sent: Wednesday, March 13, 2013 12:09 PM >> To: [email protected] >> Cc: Gary Helmling; [email protected]; lars hofhansl >> Subject: Re: Regionserver goes down while endpoint execution >> >> ** ** >> >> On Wed, Mar 13, 2013 at 8:19 AM, Kumar, Deepak8 <[email protected]> >> wrote:**** >> >> > Thanks guys for assisting. I am getting OOM exception yet. I have one >> query about Endpoints. As endpoint executes in parallel, so if I have a >> table which is distributed at 101 regions across 5 regionserver. Would it >> be 101 threads of endpoint executing in parallel?**** >> >> ** ** >> >> No and Yes.**** >> >> ** ** >> >> The endpoints are not processed as separate threads, they are processed as >> just another request (via regionserver handlers). Yes, the execution will >> be in parallel in the sense that a separate client side call will be used >> for each of the regions that are in the range you specify.**** >> >> ** ** >> >> >** ** >> >> > Regards,**** >> >> > Deepak**** >> >> >** ** >> >> > From: Gary Helmling [mailto:[email protected] <[email protected]>]** >> ** >> >> > Sent: Tuesday, March 12, 2013 2:14 PM**** >> >> > To: [email protected]**** >> >> > Cc: lars hofhansl; Kumar, Deepak8 [CCC-OT_IT NE]**** >> >> > Subject: Re: Regionserver goes down while endpoint execution**** >> >> >** ** >> >> > To expand on what Himanshu said, your endpoint is doing an unbounded >> scan on the region, so with a region with a lot of rows it's taking more >> than 60 seconds to run to the region end, which is why the client side of >> the call is timing out. In addition you're building up an in memory list >> of all the values for that qualifier in that region, which could cause you >> to bump into OOM issues, depending on how big your values are and how >> sparse the given column qualifier is. If you trigger an OOMException, then >> the region server would abort.**** >> >> >** ** >> >> > For this usage specifically, though -- scanning through a single column >> qualifier for all rows -- you would be better off just doing a normal >> client side scan, ie. HTable.getScanner(). Then you will avoid the client >> timeout and potential server-side memory issues.**** >> >> >** ** >> >> > On Tue, Mar 12, 2013 at 9:29 AM, Ted Yu < >> [email protected]<mailto:[email protected]>> wrote:**** >> >> > From region server log:**** >> >> >** ** >> >> > 2013-03-12 03:07:22,605 DEBUG org.apache.hadoop.hdfs.DFSClient: Error ** >> ** >> >> > making BlockReader. Closing stale **** >> >> > Socket[addr=/10.42.105.112<http://10.42.105.112>,port=50010,localport=** >> ** >> >> > 54114]**** >> >> > java.io.EOFException: Premature EOF: no length prefix available**** >> >> > at >> org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:162) >> **** >> >> > at **** >> >> > org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockRe** >> ** >> >> > ader2.java:407)**** >> >> >** ** >> >> > What version of HBase and hadoop are you using ?**** >> >> > Do versions of hadoop on Eclipse machine and in your cluster match ?**** >> >> >** ** >> >> > Cheers**** >> >> >** ** >> >> > On Tue, Mar 12, 2013 at 4:46 AM, Kumar, Deepak8 < >> [email protected]<mailto:[email protected]>>wrote:**** >> >> >> Lars,******** >> >> >>** ** >> >> >> I am getting following errors at datanode & region servers.******** >> >> >>** ** >> >> >> ** ****** >> >> >>** ** >> >> >> Regards,******** >> >> >>** ** >> >> >> Deepak******** >> >> >>** ** >> >> >> ** ****** >> >> >>** ** >> >> >> *From:* Kumar, Deepak8 [CCC-OT_IT NE]**** >> >> >> *Sent:* Tuesday, March 12, 2013 3:00 AM**** >> >> >> *To:* Kumar, Deepak8 [CCC-OT_IT NE]; **** >> >> >> '[email protected]<mailto:[email protected]<[email protected]>>'; >> 'lars hofhansl'**** >> >> >>** ** >> >> >> *Subject:* RE: Regionserver goes down while endpoint execution******** >> >> >>** ** >> >> >> ** ****** >> >> >>** ** >> >> >> Lars,******** >> >> >>** ** >> >> >> It is having following errors when I execute the Endpoint RPC client ** >> ** >> >> >> from eclipse. It seems some of the regions at regionserver **** >> >> >> vm-8aa9-fe74.nam.nsroot.net<http://vm-8aa9-fe74.nam.nsroot.net> is **** >> >> >> taking more time to reponse.******** >> >> >>** ** >> >> >> ** ****** >> >> >>** ** >> >> >> Could you guide how to fix it. I don't find any option to set **** >> >> >> hbase.rpc.timeout from hbase configuration menu in CDH4 CM server for * >> *** >> >> >> hbase configuration.****** >> >> >> ****** >> >> >>** ** >> >> >> ** ****** >> >> >>** ** >> >> >> Regards,******** >> >> >>** ** >> >> >> Deepak******** >> >> >>** ** >> >> >> ** ****** >> >> >>** ** >> >> >> 3/03/12 02:33:12 INFO zookeeper.ClientCnxn: Session establishment **** >> >> >> complete on server **** >> >> >> vm-15c2-3bbf.nam.nsroot.net/10.96.172.44:2181<http://vm-15c2-3bbf.nam** >> ** >> >> >> .nsroot.net/10.96.172.44:2181>, sessionid = 0x53d591b77090026, **** >> >> >> negotiated timeout = 60000******** >> >> >>** ** >> >> >> Mar 12, 2013 2:33:13 AM org.apache.hadoop.conf.Configuration**** >> >> >> warnOnceIfDeprecated******** >> >> >>** ** >> >> >> WARNING: hadoop.native.lib is deprecated. Instead, use**** >> >> >> io.native.lib.available******** >> >> >>** ** >> >> >> Mar 12, 2013 2:44:00 AM**** >> >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplemen** >> ** >> >> >> tation**** >> >> >> processExecs******** >> >> >>** ** >> >> >> WARNING: Error executing for row 153299:1362780381523:2932572079500658: >> **** >> >> >> vm-ab1f-dd21.nam.nsroot.net:******** >> >> >>** ** >> >> >> *java.util.concurrent.ExecutionException*: ***** >> >> >> org.apache.hadoop.hbase.client.RetriesExhaustedException*: Failed **** >> >> >> after attempts=10, exceptions:******** >> >> >>** ** >> >> >> Tue Mar 12 02:34:15 EDT 2013,**** >> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache** >> ** >> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, ***** >> >> >> java.net.SocketTimeoutException*: Call to **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74.na** >> ** >> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout**** >> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout **** >> >> >> while waiting for channel to be ready for read. ch :**** >> >> >> java.nio.channels.SocketChannel[connected **** >> >> >> local=/150.110.96.212:2271remote= **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020%5D****> >> <http://vm-8aa9-fe**** >> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>**** >> >> >>** ** >> >> >> Tue Mar 12 02:35:16 EDT 2013,**** >> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache** >> ** >> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, ***** >> >> >> java.net.SocketTimeoutException*: Call to **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74.na** >> ** >> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout**** >> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout **** >> >> >> while waiting for channel to be ready for read. ch :**** >> >> >> java.nio.channels.SocketChannel[connected **** >> >> >> local=/150.110.96.212:2403remote= **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020%5D****> >> <http://vm-8aa9-fe**** >> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>**** >> >> >>** ** >> >> >> Tue Mar 12 02:36:18 EDT 2013,**** >> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache** >> ** >> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, ***** >> >> >> java.net.SocketTimeoutException*: Call to **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74.na** >> ** >> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout**** >> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout **** >> >> >> while waiting for channel to be ready for read. ch :**** >> >> >> java.nio.channels.SocketChannel[connected **** >> >> >> local=/150.110.96.212:2465remote= **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020%5D****> >> <http://vm-8aa9-fe**** >> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>**** >> >> >>** ** >> >> >> Tue Mar 12 02:37:20 EDT 2013,**** >> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache** >> ** >> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, ***** >> >> >> java.net.SocketTimeoutException*: Call to **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74.na** >> ** >> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout**** >> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout **** >> >> >> while waiting for channel to be ready for read. ch :**** >> >> >> java.nio.channels.SocketChannel[connected **** >> >> >> local=/150.110.96.212:2500remote= **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020%5D****> >> <http://vm-8aa9-fe**** >> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>**** >> >> >>** ** >> >> >> Tue Mar 12 02:38:22 EDT 2013,**** >> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache** >> ** >> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, ***** >> >> >> java.net.SocketTimeoutException*: Call to **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74.na** >> ** >> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout**** >> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout **** >> >> >> while waiting for channel to be ready for read. ch :**** >> >> >> java.nio.channels.SocketChannel[connected **** >> >> >> local=/150.110.96.212:2538remote= **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020%5D****> >> <http://vm-8aa9-fe**** >> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>**** >> >> >>** ** >> >> >> Tue Mar 12 02:39:25 EDT 2013,**** >> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache** >> ** >> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, ***** >> >> >> java.net.SocketTimeoutException*: Call to **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74.na** >> ** >> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout**** >> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout **** >> >> >> while waiting for channel to be ready for read. ch :**** >> >> >> java.nio.channels.SocketChannel[connected **** >> >> >> local=/150.110.96.212:2572remote= **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020%5D****> >> <http://vm-8aa9-fe**** >> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>**** >> >> >>** ** >> >> >> Tue Mar 12 02:40:30 EDT 2013,**** >> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache** >> ** >> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, ***** >> >> >> java.net.SocketTimeoutException*: Call to **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74.na** >> ** >> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout**** >> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout **** >> >> >> while waiting for channel to be ready for read. ch :**** >> >> >> java.nio.channels.SocketChannel[connected **** >> >> >> local=/150.110.96.212:2606remote= **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020%5D****> >> <http://vm-8aa9-fe**** >> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>**** >> >> >>** ** >> >> >> Tue Mar 12 02:41:34 EDT 2013,**** >> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache** >> ** >> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, ***** >> >> >> java.net.SocketTimeoutException*: Call to **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74.na** >> ** >> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout**** >> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout **** >> >> >> while waiting for channel to be ready for read. ch :**** >> >> >> java.nio.channels.SocketChannel[connected **** >> >> >> local=/150.110.96.212:2640remote= **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020%5D****> >> <http://vm-8aa9-fe**** >> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>**** >> >> >>** ** >> >> >> Tue Mar 12 02:42:43 EDT 2013,**** >> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache** >> ** >> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, ***** >> >> >> java.net.SocketTimeoutException*: Call to **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74.na** >> ** >> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout**** >> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout **** >> >> >> while waiting for channel to be ready for read. ch :**** >> >> >> java.nio.channels.SocketChannel[connected **** >> >> >> local=/150.110.96.212:2677remote= **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020%5D****> >> <http://vm-8aa9-fe**** >> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>**** >> >> >>** ** >> >> >> Tue Mar 12 02:44:00 EDT 2013,**** >> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache** >> ** >> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, ***** >> >> >> java.net.SocketTimeoutException*: Call to **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74.na** >> ** >> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout**** >> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout **** >> >> >> while waiting for channel to be ready for read. ch :**** >> >> >> java.nio.channels.SocketChannel[connected **** >> >> >> local=/150.110.96.212:2842remote= **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020%5D****> >> <http://vm-8aa9-fe**** >> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>**** >> >> >>** ** >> >> >> ** ****** >> >> >>** ** >> >> >> at java.util.concurrent.FutureTask$Sync.innerGet(Unknown **** >> >> >> Source)******** >> >> >>** ** >> >> >> at java.util.concurrent.FutureTask.get(Unknown Source)******** >> >> >>** ** >> >> >> at**** >> >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplemen** >> ** >> >> >> tation.processExecs(**** >> >> >> *HConnectionManager.java:1466*)******** >> >> >>** ** >> >> >> at org.apache.hadoop.hbase.client.HTable.coprocessorExec(***** >> >> >> HTable.java:1577*)******** >> >> >>** ** >> >> >> at org.apache.hadoop.hbase.client.HTable.coprocessorExec(***** >> >> >> HTable.java:1557*)******** >> >> >>** ** >> >> >> at **** >> >> >> com.citi.sponge.hbase.endpoint.HBaseEndPointClientForElfLog.main(**** >> >> >> *HBaseEndPointClientForElfLog.java:33*)******** >> >> >>** ** >> >> >> Caused by: *org.apache.hadoop.hbase.client.RetriesExhaustedException*:* >> *** >> >> >> Failed after attempts=10, exceptions:******** >> >> >>** ** >> >> >> Tue Mar 12 02:34:15 EDT 2013,**** >> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache** >> ** >> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, ***** >> >> >> java.net.SocketTimeoutException*: Call to **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74.na** >> ** >> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout**** >> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout **** >> >> >> while waiting for channel to be ready for read. ch :**** >> >> >> java.nio.channels.SocketChannel[connected **** >> >> >> local=/150.110.96.212:2271remote= **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020%5D****> >> <http://vm-8aa9-fe**** >> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>**** >> >> >>** ** >> >> >> Tue Mar 12 02:35:16 EDT 2013,**** >> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache** >> ** >> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, ***** >> >> >> java.net.SocketTimeoutException*: Call to **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74.na** >> ** >> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout**** >> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout **** >> >> >> while waiting for channel to be ready for read. ch :**** >> >> >> java.nio.channels.SocketChannel[connected **** >> >> >> local=/150.110.96.212:2403remote= **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020%5D****> >> <http://vm-8aa9-fe**** >> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>**** >> >> >>** ** >> >> >> Tue Mar 12 02:36:18 EDT 2013,**** >> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache** >> ** >> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, ***** >> >> >> java.net.SocketTimeoutException*: Call to **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74.na** >> ** >> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout**** >> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout **** >> >> >> while waiting for channel to be ready for read. ch :**** >> >> >> java.nio.channels.SocketChannel[connected **** >> >> >> local=/150.110.96.212:2465remote= **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020%5D****> >> <http://vm-8aa9-fe**** >> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>**** >> >> >>** ** >> >> >> Tue Mar 12 02:37:20 EDT 2013,**** >> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache** >> ** >> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, ***** >> >> >> java.net.SocketTimeoutException*: Call to **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74.na** >> ** >> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout**** >> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout **** >> >> >> while waiting for channel to be ready for read. ch :**** >> >> >> java.nio.channels.SocketChannel[connected **** >> >> >> local=/150.110.96.212:2500remote= **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020%5D****> >> <http://vm-8aa9-fe**** >> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>**** >> >> >>** ** >> >> >> Tue Mar 12 02:38:22 EDT 2013,**** >> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache** >> ** >> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, ***** >> >> >> java.net.SocketTimeoutException*: Call to **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74.na** >> ** >> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout**** >> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout **** >> >> >> while waiting for channel to be ready for read. ch :**** >> >> >> java.nio.channels.SocketChannel[connected **** >> >> >> local=/150.110.96.212:2538remote= **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020%5D****> >> <http://vm-8aa9-fe**** >> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>**** >> >> >>** ** >> >> >> Tue Mar 12 02:39:25 EDT 2013,**** >> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache** >> ** >> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, ***** >> >> >> java.net.SocketTimeoutException*: Call to **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74.na** >> ** >> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout**** >> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout **** >> >> >> while waiting for channel to be ready for read. ch :**** >> >> >> java.nio.channels.SocketChannel[connected **** >> >> >> local=/150.110.96.212:2572remote= **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020%5D****> >> <http://vm-8aa9-fe**** >> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>**** >> >> >>** ** >> >> >> Tue Mar 12 02:40:30 EDT 2013,**** >> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache** >> ** >> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, ***** >> >> >> java.net.SocketTimeoutException*: Call to **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74.na** >> ** >> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout**** >> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout **** >> >> >> while waiting for channel to be ready for read. ch :**** >> >> >> java.nio.channels.SocketChannel[connected **** >> >> >> local=/150.110.96.212:2606remote= **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020%5D****> >> <http://vm-8aa9-fe**** >> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>**** >> >> >>** ** >> >> >> Tue Mar 12 02:41:34 EDT 2013,**** >> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache** >> ** >> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, ***** >> >> >> java.net.SocketTimeoutException*: Call to **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74.na** >> ** >> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout**** >> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout **** >> >> >> while waiting for channel to be ready for read. ch :**** >> >> >> java.nio.channels.SocketChannel[connected **** >> >> >> local=/150.110.96.212:2640remote= **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020%5D****> >> <http://vm-8aa9-fe**** >> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>**** >> >> >>** ** >> >> >> Tue Mar 12 02:42:43 EDT 2013,**** >> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache** >> ** >> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, ***** >> >> >> java.net.SocketTimeoutException*: Call to **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74.na** >> ** >> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout**** >> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout **** >> >> >> while waiting for channel to be ready for read. ch :**** >> >> >> java.nio.channels.SocketChannel[connected **** >> >> >> local=/150.110.96.212:2677remote= **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020%5D****> >> <http://vm-8aa9-fe**** >> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>**** >> >> >>** ** >> >> >> Tue Mar 12 02:44:00 EDT 2013,**** >> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache** >> ** >> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, ***** >> >> >> java.net.SocketTimeoutException*: Call to **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74.na** >> ** >> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout**** >> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout **** >> >> >> while waiting for channel to be ready for read. ch :**** >> >> >> java.nio.channels.SocketChannel[connected **** >> >> >> local=/150.110.96.212:2842remote= **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020%5D****> >> <http://vm-8aa9-fe**** >> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>**** >> >> >>** ** >> >> >> ** ****** >> >> >>** ** >> >> >> at**** >> >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplemen** >> ** >> >> >> tation.getRegionServerWithRetries(**** >> >> >> *HConnectionManager.java:1345*)******** >> >> >>** ** >> >> >> at org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(***** >> >> >> ExecRPCInvoker.java:79*)******** >> >> >>** ** >> >> >> at $Proxy8.getValues(Unknown Source)******** >> >> >>** ** >> >> >> at**** >> >> >> com.citi.sponge.hbase.endpoint.HBaseEndPointClientForElfLog$1.call(**** >> * >> >> >> HBaseEndPointClientForElfLog.java:38*)******** >> >> >>** ** >> >> >> at**** >> >> >> com.citi.sponge.hbase.endpoint.HBaseEndPointClientForElfLog$1.call(**** >> * >> >> >> HBaseEndPointClientForElfLog.java:1*)******** >> >> >>** ** >> >> >> at**** >> >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplemen** >> ** >> >> >> tation$4.call(**** >> >> >> *HConnectionManager.java:1454*)******** >> >> >>** ** >> >> >> at java.util.concurrent.FutureTask$Sync.innerRun(Unknown **** >> >> >> Source)******** >> >> >>** ** >> >> >> at java.util.concurrent.FutureTask.run(Unknown Source)******** >> >> >>** ** >> >> >> at **** >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown**** >> >> >> Source)******** >> >> >>** ** >> >> >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown *** >> * >> >> >> Source)**** >> >> >> ******** >> >> >>** ** >> >> >> at java.lang.Thread.run(Unknown Source)******** >> >> >>** ** >> >> >> *org.apache.hadoop.hbase.client.RetriesExhaustedException*: Failed **** >> >> >> after attempts=10, exceptions:******** >> >> >>** ** >> >> >> Tue Mar 12 02:34:15 EDT 2013,**** >> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache** >> ** >> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, ***** >> >> >> java.net.SocketTimeoutException*: Call to **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74.na** >> ** >> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout**** >> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout **** >> >> >> while waiting for channel to be ready for read. ch :**** >> >> >> java.nio.channels.SocketChannel[connected **** >> >> >> local=/150.110.96.212:2271remote= **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020%5D****> >> <http://vm-8aa9-fe**** >> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>**** >> >> >>** ** >> >> >> Tue Mar 12 02:35:16 EDT 2013,**** >> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache** >> ** >> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, ***** >> >> >> java.net.SocketTimeoutException*: Call to **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74.na** >> ** >> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout**** >> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout **** >> >> >> while waiting for channel to be ready for read. ch :**** >> >> >> java.nio.channels.SocketChannel[connected **** >> >> >> local=/150.110.96.212:2403remote= **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020%5D****> >> <http://vm-8aa9-fe**** >> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>**** >> >> >>** ** >> >> >> Tue Mar 12 02:36:18 EDT 2013,**** >> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache** >> ** >> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, ***** >> >> >> java.net.SocketTimeoutException*: Call to **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74.na** >> ** >> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout**** >> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout **** >> >> >> while waiting for channel to be ready for read. ch :**** >> >> >> java.nio.channels.SocketChannel[connected **** >> >> >> local=/150.110.96.212:2465remote= **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020%5D****> >> <http://vm-8aa9-fe**** >> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>**** >> >> >>** ** >> >> >> Tue Mar 12 02:37:20 EDT 2013,**** >> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache** >> ** >> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, ***** >> >> >> java.net.SocketTimeoutException*: Call to **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74.na** >> ** >> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout**** >> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout **** >> >> >> while waiting for channel to be ready for read. ch :**** >> >> >> java.nio.channels.SocketChannel[connected **** >> >> >> local=/150.110.96.212:2500remote= **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020%5D****> >> <http://vm-8aa9-fe**** >> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>**** >> >> >>** ** >> >> >> Tue Mar 12 02:38:22 EDT 2013,**** >> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache** >> ** >> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, ***** >> >> >> java.net.SocketTimeoutException*: Call to **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74.na** >> ** >> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout**** >> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout **** >> >> >> while waiting for channel to be ready for read. ch :**** >> >> >> java.nio.channels.SocketChannel[connected **** >> >> >> local=/150.110.96.212:2538remote= **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020%5D****> >> <http://vm-8aa9-fe**** >> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>**** >> >> >>** ** >> >> >> Tue Mar 12 02:39:25 EDT 2013,**** >> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache** >> ** >> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, ***** >> >> >> java.net.SocketTimeoutException*: Call to **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74.na** >> ** >> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout**** >> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout **** >> >> >> while waiting for channel to be ready for read. ch :**** >> >> >> java.nio.channels.SocketChannel[connected **** >> >> >> local=/150.110.96.212:2572remote= **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020%5D****> >> <http://vm-8aa9-fe**** >> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>**** >> >> >>** ** >> >> >> Tue Mar 12 02:40:30 EDT 2013,**** >> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache** >> ** >> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, ***** >> >> >> java.net.SocketTimeoutException*: Call to **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74.na** >> ** >> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout**** >> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout **** >> >> >> while waiting for channel to be ready for read. ch :**** >> >> >> java.nio.channels.SocketChannel[connected **** >> >> >> local=/150.110.96.212:2606remote= **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020%5D****> >> <http://vm-8aa9-fe**** >> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>**** >> >> >>** ** >> >> >> Tue Mar 12 02:41:34 EDT 2013,**** >> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache** >> ** >> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, ***** >> >> >> java.net.SocketTimeoutException*: Call to **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74.na** >> ** >> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout**** >> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout **** >> >> >> while waiting for channel to be ready for read. ch :**** >> >> >> java.nio.channels.SocketChannel[connected **** >> >> >> local=/150.110.96.212:2640remote= **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020%5D****> >> <http://vm-8aa9-fe**** >> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>**** >> >> >>** ** >> >> >> Tue Mar 12 02:42:43 EDT 2013,**** >> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache** >> ** >> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, ***** >> >> >> java.net.SocketTimeoutException*: Call to **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74.na** >> ** >> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout**** >> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout **** >> >> >> while waiting for channel to be ready for read. ch :**** >> >> >> java.nio.channels.SocketChannel[connected **** >> >> >> local=/150.110.96.212:2677remote= **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020%5D****> >> <http://vm-8aa9-fe**** >> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>**** >> >> >>** ** >> >> >> Tue Mar 12 02:44:00 EDT 2013,**** >> >> >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@39443f<mailto:org.apache** >> ** >> >> >> .hadoop.hbase.ipc.ExecRPCInvoker$1@39443f>, ***** >> >> >> java.net.SocketTimeoutException*: Call to **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020<http://vm-8aa9-fe74.na** >> ** >> >> >> m.nsroot.net/10.42.105.91:60020> failed on socket timeout**** >> >> >> exception: *java.net.SocketTimeoutException*: 60000 millis timeout **** >> >> >> while waiting for channel to be ready for read. ch :**** >> >> >> java.nio.channels.SocketChannel[connected **** >> >> >> local=/150.110.96.212:2842remote= **** >> >> >> vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020]****<http://vm-8aa9-fe74.nam.nsroot.net/10.42.105.91:60020%5D****> >> <http://vm-8aa9-fe**** >> >> >> 74.nam.nsroot.net/10.42.105.91:60020%5d****>**** >> >> >>** ** >> >> >> ** ****** >> >> >>** ** >> >> >> at**** >> >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplemen** >> ** >> >> >> tation.getRegionServerWithRetries(**** >> >> >> *HConnectionManager.java:1345*)******** >> >> >>** ** >> >> >> at org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(***** >> >> >> ExecRPCInvoker.java:79*)******** >> >> >>** ** >> >> >> at $Proxy8.getValues(Unknown Source)******** >> >> >>** ** >> >> >> at**** >> >> >> com.citi.sponge.hbase.endpoint.HBaseEndPointClientForElfLog$1.call(**** >> * >> >> >> HBaseEndPointClientForElfLog.java:38*)******** >> >> >>** ** >> >> >> at**** >> >> >> com.citi.sponge.hbase.endpoint.HBaseEndPointClientForElfLog$1.call(**** >> * >> >> >> HBaseEndPointClientForElfLog.java:1*)******** >> >> >>** ** >> >> >> at**** >> >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplemen** >> ** >> >> >> tation$4.call(**** >> >> >> *HConnectionManager.java:1454*)******** >> >> >>** ** >> >> >> at java.util.concurrent.FutureTask$Sync.innerRun(Unknown **** >> >> >> Source)******** >> >> >>** ** >> >> >> at java.util.concurrent.FutureTask.run(Unknown Source)******** >> >> >>** ** >> >> >> at **** >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown**** >> >> >> Source)******** >> >> >>** ** >> >> >> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown *** >> * >> >> >> Source)**** >> >> >> ******** >> >> >>** ** >> >> >> at java.lang.Thread.run(Unknown Source)******** >> >> >>** ** >> >> >> ** ****** >> >> >>** ** >> >> >> ** ****** >> >> >>** ** >> >> >> *From:* Kumar, Deepak8 [CCC-OT_IT NE]**** >> >> >> *Sent:* Tuesday, March 12, 2013 2:27 AM**** >> >> >> *To:* >> >> '[email protected]<mailto:[email protected]<[email protected]>>'; >> 'lars hofhansl'**** >> >> >> *Subject:* RE: Regionserver goes down while endpoint execution******** >> >> >>** ** >> >> >> ** ****** >> >> >>** ** >> >> >> Lars,******** >> >> >>** ** >> >> >> Thanks for your quick response.There is not much info in region **** >> >> >> server log. I am again executing it with DEBUG log level in region **** >> >> >> servers.******** >> >> >>** ** >> >> >> ** ****** >> >> >>** ** >> >> >> *Here is the endpoint code***** >> >> >>** ** >> >> >> ** ****** >> >> >>** ** >> >> >> public class ColumnAggregationEndpoint extends **** >> >> >> BaseEndpointCoprocessor******** >> >> >>** ** >> >> >> implements ColumnAggregationProtocol {******** >> >> >>** ** >> >> >> ******** >> >> >>** ** >> >> >> @Override******** >> >> >>** ** >> >> >> public List<String> getValues(byte[] family, byte[] **** >> >> >> qualifier, int batchSize, int cacheSize)******** >> >> >>** ** >> >> >> throws IOException {******** >> >> >>** ** >> >> >> // aggregate at each region******** >> >> >>** ** >> >> >> Scan scan = new Scan();******** >> >> >>** ** >> >> >> scan.addColumn(family, qualifier);******** >> >> >>** ** >> >> >> scan.setCaching(cacheSize);******** >> >> >>** ** >> >> >> scan.setBatch(batchSize);******** >> >> >>** ** >> >> >> List<String> values = new ArrayList<String>();******** >> >> >>** ** >> >> >> RegionCoprocessorEnvironment environment =******** >> >> >>** ** >> >> >> (RegionCoprocessorEnvironment) ** ** >> >> >> getEnvironment();******** >> >> >>** ** >> >> >> ******** >> >> >>** ** >> >> >> InternalScanner scanner =**** >> >> >> environment.getRegion().getScanner(scan);******** >> >> >>** ** >> >> >> try {******** >> >> >>** ** >> >> >> List<KeyValue> curVals = new ArrayList<KeyValue>();******** >> >> >>** ** >> >> >> boolean hasMore = false;******** >> >> >>** ** >> >> >> do {******** >> >> >>** ** >> >> >> curVals.clear();******** >> >> >>** ** >> >> >> hasMore = scanner.next(curVals);******** >> >> >>** ** >> >> >> KeyValue kv = curVals.get(0);******** >> >> >>** ** >> >> >> values.add(Bytes.toString(kv.getValue()));******** >> >> >>** ** >> >> >> } while (hasMore);******** >> >> >>** ** >> >> >> } finally {******** >> >> >>** ** >> >> >> scanner.close();******** >> >> >>** ** >> >> >> }******** >> >> >>** ** >> >> >> return values;******** >> >> >>** ** >> >> >> }******** >> >> >>** ** >> >> >> }******** >> >> >>** ** >> >> >> ** ****** >> >> >>** ** >> >> >> ** ****** >> >> >>** ** >> >> >> ** ****** >> >> >>** ** >> >> >> *The RPC client to invoke the Endpoint is as follows:***** >> >> >>** ** >> >> >> ** ****** >> >> >>** ** >> >> >> public class HBaseEndPointClientForElfLog {******** >> >> >>** ** >> >> >> public static void main(String[] args) {******** >> >> >>** ** >> >> >> try {******** >> >> >>** ** >> >> >> Configuration conf = **** >> >> >> HBaseConfiguration.create();******** >> >> >>** ** >> >> >> conf.set(******** >> >> >>** ** >> >> >> "hbase.zookeeper.quorum",******** >> >> >>** ** >> >> >> **** >> >> >> "vm-ab1f-dd21.nam.nsroot.net<http://vm-ab1f-dd21.nam.nsroot.net>,**** >> >> >> vm-cb03-2277.nam.nsroot.net<http://vm-cb03-2277.nam.nsroot.net>,vm-15** >> ** >> >> >> c2-3bbf.nam.nsroot.net<http://vm-15c2-3bbf.nam.nsroot.net>");******** >> >> >>** ** >> >> >> String tableName = "elf_log";******** >> >> >>** ** >> >> >> final String columnFamily = "content";******** >> >> >>** ** >> >> >> final String columnQualifier = "logFileName";******** >> >> >>** ** >> >> >> final String startRowKey =**** >> >> >> "153299:1362780381523:2932572079500658:vm-ab1f-dd21.nam.nsroot.net<ht** >> ** >> >> >> tp://vm-ab1f-dd21.nam.nsroot.net>:";******** >> >> >>** ** >> >> >> final String endRowKey = **** >> >> >> "153299:1362953388000";******** >> >> >>** ** >> >> >> HTableInterface table = new HTable(conf, **** >> >> >> tableName);******** >> >> >>** ** >> >> >> Scan scan;******** >> >> >>** ** >> >> >> Map<byte[], List<String>> results;******** >> >> >>** ** >> >> >> ******** >> >> >>** ** >> >> >> // scan: for all regions******** >> >> >>** ** >> >> >> scan = new Scan();******** >> >> >>** ** >> >> >> ******** >> >> >>** ** >> >> >> results =**** >> >> >> table.coprocessorExec(ColumnAggregationProtocol.class,******** >> >> >>** ** >> >> >> startRowKey.getBytes(), ** ** >> >> >> endRowKey.getBytes(),**** >> >> >> ******** >> >> >>** ** >> >> >> new **** >> >> >> Batch.Call<ColumnAggregationProtocol,**** >> >> >> List<String>>() {******** >> >> >>** ** >> >> >> public List<String> **** >> >> >> call(ColumnAggregationProtocol instance)******** >> >> >>** ** >> >> >> throws IOException **** >> >> >> {******** >> >> >>** ** >> >> >> return**** >> >> >> instance.getValues(columnFamily.getBytes(),******** >> >> >>** ** >> >> >>** ** >> >> >> columnQualifier.getBytes(),2,5);******** >> >> >>** ** >> >> >> }******** >> >> >>** ** >> >> >> });******** >> >> >>** ** >> >> >> ******** >> >> >>** ** >> >> >> for (Map.Entry<byte[], List<String>> e :**** >> >> >> results.entrySet()) {******** >> >> >>** ** >> >> >> System.out.println("Size of list returned:**** >> >> >> "+e.getValue().size());******** >> >> >>** ** >> >> >> for(String singleVal: e.getValue()){******** >> >> >>** ** >> >> >> System.out.println(singleVal);******** >> >> >>** ** >> >> >> }******** >> >> >>** ** >> >> >> ******** >> >> >>** ** >> >> >> ******** >> >> >>** ** >> >> >> }******** >> >> >>** ** >> >> >> } catch (Throwable throwable) {******** >> >> >>** ** >> >> >> throwable.printStackTrace();******** >> >> >>** ** >> >> >> }******** >> >> >>** ** >> >> >> }******** >> >> >>** ** >> >> >> }******** >> >> >>** ** >> >> >> ** ****** >> >> >>** ** >> >> >> Regards,******** >> >> >>** ** >> >> >> Deepak******** >> >> >>** ** >> >> >> ** ****** >> >> >>** ** >> >> >> -----Original Message-----**** >> >> >> From: lars hofhansl [mailto:[email protected]<mailto:[email protected]> * >> *** >> >> >> <[email protected]<mailto:[email protected]>>]**** >> >> >> Sent: Tuesday, March 12, 2013 2:01 AM**** >> >> >> To: [email protected]<mailto:[email protected]>**** >> >> >> Subject: Re: Regionserver goes down while endpoint execution******** >> >> >>** ** >> >> >> ** ****** >> >> >>** ** >> >> >> What does the region server log say?******** >> >> >>** ** >> >> >> ** ****** >> >> >>** ** >> >> >> ** ****** >> >> >>** ** >> >> >> Endpoints do not run in a sandbox. You can call System.exit(...) and ** >> ** >> >> >> your RegionServer will happily exit.******** >> >> >>** ** >> >> >> If you can, please show us your endpoint code.******** >> >> >>** ** >> >> >> ** ****** >> >> >>** ** >> >> >> -- Lars******** >> >> >>** ** >> >> >> ** ****** >> >> >>** ** >> >> >> ** ****** >> >> >>** ** >> >> >> ** ****** >> >> >>** ** >> >> >> ________________________________******** >> >> >>** ** >> >> >> From: "Kumar, Deepak8 " **** >> >> >> <[email protected]<mailto:[email protected]>>******** >> >> >>** ** >> >> >> To: >> >> "'[email protected]<mailto:[email protected]<[email protected]>>'" >> **** >> >> >> <[email protected]<mailto:[email protected]>> ******** >> >> >>** ** >> >> >> Sent: Monday, March 11, 2013 10:51 PM******** >> >> >>** ** >> >> >> Subject: Regionserver goes down while endpoint execution******** >> >> >>** ** >> >> >> ** ****** >> >> >>** ** >> >> >> Hi,******** >> >> >>** ** >> >> >> I have a table in hbase which has more than 5GB of data, it is **** >> >> >> distributed at 101 regions at 5 regionservers.******** >> >> >>** ** >> >> >> ** ****** >> >> >>** ** >> >> >> When I execute an endpoint which is supposed to fetch a column **** >> >> >> qualifier value using an endpoint RPC client, the region server goes ** >> ** >> >> >> down. The hbase master log says "Can't connect to region, retrying.." * >> *** >> >> >> The same endpoint works fine for tables which has 300 records.******** >> >> >>** ** >> >> >> ** ****** >> >> >>** ** >> >> >> Could you please guide me the reason for being regionserver down?****** >> ** >> >> >>** ** >> >> >> ** ****** >> >> >>** ** >> >> >> Regards,******** >> >> >>** ** >> >> >> Deepak******** >> >> >>** ** >> >> >** ** >>
