One thing I don't get: If the RS went down, then the RPC connection should have been reset, thus causing the client to interrupt, right? It shouldn't be a matter of timeout at all.
On Dec 17, 2012, at 7:18 PM, Bryan Keller wrote: > It seems there was a cascading effect. The regionservers were busy with > scanning a table, which resulted in some long GC's. The GC's were long enough > to trigger the Zookeeper timeout on at least one regionserver, which resulted > in the regionserver shutting itself down. This then caused the Object.wait() > call which got stuck, and only exited after the very long RPC timeout. > > I have done a fair amount of work optimizing the GCs, and I increased the > regionserver timeouts, which should help with the regionserver shutdowns. But > if a regionserver does shut down for some other reason, this will still > result in the Object.wait() hang. > > One approach might be to have the regionservers send back a keep-alive, or > progress, message during a scan, and that message would reset the RPC timer. > The regionserver could do this every x number of rows processed server-side. > Then the RPC timeout could be something more sensible rather than being set > to the longest time it takes to scan a region. > > HBASE-5416 looks useful, it will make scans faster, but the problem I'm > encountering will still be present, but perhaps I could set the RPC timeout a > bit lower. HBASE-6313 might fix the hang, in which case I could live with the > longer RPC timeout setting. > > > On Dec 14, 2012, at 9:49 PM, Ted Yu <[email protected]> wrote: > >> Bryan: >> >> bq. My only thought would be to forego using filters >> Please keep using filters. >> >> I and Sergey are working on HBASE-5416: Improve performance of scans with >> some kind of filters >> This feature allows you to specify one column family as being essential. >> The other column family is only returned to client when essential column >> family matches. I wonder if this may be of help to you. >> >> You mentioned regionserver going down or being busy. I assume it was not >> often that regionserver(s) went down. For busy region server, did you try >> jstack'ing regionserver process ? >> >> Thanks >> >> On Fri, Dec 14, 2012 at 2:59 PM, Bryan Keller <[email protected]> wrote: >> >>> I have encountered a problem with HBaseClient.call() hanging. This occurs >>> when one of my regionservers goes down while performing a table scan. >>> >>> What exacerbates this problem is that the scan I am performing uses >>> filters, and the region size of the table is large (4gb). Because of this, >>> it can take several minutes for a row to be returned when calling >>> scanner.next(). Apparently there is no keep alive message being sent back >>> to the scanner while the region server is busy, so I had to increase the >>> hbase.rpc.timeout value to a large number (60 min), otherwise the next() >>> call will timeout waiting for the regionserver to send something back. >>> >>> The result is that this HBaseClient.call() hang is made much worse, >>> because it won't time out for 60 minutes. >>> >>> I have a couple of questions: >>> >>> 1. Any thoughts on why the HBaseClient.call() is getting stuck? I noticed >>> that call.wait() is not using any timeout so it will wait indefinitely >>> until interrupted externally >>> >>> 2. Is there a solution where I do not need to set hbase.rpc.timeout to a >>> very large number? My only thought would be to forego using filters and do >>> the filtering client side, which seems pretty inefficient >>> >>> Here is a stack dump of the thread that was hung: >>> >>> Thread 10609: (state = BLOCKED) >>> - java.lang.Object.wait(long) @bci=0 (Interpreted frame) >>> - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) >>> - >>> org.apache.hadoop.hbase.ipc.HBaseClient.call(org.apache.hadoop.io.Writable, >>> java.net.InetSocketAddress, java.lang.Class, >>> org.apache.hadoop.hbase.security.User, int) @bci=51, line=904 (Interpreted >>> frame) >>> - >>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(java.lang.Object, >>> java.lang.reflect.Method, java.lang.Object[]) @bci=52, line=150 >>> (Interpreted frame) >>> - $Proxy12.next(long, int) @bci=26 (Interpreted frame) >>> - org.apache.hadoop.hbase.client.ScannerCallable.call() @bci=72, line=92 >>> (Interpreted frame) >>> - org.apache.hadoop.hbase.client.ScannerCallable.call() @bci=1, line=42 >>> (Interpreted frame) >>> - >>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(org.apache.hadoop.hbase.client.ServerCallable) >>> @bci=36, line=1325 (Interpreted frame) >>> - org.apache.hadoop.hbase.client.HTable$ClientScanner.next() @bci=117, >>> line=1299 (Compiled frame) >>> - org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue() >>> @bci=41, line=150 (Interpreted frame) >>> - org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue() >>> @bci=4, line=142 (Interpreted frame) >>> - org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue() >>> @bci=4, line=458 (Interpreted frame) >>> - org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue() @bci=4, >>> line=76 (Interpreted frame) >>> - >>> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue() >>> @bci=4, line=85 (Interpreted frame) >>> - >>> org.apache.hadoop.mapreduce.Mapper.run(org.apache.hadoop.mapreduce.Mapper$Context) >>> @bci=6, line=139 (Interpreted frame) >>> - >>> org.apache.hadoop.mapred.MapTask.runNewMapper(org.apache.hadoop.mapred.JobConf, >>> org.apache.hadoop.mapreduce.split.JobSplit$TaskSplitIndex, >>> org.apache.hadoop.mapred.TaskUmbilicalProtocol, >>> org.apache.hadoop.mapred.Task$TaskReporter) @bci=201, line=645 (Interpreted >>> frame) >>> - org.apache.hadoop.mapred.MapTask.run(org.apache.hadoop.mapred.JobConf, >>> org.apache.hadoop.mapred.TaskUmbilicalProtocol) @bci=100, line=325 >>> (Interpreted frame) >>> - org.apache.hadoop.mapred.Child$4.run() @bci=29, line=268 (Interpreted >>> frame) >>> - >>> java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction, >>> java.security.AccessControlContext) @bci=0 (Interpreted frame) >>> - javax.security.auth.Subject.doAs(javax.security.auth.Subject, >>> java.security.PrivilegedExceptionAction) @bci=42, line=396 (Interpreted >>> frame) >>> - >>> org.apache.hadoop.security.UserGroupInformation.doAs(java.security.PrivilegedExceptionAction) >>> @bci=14, line=1332 (Interpreted frame) >>> - org.apache.hadoop.mapred.Child.main(java.lang.String[]) @bci=776, >>> line=262 (Interpreted frame) >>> >>> >
