I think this is expected. The caching means that you only get blocks of 2000 rows. And if you go for longer than 60 seconds between blocks, then the scanner will time out. You could try tuning your caching down to 100 to see if that works for a bit (although, due to variance in the time you take for processing, you might want to give yourself a bit more cushion than that).
The next question, of course, is whether or not this is desirable behavior. Dave -----Original Message----- From: Joseph Boyd [mailto:[email protected]] Sent: Thursday, March 31, 2011 4:59 PM To: [email protected] Subject: ScannerTimeoutException when a scan enables caching, no exception when it doesn't We're using hbase 0.90.0 here, and I'm seeing a curious behavior with my scans. I have some code that does a scan over a table, and for each row returned some work to verify the data... I set the scan up like so : byte[] family = Bytes.toBytes("mytable"); Scan scan = new Scan(); scan.setCaching(2000); scan.addFamily(family); and then scan using a fairly normal looking loop: ResultScanner scanner = table.getScanner(scan); for (Result userInfoResult : scanner) { // do some work that takes about half a second } After this code runs for 60 seconds, I get the exception below: Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.hbase.client.ScannerTimeoutException: 78850ms passed since the last invocation, timeout is currently set to 60000 at org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1213) at cnwk.tridentp.firehose.ShowAssetMatchStatsMain.scanForProperties(ShowAssetMatchStatsMain.java:82) at cnwk.tridentp.firehose.ShowAssetMatchStatsMain.generateAssetMatchStats(ShowAssetMatchStatsMain.java:56) at cnwk.tridentp.firehose.ShowAssetMatchStatsMain.main(ShowAssetMatchStatsMain.java:33) Caused by: org.apache.hadoop.hbase.client.ScannerTimeoutException: 78850ms passed since the last invocation, timeout is currently set to 60000 at org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1114) at org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1210) ... 3 more Caused by: org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Name: -3447615741604242626 at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:1793) at sun.reflect.GeneratedMethodAccessor31.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1036) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:96) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:83) at org.apache.hadoop.hbase.client.ScannerCallable.call(ScannerCallable.java:38) at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:999) at org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1100) ... 4 more The message seems to indicate that I haven't called scanner.next() in a long time (> 60 seconds), which isn't true, scanner.next() is being called more than once per second. However, if I disable caching (comment out the scan.setCaching() line above), or set caching to a smaller value (1000, instead of 2000), the code runs as expected. I'm guessing that a cache-hit during my scan isn't resetting the timeout that leads to a ScannerTimeoutException, but that's just a guess... Is this expected behavior? ...joe
