Re: Getting ScannerTimeoutException even after several calls in the specified time limit
Hi, Are u trying to do parallel scans. If yes, check the time taken for GC and the number of calls that can be served at your end point. Best Regards N.Hari Kumar On Tue, Sep 11, 2012 at 8:22 AM, Dhirendra Singh dps...@gmail.com wrote: i tried with a smaller caching i.e 10, it failed again, not its not really a big cell. this small cluster(4 nodes) is only used for Hbase, i am currently using hbase-0.92.1-cdh4.0.1. , could you just let me know how could i debug this issue ? aused by: org.apache.hadoop.hbase.client.ScannerTimeoutException: 99560ms passed since the last invocation, timeout is currently set to 6 at org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1302) at org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1399) ... 5 more Caused by: org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Name: -8889369042827960647 at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2114) at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336) On Mon, Sep 10, 2012 at 10:53 PM, Stack st...@duboce.net wrote: On Mon, Sep 10, 2012 at 10:13 AM, Dhirendra Singh dps...@gmail.com wrote: I am facing this exception while iterating over a big table, by default i have specified caching as 100, i am getting the below exception, even though i checked there are several calls made to the scanner before it threw this exception, but somehow its saying 86095ms were passed since last invocation. i also observed that if it set scan.setCaching(false), it succeeds, could some one please explain or point me to some document as if what's happening here and what's the best practices to avoid it. Try again cachine 100. See if it works. A big cell? A GC pause? You should be able to tell roughly which server is being traversed when you get the timeout. Anything else going on on that server at the time? What version of HBase? St.Ack -- Warm Regards, Dhirendra Pratap +91. 9717394713 -- FROM HARI KUMAR.N
Re: Getting ScannerTimeoutException even after several calls in the specified time limit
For GC Monitoring, Add Parameters export HBASE_OPTS=$HBASE_OPTS -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:$HBASE_HOME/logs/gc-hbase.log to hbase-env.sh and try to view the file using tools like GCViewer. or use tools like VisualVM to look at your GC Consumption. ./hari Add On Tue, Sep 11, 2012 at 2:11 PM, Dhirendra Singh dps...@gmail.com wrote: No i am not doing parallel scans, * If yes, check the time taken for GC and the number of calls that can be served at your end point*. could you please tell me how to do that, where can i see the GC logs? On Tue, Sep 11, 2012 at 12:54 PM, HARI KUMAR harikum2...@gmail.comwrote: Hi, Are u trying to do parallel scans. If yes, check the time taken for GC and the number of calls that can be served at your end point. Best Regards N.Hari Kumar On Tue, Sep 11, 2012 at 8:22 AM, Dhirendra Singh dps...@gmail.com wrote: i tried with a smaller caching i.e 10, it failed again, not its not really a big cell. this small cluster(4 nodes) is only used for Hbase, i am currently using hbase-0.92.1-cdh4.0.1. , could you just let me know how could i debug this issue ? aused by: org.apache.hadoop.hbase.client.ScannerTimeoutException: 99560ms passed since the last invocation, timeout is currently set to 6 at org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1302) at org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1399) ... 5 more Caused by: org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Name: -8889369042827960647 at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2114) at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336) On Mon, Sep 10, 2012 at 10:53 PM, Stack st...@duboce.net wrote: On Mon, Sep 10, 2012 at 10:13 AM, Dhirendra Singh dps...@gmail.com wrote: I am facing this exception while iterating over a big table, by default i have specified caching as 100, i am getting the below exception, even though i checked there are several calls made to the scanner before it threw this exception, but somehow its saying 86095ms were passed since last invocation. i also observed that if it set scan.setCaching(false), it succeeds, could some one please explain or point me to some document as if what's happening here and what's the best practices to avoid it. Try again cachine 100. See if it works. A big cell? A GC pause? You should be able to tell roughly which server is being traversed when you get the timeout. Anything else going on on that server at the time? What version of HBase? St.Ack -- Warm Regards, Dhirendra Pratap +91. 9717394713 -- FROM HARI KUMAR.N -- Warm Regards, Dhirendra Pratap +91. 9717394713 -- FROM HARI KUMAR.N
Re: Getting ScannerTimeoutException even after several calls in the specified time limit
For pretty graphs with JVM GC info + system + HBase metrics you could also easily hook up SPM to your cluster. See URL in signature. Otis -- Performance Monitoring - http://sematext.com/spm On Sep 11, 2012 6:30 AM, HARI KUMAR harikum2...@gmail.com wrote: For GC Monitoring, Add Parameters export HBASE_OPTS=$HBASE_OPTS -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:$HBASE_HOME/logs/gc-hbase.log to hbase-env.sh and try to view the file using tools like GCViewer. or use tools like VisualVM to look at your GC Consumption. ./hari Add On Tue, Sep 11, 2012 at 2:11 PM, Dhirendra Singh dps...@gmail.com wrote: No i am not doing parallel scans, * If yes, check the time taken for GC and the number of calls that can be served at your end point*. could you please tell me how to do that, where can i see the GC logs? On Tue, Sep 11, 2012 at 12:54 PM, HARI KUMAR harikum2...@gmail.com wrote: Hi, Are u trying to do parallel scans. If yes, check the time taken for GC and the number of calls that can be served at your end point. Best Regards N.Hari Kumar On Tue, Sep 11, 2012 at 8:22 AM, Dhirendra Singh dps...@gmail.com wrote: i tried with a smaller caching i.e 10, it failed again, not its not really a big cell. this small cluster(4 nodes) is only used for Hbase, i am currently using hbase-0.92.1-cdh4.0.1. , could you just let me know how could i debug this issue ? aused by: org.apache.hadoop.hbase.client.ScannerTimeoutException: 99560ms passed since the last invocation, timeout is currently set to 6 at org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1302) at org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1399) ... 5 more Caused by: org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Name: -8889369042827960647 at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2114) at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336) On Mon, Sep 10, 2012 at 10:53 PM, Stack st...@duboce.net wrote: On Mon, Sep 10, 2012 at 10:13 AM, Dhirendra Singh dps...@gmail.com wrote: I am facing this exception while iterating over a big table, by default i have specified caching as 100, i am getting the below exception, even though i checked there are several calls made to the scanner before it threw this exception, but somehow its saying 86095ms were passed since last invocation. i also observed that if it set scan.setCaching(false), it succeeds, could some one please explain or point me to some document as if what's happening here and what's the best practices to avoid it. Try again cachine 100. See if it works. A big cell? A GC pause? You should be able to tell roughly which server is being traversed when you get the timeout. Anything else going on on that server at the time? What version of HBase? St.Ack -- Warm Regards, Dhirendra Pratap +91. 9717394713 -- FROM HARI KUMAR.N -- Warm Regards, Dhirendra Pratap +91. 9717394713 -- FROM HARI KUMAR.N
Re: Getting ScannerTimeoutException even after several calls in the specified time limit
could someone please clarify, when i say caching 100 or any number, where does this actually happen on server (cluster ) or client. if i assume it happens on cluster, so does this ScannerTimeOut is because of caching as the server might have run out of memory and hence not able to respond within the specified timeout? any link related to caching mechanism in HBase would be of great help Thanks, On Wed, Sep 12, 2012 at 7:41 AM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: For pretty graphs with JVM GC info + system + HBase metrics you could also easily hook up SPM to your cluster. See URL in signature. Otis -- Performance Monitoring - http://sematext.com/spm On Sep 11, 2012 6:30 AM, HARI KUMAR harikum2...@gmail.com wrote: For GC Monitoring, Add Parameters export HBASE_OPTS=$HBASE_OPTS -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:$HBASE_HOME/logs/gc-hbase.log to hbase-env.sh and try to view the file using tools like GCViewer. or use tools like VisualVM to look at your GC Consumption. ./hari Add On Tue, Sep 11, 2012 at 2:11 PM, Dhirendra Singh dps...@gmail.com wrote: No i am not doing parallel scans, * If yes, check the time taken for GC and the number of calls that can be served at your end point*. could you please tell me how to do that, where can i see the GC logs? On Tue, Sep 11, 2012 at 12:54 PM, HARI KUMAR harikum2...@gmail.com wrote: Hi, Are u trying to do parallel scans. If yes, check the time taken for GC and the number of calls that can be served at your end point. Best Regards N.Hari Kumar On Tue, Sep 11, 2012 at 8:22 AM, Dhirendra Singh dps...@gmail.com wrote: i tried with a smaller caching i.e 10, it failed again, not its not really a big cell. this small cluster(4 nodes) is only used for Hbase, i am currently using hbase-0.92.1-cdh4.0.1. , could you just let me know how could i debug this issue ? aused by: org.apache.hadoop.hbase.client.ScannerTimeoutException: 99560ms passed since the last invocation, timeout is currently set to 6 at org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1302) at org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1399) ... 5 more Caused by: org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Name: -8889369042827960647 at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2114) at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336) On Mon, Sep 10, 2012 at 10:53 PM, Stack st...@duboce.net wrote: On Mon, Sep 10, 2012 at 10:13 AM, Dhirendra Singh dps...@gmail.com wrote: I am facing this exception while iterating over a big table, by default i have specified caching as 100, i am getting the below exception, even though i checked there are several calls made to the scanner before it threw this exception, but somehow its saying 86095ms were passed since last invocation. i also observed that if it set scan.setCaching(false), it succeeds, could some one please explain or point me to some document as if what's happening here and what's the best practices to avoid it. Try again cachine 100. See if it works. A big cell? A GC pause? You should be able to tell roughly which server is being traversed when you get the timeout. Anything else going on on that server at the time? What version of HBase? St.Ack -- Warm Regards, Dhirendra Pratap +91. 9717394713 -- FROM HARI KUMAR.N -- Warm Regards, Dhirendra Pratap +91. 9717394713 -- FROM HARI KUMAR.N -- Warm Regards, Dhirendra Pratap +91. 9717394713
RE: Getting ScannerTimeoutException even after several calls in the specified time limit
could someone please clarify, when i say caching 100 or any number, where does this actually happen on server (cluster ) or client This happens at both places. When the scan calls with caching= N, the client will pass this number N to the 1st region which is under scan for this specific scan. Server side (RS) will try to find as much results(rows) from this region with max rows=N. If it is able to find the client got the results for that next() call. If it gets rows less than N, client will try to get the remaining number of rows from the next region and so on.. Mostly this will happen in server side alone.[It might be finding N rows from one region itself] But when you have some Filter conditions it might not be finding N rows from one region... Note : Client will try to find N rows with one next() call as N is specified as caching. So it might be contacting many regions across different RSs. There is a max result size config param also available at client side.. If the total size of the results exceeds this value and there are less than N results, then client will stop scanning even it has not got N results... If this cross of size is not happening well one call of next() might go through all the regions.. [You may be getting ScannerTimeouts due to RPC time outs] Hope I have answered your question.. :) -Anoop- From: Dhirendra Singh [dps...@gmail.com] Sent: Wednesday, September 12, 2012 7:55 AM To: user@hbase.apache.org Subject: Re: Getting ScannerTimeoutException even after several calls in the specified time limit could someone please clarify, when i say caching 100 or any number, where does this actually happen on server (cluster ) or client. if i assume it happens on cluster, so does this ScannerTimeOut is because of caching as the server might have run out of memory and hence not able to respond within the specified timeout? any link related to caching mechanism in HBase would be of great help Thanks, On Wed, Sep 12, 2012 at 7:41 AM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: For pretty graphs with JVM GC info + system + HBase metrics you could also easily hook up SPM to your cluster. See URL in signature. Otis -- Performance Monitoring - http://sematext.com/spm On Sep 11, 2012 6:30 AM, HARI KUMAR harikum2...@gmail.com wrote: For GC Monitoring, Add Parameters export HBASE_OPTS=$HBASE_OPTS -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:$HBASE_HOME/logs/gc-hbase.log to hbase-env.sh and try to view the file using tools like GCViewer. or use tools like VisualVM to look at your GC Consumption. ./hari Add On Tue, Sep 11, 2012 at 2:11 PM, Dhirendra Singh dps...@gmail.com wrote: No i am not doing parallel scans, * If yes, check the time taken for GC and the number of calls that can be served at your end point*. could you please tell me how to do that, where can i see the GC logs? On Tue, Sep 11, 2012 at 12:54 PM, HARI KUMAR harikum2...@gmail.com wrote: Hi, Are u trying to do parallel scans. If yes, check the time taken for GC and the number of calls that can be served at your end point. Best Regards N.Hari Kumar On Tue, Sep 11, 2012 at 8:22 AM, Dhirendra Singh dps...@gmail.com wrote: i tried with a smaller caching i.e 10, it failed again, not its not really a big cell. this small cluster(4 nodes) is only used for Hbase, i am currently using hbase-0.92.1-cdh4.0.1. , could you just let me know how could i debug this issue ? aused by: org.apache.hadoop.hbase.client.ScannerTimeoutException: 99560ms passed since the last invocation, timeout is currently set to 6 at org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1302) at org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1399) ... 5 more Caused by: org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Name: -8889369042827960647 at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2114) at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336) On Mon, Sep 10, 2012 at 10:53 PM, Stack st...@duboce.net wrote: On Mon, Sep 10, 2012 at 10:13 AM, Dhirendra Singh dps...@gmail.com wrote: I am facing this exception while iterating over a big table, by default i have
Re: Getting ScannerTimeoutException even after several calls in the specified time limit
On Mon, Sep 10, 2012 at 10:13 AM, Dhirendra Singh dps...@gmail.com wrote: I am facing this exception while iterating over a big table, by default i have specified caching as 100, i am getting the below exception, even though i checked there are several calls made to the scanner before it threw this exception, but somehow its saying 86095ms were passed since last invocation. i also observed that if it set scan.setCaching(false), it succeeds, could some one please explain or point me to some document as if what's happening here and what's the best practices to avoid it. Try again cachine 100. See if it works. A big cell? A GC pause? You should be able to tell roughly which server is being traversed when you get the timeout. Anything else going on on that server at the time? What version of HBase? St.Ack
Re: Getting ScannerTimeoutException even after several calls in the specified time limit
i tried with a smaller caching i.e 10, it failed again, not its not really a big cell. this small cluster(4 nodes) is only used for Hbase, i am currently using hbase-0.92.1-cdh4.0.1. , could you just let me know how could i debug this issue ? aused by: org.apache.hadoop.hbase.client.ScannerTimeoutException: 99560ms passed since the last invocation, timeout is currently set to 6 at org.apache.hadoop.hbase.client.HTable$ClientScanner.next(HTable.java:1302) at org.apache.hadoop.hbase.client.HTable$ClientScanner$1.hasNext(HTable.java:1399) ... 5 more Caused by: org.apache.hadoop.hbase.UnknownScannerException: org.apache.hadoop.hbase.UnknownScannerException: Name: -8889369042827960647 at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2114) at sun.reflect.GeneratedMethodAccessor38.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1336) On Mon, Sep 10, 2012 at 10:53 PM, Stack st...@duboce.net wrote: On Mon, Sep 10, 2012 at 10:13 AM, Dhirendra Singh dps...@gmail.com wrote: I am facing this exception while iterating over a big table, by default i have specified caching as 100, i am getting the below exception, even though i checked there are several calls made to the scanner before it threw this exception, but somehow its saying 86095ms were passed since last invocation. i also observed that if it set scan.setCaching(false), it succeeds, could some one please explain or point me to some document as if what's happening here and what's the best practices to avoid it. Try again cachine 100. See if it works. A big cell? A GC pause? You should be able to tell roughly which server is being traversed when you get the timeout. Anything else going on on that server at the time? What version of HBase? St.Ack -- Warm Regards, Dhirendra Pratap +91. 9717394713
