Thanks Tom for the investigation. I will apply the null check in an addendum to HBASE-5139.
Operation.toJSON() does bring in Jackson JSON processor Please confirm that the jackson jars are in your CLASSPATH: $ ls lib/*jackson* lib/jackson-core-asl-1.5.5.jar lib/jackson-jaxrs-1.5.5.jar lib/jackson-mapper-asl-1.5.5.jar lib/jackson-xc-1.5.5.jar Looks like you haven't encountered the need to print contents of Scan so far. If you feel the dependency on jackson is burdensome, please file a JIRA so that we can relax this dependency. Cheers On Mon, Jan 23, 2012 at 8:35 AM, Tom Wilcox <[email protected]> wrote: > Hi Ted, > > Following from what you have said, we have edited AggregateClient.java > with the following modification to the median() method: > > ... > > // scan the region with median and find it > Scan scan2 = new Scan(scan); > // inherit stop row from method parameter > if (startRow != null) > scan2.setStartRow(startRow); > HTable table = new HTable(conf, tableName); > int cacheSize = scan2.getCaching(); > if (!scan2.getCacheBlocks() || scan2.getCaching() < 2) { > scan2.setCacheBlocks(true); > cacheSize = 5; > scan2.setCaching(cacheSize); > } > > // log.debug("scan:\t" + scan); > // log.debug("scan2:\t" + scan2); > > ResultScanner scanner = table.getScanner(scan2); > > ... > > The addition of "if (startRow != null) scan2.setStartRow(startRow);" has > fixed our NPE/Memory issue and we now see a sensible result successfully > returned. > > However, in the process of debugging this, I added some > 'log.debug("scan:\t" + scan);' statements as you can see in the snippet > above. These lines (when uncommented) were causing the following exception: > > [sshexec] java.lang.NoSuchMethodError: > org.codehaus.jackson.map.ObjectMapper.writeValueAsString(Ljava/lang/Object;)Ljava/lang/String; > [sshexec] > [sshexec] at > org.apache.hadoop.hbase.client.Operation.toJSON(Operation.java:67) > [sshexec] > [sshexec] at > org.apache.hadoop.hbase.client.Operation.toString(Operation.java:93) > [sshexec] > [sshexec] at > org.apache.hadoop.hbase.client.Operation.toString(Operation.java:107) > [sshexec] > [sshexec] at java.lang.String.valueOf(String.java:2826) > [sshexec] at java.lang.StringBuilder.append(StringBuilder.java:115) > [sshexec] at > org.apache.hadoop.hbase.client.coprocessor.AggregationClient.median(AggregationClient.java:473) > [sshexec] at > uk.org.cse.aggregation.EDRPAggregator.testSumWithValidRange(EDRPAggregator.java:55) > [sshexec] at > uk.org.cse.aggregation.EDRPAggregator.main(EDRPAggregator.java:85) > [sshexec] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > [sshexec] at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > [sshexec] at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > [sshexec] at java.lang.reflect.Method.invoke(Method.java:597) > [sshexec] at org.apache.hadoop.util.RunJar.main(RunJar.java:156) > > Do you have any idea why we got this error above? It seems the scanner > toString() method requires the Jackson JSON processor, which is failing for > some reason... > > Also, please can you tell me where log.debug() output can be found? (e.g. > $HBASE_HOME/logs/xxx-region-....log?) > > Please find a patch attached with the fixed median method for HBase 0.92 > branch. Note that I have removed the log.debug() statements. > > Thanks, > Tom > > -----Original Message----- > From: Ted Yu [mailto:[email protected]] > Sent: 23 January 2012 15:24 > To: [email protected] > Subject: median through AggregationProtocol > > Royston: > The exception came from this line: > ResultScanner scanner = table.getScanner(scan2); Can you help me review > the logic starting with: > // scan the region with median and find it > Scan scan2 = new Scan(scan); > You can log the String form of scan and scan2 before the > table.getScanner() call. > > I think the NPE below reveals that startRow is null (median is in first > region). > If that is the case, the following should help: > if (startRow != null) scan2.setStartRow(startRow); > > Thanks > > On Mon, Jan 23, 2012 at 5:50 AM, Royston Sellman < > [email protected]> wrote: > > > Hi Ted, > > > > Finally rebuilt branch/0.92 and applied your patch and rebuilt my code. > > Using AggregationClient.sum() on my test table I get the correct result. > > Just swapping to AggregationClient.median() I get the following error: > > > > [sshexec] org.apache.hadoop.hbase.client.RetriesExhaustedException: > > Failed after attempts=10, exceptions: > > [sshexec] Mon Jan 23 13:44:12 GMT 2012, > > org.apache.hadoop.hbase.client.ScannerCallable@219ba640, > > java.lang.NullPointerException > > [sshexec] Mon Jan 23 13:44:13 GMT 2012, > > org.apache.hadoop.hbase.client.ScannerCallable@219ba640, > > java.lang.NullPointerException > > [sshexec] Mon Jan 23 13:44:14 GMT 2012, > > org.apache.hadoop.hbase.client.ScannerCallable@219ba640, > > java.lang.NullPointerException > > [sshexec] Mon Jan 23 13:44:15 GMT 2012, > > org.apache.hadoop.hbase.client.ScannerCallable@219ba640, > > java.lang.NullPointerException > > [sshexec] Mon Jan 23 13:44:17 GMT 2012, > > org.apache.hadoop.hbase.client.ScannerCallable@219ba640, > > java.lang.NullPointerException > > [sshexec] Mon Jan 23 13:44:19 GMT 2012, > > org.apache.hadoop.hbase.client.ScannerCallable@219ba640, > > java.lang.NullPointerException > > [sshexec] Mon Jan 23 13:44:23 GMT 2012, > > org.apache.hadoop.hbase.client.ScannerCallable@219ba640, > > java.lang.NullPointerException > > [sshexec] Mon Jan 23 13:44:27 GMT 2012, > > org.apache.hadoop.hbase.client.ScannerCallable@219ba640, > > java.lang.NullPointerException > > [sshexec] Mon Jan 23 13:44:35 GMT 2012, > > org.apache.hadoop.hbase.client.ScannerCallable@219ba640, > > java.lang.NullPointerException > > [sshexec] Mon Jan 23 13:44:51 GMT 2012, > > org.apache.hadoop.hbase.client.ScannerCallable@219ba640, > > java.lang.NullPointerException > > [sshexec] > > [sshexec] Result = -1 > > [sshexec] at > > > > > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation. > > getRegionServerWithRetries(HConnectionManager.java:1345) > > [sshexec] at > > > > > org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java: > > 1203) > > [sshexec] at > > > > org.apache.hadoop.hbase.client.HTable$ClientScanner.initialize(HTable. > > java:1 > > 126) > > [sshexec] at > > org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:627) > > [sshexec] at > > > > org.apache.hadoop.hbase.client.coprocessor.AggregationClient.median(Ag > > gregat > > ionClient.java:469) > > [sshexec] at > > > > uk.org.cse.aggregation.EDRPAggregator.testSumWithValidRange(EDRPAggreg > > ator.j > > ava:55) > > [sshexec] at > > uk.org.cse.aggregation.EDRPAggregator.main(EDRPAggregator.java:85) > > [sshexec] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > > Method) > > [sshexec] at > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j > > ava:39 > > ) > > [sshexec] at > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess > > orImpl > > .java:25) > > [sshexec] at java.lang.reflect.Method.invoke(Method.java:597) > > [sshexec] at org.apache.hadoop.util.RunJar.main(RunJar.java:156) > > > > Something wrong with Scan setup? > > > > Cheers, > > Royston > > > > > > > > -----Original Message----- > > From: [email protected] [mailto:[email protected]] > > Sent: 21 January 2012 17:14 > > To: [email protected] > > Subject: Re: Hbase out of memory error > > > > Benoit's patches are already in 0.92 > > > > Thanks > > > > > > On Jan 21, 2012, at 9:11 AM, Royston Sellman > > <[email protected]> wrote: > > > > > So should I try applying Benoit Sigoure's patch for HBASE-5204? Will > > > this > > patch be in the 0.92 branch soon? > > > > > > Cheers, > > > Royston > > > > > > > > > > > > On 21 Jan 2012, at 16:58, [email protected] wrote: > > > > > >> That is the correct branch. > > >> > > >> Thanks > > >> > > >> > > >> > > >> On Jan 21, 2012, at 8:50 AM, Royston Sellman > > <[email protected]> wrote: > > >> > > >>> Hi Ted, > > >>> > > >>> Yes, I am compiling with the same HBase jars. I wasn't aware of > > HBASE-5204, thanks, it sounds possible this is my problem. Can you > > think of anything else I should check? > > >>> > > >>> Just to make sure: I am checking out the code from > > svn.apache.org/repos/asf/hbase/branches/0.92 Is this the correct > branch? > > >>> > > >>> Thanks, > > >>> Royston > > >>> > > >>> > > >>> On 20 Jan 2012, at 18:45, Ted Yu wrote: > > >>> > > >>>> Royston: > > >>>> I guess you have seen HBASE-5204. In particular: > > >>>>>> when a 0.92 server fails to deserialize a 0.90-style RPC, it > > >>>>>> attempts to > > >>>> allocate a large buffer because it doesn't read fields of > > >>>> 0.90-style RPCs properly. > > >>>> > > >>>> Were your client code compiled with the same version of HBase as > > >>>> what was running on your cluster ? > > >>>> > > >>>> Thanks > > >>>> > > >>>> On Fri, Jan 20, 2012 at 9:20 AM, Royston Sellman < > > >>>> [email protected]> wrote: > > >>>> > > >>>>> Trying to run my code (a test of Aggregation Protocol and an MR > > >>>>> HBase table > > >>>>> loader) on latest build of 0.92.0 (r1232715) I get an 'old server' > > >>>>> warning (I've seen this before and it's always been non-fatal) > > >>>>> then an out of memory exception then job hangs: > > >>>>> > > >>>>> > > >>>>> > > >>>>> [sshexec] 12/01/20 16:56:48 WARN zookeeper.ClientCnxnSocket: > > >>>>> Connected to an old server; r-o mode will be unavailable > > >>>>> > > >>>>> [sshexec] 12/01/20 16:56:48 INFO zookeeper.ClientCnxn: Session > > >>>>> establishment complete on server namenode/10.0.0.235:2181, > > >>>>> sessionid = 0x34cda4e5d000e5, negotiated timeout = 40000 > > >>>>> > > >>>>> [sshexec] 12/01/20 16:56:49 WARN ipc.HBaseClient: Unexpected > > >>>>> exception receiving call responses > > >>>>> > > >>>>> [sshexec] java.lang.OutOfMemoryError: Java heap space > > >>>>> > > >>>>> [sshexec] at java.lang.reflect.Array.newArray(Native Method) > > >>>>> > > >>>>> [sshexec] at > java.lang.reflect.Array.newInstance(Array.java:52) > > >>>>> > > >>>>> [sshexec] at > > >>>>> > > >>>>> org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseO > > >>>>> bj > > >>>>> ectWritabl > > >>>>> e.java:542) > > >>>>> > > >>>>> [sshexec] at > > >>>>> > > >>>>> org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseO > > >>>>> bj > > >>>>> ectWritabl > > >>>>> e.java:289) > > >>>>> > > >>>>> [sshexec] at > > >>>>> > > >>>>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveRespon > > >>>>> se > > >>>>> (HBaseClie > > >>>>> nt.java:593) > > >>>>> > > >>>>> [sshexec] at > > >>>>> > > >>>>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClie > > >>>>> nt > > >>>>> .java:505) > > >>>>> > > >>>>> > > >>>>> > > >>>>> Hbase shell seems to work (I can list and scan my tables). > > >>>>> > > >>>>> > > >>>>> > > >>>>> If I svn roll back to 12 Jan 0.92 and rebuild my code works. > > >>>>> > > >>>>> > > >>>>> > > >>>>> Tried setting export HBASE_HEAPSIZE=1500 but got same error. > > >>>>> > > >>>>> > > >>>>> > > >>>>> Nothing significant in logs. > > >>>>> > > >>>>> > > >>>>> > > >>>>> [Note to Ted Yu: I need to fix this so I can carry on testing on > > >>>>> Aggregation Protocol] > > >>>>> > > >>>>> > > >>>>> > > >>>>> Best Regards, > > >>>>> > > >>>>> Royston > > >>>>> > > >>>>> > > >>> > > > > > > > >
