Royston / Tom: I would encourage you to explore other aggregations where AggregationProtocol is of help.
Feel free to discuss any limitation in current implementation, propose suggestions, etc. Thanks On Mon, Jan 23, 2012 at 9:03 AM, Ted Yu <[email protected]> wrote: > Thanks Tom for the investigation. > I will apply the null check in an addendum to HBASE-5139. > > Operation.toJSON() does bring in Jackson JSON processor > Please confirm that the jackson jars are in your CLASSPATH: > $ ls lib/*jackson* > lib/jackson-core-asl-1.5.5.jar lib/jackson-jaxrs-1.5.5.jar > lib/jackson-mapper-asl-1.5.5.jar lib/jackson-xc-1.5.5.jar > > Looks like you haven't encountered the need to print contents of Scan so > far. > If you feel the dependency on jackson is burdensome, please file a JIRA so > that we can relax this dependency. > > Cheers > > > On Mon, Jan 23, 2012 at 8:35 AM, Tom Wilcox <[email protected]> wrote: > >> Hi Ted, >> >> Following from what you have said, we have edited AggregateClient.java >> with the following modification to the median() method: >> >> ... >> >> // scan the region with median and find it >> Scan scan2 = new Scan(scan); >> // inherit stop row from method parameter >> if (startRow != null) >> scan2.setStartRow(startRow); >> HTable table = new HTable(conf, tableName); >> int cacheSize = scan2.getCaching(); >> if (!scan2.getCacheBlocks() || scan2.getCaching() < 2) { >> scan2.setCacheBlocks(true); >> cacheSize = 5; >> scan2.setCaching(cacheSize); >> } >> >> // log.debug("scan:\t" + scan); >> // log.debug("scan2:\t" + scan2); >> >> ResultScanner scanner = table.getScanner(scan2); >> >> ... >> >> The addition of "if (startRow != null) scan2.setStartRow(startRow);" has >> fixed our NPE/Memory issue and we now see a sensible result successfully >> returned. >> >> However, in the process of debugging this, I added some >> 'log.debug("scan:\t" + scan);' statements as you can see in the snippet >> above. These lines (when uncommented) were causing the following exception: >> >> [sshexec] java.lang.NoSuchMethodError: >> org.codehaus.jackson.map.ObjectMapper.writeValueAsString(Ljava/lang/Object;)Ljava/lang/String; >> [sshexec] >> [sshexec] at >> org.apache.hadoop.hbase.client.Operation.toJSON(Operation.java:67) >> [sshexec] >> [sshexec] at >> org.apache.hadoop.hbase.client.Operation.toString(Operation.java:93) >> [sshexec] >> [sshexec] at >> org.apache.hadoop.hbase.client.Operation.toString(Operation.java:107) >> [sshexec] >> [sshexec] at java.lang.String.valueOf(String.java:2826) >> [sshexec] at java.lang.StringBuilder.append(StringBuilder.java:115) >> [sshexec] at >> org.apache.hadoop.hbase.client.coprocessor.AggregationClient.median(AggregationClient.java:473) >> [sshexec] at >> uk.org.cse.aggregation.EDRPAggregator.testSumWithValidRange(EDRPAggregator.java:55) >> [sshexec] at >> uk.org.cse.aggregation.EDRPAggregator.main(EDRPAggregator.java:85) >> [sshexec] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> Method) >> [sshexec] at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> [sshexec] at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> [sshexec] at java.lang.reflect.Method.invoke(Method.java:597) >> [sshexec] at org.apache.hadoop.util.RunJar.main(RunJar.java:156) >> >> Do you have any idea why we got this error above? It seems the scanner >> toString() method requires the Jackson JSON processor, which is failing for >> some reason... >> >> Also, please can you tell me where log.debug() output can be found? (e.g. >> $HBASE_HOME/logs/xxx-region-....log?) >> >> Please find a patch attached with the fixed median method for HBase 0.92 >> branch. Note that I have removed the log.debug() statements. >> >> Thanks, >> Tom >> >> -----Original Message----- >> From: Ted Yu [mailto:[email protected]] >> Sent: 23 January 2012 15:24 >> To: [email protected] >> Subject: median through AggregationProtocol >> >> Royston: >> The exception came from this line: >> ResultScanner scanner = table.getScanner(scan2); Can you help me >> review the logic starting with: >> // scan the region with median and find it >> Scan scan2 = new Scan(scan); >> You can log the String form of scan and scan2 before the >> table.getScanner() call. >> >> I think the NPE below reveals that startRow is null (median is in first >> region). >> If that is the case, the following should help: >> if (startRow != null) scan2.setStartRow(startRow); >> >> Thanks >> >> On Mon, Jan 23, 2012 at 5:50 AM, Royston Sellman < >> [email protected]> wrote: >> >> > Hi Ted, >> > >> > Finally rebuilt branch/0.92 and applied your patch and rebuilt my code. >> > Using AggregationClient.sum() on my test table I get the correct result. >> > Just swapping to AggregationClient.median() I get the following error: >> > >> > [sshexec] org.apache.hadoop.hbase.client.RetriesExhaustedException: >> > Failed after attempts=10, exceptions: >> > [sshexec] Mon Jan 23 13:44:12 GMT 2012, >> > org.apache.hadoop.hbase.client.ScannerCallable@219ba640, >> > java.lang.NullPointerException >> > [sshexec] Mon Jan 23 13:44:13 GMT 2012, >> > org.apache.hadoop.hbase.client.ScannerCallable@219ba640, >> > java.lang.NullPointerException >> > [sshexec] Mon Jan 23 13:44:14 GMT 2012, >> > org.apache.hadoop.hbase.client.ScannerCallable@219ba640, >> > java.lang.NullPointerException >> > [sshexec] Mon Jan 23 13:44:15 GMT 2012, >> > org.apache.hadoop.hbase.client.ScannerCallable@219ba640, >> > java.lang.NullPointerException >> > [sshexec] Mon Jan 23 13:44:17 GMT 2012, >> > org.apache.hadoop.hbase.client.ScannerCallable@219ba640, >> > java.lang.NullPointerException >> > [sshexec] Mon Jan 23 13:44:19 GMT 2012, >> > org.apache.hadoop.hbase.client.ScannerCallable@219ba640, >> > java.lang.NullPointerException >> > [sshexec] Mon Jan 23 13:44:23 GMT 2012, >> > org.apache.hadoop.hbase.client.ScannerCallable@219ba640, >> > java.lang.NullPointerException >> > [sshexec] Mon Jan 23 13:44:27 GMT 2012, >> > org.apache.hadoop.hbase.client.ScannerCallable@219ba640, >> > java.lang.NullPointerException >> > [sshexec] Mon Jan 23 13:44:35 GMT 2012, >> > org.apache.hadoop.hbase.client.ScannerCallable@219ba640, >> > java.lang.NullPointerException >> > [sshexec] Mon Jan 23 13:44:51 GMT 2012, >> > org.apache.hadoop.hbase.client.ScannerCallable@219ba640, >> > java.lang.NullPointerException >> > [sshexec] >> > [sshexec] Result = -1 >> > [sshexec] at >> > >> > >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation. >> > getRegionServerWithRetries(HConnectionManager.java:1345) >> > [sshexec] at >> > >> > >> org.apache.hadoop.hbase.client.HTable$ClientScanner.nextScanner(HTable.java: >> > 1203) >> > [sshexec] at >> > >> > org.apache.hadoop.hbase.client.HTable$ClientScanner.initialize(HTable. >> > java:1 >> > 126) >> > [sshexec] at >> > org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:627) >> > [sshexec] at >> > >> > org.apache.hadoop.hbase.client.coprocessor.AggregationClient.median(Ag >> > gregat >> > ionClient.java:469) >> > [sshexec] at >> > >> > uk.org.cse.aggregation.EDRPAggregator.testSumWithValidRange(EDRPAggreg >> > ator.j >> > ava:55) >> > [sshexec] at >> > uk.org.cse.aggregation.EDRPAggregator.main(EDRPAggregator.java:85) >> > [sshexec] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> > Method) >> > [sshexec] at >> > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j >> > ava:39 >> > ) >> > [sshexec] at >> > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess >> > orImpl >> > .java:25) >> > [sshexec] at java.lang.reflect.Method.invoke(Method.java:597) >> > [sshexec] at org.apache.hadoop.util.RunJar.main(RunJar.java:156) >> > >> > Something wrong with Scan setup? >> > >> > Cheers, >> > Royston >> > >> > >> > >> > -----Original Message----- >> > From: [email protected] [mailto:[email protected]] >> > Sent: 21 January 2012 17:14 >> > To: [email protected] >> > Subject: Re: Hbase out of memory error >> > >> > Benoit's patches are already in 0.92 >> > >> > Thanks >> > >> > >> > On Jan 21, 2012, at 9:11 AM, Royston Sellman >> > <[email protected]> wrote: >> > >> > > So should I try applying Benoit Sigoure's patch for HBASE-5204? Will >> > > this >> > patch be in the 0.92 branch soon? >> > > >> > > Cheers, >> > > Royston >> > > >> > > >> > > >> > > On 21 Jan 2012, at 16:58, [email protected] wrote: >> > > >> > >> That is the correct branch. >> > >> >> > >> Thanks >> > >> >> > >> >> > >> >> > >> On Jan 21, 2012, at 8:50 AM, Royston Sellman >> > <[email protected]> wrote: >> > >> >> > >>> Hi Ted, >> > >>> >> > >>> Yes, I am compiling with the same HBase jars. I wasn't aware of >> > HBASE-5204, thanks, it sounds possible this is my problem. Can you >> > think of anything else I should check? >> > >>> >> > >>> Just to make sure: I am checking out the code from >> > svn.apache.org/repos/asf/hbase/branches/0.92 Is this the correct >> branch? >> > >>> >> > >>> Thanks, >> > >>> Royston >> > >>> >> > >>> >> > >>> On 20 Jan 2012, at 18:45, Ted Yu wrote: >> > >>> >> > >>>> Royston: >> > >>>> I guess you have seen HBASE-5204. In particular: >> > >>>>>> when a 0.92 server fails to deserialize a 0.90-style RPC, it >> > >>>>>> attempts to >> > >>>> allocate a large buffer because it doesn't read fields of >> > >>>> 0.90-style RPCs properly. >> > >>>> >> > >>>> Were your client code compiled with the same version of HBase as >> > >>>> what was running on your cluster ? >> > >>>> >> > >>>> Thanks >> > >>>> >> > >>>> On Fri, Jan 20, 2012 at 9:20 AM, Royston Sellman < >> > >>>> [email protected]> wrote: >> > >>>> >> > >>>>> Trying to run my code (a test of Aggregation Protocol and an MR >> > >>>>> HBase table >> > >>>>> loader) on latest build of 0.92.0 (r1232715) I get an 'old server' >> > >>>>> warning (I've seen this before and it's always been non-fatal) >> > >>>>> then an out of memory exception then job hangs: >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> [sshexec] 12/01/20 16:56:48 WARN zookeeper.ClientCnxnSocket: >> > >>>>> Connected to an old server; r-o mode will be unavailable >> > >>>>> >> > >>>>> [sshexec] 12/01/20 16:56:48 INFO zookeeper.ClientCnxn: Session >> > >>>>> establishment complete on server namenode/10.0.0.235:2181, >> > >>>>> sessionid = 0x34cda4e5d000e5, negotiated timeout = 40000 >> > >>>>> >> > >>>>> [sshexec] 12/01/20 16:56:49 WARN ipc.HBaseClient: Unexpected >> > >>>>> exception receiving call responses >> > >>>>> >> > >>>>> [sshexec] java.lang.OutOfMemoryError: Java heap space >> > >>>>> >> > >>>>> [sshexec] at java.lang.reflect.Array.newArray(Native Method) >> > >>>>> >> > >>>>> [sshexec] at >> java.lang.reflect.Array.newInstance(Array.java:52) >> > >>>>> >> > >>>>> [sshexec] at >> > >>>>> >> > >>>>> org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseO >> > >>>>> bj >> > >>>>> ectWritabl >> > >>>>> e.java:542) >> > >>>>> >> > >>>>> [sshexec] at >> > >>>>> >> > >>>>> org.apache.hadoop.hbase.io.HbaseObjectWritable.readFields(HbaseO >> > >>>>> bj >> > >>>>> ectWritabl >> > >>>>> e.java:289) >> > >>>>> >> > >>>>> [sshexec] at >> > >>>>> >> > >>>>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveRespon >> > >>>>> se >> > >>>>> (HBaseClie >> > >>>>> nt.java:593) >> > >>>>> >> > >>>>> [sshexec] at >> > >>>>> >> > >>>>> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClie >> > >>>>> nt >> > >>>>> .java:505) >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> Hbase shell seems to work (I can list and scan my tables). >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> If I svn roll back to 12 Jan 0.92 and rebuild my code works. >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> Tried setting export HBASE_HEAPSIZE=1500 but got same error. >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> Nothing significant in logs. >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> [Note to Ted Yu: I need to fix this so I can carry on testing on >> > >>>>> Aggregation Protocol] >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> Best Regards, >> > >>>>> >> > >>>>> Royston >> > >>>>> >> > >>>>> >> > >>> >> > > >> > >> > >> > >
