Royston: For the median aggregate, is the following what you're looking for ? Find the median among the values of all the keyvalue for cf:qualifier column.
There is a well known distributed method of computing median that involves multiple roundtrips (to the region servers). Just want to confirm the use case. Thanks On Wed, Jan 4, 2012 at 10:57 AM, Royston Sellman < [email protected]> wrote: > Great ideas. Thanks. > > w.r.t. 5123: I'll think about it for a day or two then make some comments. > > 5122 is very desirable. > > Best Regards, > Royston > > On 4 Jan 2012, at 15:01, Ted Yu wrote: > > > I can see room for improvement w.r.t. ColumnInterpreters > > I logged two JIRAs: > > https://issues.apache.org/jira/browse/HBASE-5122 is for loading > > ColumnInterpreters dynamically > > > > https://issues.apache.org/jira/browse/HBASE-5123 is for adding more > > aggregation functions. > > > > Royston: > > Feel free to elaborate on 5213 and explain what Mult aggregate should do. > > > > Cheers > > > > On Wed, Jan 4, 2012 at 3:43 AM, Royston Sellman < > > [email protected]> wrote: > > > >> Ted, Himanshu and Gary, > >> > >> It works now! I re-created my HBase table to contain Bytes.toBytes(Long) > >> values and that fixed it. > >> > >> For the time being we can convert everything to Longs and work with > that, > >> but we will probably write our own ColumnInterpreters soon for our data > >> types, so thanks for the pointer to HBASE-4946. There are also > Functions we > >> need (e.g. Median, Weighted Median, Mult) which might best be placed in > the > >> Aggregations Protocol. We'll be sure to discuss this with you when we > start. > >> > >> Meanwhile, thanks again for all your help! > >> > >> Royston > >> > >> > >> On 3 Jan 2012, at 18:58, Ted Yu wrote: > >> > >>> I like long messages :-) because they provide more clues. > >>> > >>> For part 1, you don't have to call Bytes.toxxx as long as the > interpreter > >>> uses method consistent with the way you write values into HBase tables. > >>> > >>> For part 2, HBASE-4946 is related. > >>> Basically you need to place the jar containing your coprocessor and > >>> interpreter code on hdfs so that you can load it into your HBase table. > >>> Look at this for details: > >>> https://issues.apache.org/jira/browse/HBASE-4554 > >>> > >>> Cheers > >>> > >>> On Tue, Jan 3, 2012 at 10:42 AM, Royston Sellman < > >>> [email protected]> wrote: > >>> > >>>> Hi Ted, > >>>> > >>>> PART 1 > >>>> ===== > >>>> Thanks for the hint. I think maybe you have given me some inspiration! > >>>> > >>>> It looks like getValue will return null if the table value is not the > >>>> length > >>>> of a long. When we created our table (batch loading CSVs using the > >>>> SampleUploader example) we simply have this as our put(): > >>>> put.add(family, Bytes.toBytes("advanceKWh"), advanceKWh); > >>>> [note we do no Bytes.toxxx casts to the advanceKWh value. The values > >> look > >>>> OK > >>>> from HBase shell though :-)] > >>>> > >>>> but I looked at TestAggregateProtocol.java again and I see there puts > >> like: > >>>> p2.add(TEST_FAMILY, Bytes.add(TEST_MULTI_CQ, Bytes.toBytes(l)), > >>>> Bytes.toBytes(l * 10)); > >>>> > >>>> So my hypothesis is that we need to do something like: > >>>> Long l = new Long(1); > >>>> put.add(family, Bytes.toBytes("advanceKWh"), Bytes.toBytes(l * > >>>> advanceKWh)); > >>>> when we create the table. > >>>> > >>>> Do you think my hypothesis is correct? Did we build our table > >> incorrectly > >>>> for reading longs from it? > >>>> > >>>> PART 2 > >>>> ===== > >>>> Anyway we will obviously need to make our own interpreters. but we > >> failed > >>>> at > >>>> this task so far: > >>>> In order to implement our own ColumnInterpretter, we first attempted > >> simply > >>>> extending the LongColumnInterpreter and passing that as a parameter to > >>>> aClient.sum(). > >>>> import > org.apache.hadoop.hbase.client.coprocessor.LongColumnInterpreter; > >>>> > >>>> public class LCI extends LongColumnInterpreter { > >>>> public Long add(Long a, Long b) { > >>>> System.out.print("LCI.add("+a+","+b+")"); > >>>> Long c = super.add(a,b); > >>>> System.out.println(" -> "+c); > >>>> return c; > >>>> } > >>>> }; > >>>> .... > >>>> final ColumnInterpreter<Long, Long> ci = new LCI(); > >>>> long sum=-1; > >>>> try { > >>>> sum = aClient.sum(EDRP_TABLE, ci, scan); > >>>> ... > >>>> However, we received class not found errors in our regionserver log > >> when we > >>>> ran this code > >>>> Clearly we are missing something > >>>> We've started looking at modifying the HBase source and rebuilding > that > >>>> But I think this is not the way we should be working > >>>> It seems that we should be implementing the ColumnInterpreter > interface > >> and > >>>> passing an instance of that implementation class too as a parameter. > Is > >>>> this correct? Could you provide us with an example? Any ideas why we > >>>> might > >>>> be getting class not found exception from the regionserver? > >>>> > >>>> Sorry for the long message! > >>>> > >>>> Royston (and Tom) > >>>> > >>>> > >>>> -----Original Message----- > >>>> From: Ted Yu [mailto:[email protected]] > >>>> Sent: 03 January 2012 18:00 > >>>> To: [email protected] > >>>> Subject: Re: AggregateProtocol Help > >>>> > >>>> My previous email might not be hitting the root cause. > >>>> I think the following method in LCI may be giving you the null: > >>>> > >>>> public Long getValue(byte[] colFamily, byte[] colQualifier, KeyValue > kv) > >>>> throws IOException { > >>>> if (kv == null || kv.getValueLength() != Bytes.SIZEOF_LONG) > >>>> return null; > >>>> return Bytes.toLong(kv.getBuffer(), kv.getValueOffset()); > >>>> > >>>> Look at the if statement above carefully. > >>>> If it doesn't match how you store values in HBase, feel free to > subclass > >>>> LongColumnInterpreter and provide the correct interpretation. > >>>> > >>>> BTW you don't need to restart cluster just because you need to use > your > >> own > >>>> interpreter :-) > >>>> > >>>> On Tue, Jan 3, 2012 at 9:48 AM, Royston Sellman < > >>>> [email protected]> wrote: > >>>> > >>>>> Hi Ted, > >>>>> > >>>>> Here is the output. As you can see aClient is not nul: > >>>>> > >>>>> AggregationClient aClient = new AggregationClient(conf); > >>>>> System.err.println("aClient: "+aClient); > >>>>> > >>>>> <<< aClient: > >>>>> org.apache.hadoop.hbase.client.coprocessor.AggregationClient@28787c16 > >>>>> > >>>>> It will take us a little while to add log code to LCI... we have to > >>>>> edit the source, rebuild 0.92, redistribute round our cluster, > restart > >>>>> ;) We'll get back to you when this is done. > >>>>> > >>>>> Royston > >>>>> > >>>>> -----Original Message----- > >>>>> From: Ted Yu [mailto:[email protected]] > >>>>> Sent: 03 January 2012 17:10 > >>>>> To: [email protected] > >>>>> Subject: Re: AggregateProtocol Help > >>>>> > >>>>> Royston: > >>>>> Thanks for your effort trying to hunt down the problem. > >>>>> > >>>>> Can you add a log after this line to see if aClient is null ? > >>>>> AggregationClient aClient = new AggregationClient(conf); > >>>>> > >>>>> I was looking at LongColumnInterpreter.add() which is called by > >>>>> aClient.sum() > >>>>> Can you add a few log statements in LongColumnInterpreter.add() to > see > >>>>> what parameters are passed to it ? > >>>>> > >>>>> Cheers > >>>>> > >>>>> On Tue, Jan 3, 2012 at 8:32 AM, Royston Sellman < > >>>>> [email protected]> wrote: > >>>>> > >>>>>> Hi Ted, Himanshu, Gary, > >>>>>> > >>>>>> Thanks again for your attention. I experimented with a shorter table > >>>>>> and it looks like the timeout error was spurious... > >>>>>> > >>>>>> With the shorter table I now get an NPE when I call > >>>>>> AggregationClient.sum(). > >>>>>> Here's the code snippet: > >>>>>> > >>>>>> // Test the table > >>>>>> HTable table = new HTable(EDRP_TABLE); > >>>>>> Get get = new Get(Bytes.toBytes("row-aa")); > >>>>>> get.addColumn(Bytes.toBytes("EDRP"), > >>>>>> Bytes.toBytes("advanceKWh")); > >>>>>> Result result = table.get(get); > >>>>>> byte [] val = result.getValue(Bytes.toBytes("EDRP"), > >>>>>> Bytes.toBytes("advanceKWh")); > >>>>>> System.out.println("Row aa = " + > >>>>>> Bytes.toString(val)); > >>>>>> > >>>>>> AggregationClient aClient = new > AggregationClient(conf); > >>>>>> Scan scan = new Scan(); > >>>>>> scan.addColumn(EDRP_FAMILY, EDRP_QUALIFIER); > >>>>>> scan.setStartRow(Bytes.toBytes("row-ab")); > >>>>>> scan.setStopRow(Bytes.toBytes("row-az")); > >>>>>> System.out.println(Bytes.toString(EDRP_FAMILY) + ":" > >>>>>> + Bytes.toString(EDRP_QUALIFIER)); > >>>>>> final ColumnInterpreter<Long, Long> ci = new > >>>>>> LongColumnInterpreter(); > >>>>>> long sum=-1; > >>>>>> try { > >>>>>> sum = aClient.sum(EDRP_TABLE, ci, scan); > >>>>>> } catch (Throwable e) { > >>>>>> // TODO Auto-generated catch block > >>>>>> e.printStackTrace(); > >>>>>> } > >>>>>> System.out.println(sum); > >>>>>> > >>>>>> The first part is just to check that my table is OK. It prints the > >>>>>> correct value for row aa. Then I check CF:CQ is correct. Then I get > >>>>>> the -1 that sum was defined as. Then the NPE. Here is the run > output: > >>>>>> > >>>>>> Row aa = 3.0 > >>>>>> EDRP:advanceKWh > >>>>>> -1 > >>>>>> java.lang.NullPointerException > >>>>>> at EDRPAggregator.testSumWithValidRange(EDRPAggregator.java:66) > >>>>>> at EDRPAggregator.main(EDRPAggregator.java:96) > >>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at > >>>>>> > >>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl > >>>>>> .j > >>>>>> ava:39 > >>>>>> ) > >>>>>> at > >>>>>> > >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcce > >>>>>> ss > >>>>>> orImpl > >>>>>> .java:25) > >>>>>> at java.lang.reflect.Method.invoke(Method.java:597) > >>>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156) > >>>>>> > >>>>>> Line 66 is the line: > >>>>>> sum = aClient.sum(EDRP_TABLE, ci, scan); > >>>>>> > >>>>>> But I'm not sure that is really the line generating the NPE because > >>>>>> I get the -1 output (same result when line 66 is not wrapped in > >>>>>> try-catch > >>>>> BTW). > >>>>>> Anyway, -1 is clearly not the correct value for the sum. > >>>>>> > >>>>>> [Note: I'm setting a start and end row on the scan because I saw a > >>>>>> comment in AggregationClient.java saying they must be set. However, > >>>>>> I think this comment is obsolete as the test code in > >>>>>> TestAggregateProtocol.java sometimes does not include start & end > >>>>>> rows. The run is exactly the same without the start & stop rows ] > >>>>>> > >>>>>> We are struggling with Aggregations not least because > >>>>>> TestAggregateProtocol.java is the only source of example code we > >>>>>> could find and it is not ideal because it is designed to work with > >>>>>> MiniCluster rather than a real cluster like ours. Is there any other > >>>>>> example code in existence? > >>>>>> I think that would really help us. > >>>>>> > >>>>>> Meanwhile, thanks for your patience with our questions. We are > >>>>>> really impressed with the AggregationProtocol > >>>>>> architecture/functionality and very keen to use it. > >>>>>> > >>>>>> Royston (and Tom) > >>>>>> > >>>>>> -----Original Message----- > >>>>>> From: Gary Helmling [mailto:[email protected]] > >>>>>> Sent: 02 January 2012 06:23 > >>>>>> To: [email protected] > >>>>>> Subject: Re: AggregateProtocol Help > >>>>>> > >>>>>> Hi Royston, > >>>>>> > >>>>>> Try increasing the value set for hbase.rpc.timeout (by default 60 > >>>>> seconds). > >>>>>> Add something like this to hbase-site.xml: > >>>>>> > >>>>>> <property> > >>>>>> <name>hbase.rpc.timeout</name> > >>>>>> <value>180000</value> > >>>>>> </property> > >>>>>> > >>>>>> This would increase the timeout value to 3 minutes, for example. > >>>>>> > >>>>>> But as Ted and Himanshu mention, 557k rows is not a large table and > >>>>>> should not be taking very long to process, unless these happen to be > >>>>>> very, very wide rows. It sounds like there could be something else > >>>>>> going on. Does debug level logging show anything else in the region > >>>>> server log? > >>>>>> > >>>>>> --gh > >>>>>> > >>>>>> On Sun, Jan 1, 2012 at 5:53 PM, Ted Yu <[email protected]> wrote: > >>>>>>> Thanks for the reminder Himanshu. > >>>>>>> > >>>>>>> Royston: > >>>>>>> From this blog you can get some history on this subject: > >>>>>>> http://zhihongyu.blogspot.com/2011/03/genericizing-endpointcoproce > >>>>>>> ss > >>>>>>> or > >>>>>>> .html > >>>>>>> > >>>>>>> > >>>>>>> On Sun, Jan 1, 2012 at 5:18 PM, Himanshu Vashishtha > >>>>>>> <[email protected] > >>>>>>>> wrote: > >>>>>>> > >>>>>>>> Hello Royston, > >>>>>>>> > >>>>>>>> Sorry to hear that you are getting trouble while using > >>>>>>>> Aggregation functionalities. > >>>>>>>> > >>>>>>>> 557k rows seems to be a small table and a SocketTimeout does not > >>>>>>>> seem to be an ok response. > >>>>>>>> It will be good to know the region distribution as such. (how > >>>>>>>> many > >>>>>> regions? > >>>>>>>> Is it a full table scan?) > >>>>>>>> > >>>>>>>> You are using the sum function; how are you using the > >>>>> ColumnInterpreter. > >>>>>>>> Can you enable the log level to debug to see why the RS is taking > >>>>>>>> that long to respond (more than 113 sec). > >>>>>>>> The 0 return value is the default result. > >>>>>>>> > >>>>>>>> Thanks for trying this out. > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> Himanshu > >>>>>>>> > >>>>>>>> On Sun, Jan 1, 2012 at 12:26 PM, Royston Sellman < > >>>>>>>> [email protected]> wrote: > >>>>>>>> > >>>>>>>>> Hi Ted, > >>>>>>>>> > >>>>>>>>> I think 0 is the only value we ever see (I'll check tomorrow: > >>>>>>>>> the server is down right now). Our table has 557,000 rows. I'll > >>>>>>>>> try a much shorter table tomorrow. > >>>>>>>>> > >>>>>>>>> Yes, we have RS running on the NN, but it's a test cluster and > >>>>>>>>> we are > >>>>>>>> used > >>>>>>>>> to it :) > >>>>>>>>> > >>>>>>>>> Do you think using AggregationProtocol is the best strategy for > >>>>>>>>> the case where we want to use basic SQL-style functions like > >>>>>>>>> SUM, AVG, STD, MIN, MAX? Do you think there is a better strategy? > >>>>>>>>> > >>>>>>>>> Many thanks, > >>>>>>>>> Royston > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On 1 Jan 2012, at 17:58, Ted Yu wrote: > >>>>>>>>> > >>>>>>>>>> Royston: > >>>>>>>>>> Happy New Year to you too. > >>>>>>>>>> > >>>>>>>>>>>> java.net.SocketTimeoutException: Call to > >>>>>>>>>>>> namenode/10.0.0.235:60020 > >>>>>>>> failed > >>>>>>>>> on > >>>>>>>>>> > >>>>>>>>>> It seems the namenode above actually refers to a region server. > >>>>>>>>>> This > >>>>>>>> is a > >>>>>>>>>> little bit confusing :-) > >>>>>>>>>> > >>>>>>>>>> The sum value below is 0. > >>>>>>>>>> Have you ever seen a value greater than 0 ? > >>>>>>>>>> > >>>>>>>>>> How many rows are there in this CF:CQ ? > >>>>>>>>>> The timeout was reported earlier by other people where > >>>>>>>>>> there're many > >>>>>>>> rows > >>>>>>>>>> in the table. > >>>>>>>>>> > >>>>>>>>>> There is a JIRA to provide streaming support for coprocessor > >>>>>>>>>> but the development there has stalled. > >>>>>>>>>> > >>>>>>>>>> Cheers > >>>>>>>>>> > >>>>>>>>>> On Sun, Jan 1, 2012 at 9:35 AM, Royston Sellman < > >>>>>>>>>> [email protected]> wrote: > >>>>>>>>>> > >>>>>>>>>>> Hi Gary and Ted, > >>>>>>>>>>> > >>>>>>>>>>> Royston (Tom's colleague) here. Back onto this after the > >>>>>>>>>>> Christmas/New > >>>>>>>>> Year > >>>>>>>>>>> break. > >>>>>>>>>>> > >>>>>>>>>>> Many thanks for your help so far. We enabled our database > >>>>>>>>>>> via your hbase-site.xml mod and were able to move on. to > >>>>>>>>>>> other errors. But I > >>>>>>>>> think > >>>>>>>>>>> we > >>>>>>>>>>> are now actually getting an aggregation partially calculated > >>>>>>>>>>> on our > >>>>>>>>> table > >>>>>>>>>>> (this feels like progress). The details: > >>>>>>>>>>> > >>>>>>>>>>> On running our client we now get this exception: > >>>>>>>>>>> 11/12/31 17:51:09 WARN > >>>>>>>>>>> client.HConnectionManager$HConnectionImplementation: Error > >>>>>>>>>>> executing > >>>>>>>> for > >>>>>>>>>>> row > >>>>>>>>>>> > >>>>>>>>>>> java.util.concurrent.ExecutionException: > >>>>>>>>>>> org.apache.hadoop.hbase.client.RetriesExhaustedException: > >>>>>>>>>>> Failed after attempts=10, exceptions: > >>>>>>>>>>> Sat Dec 31 17:41:30 GMT 2011, > >>>>>>>>>>> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@1fc4f0f8, > >>>>>>>>>>> java.net.SocketTimeoutException: Call to > >>>>>>>>>>> namenode/10.0.0.235:60020 > >>>>>>>> failed > >>>>>>>>>>> on > >>>>>>>>>>> socket timeout exception: java.net.SocketTimeoutException: > >>>>>>>>>>> 60000 > >>>>>>>> millis > >>>>>>>>>>> timeout while waiting for channel to be ready for read. ch : > >>>>>>>>>>> java.nio.channels.SocketChannel[connected > >>>>>>>>>>> local=/10.0.0.235:59999 remote=namenode/10.0.0.235:60020] > >>>>>>>>>>> (8 more of these, making for 10 tries) Sat Dec 31 17:51:09 > >>>>>>>>>>> GMT 2011, > >>>>>>>>>>> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@1fc4f0f8, > >>>>>>>>>>> java.net.SocketTimeoutException: Call to > >>>>>>>>>>> namenode/10.0.0.235:60020 > >>>>>>>> failed > >>>>>>>>>>> on > >>>>>>>>>>> socket timeout exception: java.net.SocketTimeoutException: > >>>>>>>>>>> 60000 > >>>>>>>> millis > >>>>>>>>>>> timeout while waiting for channel to be ready for read. ch : > >>>>>>>>>>> java.nio.channels.SocketChannel[connected > >>>>>>>>>>> local=/10.0.0.235:59364 remote=namenode/10.0.0.235:60020] > >>>>>>>>>>> > >>>>>>>>>>> at > >>>>>>>>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java: > >>>>>>>>>>> 22 > >>>>>>>>>>> 2) > >>>>>>>>>>> at > >>>> java.util.concurrent.FutureTask.get(FutureTask.java:83) > >>>>>>>>>>> at > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation. > >>>>>>>>>>> processExecs(HConnectionManager.java:1465) > >>>>>>>>>>> at > >>>>>>>>>>> > >>>>>>>> org.apache.hadoop.hbase.client.HTable.coprocessorExec(HTable.java > >>>>>>>> :1 > >>>>>>>> 55 > >>>>>>>> 5) > >>>>>>>>>>> at > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>> org.apache.hadoop.hbase.client.coprocessor.AggregationClient.sum( > >>>>>>>> Ag > >>>>>>>> gr > >>>>>>>> egation > >>>>>>>>>>> Client.java:229) > >>>>>>>>>>> at > >>>>>>>>>>> EDRPAggregator.testSumWithValidRange(EDRPAggregator.java:51) > >>>>>>>>>>> at EDRPAggregator.main(EDRPAggregator.java:77) > >>>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > >>>>>>>>>>> Method) > >>>>>>>>>>> at > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. > >>>>>>>> java:39 > >>>>>>>>>>> ) > >>>>>>>>>>> at > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodA > >>>>>>>> cc > >>>>>>>> es > >>>>>>>> sorImpl > >>>>>>>>>>> .java:25) > >>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597) > >>>>>>>>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156) > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Looking at the log (.regionserver-namenode.log) I see this > >>>>>>>>>>> debug > >>>>>>>>> message: > >>>>>>>>>>> > >>>>>>>>>>> 2011-12-31 17:42:23,472 DEBUG > >>>>>>>>>>> org.apache.hadoop.hbase.coprocessor.AggregateImplementation: > >>>>>>>>>>> Sum from > >>>>>>>>> this > >>>>>>>>>>> region is > >>>>>>>> EDRPTestTbl,,1324485124322.7b9ee0d113db9b24ea9fdde90702d006.: > >>>>>>>>> 0 > >>>>>>>>>>> > >>>>>>>>>>> Where the sum value looks reasonable which makes me think > >>>>>>>>>>> the sum of a CF:CQ worked. But I never see this value on > >>>> stdout. > >>>>>>>>>>> Then I see this warning: > >>>>>>>>>>> > >>>>>>>>>>> 2011-12-31 17:42:23,476 WARN org.apache.hadoop.ipc.HBaseServer: > >>>>>>>>>>> (responseTooSlow): > >>>>>>>>>>> {"processingtimems":113146,"call":"execCoprocess$ > >>>>>>>>>>> 2011-12-31 17:42:23,511 WARN org.apache.hadoop.ipc.HBaseServer: > >>>>>>>>>>> IPC > >>>>>>>>> Server > >>>>>>>>>>> Responder, call execCoprocessor([B@4b22fad6, getSum(org.$ > >>>>>>>>>>> 2011-12-31 17:42:23,515 WARN org.apache.hadoop.ipc.HBaseServer: > >>>>>>>>>>> IPC > >>>>>>>>> Server > >>>>>>>>>>> handler 1 on 60020 caught: java.nio.channels.ClosedChann$ > >>>>>>>>>>> at > >>>>>>>>>>> > >>>>>>>> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.ja > >>>>>>>> va > >>>>>>>> :1 > >>>>>>>> 33) > >>>>>>>>>>> at > >>>>>>>> sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324) > >>>>>>>>>>> at > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>> org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer. > >>>>>>>> ja > >>>>>>>> va > >>>>>>>> :1651) > >>>>>>>>>>> at > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse > >>>>>>>> (H > >>>>>>>> Ba > >>>>>>>> seServe > >>>>>>>>>>> r.java:924) > >>>>>>>>>>> at > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBase > >>>>>>>> Se > >>>>>>>> rv > >>>>>>>> er.java > >>>>>>>>>>> :1003) > >>>>>>>>>>> at > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>> org.apache.hadoop.hbase.ipc.HBaseServer$Call.sendResponseIfReady( > >>>>>>>> HB > >>>>>>>> as > >>>>>>>> eServer > >>>>>>>>>>> .java:409) > >>>>>>>>>>> at > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java: > >>>>>>>> 1345) > >>>>>>>>>>> > >>>>>>>>>>> Have we missed out some step in the HBase/RegionServerconfig? > >>>>>>>>>>> Or is > >>>>>>>> our > >>>>>>>>>>> client code still deficient? > >>>>>>>>>>> > >>>>>>>>>>> Can you offer any suggestions? Is there any example code for > >>>>>>>>>>> the new Aggregations stuff. > >>>>>>>>>>> > >>>>>>>>>>> Thanks and Happy New Year to you guys, > >>>>>>>>>>> > >>>>>>>>>>> Royston (and Tom). > >>>>>>>>>>> > >>>>>>>>>>> (HBase 0.92, Hadoop 1.0) > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> -----Original Message----- > >>>>>>>>>>> From: Gary Helmling [mailto:[email protected]] > >>>>>>>>>>> Sent: 23 December 2011 18:06 > >>>>>>>>>>> To: [email protected] > >>>>>>>>>>> Subject: Re: AggregateProtocol Help > >>>>>>>>>>> > >>>>>>>>>>> Hi Tom, > >>>>>>>>>>> > >>>>>>>>>>> The test code is not really the best guide for configuration. > >>>>>>>>>>> > >>>>>>>>>>> To enable the AggregateProtocol on all of your tables, add > >>>>>>>>>>> this to the hbase-site.xml for the servers in your cluster: > >>>>>>>>>>> > >>>>>>>>>>> <property> > >>>>>>>>>>> <name>hbase.coprocessor.user.region.classes</name> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>> <value>org.apache.hadoop.hbase.coprocessor.AggregateImplementatio > >>>>>>>> n< > >>>>>>>> /v > >>>>>>>> alue> > >>>>>>>>>>> </property> > >>>>>>>>>>> > >>>>>>>>>>> If you only want to use the aggregate functions on a > >>>>>>>>>>> specific table > >>>>>>>> (or > >>>>>>>>>>> tables), then you can enable that individually for the table > >>>>>>>>>>> from the > >>>>>>>>>>> shell: > >>>>>>>>>>> > >>>>>>>>>>> 1) disable the table > >>>>>>>>>>> hbase> disable 'EDRP7' > >>>>>>>>>>> > >>>>>>>>>>> 2) add the coprocessor > >>>>>>>>>>> hbase> alter 'EDRP7', METHOD => 'table_att', > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>> 'coprocessor'=>'|org.apache.hadoop.hbase.coprocessor.AggregateImp > >>>>>>>> le > >>>>>>>> me > >>>>>>>> ntation > >>>>>>>>>>> ||' > >>>>>>>>>>> > >>>>>>>>>>> (Note that the pipes in the value string are required) > >>>>>>>>>>> > >>>>>>>>>>> 3) re-enable the table > >>>>>>>>>>> hbase> enable 'EDRP7' > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Either way should work. With the second approach you will > >>>>>>>>>>> see the coprocessor listed when you describe the table from > >>>>>>>>>>> the shell, as Ted mentioned. With the first approach you > >>>>>>>>>>> will not, but it should be > >>>>>>>>> loaded > >>>>>>>>>>> all the same. > >>>>>>>>>>> > >>>>>>>>>>> --gh > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> On Fri, Dec 23, 2011 at 7:04 AM, Ted Yu > >>>>>>>>>>> <[email protected]> > >>>>>> wrote: > >>>>>>>>>>>> I don't know why you chose HBaseTestingUtility to create > >>>>>>>>>>>> the > >>>>>> table. > >>>>>>>>>>>> I guess you followed test code example. > >>>>>>>>>>>> > >>>>>>>>>>>> At least you should pass the conf to this ctor: > >>>>>>>>>>>> public HBaseTestingUtility(Configuration conf) { > >>>>>>>>>>>> > >>>>>>>>>>>> If coprocessor was installed correctly, you should see > >>>>>>>>>>>> something like(from > >>>>>>>>>>>> HBASE-5070): > >>>>>>>>>>>> coprocessor$1 => > >>>>>>>>>>>> > >>>>>> > '|org.apache.hadoop.hbase.constraint.ConstraintProcessor|1073741823|' > >>>>>>>>>>>> > >>>>>>>>>>>> Cheers > >>>>>>>>>>>> > >>>>>>>>>>>> On Fri, Dec 23, 2011 at 3:02 AM, Tom Wilcox > >>>>>>>>>>>> <[email protected]> > >>>>>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> Hi, > >>>>>>>>>>>>> > >>>>>>>>>>>>> I am not sure how we load the AggregateImplementation into > >>>>>>>>>>>>> the > >>>>>>>> table. > >>>>>>>>>>>>> When we are creating a table, we use the same functions as > >>>>>>>>>>>>> the test > >>>>>>>> as > >>>>>>>>>>> follows... > >>>>>>>>>>>>> > >>>>>>>>>>>>> ... > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> conf.set(CoprocessorHost.REGION_COPROCESSOR_CONF_KEY, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>> "org.apache.hadoop.hbase.coprocessor.AggregateImplementation" > >>>>>>>>>>>>>> ); > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> // Utility.CreateHBaseTable(conf, > >>>>>>>>>>>>>> otherArgs[1], > >>>>>>>>>>>>> otherArgs[2], > >>>>>>>>>>>>>> true); > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> HBaseTestingUtility util = new > >>>>>>>>>>>>>> HBaseTestingUtility(); > >>>>>>>>>>>>>> HTable table = util.createTable(EDRP_TABLE, > >>>>>>>>>>>>>> EDRP_FAMILY); > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> AggregationClient aClient = new > >>>>>>>>>>>>>> AggregationClient(conf); > >>>>>>>>>>>>> ... > >>>>>>>>>>>>> > >>>>>>>>>>>>> Running DESCRIBE on a table produced shows the following > >>>>> output: > >>>>>>>>>>>>> > >>>>>>>>>>>>> hbase(main):002:0> describe 'EDRP7' > >>>>>>>>>>>>> DESCRIPTION > >>>>>>>>>>>>> ENABLED {NAME => > >>>>>>>>>>>>> 'EDRP7', FAMILIES => [{NAME => 'advanceKWh', BLOOMFILTER > >>>>>>>>>>>>> => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => true > >>>>>>>>>>>>> '3', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => > >>>>>>>>>>>>> '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', > >>>>>>>>>>>>> B LOCKCACHE => 'true'}]} > >>>>>>>>>>>>> > >>>>>>>>>>>>> We are using the tip of 0.92 (cloned from the Git repo). > >>>>>>>>>>>>> See the version string below: > >>>>>>>>>>>>> > >>>>>>>>>>>>> hbase(main):005:0> version 0.92.0, r1208286, Thu Dec 15 > >>>>>>>>>>>>> 13:16:03 GMT 2011 > >>>>>>>>>>>>> > >>>>>>>>>>>>> We would really appreciate an example of how to create a > >>>>>>>>>>>>> table that is enabled to handle Aggregation). > >>>>>>>>>>>>> > >>>>>>>>>>>>> Thanks > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> ________________________________________ > >>>>>>>>>>>>> From: Ted Yu [[email protected]] > >>>>>>>>>>>>> Sent: 22 December 2011 17:03 > >>>>>>>>>>>>> To: [email protected] > >>>>>>>>>>>>> Subject: Re: AggregateProtocol Help > >>>>>>>>>>>>> > >>>>>>>>>>>>> Have you loaded AggregateImplementation into your table ? > >>>>>>>>>>>>> Can you show us the contents of the following command in > >>>>>>>>>>>>> hbase > >>>>>>>> shell: > >>>>>>>>>>>>> describe 'your-table' > >>>>>>>>>>>>> > >>>>>>>>>>>>> BTW are you using the tip of 0.92 ? > >>>>>>>>>>>>> HBASE-4946 would be of help for dynamically loaded > >>>>>>>>>>>>> coprocessors > >>>>>>>> which > >>>>>>>>>>>>> you might use in the future. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Cheers > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Thu, Dec 22, 2011 at 8:09 AM, Tom Wilcox > >>>>>>>>>>>>> <[email protected]> > >>>>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> Hi, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> We are trying to use the aggregation functionality in > >>>>>>>>>>>>>> HBase > >>>>>>>>>>>>>> 0.92 and we have managed to get the test code working > >>>>>>>>>>>>>> using the > >>>>>>>> following > >>>>>>>>>>> command: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> java -classpath junit-4.10.jar:build/*:$HBASELIBS/* > >>>>>>>>>>>>>> org.junit.runner.JUnitCore > >>>>>>>>>>>>>> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Closer inspection of this test class has revealed that it > >>>>>>>>>>>>>> uses a mini DFS cluster to populate and run the tests. > >>>>>>>>>>>>>> These tests return > >>>>>>>>>>> successfully. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> However, when we attempt to run similar code on our > >>>>>>>>>>>>>> development HDFS cluster we experience the following error: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 11/12/22 15:46:28 WARN > >>>>>>>>>>>>>> client.HConnectionManager$HConnectionImplementation: > >>>>>>>>>>>>>> Error executing for > >>>>>>>>>>>>> row > >>>>>>>>>>>>>> java.util.concurrent.ExecutionException: > >>>>>>>>>>>>>> > >>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException: > >>>>>>>>>>>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolExcep > >>>>>>>>>>>>>> ti > >>>>>>>>>>>>>> on > >>>>>>>>>>>>>> : No > >>>>>>>>>>>>> matching > >>>>>>>>>>>>>> handler for protocol > >>>>>>>>>>>>> org.apache.hadoop.hbase.coprocessor.AggregateProtocol > >>>>>>>>>>>>>> in region > >>>>>>>>> EDRPTestTbl,,1324485124322.7b9ee0d113db9b24ea9fdde90702d006. > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegion.exec(HRegion > >>>>>>>>>>>>>> .j > >>>>>>>>>>>>>> av > >>>>>>>>>>>>>> a:4010 > >>>>>>>>>>>>>> ) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.execCoprocesso > >>>>>>>> r( > >>>>>>>> HR > >>>>>>>>>>>>> egionServer.java:3040) > >>>>>>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > >>>>>>>>>>>>> Method) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> > >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. > >>>>>>>>>>>>> java:39) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodA > >>>>>>>> cc > >>>>>>>> es > >>>>>>>>>>>>> sorImpl.java:25) > >>>>>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(Writabl > >>>>>>>> eR > >>>>>>>> pc > >>>>>>>>>>>>> Engine.java:364) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> > >>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java: > >>>>>>>>>>>>> 1325) > >>>>>>>>>>>>>> [sshexec] > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask. > >>>>>>>>>>>>>> ja > >>>>>>>>>>>>>> va > >>>>>>>>>>>>>> :222) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> java.util.concurrent.FutureTask.get(FutureTask.java:83) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl > >>>>>>>> em > >>>>>>>> en > >>>>>>>>>>>>> tation.processExecs(HConnectionManager.java:1465) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>> org.apache.hadoop.hbase.client.HTable.coprocessorExec(HTable. > >>>>>>>>>>>>>> java:1 > >>>>>>>>>>>>>> 555) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> org.apache.hadoop.hbase.client.coprocessor.AggregationClient.sum( > >>>>>>>> Ag > >>>>>>>> gr > >>>>>>>>>>>>> egationClient.java:229) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> EDRPAggregator.testSumWithValidRange(EDRPAggregator.java:51) > >>>>>>>>>>>>>> at EDRPAggregator.main(EDRPAggregator.java:77) > >>>>>>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > >>>>>>>>>>>>> Method) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> > >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. > >>>>>>>>>>>>> java:39) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodA > >>>>>>>> cc > >>>>>>>> es > >>>>>>>>>>>>> sorImpl.java:25) > >>>>>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597) > >>>>>>>>>>>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156) > >>>>>>>>>>>>>> Caused by: > >>>>>>>>>>>>>> > >>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException: > >>>>>>>>>>>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolExcep > >>>>>>>>>>>>>> ti > >>>>>>>>>>>>>> on > >>>>>>>>>>>>>> : No > >>>>>>>>>>>>> matching > >>>>>>>>>>>>>> handler for protocol > >>>>>>>>>>>>> org.apache.hadoop.hbase.coprocessor.AggregateProtocol > >>>>>>>>>>>>>> in region > >>>>>>>>> EDRPTestTbl,,1324485124322.7b9ee0d113db9b24ea9fdde90702d006. > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegion.exec(HRegion > >>>>>>>>>>>>>> .j > >>>>>>>>>>>>>> av > >>>>>>>>>>>>>> a:4010 > >>>>>>>>>>>>>> ) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.execCoprocesso > >>>>>>>> r( > >>>>>>>> HR > >>>>>>>>>>>>> egionServer.java:3040) > >>>>>>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > >>>>>>>>>>>>> Method) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> > >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. > >>>>>>>>>>>>> java:39) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodA > >>>>>>>> cc > >>>>>>>> es > >>>>>>>>>>>>> sorImpl.java:25) > >>>>>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(Writabl > >>>>>>>> eR > >>>>>>>> pc > >>>>>>>>>>>>> Engine.java:364) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> > >>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java: > >>>>>>>>>>>>> 1325) > >>>>>>>>>>>>>> [sshexec] > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Na > >>>>>>>>>>>>>> ti > >>>>>>>>>>>>>> ve > >>>>>>>>>>>>>> Method) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConst > >>>>>>>> ru > >>>>>>>> ct > >>>>>>>>>>>>> orAccessorImpl.java:39) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Delegat > >>>>>>>> in > >>>>>>>> gC > >>>>>>>>>>>>> onstructorAccessorImpl.java:27) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> java.lang.reflect.Constructor.newInstance(Constructor.java: > >>>>>>>>>>>>>> 51 > >>>>>>>>>>>>>> 3) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteExcept > >>>>>>>> io > >>>>>>>> n( > >>>>>>>>>>>>> RemoteExceptionHandler.java:96) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl > >>>>>>>> em > >>>>>>>> en > >>>>>>>>>>>>> tation.translateException(HConnectionManager.java:1651) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl > >>>>>>>> em > >>>>>>>> en > >>>>>>>>>>>>> tation.getRegionServerWithRetries(HConnectionManager.java: > >>>>>>>>>>>>> 13 > >>>>>>>>>>>>> 27 > >>>>>>>>>>>>> ) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(ExecRPC > >>>>>>>>>>>>>> In > >>>>>>>>>>>>>> vo > >>>>>>>>>>>>>> ker.ja > >>>>>>>>>>>>>> va:79) > >>>>>>>>>>>>>> at $Proxy3.getSum(Unknown Source) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.ca > >>>>>>>> ll > >>>>>>>> (A > >>>>>>>>>>>>> ggregationClient.java:233) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.ca > >>>>>>>> ll > >>>>>>>> (A > >>>>>>>>>>>>> ggregationClient.java:230) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl > >>>>>>>> em > >>>>>>>> en > >>>>>>>>>>>>> tation$4.call(HConnectionManager.java:1453) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask. > >>>>>>>>>>>>>> ja > >>>>>>>>>>>>>> va > >>>>>>>>>>>>>> :303) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPool > >>>>>>>> Ex > >>>>>>>> ec > >>>>>>>>>>>>> utor.java:886) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExec > >>>>>>>> ut > >>>>>>>> or > >>>>>>>>>>>>> .java:908) > >>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:662) > >>>>>>>>>>>>>> > >>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException: > >>>>>>>>>>>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolExcep > >>>>>>>>>>>>>> ti > >>>>>>>>>>>>>> on > >>>>>>>>>>>>>> : No > >>>>>>>>>>>>> matching > >>>>>>>>>>>>>> handler for protocol > >>>>>>>>>>>>> org.apache.hadoop.hbase.coprocessor.AggregateProtocol > >>>>>>>>>>>>>> in region > >>>>>>>>> EDRPTestTbl,,1324485124322.7b9ee0d113db9b24ea9fdde90702d006. > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> org.apache.hadoop.hbase.regionserver.HRegion.exec(HRegion > >>>>>>>>>>>>>> .j > >>>>>>>>>>>>>> av > >>>>>>>>>>>>>> a:4010 > >>>>>>>>>>>>>> ) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> org.apache.hadoop.hbase.regionserver.HRegionServer.execCoprocesso > >>>>>>>> r( > >>>>>>>> HR > >>>>>>>>>>>>> egionServer.java:3040) > >>>>>>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > >>>>>>>>>>>>> Method) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> > >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. > >>>>>>>>>>>>> java:39) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodA > >>>>>>>> cc > >>>>>>>> es > >>>>>>>>>>>>> sorImpl.java:25) > >>>>>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(Writabl > >>>>>>>> eR > >>>>>>>> pc > >>>>>>>>>>>>> Engine.java:364) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> > >>>> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java: > >>>>>>>>>>>>> 1325) > >>>>>>>>>>>>>> [sshexec] > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Na > >>>>>>>>>>>>>> ti > >>>>>>>>>>>>>> ve > >>>>>>>>>>>>>> Method) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConst > >>>>>>>> ru > >>>>>>>> ct > >>>>>>>>>>>>> orAccessorImpl.java:39) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Delegat > >>>>>>>> in > >>>>>>>> gC > >>>>>>>>>>>>> onstructorAccessorImpl.java:27) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> java.lang.reflect.Constructor.newInstance(Constructor.java: > >>>>>>>>>>>>>> 51 > >>>>>>>>>>>>>> 3) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteExcept > >>>>>>>> io > >>>>>>>> n( > >>>>>>>>>>>>> RemoteExceptionHandler.java:96) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl > >>>>>>>> em > >>>>>>>> en > >>>>>>>>>>>>> tation.translateException(HConnectionManager.java:1651) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl > >>>>>>>> em > >>>>>>>> en > >>>>>>>>>>>>> tation.getRegionServerWithRetries(HConnectionManager.java: > >>>>>>>>>>>>> 13 > >>>>>>>>>>>>> 27 > >>>>>>>>>>>>> ) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(ExecRPC > >>>>>>>>>>>>>> In > >>>>>>>>>>>>>> vo > >>>>>>>>>>>>>> ker.ja > >>>>>>>>>>>>>> va:79) > >>>>>>>>>>>>>> at $Proxy3.getSum(Unknown Source) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.ca > >>>>>>>> ll > >>>>>>>> (A > >>>>>>>>>>>>> ggregationClient.java:233) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.ca > >>>>>>>> ll > >>>>>>>> (A > >>>>>>>>>>>>> ggregationClient.java:230) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl > >>>>>>>> em > >>>>>>>> en > >>>>>>>>>>>>> tation$4.call(HConnectionManager.java:1453) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask. > >>>>>>>>>>>>>> ja > >>>>>>>>>>>>>> va > >>>>>>>>>>>>>> :303) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPool > >>>>>>>> Ex > >>>>>>>> ec > >>>>>>>>>>>>> utor.java:886) > >>>>>>>>>>>>>> at > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExec > >>>>>>>> ut > >>>>>>>> or > >>>>>>>>>>>>> .java:908) > >>>>>>>>>>>>>> at java.lang.Thread.run(Thread.java:662) > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> The source for our class is: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> import static org.junit.Assert.assertEquals; > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> import java.io.IOException; > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> import org.apache.hadoop.conf.Configuration; > >>>>>>>>>>>>>> import org.apache.hadoop.hbase.HBaseConfiguration; > >>>>>>>>>>>>>> import org.apache.hadoop.hbase.HBaseTestingUtility; > >>>>>>>>>>>>>> import org.apache.hadoop.hbase.HConstants; > >>>>>>>>>>>>>> import org.apache.hadoop.hbase.HTableDescriptor; > >>>>>>>>>>>>>> import org.apache.hadoop.hbase.client.HTable; > >>>>>>>>>>>>>> import org.apache.hadoop.hbase.client.Put; > >>>>>>>>>>>>>> import org.apache.hadoop.hbase.client.Scan; > >>>>>>>>>>>>>> import > >>>>>>>>>>>>>> org.apache.hadoop.hbase.client.coprocessor.AggregationCli > >>>>>>>>>>>>>> en > >>>>>>>>>>>>>> t; > >>>>>>>>>>>>>> import > >>>>>>>>>>>>>> org.apache.hadoop.hbase.client.coprocessor.LongColumnInte > >>>>>>>>>>>>>> rp re ter; import org.apache.hadoop.hbase.util.Bytes; > >>>>>>>>>>>>>> import org.apache.hadoop.util.GenericOptionsParser; > >>>>>>>>>>>>>> import > >>>>>>>>>>>>>> org.apache.hadoop.hbase.coprocessor.ColumnInterpreter; > >>>>>>>>>>>>>> import > >>>>>>>>>>>>>> org.apache.hadoop.hbase.coprocessor.CoprocessorHost; > >>>>>>>>>>>>>> import org.junit.Test; > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> public class EDRPAggregator { > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> // private static final byte[] EDRP_FAMILY = > >>>>>>>>>>>>>> Bytes.toBytes("EDRP"); > >>>>>>>>>>>>>> // private static final byte[] EDRP_QUALIFIER = > >>>>>>>>>>>>>> Bytes.toBytes("advanceKWh"); > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> private static byte[] ROW = Bytes.toBytes("testRow"); > >>>>>>>>>>>>>> private static final int ROWSIZE = 20; > >>>>>>>>>>>>>> private static byte[][] ROWS = makeN(ROW, ROWSIZE); > >>>>>>>>>>>>>> private static final byte[] TEST_QUALIFIER = > >>>>>>>>>>>>>> Bytes.toBytes("TestQualifier"); > >>>>>>>>>>>>>> private static final byte[] TEST_MULTI_CQ = > >>>>>>>>>>>>>> Bytes.toBytes("TestMultiCQ"); > >>>>>>>>>>>>>> private static final int rowSeperator1 = 5; > >>>>>>>>>>>>>> private static final int rowSeperator2 = 12; > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> public static void > >>>>>>>>>>>>>> testSumWithValidRange(Configuration > >>>>>>>>>>>>>> conf, > >>>>>>>>>>>>>> String[] otherArgs) throws Throwable { > >>>>>>>>>>>>>> byte[] EDRP_TABLE = Bytes.toBytes(otherArgs[1]); > >>>>>>>>>>>>>> byte[] EDRP_FAMILY = > >>>>>>>>>>>>>> Bytes.toBytes(otherArgs[2]); > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> conf.set(CoprocessorHost.REGION_COPROCESSOR_CONF_KEY, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>> "org.apache.hadoop.hbase.coprocessor.AggregateImplementation" > >>>>>>>>>>>>>> ); > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> // Utility.CreateHBaseTable(conf, > >>>>>>>>>>>>>> otherArgs[1], > >>>>>>>>>>>>> otherArgs[2], > >>>>>>>>>>>>>> true); > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> HBaseTestingUtility util = new > >>>>>>>>>>>>>> HBaseTestingUtility(); > >>>>>>>>>>>>>> HTable table = util.createTable(EDRP_TABLE, > >>>>>>>>>>>>>> EDRP_FAMILY); > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> AggregationClient aClient = new > >>>>>>>>>>>>>> AggregationClient(conf); > >>>>>>>>>>>>>> Scan scan = new Scan(); > >>>>>>>>>>>>>> scan.addColumn(EDRP_TABLE, EDRP_FAMILY); > >>>>>>>>>>>>>> final ColumnInterpreter<Long, Long> ci = new > >>>>>>>>>>>>>> LongColumnInterpreter(); > >>>>>>>>>>>>>> long sum = > >>>>>>>>>>>>>> aClient.sum(Bytes.toBytes(otherArgs[0]), > >>>>>>>>>>>>>> ci, > >>>>>>>>>>>>> scan); > >>>>>>>>>>>>>> System.out.println(sum); > >>>>>>>>>>>>>> } > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> /** > >>>>>>>>>>>>>> * Main entry point. > >>>>>>>>>>>>>> * > >>>>>>>>>>>>>> * @param argsThe > >>>>>>>>>>>>>> * command line parameters. > >>>>>>>>>>>>>> * @throws Exception > >>>>>>>>>>>>>> * When running the job fails. > >>>>>>>>>>>>>> */ > >>>>>>>>>>>>>> public static void main(String[] args) throws > >>>>>>>>>>>>>> Exception { > >>>>>>>>>>>>>> Configuration conf = > >>>>>>>>>>>>>> HBaseConfiguration.create(); > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> String[] otherArgs = new > >>>>>>>>>>>>>> GenericOptionsParser(conf, > >>>>>>>>>>>>>> args) > >>>>>>>>>>>>>> .getRemainingArgs(); > >>>>>>>>>>>>>> if (otherArgs.length != 3) { > >>>>>>>>>>>>>> System.err > >>>>>>>>>>>>>> .println("Wrong number of > >>>>>>>>>>>>>> arguments: " + otherArgs.length); > >>>>>>>>>>>>>> System.err.println("Usage: " + > >>>>>>>>>>>>>> "<tablename> <colfam> <qualifier>"); > >>>>>>>>>>>>>> System.exit(-1); > >>>>>>>>>>>>>> } > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> try { > >>>>>>>>>>>>>> testSumWithValidRange(conf, otherArgs); > >>>>>>>>>>>>>> } catch (Throwable e) { > >>>>>>>>>>>>>> e.printStackTrace(); > >>>>>>>>>>>>>> } > >>>>>>>>>>>>>> } > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> /** > >>>>>>>>>>>>>> * an infrastructure method to prepare rows for the > >>>>>> testtable. > >>>>>>>>>>>>>> * > >>>>>>>>>>>>>> * @param base > >>>>>>>>>>>>>> * @param n > >>>>>>>>>>>>>> * @return > >>>>>>>>>>>>>> */ > >>>>>>>>>>>>>> private static byte[][] makeN(byte[] base, int n) { > >>>>>>>>>>>>>> byte[][] ret = new byte[n][]; > >>>>>>>>>>>>>> for (int i = 0; i < n; i++) { > >>>>>>>>>>>>>> ret[i] = Bytes.add(base, > >>>>>>>>>>>>>> Bytes.toBytes(i)); > >>>>>>>>>>>>>> } > >>>>>>>>>>>>>> return ret; > >>>>>>>>>>>>>> } > >>>>>>>>>>>>>> } > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Please can you suggest what might be causing and/or how > >>>>>>>>>>>>>> we might fix this UnknownProtocolException? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Also, does anyone have any working examples using the > >>>>>>>>>>>>>> aggregation > >>>>>>>>>>>>> protocol > >>>>>>>>>>>>>> other than the test code? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Thanks, > >>>>>>>>>>>>>> Tom > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>>> > >> > >> > >
