I realized that HBASE-4554 may be heavy-weight w.r.t. loading custom interpreter into region servers.
Royston: If your use case doesn't require writing coprocessor, we should create a JIRA for the loading of interpreter implementations. Cheers On Tue, Jan 3, 2012 at 10:58 AM, Ted Yu <[email protected]> wrote: > I like long messages :-) because they provide more clues. > > For part 1, you don't have to call Bytes.toxxx as long as the interpreter > uses method consistent with the way you write values into HBase tables. > > For part 2, HBASE-4946 is related. > Basically you need to place the jar containing your coprocessor and > interpreter code on hdfs so that you can load it into your HBase table. > Look at this for details: > https://issues.apache.org/jira/browse/HBASE-4554 > > Cheers > > > On Tue, Jan 3, 2012 at 10:42 AM, Royston Sellman < > [email protected]> wrote: > >> Hi Ted, >> >> PART 1 >> ===== >> Thanks for the hint. I think maybe you have given me some inspiration! >> >> It looks like getValue will return null if the table value is not the >> length >> of a long. When we created our table (batch loading CSVs using the >> SampleUploader example) we simply have this as our put(): >> put.add(family, Bytes.toBytes("advanceKWh"), advanceKWh); >> [note we do no Bytes.toxxx casts to the advanceKWh value. The values look >> OK >> from HBase shell though :-)] >> >> but I looked at TestAggregateProtocol.java again and I see there puts >> like: >> p2.add(TEST_FAMILY, Bytes.add(TEST_MULTI_CQ, Bytes.toBytes(l)), >> Bytes.toBytes(l * 10)); >> >> So my hypothesis is that we need to do something like: >> Long l = new Long(1); >> put.add(family, Bytes.toBytes("advanceKWh"), Bytes.toBytes(l * >> advanceKWh)); >> when we create the table. >> >> Do you think my hypothesis is correct? Did we build our table incorrectly >> for reading longs from it? >> >> PART 2 >> ===== >> Anyway we will obviously need to make our own interpreters. but we failed >> at >> this task so far: >> In order to implement our own ColumnInterpretter, we first attempted >> simply >> extending the LongColumnInterpreter and passing that as a parameter to >> aClient.sum(). >> import org.apache.hadoop.hbase.client.coprocessor.LongColumnInterpreter; >> >> public class LCI extends LongColumnInterpreter { >> public Long add(Long a, Long b) { >> System.out.print("LCI.add("+a+","+b+")"); >> Long c = super.add(a,b); >> System.out.println(" -> "+c); >> return c; >> } >> }; >> .... >> final ColumnInterpreter<Long, Long> ci = new LCI(); >> long sum=-1; >> try { >> sum = aClient.sum(EDRP_TABLE, ci, scan); >> ... >> However, we received class not found errors in our regionserver log when >> we >> ran this code >> Clearly we are missing something >> We've started looking at modifying the HBase source and rebuilding that >> But I think this is not the way we should be working >> It seems that we should be implementing the ColumnInterpreter interface >> and >> passing an instance of that implementation class too as a parameter. Is >> this correct? Could you provide us with an example? Any ideas why we >> might >> be getting class not found exception from the regionserver? >> >> Sorry for the long message! >> >> Royston (and Tom) >> >> >> -----Original Message----- >> From: Ted Yu [mailto:[email protected]] >> Sent: 03 January 2012 18:00 >> To: [email protected] >> Subject: Re: AggregateProtocol Help >> >> My previous email might not be hitting the root cause. >> I think the following method in LCI may be giving you the null: >> >> public Long getValue(byte[] colFamily, byte[] colQualifier, KeyValue kv) >> throws IOException { >> if (kv == null || kv.getValueLength() != Bytes.SIZEOF_LONG) >> return null; >> return Bytes.toLong(kv.getBuffer(), kv.getValueOffset()); >> >> Look at the if statement above carefully. >> If it doesn't match how you store values in HBase, feel free to subclass >> LongColumnInterpreter and provide the correct interpretation. >> >> BTW you don't need to restart cluster just because you need to use your >> own >> interpreter :-) >> >> On Tue, Jan 3, 2012 at 9:48 AM, Royston Sellman < >> [email protected]> wrote: >> >> > Hi Ted, >> > >> > Here is the output. As you can see aClient is not nul: >> > >> > AggregationClient aClient = new AggregationClient(conf); >> > System.err.println("aClient: "+aClient); >> > >> > <<< aClient: >> > org.apache.hadoop.hbase.client.coprocessor.AggregationClient@28787c16 >> > >> > It will take us a little while to add log code to LCI... we have to >> > edit the source, rebuild 0.92, redistribute round our cluster, restart >> > ;) We'll get back to you when this is done. >> > >> > Royston >> > >> > -----Original Message----- >> > From: Ted Yu [mailto:[email protected]] >> > Sent: 03 January 2012 17:10 >> > To: [email protected] >> > Subject: Re: AggregateProtocol Help >> > >> > Royston: >> > Thanks for your effort trying to hunt down the problem. >> > >> > Can you add a log after this line to see if aClient is null ? >> > AggregationClient aClient = new AggregationClient(conf); >> > >> > I was looking at LongColumnInterpreter.add() which is called by >> > aClient.sum() >> > Can you add a few log statements in LongColumnInterpreter.add() to see >> > what parameters are passed to it ? >> > >> > Cheers >> > >> > On Tue, Jan 3, 2012 at 8:32 AM, Royston Sellman < >> > [email protected]> wrote: >> > >> > > Hi Ted, Himanshu, Gary, >> > > >> > > Thanks again for your attention. I experimented with a shorter table >> > > and it looks like the timeout error was spurious... >> > > >> > > With the shorter table I now get an NPE when I call >> > > AggregationClient.sum(). >> > > Here's the code snippet: >> > > >> > > // Test the table >> > > HTable table = new HTable(EDRP_TABLE); >> > > Get get = new Get(Bytes.toBytes("row-aa")); >> > > get.addColumn(Bytes.toBytes("EDRP"), >> > > Bytes.toBytes("advanceKWh")); >> > > Result result = table.get(get); >> > > byte [] val = result.getValue(Bytes.toBytes("EDRP"), >> > > Bytes.toBytes("advanceKWh")); >> > > System.out.println("Row aa = " + >> > > Bytes.toString(val)); >> > > >> > > AggregationClient aClient = new >> AggregationClient(conf); >> > > Scan scan = new Scan(); >> > > scan.addColumn(EDRP_FAMILY, EDRP_QUALIFIER); >> > > scan.setStartRow(Bytes.toBytes("row-ab")); >> > > scan.setStopRow(Bytes.toBytes("row-az")); >> > > System.out.println(Bytes.toString(EDRP_FAMILY) + ":" >> > > + Bytes.toString(EDRP_QUALIFIER)); >> > > final ColumnInterpreter<Long, Long> ci = new >> > > LongColumnInterpreter(); >> > > long sum=-1; >> > > try { >> > > sum = aClient.sum(EDRP_TABLE, ci, scan); >> > > } catch (Throwable e) { >> > > // TODO Auto-generated catch block >> > > e.printStackTrace(); >> > > } >> > > System.out.println(sum); >> > > >> > > The first part is just to check that my table is OK. It prints the >> > > correct value for row aa. Then I check CF:CQ is correct. Then I get >> > > the -1 that sum was defined as. Then the NPE. Here is the run output: >> > > >> > > Row aa = 3.0 >> > > EDRP:advanceKWh >> > > -1 >> > > java.lang.NullPointerException >> > > at EDRPAggregator.testSumWithValidRange(EDRPAggregator.java:66) >> > > at EDRPAggregator.main(EDRPAggregator.java:96) >> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at >> > > >> > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl >> > > .j >> > > ava:39 >> > > ) >> > > at >> > > >> > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcce >> > > ss >> > > orImpl >> > > .java:25) >> > > at java.lang.reflect.Method.invoke(Method.java:597) >> > > at org.apache.hadoop.util.RunJar.main(RunJar.java:156) >> > > >> > > Line 66 is the line: >> > > sum = aClient.sum(EDRP_TABLE, ci, scan); >> > > >> > > But I'm not sure that is really the line generating the NPE because >> > > I get the -1 output (same result when line 66 is not wrapped in >> > > try-catch >> > BTW). >> > > Anyway, -1 is clearly not the correct value for the sum. >> > > >> > > [Note: I'm setting a start and end row on the scan because I saw a >> > > comment in AggregationClient.java saying they must be set. However, >> > > I think this comment is obsolete as the test code in >> > > TestAggregateProtocol.java sometimes does not include start & end >> > > rows. The run is exactly the same without the start & stop rows ] >> > > >> > > We are struggling with Aggregations not least because >> > > TestAggregateProtocol.java is the only source of example code we >> > > could find and it is not ideal because it is designed to work with >> > > MiniCluster rather than a real cluster like ours. Is there any other >> > > example code in existence? >> > > I think that would really help us. >> > > >> > > Meanwhile, thanks for your patience with our questions. We are >> > > really impressed with the AggregationProtocol >> > > architecture/functionality and very keen to use it. >> > > >> > > Royston (and Tom) >> > > >> > > -----Original Message----- >> > > From: Gary Helmling [mailto:[email protected]] >> > > Sent: 02 January 2012 06:23 >> > > To: [email protected] >> > > Subject: Re: AggregateProtocol Help >> > > >> > > Hi Royston, >> > > >> > > Try increasing the value set for hbase.rpc.timeout (by default 60 >> > seconds). >> > > Add something like this to hbase-site.xml: >> > > >> > > <property> >> > > <name>hbase.rpc.timeout</name> >> > > <value>180000</value> >> > > </property> >> > > >> > > This would increase the timeout value to 3 minutes, for example. >> > > >> > > But as Ted and Himanshu mention, 557k rows is not a large table and >> > > should not be taking very long to process, unless these happen to be >> > > very, very wide rows. It sounds like there could be something else >> > > going on. Does debug level logging show anything else in the region >> > server log? >> > > >> > > --gh >> > > >> > > On Sun, Jan 1, 2012 at 5:53 PM, Ted Yu <[email protected]> wrote: >> > > > Thanks for the reminder Himanshu. >> > > > >> > > > Royston: >> > > > From this blog you can get some history on this subject: >> > > > http://zhihongyu.blogspot.com/2011/03/genericizing-endpointcoproce >> > > > ss >> > > > or >> > > > .html >> > > > >> > > > >> > > > On Sun, Jan 1, 2012 at 5:18 PM, Himanshu Vashishtha >> > > > <[email protected] >> > > >> wrote: >> > > > >> > > >> Hello Royston, >> > > >> >> > > >> Sorry to hear that you are getting trouble while using >> > > >> Aggregation functionalities. >> > > >> >> > > >> 557k rows seems to be a small table and a SocketTimeout does not >> > > >> seem to be an ok response. >> > > >> It will be good to know the region distribution as such. (how >> > > >> many >> > > regions? >> > > >> Is it a full table scan?) >> > > >> >> > > >> You are using the sum function; how are you using the >> > ColumnInterpreter. >> > > >> Can you enable the log level to debug to see why the RS is taking >> > > >> that long to respond (more than 113 sec). >> > > >> The 0 return value is the default result. >> > > >> >> > > >> Thanks for trying this out. >> > > >> >> > > >> Thanks, >> > > >> Himanshu >> > > >> >> > > >> On Sun, Jan 1, 2012 at 12:26 PM, Royston Sellman < >> > > >> [email protected]> wrote: >> > > >> >> > > >> > Hi Ted, >> > > >> > >> > > >> > I think 0 is the only value we ever see (I'll check tomorrow: >> > > >> > the server is down right now). Our table has 557,000 rows. I'll >> > > >> > try a much shorter table tomorrow. >> > > >> > >> > > >> > Yes, we have RS running on the NN, but it's a test cluster and >> > > >> > we are >> > > >> used >> > > >> > to it :) >> > > >> > >> > > >> > Do you think using AggregationProtocol is the best strategy for >> > > >> > the case where we want to use basic SQL-style functions like >> > > >> > SUM, AVG, STD, MIN, MAX? Do you think there is a better strategy? >> > > >> > >> > > >> > Many thanks, >> > > >> > Royston >> > > >> > >> > > >> > >> > > >> > On 1 Jan 2012, at 17:58, Ted Yu wrote: >> > > >> > >> > > >> > > Royston: >> > > >> > > Happy New Year to you too. >> > > >> > > >> > > >> > >>> java.net.SocketTimeoutException: Call to >> > > >> > >>> namenode/10.0.0.235:60020 >> > > >> failed >> > > >> > on >> > > >> > > >> > > >> > > It seems the namenode above actually refers to a region server. >> > > >> > > This >> > > >> is a >> > > >> > > little bit confusing :-) >> > > >> > > >> > > >> > > The sum value below is 0. >> > > >> > > Have you ever seen a value greater than 0 ? >> > > >> > > >> > > >> > > How many rows are there in this CF:CQ ? >> > > >> > > The timeout was reported earlier by other people where >> > > >> > > there're many >> > > >> rows >> > > >> > > in the table. >> > > >> > > >> > > >> > > There is a JIRA to provide streaming support for coprocessor >> > > >> > > but the development there has stalled. >> > > >> > > >> > > >> > > Cheers >> > > >> > > >> > > >> > > On Sun, Jan 1, 2012 at 9:35 AM, Royston Sellman < >> > > >> > > [email protected]> wrote: >> > > >> > > >> > > >> > >> Hi Gary and Ted, >> > > >> > >> >> > > >> > >> Royston (Tom's colleague) here. Back onto this after the >> > > >> > >> Christmas/New >> > > >> > Year >> > > >> > >> break. >> > > >> > >> >> > > >> > >> Many thanks for your help so far. We enabled our database >> > > >> > >> via your hbase-site.xml mod and were able to move on. to >> > > >> > >> other errors. But I >> > > >> > think >> > > >> > >> we >> > > >> > >> are now actually getting an aggregation partially calculated >> > > >> > >> on our >> > > >> > table >> > > >> > >> (this feels like progress). The details: >> > > >> > >> >> > > >> > >> On running our client we now get this exception: >> > > >> > >> 11/12/31 17:51:09 WARN >> > > >> > >> client.HConnectionManager$HConnectionImplementation: Error >> > > >> > >> executing >> > > >> for >> > > >> > >> row >> > > >> > >> >> > > >> > >> java.util.concurrent.ExecutionException: >> > > >> > >> org.apache.hadoop.hbase.client.RetriesExhaustedException: >> > > >> > >> Failed after attempts=10, exceptions: >> > > >> > >> Sat Dec 31 17:41:30 GMT 2011, >> > > >> > >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@1fc4f0f8, >> > > >> > >> java.net.SocketTimeoutException: Call to >> > > >> > >> namenode/10.0.0.235:60020 >> > > >> failed >> > > >> > >> on >> > > >> > >> socket timeout exception: java.net.SocketTimeoutException: >> > > >> > >> 60000 >> > > >> millis >> > > >> > >> timeout while waiting for channel to be ready for read. ch : >> > > >> > >> java.nio.channels.SocketChannel[connected >> > > >> > >> local=/10.0.0.235:59999 remote=namenode/10.0.0.235:60020] >> > > >> > >> (8 more of these, making for 10 tries) Sat Dec 31 17:51:09 >> > > >> > >> GMT 2011, >> > > >> > >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@1fc4f0f8, >> > > >> > >> java.net.SocketTimeoutException: Call to >> > > >> > >> namenode/10.0.0.235:60020 >> > > >> failed >> > > >> > >> on >> > > >> > >> socket timeout exception: java.net.SocketTimeoutException: >> > > >> > >> 60000 >> > > >> millis >> > > >> > >> timeout while waiting for channel to be ready for read. ch : >> > > >> > >> java.nio.channels.SocketChannel[connected >> > > >> > >> local=/10.0.0.235:59364 remote=namenode/10.0.0.235:60020] >> > > >> > >> >> > > >> > >> at >> > > >> > >> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java: >> > > >> > >> 22 >> > > >> > >> 2) >> > > >> > >> at >> java.util.concurrent.FutureTask.get(FutureTask.java:83) >> > > >> > >> at >> > > >> > >> >> > > >> > >> >> > > >> > >> > > >> >> > > >> > > >> > >> > >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation. >> > > >> > >> processExecs(HConnectionManager.java:1465) >> > > >> > >> at >> > > >> > >> >> > > >> org.apache.hadoop.hbase.client.HTable.coprocessorExec(HTable.java >> > > >> :1 >> > > >> 55 >> > > >> 5) >> > > >> > >> at >> > > >> > >> >> > > >> > >> >> > > >> > >> > > >> org.apache.hadoop.hbase.client.coprocessor.AggregationClient.sum( >> > > >> Ag >> > > >> gr >> > > >> egation >> > > >> > >> Client.java:229) >> > > >> > >> at >> > > >> > >> EDRPAggregator.testSumWithValidRange(EDRPAggregator.java:51) >> > > >> > >> at EDRPAggregator.main(EDRPAggregator.java:77) >> > > >> > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> > > >> > >> Method) >> > > >> > >> at >> > > >> > >> >> > > >> > >> >> > > >> > >> > > >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. >> > > >> java:39 >> > > >> > >> ) >> > > >> > >> at >> > > >> > >> >> > > >> > >> >> > > >> > >> > > >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodA >> > > >> cc >> > > >> es >> > > >> sorImpl >> > > >> > >> .java:25) >> > > >> > >> at java.lang.reflect.Method.invoke(Method.java:597) >> > > >> > >> at org.apache.hadoop.util.RunJar.main(RunJar.java:156) >> > > >> > >> >> > > >> > >> >> > > >> > >> Looking at the log (.regionserver-namenode.log) I see this >> > > >> > >> debug >> > > >> > message: >> > > >> > >> >> > > >> > >> 2011-12-31 17:42:23,472 DEBUG >> > > >> > >> org.apache.hadoop.hbase.coprocessor.AggregateImplementation: >> > > >> > >> Sum from >> > > >> > this >> > > >> > >> region is >> > > >> EDRPTestTbl,,1324485124322.7b9ee0d113db9b24ea9fdde90702d006.: >> > > >> > 0 >> > > >> > >> >> > > >> > >> Where the sum value looks reasonable which makes me think >> > > >> > >> the sum of a CF:CQ worked. But I never see this value on >> stdout. >> > > >> > >> Then I see this warning: >> > > >> > >> >> > > >> > >> 2011-12-31 17:42:23,476 WARN >> org.apache.hadoop.ipc.HBaseServer: >> > > >> > >> (responseTooSlow): >> > > >> > >> {"processingtimems":113146,"call":"execCoprocess$ >> > > >> > >> 2011-12-31 17:42:23,511 WARN >> org.apache.hadoop.ipc.HBaseServer: >> > > >> > >> IPC >> > > >> > Server >> > > >> > >> Responder, call execCoprocessor([B@4b22fad6, getSum(org.$ >> > > >> > >> 2011-12-31 17:42:23,515 WARN >> org.apache.hadoop.ipc.HBaseServer: >> > > >> > >> IPC >> > > >> > Server >> > > >> > >> handler 1 on 60020 caught: java.nio.channels.ClosedChann$ >> > > >> > >> at >> > > >> > >> >> > > >> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.ja >> > > >> va >> > > >> :1 >> > > >> 33) >> > > >> > >> at >> > > >> sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324) >> > > >> > >> at >> > > >> > >> >> > > >> > >> > > >> org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer. >> > > >> ja >> > > >> va >> > > >> :1651) >> > > >> > >> at >> > > >> > >> >> > > >> > >> >> > > >> > >> > > >> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse >> > > >> (H >> > > >> Ba >> > > >> seServe >> > > >> > >> r.java:924) >> > > >> > >> at >> > > >> > >> >> > > >> > >> >> > > >> > >> > > >> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBase >> > > >> Se >> > > >> rv >> > > >> er.java >> > > >> > >> :1003) >> > > >> > >> at >> > > >> > >> >> > > >> > >> >> > > >> > >> > > >> org.apache.hadoop.hbase.ipc.HBaseServer$Call.sendResponseIfReady( >> > > >> HB >> > > >> as >> > > >> eServer >> > > >> > >> .java:409) >> > > >> > >> at >> > > >> > >> >> > > >> > >> > > >> >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java: >> > > >> 1345) >> > > >> > >> >> > > >> > >> Have we missed out some step in the HBase/RegionServerconfig? >> > > >> > >> Or is >> > > >> our >> > > >> > >> client code still deficient? >> > > >> > >> >> > > >> > >> Can you offer any suggestions? Is there any example code for >> > > >> > >> the new Aggregations stuff. >> > > >> > >> >> > > >> > >> Thanks and Happy New Year to you guys, >> > > >> > >> >> > > >> > >> Royston (and Tom). >> > > >> > >> >> > > >> > >> (HBase 0.92, Hadoop 1.0) >> > > >> > >> >> > > >> > >> >> > > >> > >> -----Original Message----- >> > > >> > >> From: Gary Helmling [mailto:[email protected]] >> > > >> > >> Sent: 23 December 2011 18:06 >> > > >> > >> To: [email protected] >> > > >> > >> Subject: Re: AggregateProtocol Help >> > > >> > >> >> > > >> > >> Hi Tom, >> > > >> > >> >> > > >> > >> The test code is not really the best guide for configuration. >> > > >> > >> >> > > >> > >> To enable the AggregateProtocol on all of your tables, add >> > > >> > >> this to the hbase-site.xml for the servers in your cluster: >> > > >> > >> >> > > >> > >> <property> >> > > >> > >> <name>hbase.coprocessor.user.region.classes</name> >> > > >> > >> >> > > >> > >> >> > > >> > >> > > >> <value>org.apache.hadoop.hbase.coprocessor.AggregateImplementatio >> > > >> n< >> > > >> /v >> > > >> alue> >> > > >> > >> </property> >> > > >> > >> >> > > >> > >> If you only want to use the aggregate functions on a >> > > >> > >> specific table >> > > >> (or >> > > >> > >> tables), then you can enable that individually for the table >> > > >> > >> from the >> > > >> > >> shell: >> > > >> > >> >> > > >> > >> 1) disable the table >> > > >> > >> hbase> disable 'EDRP7' >> > > >> > >> >> > > >> > >> 2) add the coprocessor >> > > >> > >> hbase> alter 'EDRP7', METHOD => 'table_att', >> > > >> > >> >> > > >> > >> >> > > >> > >> >> > > >> > >> > > >> 'coprocessor'=>'|org.apache.hadoop.hbase.coprocessor.AggregateImp >> > > >> le >> > > >> me >> > > >> ntation >> > > >> > >> ||' >> > > >> > >> >> > > >> > >> (Note that the pipes in the value string are required) >> > > >> > >> >> > > >> > >> 3) re-enable the table >> > > >> > >> hbase> enable 'EDRP7' >> > > >> > >> >> > > >> > >> >> > > >> > >> Either way should work. With the second approach you will >> > > >> > >> see the coprocessor listed when you describe the table from >> > > >> > >> the shell, as Ted mentioned. With the first approach you >> > > >> > >> will not, but it should be >> > > >> > loaded >> > > >> > >> all the same. >> > > >> > >> >> > > >> > >> --gh >> > > >> > >> >> > > >> > >> >> > > >> > >> On Fri, Dec 23, 2011 at 7:04 AM, Ted Yu >> > > >> > >> <[email protected]> >> > > wrote: >> > > >> > >>> I don't know why you chose HBaseTestingUtility to create >> > > >> > >>> the >> > > table. >> > > >> > >>> I guess you followed test code example. >> > > >> > >>> >> > > >> > >>> At least you should pass the conf to this ctor: >> > > >> > >>> public HBaseTestingUtility(Configuration conf) { >> > > >> > >>> >> > > >> > >>> If coprocessor was installed correctly, you should see >> > > >> > >>> something like(from >> > > >> > >>> HBASE-5070): >> > > >> > >>> coprocessor$1 => >> > > >> > >>> >> > > '|org.apache.hadoop.hbase.constraint.ConstraintProcessor|1073741823|' >> > > >> > >>> >> > > >> > >>> Cheers >> > > >> > >>> >> > > >> > >>> On Fri, Dec 23, 2011 at 3:02 AM, Tom Wilcox >> > > >> > >>> <[email protected]> >> > > >> > >> wrote: >> > > >> > >>> >> > > >> > >>>> Hi, >> > > >> > >>>> >> > > >> > >>>> I am not sure how we load the AggregateImplementation into >> > > >> > >>>> the >> > > >> table. >> > > >> > >>>> When we are creating a table, we use the same functions as >> > > >> > >>>> the test >> > > >> as >> > > >> > >> follows... >> > > >> > >>>> >> > > >> > >>>> ... >> > > >> > >>>>> >> > > >> > >>>>> conf.set(CoprocessorHost.REGION_COPROCESSOR_CONF_KEY, >> > > >> > >>>>> >> > > >> > >>>>> >> "org.apache.hadoop.hbase.coprocessor.AggregateImplementation" >> > > >> > >>>>> ); >> > > >> > >>>>> >> > > >> > >>>>> // Utility.CreateHBaseTable(conf, >> > > >> > >>>>> otherArgs[1], >> > > >> > >>>> otherArgs[2], >> > > >> > >>>>> true); >> > > >> > >>>>> >> > > >> > >>>>> HBaseTestingUtility util = new >> > > >> > >>>>> HBaseTestingUtility(); >> > > >> > >>>>> HTable table = util.createTable(EDRP_TABLE, >> > > >> > >>>>> EDRP_FAMILY); >> > > >> > >>>>> >> > > >> > >>>>> AggregationClient aClient = new >> > > >> > >>>>> AggregationClient(conf); >> > > >> > >>>> ... >> > > >> > >>>> >> > > >> > >>>> Running DESCRIBE on a table produced shows the following >> > output: >> > > >> > >>>> >> > > >> > >>>> hbase(main):002:0> describe 'EDRP7' >> > > >> > >>>> DESCRIPTION >> > > >> > >>>> ENABLED {NAME => >> > > >> > >>>> 'EDRP7', FAMILIES => [{NAME => 'advanceKWh', BLOOMFILTER >> > > >> > >>>> => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => true >> > > >> > >>>> '3', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => >> > > >> > >>>> '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', >> > > >> > >>>> B LOCKCACHE => 'true'}]} >> > > >> > >>>> >> > > >> > >>>> We are using the tip of 0.92 (cloned from the Git repo). >> > > >> > >>>> See the version string below: >> > > >> > >>>> >> > > >> > >>>> hbase(main):005:0> version 0.92.0, r1208286, Thu Dec 15 >> > > >> > >>>> 13:16:03 GMT 2011 >> > > >> > >>>> >> > > >> > >>>> We would really appreciate an example of how to create a >> > > >> > >>>> table that is enabled to handle Aggregation). >> > > >> > >>>> >> > > >> > >>>> Thanks >> > > >> > >>>> >> > > >> > >>>> >> > > >> > >>>> ________________________________________ >> > > >> > >>>> From: Ted Yu [[email protected]] >> > > >> > >>>> Sent: 22 December 2011 17:03 >> > > >> > >>>> To: [email protected] >> > > >> > >>>> Subject: Re: AggregateProtocol Help >> > > >> > >>>> >> > > >> > >>>> Have you loaded AggregateImplementation into your table ? >> > > >> > >>>> Can you show us the contents of the following command in >> > > >> > >>>> hbase >> > > >> shell: >> > > >> > >>>> describe 'your-table' >> > > >> > >>>> >> > > >> > >>>> BTW are you using the tip of 0.92 ? >> > > >> > >>>> HBASE-4946 would be of help for dynamically loaded >> > > >> > >>>> coprocessors >> > > >> which >> > > >> > >>>> you might use in the future. >> > > >> > >>>> >> > > >> > >>>> Cheers >> > > >> > >>>> >> > > >> > >>>> On Thu, Dec 22, 2011 at 8:09 AM, Tom Wilcox >> > > >> > >>>> <[email protected]> >> > > >> > >> wrote: >> > > >> > >>>> >> > > >> > >>>>> Hi, >> > > >> > >>>>> >> > > >> > >>>>> We are trying to use the aggregation functionality in >> > > >> > >>>>> HBase >> > > >> > >>>>> 0.92 and we have managed to get the test code working >> > > >> > >>>>> using the >> > > >> following >> > > >> > >> command: >> > > >> > >>>>> >> > > >> > >>>>> java -classpath junit-4.10.jar:build/*:$HBASELIBS/* >> > > >> > >>>>> org.junit.runner.JUnitCore >> > > >> > >>>>> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol >> > > >> > >>>>> >> > > >> > >>>>> Closer inspection of this test class has revealed that it >> > > >> > >>>>> uses a mini DFS cluster to populate and run the tests. >> > > >> > >>>>> These tests return >> > > >> > >> successfully. >> > > >> > >>>>> >> > > >> > >>>>> However, when we attempt to run similar code on our >> > > >> > >>>>> development HDFS cluster we experience the following error: >> > > >> > >>>>> >> > > >> > >>>>> 11/12/22 15:46:28 WARN >> > > >> > >>>>> client.HConnectionManager$HConnectionImplementation: >> > > >> > >>>>> Error executing for >> > > >> > >>>> row >> > > >> > >>>>> java.util.concurrent.ExecutionException: >> > > >> > >>>>> >> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException: >> > > >> > >>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolExcep >> > > >> > >>>>> ti >> > > >> > >>>>> on >> > > >> > >>>>> : No >> > > >> > >>>> matching >> > > >> > >>>>> handler for protocol >> > > >> > >>>> org.apache.hadoop.hbase.coprocessor.AggregateProtocol >> > > >> > >>>>> in region >> > > >> > EDRPTestTbl,,1324485124322.7b9ee0d113db9b24ea9fdde90702d006. >> > > >> > >>>>> at >> > > >> > >>>>> org.apache.hadoop.hbase.regionserver.HRegion.exec(HRegion >> > > >> > >>>>> .j >> > > >> > >>>>> av >> > > >> > >>>>> a:4010 >> > > >> > >>>>> ) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> org.apache.hadoop.hbase.regionserver.HRegionServer.execCoprocesso >> > > >> r( >> > > >> HR >> > > >> > >>>> egionServer.java:3040) >> > > >> > >>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> > > >> > >>>> Method) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. >> > > >> > >>>> java:39) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodA >> > > >> cc >> > > >> es >> > > >> > >>>> sorImpl.java:25) >> > > >> > >>>>> at java.lang.reflect.Method.invoke(Method.java:597) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(Writabl >> > > >> eR >> > > >> pc >> > > >> > >>>> Engine.java:364) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java: >> > > >> > >>>> 1325) >> > > >> > >>>>> [sshexec] >> > > >> > >>>>> at >> > > >> > >>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask. >> > > >> > >>>>> ja >> > > >> > >>>>> va >> > > >> > >>>>> :222) >> > > >> > >>>>> at >> > > >> > >>>>> java.util.concurrent.FutureTask.get(FutureTask.java:83) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl >> > > >> em >> > > >> en >> > > >> > >>>> tation.processExecs(HConnectionManager.java:1465) >> > > >> > >>>>> at >> > > >> > >>>>> >> org.apache.hadoop.hbase.client.HTable.coprocessorExec(HTable. >> > > >> > >>>>> java:1 >> > > >> > >>>>> 555) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> org.apache.hadoop.hbase.client.coprocessor.AggregationClient.sum( >> > > >> Ag >> > > >> gr >> > > >> > >>>> egationClient.java:229) >> > > >> > >>>>> at >> > > >> > >>>>> >> EDRPAggregator.testSumWithValidRange(EDRPAggregator.java:51) >> > > >> > >>>>> at EDRPAggregator.main(EDRPAggregator.java:77) >> > > >> > >>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> > > >> > >>>> Method) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. >> > > >> > >>>> java:39) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodA >> > > >> cc >> > > >> es >> > > >> > >>>> sorImpl.java:25) >> > > >> > >>>>> at java.lang.reflect.Method.invoke(Method.java:597) >> > > >> > >>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156) >> > > >> > >>>>> Caused by: >> > > >> > >>>>> >> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException: >> > > >> > >>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolExcep >> > > >> > >>>>> ti >> > > >> > >>>>> on >> > > >> > >>>>> : No >> > > >> > >>>> matching >> > > >> > >>>>> handler for protocol >> > > >> > >>>> org.apache.hadoop.hbase.coprocessor.AggregateProtocol >> > > >> > >>>>> in region >> > > >> > EDRPTestTbl,,1324485124322.7b9ee0d113db9b24ea9fdde90702d006. >> > > >> > >>>>> at >> > > >> > >>>>> org.apache.hadoop.hbase.regionserver.HRegion.exec(HRegion >> > > >> > >>>>> .j >> > > >> > >>>>> av >> > > >> > >>>>> a:4010 >> > > >> > >>>>> ) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> org.apache.hadoop.hbase.regionserver.HRegionServer.execCoprocesso >> > > >> r( >> > > >> HR >> > > >> > >>>> egionServer.java:3040) >> > > >> > >>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> > > >> > >>>> Method) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. >> > > >> > >>>> java:39) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodA >> > > >> cc >> > > >> es >> > > >> > >>>> sorImpl.java:25) >> > > >> > >>>>> at java.lang.reflect.Method.invoke(Method.java:597) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(Writabl >> > > >> eR >> > > >> pc >> > > >> > >>>> Engine.java:364) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java: >> > > >> > >>>> 1325) >> > > >> > >>>>> [sshexec] >> > > >> > >>>>> at >> > > >> > >>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Na >> > > >> > >>>>> ti >> > > >> > >>>>> ve >> > > >> > >>>>> Method) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConst >> > > >> ru >> > > >> ct >> > > >> > >>>> orAccessorImpl.java:39) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Delegat >> > > >> in >> > > >> gC >> > > >> > >>>> onstructorAccessorImpl.java:27) >> > > >> > >>>>> at >> > > >> > >>>>> java.lang.reflect.Constructor.newInstance(Constructor.java: >> > > >> > >>>>> 51 >> > > >> > >>>>> 3) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteExcept >> > > >> io >> > > >> n( >> > > >> > >>>> RemoteExceptionHandler.java:96) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl >> > > >> em >> > > >> en >> > > >> > >>>> tation.translateException(HConnectionManager.java:1651) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl >> > > >> em >> > > >> en >> > > >> > >>>> tation.getRegionServerWithRetries(HConnectionManager.java: >> > > >> > >>>> 13 >> > > >> > >>>> 27 >> > > >> > >>>> ) >> > > >> > >>>>> at >> > > >> > >>>>> org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(ExecRPC >> > > >> > >>>>> In >> > > >> > >>>>> vo >> > > >> > >>>>> ker.ja >> > > >> > >>>>> va:79) >> > > >> > >>>>> at $Proxy3.getSum(Unknown Source) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.ca >> > > >> ll >> > > >> (A >> > > >> > >>>> ggregationClient.java:233) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.ca >> > > >> ll >> > > >> (A >> > > >> > >>>> ggregationClient.java:230) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl >> > > >> em >> > > >> en >> > > >> > >>>> tation$4.call(HConnectionManager.java:1453) >> > > >> > >>>>> at >> > > >> > >>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask. >> > > >> > >>>>> ja >> > > >> > >>>>> va >> > > >> > >>>>> :303) >> > > >> > >>>>> at >> > > >> > >>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPool >> > > >> Ex >> > > >> ec >> > > >> > >>>> utor.java:886) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExec >> > > >> ut >> > > >> or >> > > >> > >>>> .java:908) >> > > >> > >>>>> at java.lang.Thread.run(Thread.java:662) >> > > >> > >>>>> >> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException: >> > > >> > >>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolExcep >> > > >> > >>>>> ti >> > > >> > >>>>> on >> > > >> > >>>>> : No >> > > >> > >>>> matching >> > > >> > >>>>> handler for protocol >> > > >> > >>>> org.apache.hadoop.hbase.coprocessor.AggregateProtocol >> > > >> > >>>>> in region >> > > >> > EDRPTestTbl,,1324485124322.7b9ee0d113db9b24ea9fdde90702d006. >> > > >> > >>>>> at >> > > >> > >>>>> org.apache.hadoop.hbase.regionserver.HRegion.exec(HRegion >> > > >> > >>>>> .j >> > > >> > >>>>> av >> > > >> > >>>>> a:4010 >> > > >> > >>>>> ) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> org.apache.hadoop.hbase.regionserver.HRegionServer.execCoprocesso >> > > >> r( >> > > >> HR >> > > >> > >>>> egionServer.java:3040) >> > > >> > >>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> > > >> > >>>> Method) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl. >> > > >> > >>>> java:39) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodA >> > > >> cc >> > > >> es >> > > >> > >>>> sorImpl.java:25) >> > > >> > >>>>> at java.lang.reflect.Method.invoke(Method.java:597) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(Writabl >> > > >> eR >> > > >> pc >> > > >> > >>>> Engine.java:364) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java: >> > > >> > >>>> 1325) >> > > >> > >>>>> [sshexec] >> > > >> > >>>>> at >> > > >> > >>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Na >> > > >> > >>>>> ti >> > > >> > >>>>> ve >> > > >> > >>>>> Method) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConst >> > > >> ru >> > > >> ct >> > > >> > >>>> orAccessorImpl.java:39) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Delegat >> > > >> in >> > > >> gC >> > > >> > >>>> onstructorAccessorImpl.java:27) >> > > >> > >>>>> at >> > > >> > >>>>> java.lang.reflect.Constructor.newInstance(Constructor.java: >> > > >> > >>>>> 51 >> > > >> > >>>>> 3) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteExcept >> > > >> io >> > > >> n( >> > > >> > >>>> RemoteExceptionHandler.java:96) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl >> > > >> em >> > > >> en >> > > >> > >>>> tation.translateException(HConnectionManager.java:1651) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl >> > > >> em >> > > >> en >> > > >> > >>>> tation.getRegionServerWithRetries(HConnectionManager.java: >> > > >> > >>>> 13 >> > > >> > >>>> 27 >> > > >> > >>>> ) >> > > >> > >>>>> at >> > > >> > >>>>> org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(ExecRPC >> > > >> > >>>>> In >> > > >> > >>>>> vo >> > > >> > >>>>> ker.ja >> > > >> > >>>>> va:79) >> > > >> > >>>>> at $Proxy3.getSum(Unknown Source) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.ca >> > > >> ll >> > > >> (A >> > > >> > >>>> ggregationClient.java:233) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.ca >> > > >> ll >> > > >> (A >> > > >> > >>>> ggregationClient.java:230) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImpl >> > > >> em >> > > >> en >> > > >> > >>>> tation$4.call(HConnectionManager.java:1453) >> > > >> > >>>>> at >> > > >> > >>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask. >> > > >> > >>>>> ja >> > > >> > >>>>> va >> > > >> > >>>>> :303) >> > > >> > >>>>> at >> > > >> > >>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPool >> > > >> Ex >> > > >> ec >> > > >> > >>>> utor.java:886) >> > > >> > >>>>> at >> > > >> > >>>>> >> > > >> > >>>> >> > > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExec >> > > >> ut >> > > >> or >> > > >> > >>>> .java:908) >> > > >> > >>>>> at java.lang.Thread.run(Thread.java:662) >> > > >> > >>>>> >> > > >> > >>>>> The source for our class is: >> > > >> > >>>>> >> > > >> > >>>>> import static org.junit.Assert.assertEquals; >> > > >> > >>>>> >> > > >> > >>>>> import java.io.IOException; >> > > >> > >>>>> >> > > >> > >>>>> import org.apache.hadoop.conf.Configuration; >> > > >> > >>>>> import org.apache.hadoop.hbase.HBaseConfiguration; >> > > >> > >>>>> import org.apache.hadoop.hbase.HBaseTestingUtility; >> > > >> > >>>>> import org.apache.hadoop.hbase.HConstants; >> > > >> > >>>>> import org.apache.hadoop.hbase.HTableDescriptor; >> > > >> > >>>>> import org.apache.hadoop.hbase.client.HTable; >> > > >> > >>>>> import org.apache.hadoop.hbase.client.Put; >> > > >> > >>>>> import org.apache.hadoop.hbase.client.Scan; >> > > >> > >>>>> import >> > > >> > >>>>> org.apache.hadoop.hbase.client.coprocessor.AggregationCli >> > > >> > >>>>> en >> > > >> > >>>>> t; >> > > >> > >>>>> import >> > > >> > >>>>> org.apache.hadoop.hbase.client.coprocessor.LongColumnInte >> > > >> > >>>>> rp re ter; import org.apache.hadoop.hbase.util.Bytes; >> > > >> > >>>>> import org.apache.hadoop.util.GenericOptionsParser; >> > > >> > >>>>> import >> > > >> > >>>>> org.apache.hadoop.hbase.coprocessor.ColumnInterpreter; >> > > >> > >>>>> import >> > > >> > >>>>> org.apache.hadoop.hbase.coprocessor.CoprocessorHost; >> > > >> > >>>>> import org.junit.Test; >> > > >> > >>>>> >> > > >> > >>>>> public class EDRPAggregator { >> > > >> > >>>>> >> > > >> > >>>>> // private static final byte[] EDRP_FAMILY = >> > > >> > >>>>> Bytes.toBytes("EDRP"); >> > > >> > >>>>> // private static final byte[] EDRP_QUALIFIER = >> > > >> > >>>>> Bytes.toBytes("advanceKWh"); >> > > >> > >>>>> >> > > >> > >>>>> private static byte[] ROW = Bytes.toBytes("testRow"); >> > > >> > >>>>> private static final int ROWSIZE = 20; >> > > >> > >>>>> private static byte[][] ROWS = makeN(ROW, ROWSIZE); >> > > >> > >>>>> private static final byte[] TEST_QUALIFIER = >> > > >> > >>>>> Bytes.toBytes("TestQualifier"); >> > > >> > >>>>> private static final byte[] TEST_MULTI_CQ = >> > > >> > >>>>> Bytes.toBytes("TestMultiCQ"); >> > > >> > >>>>> private static final int rowSeperator1 = 5; >> > > >> > >>>>> private static final int rowSeperator2 = 12; >> > > >> > >>>>> >> > > >> > >>>>> public static void >> > > >> > >>>>> testSumWithValidRange(Configuration >> > > >> > >>>>> conf, >> > > >> > >>>>> String[] otherArgs) throws Throwable { >> > > >> > >>>>> byte[] EDRP_TABLE = >> Bytes.toBytes(otherArgs[1]); >> > > >> > >>>>> byte[] EDRP_FAMILY = >> > > >> > >>>>> Bytes.toBytes(otherArgs[2]); >> > > >> > >>>>> >> > > >> > >>>>> >> > > >> > >>>>> conf.set(CoprocessorHost.REGION_COPROCESSOR_CONF_KEY, >> > > >> > >>>>> >> > > >> > >>>>> >> "org.apache.hadoop.hbase.coprocessor.AggregateImplementation" >> > > >> > >>>>> ); >> > > >> > >>>>> >> > > >> > >>>>> // Utility.CreateHBaseTable(conf, >> > > >> > >>>>> otherArgs[1], >> > > >> > >>>> otherArgs[2], >> > > >> > >>>>> true); >> > > >> > >>>>> >> > > >> > >>>>> HBaseTestingUtility util = new >> > > >> > >>>>> HBaseTestingUtility(); >> > > >> > >>>>> HTable table = util.createTable(EDRP_TABLE, >> > > >> > >>>>> EDRP_FAMILY); >> > > >> > >>>>> >> > > >> > >>>>> AggregationClient aClient = new >> > > >> > >>>>> AggregationClient(conf); >> > > >> > >>>>> Scan scan = new Scan(); >> > > >> > >>>>> scan.addColumn(EDRP_TABLE, EDRP_FAMILY); >> > > >> > >>>>> final ColumnInterpreter<Long, Long> ci = new >> > > >> > >>>>> LongColumnInterpreter(); >> > > >> > >>>>> long sum = >> > > >> > >>>>> aClient.sum(Bytes.toBytes(otherArgs[0]), >> > > >> > >>>>> ci, >> > > >> > >>>> scan); >> > > >> > >>>>> System.out.println(sum); >> > > >> > >>>>> } >> > > >> > >>>>> >> > > >> > >>>>> /** >> > > >> > >>>>> * Main entry point. >> > > >> > >>>>> * >> > > >> > >>>>> * @param argsThe >> > > >> > >>>>> * command line parameters. >> > > >> > >>>>> * @throws Exception >> > > >> > >>>>> * When running the job fails. >> > > >> > >>>>> */ >> > > >> > >>>>> public static void main(String[] args) throws >> > > >> > >>>>> Exception { >> > > >> > >>>>> Configuration conf = >> > > >> > >>>>> HBaseConfiguration.create(); >> > > >> > >>>>> >> > > >> > >>>>> String[] otherArgs = new >> > > >> > >>>>> GenericOptionsParser(conf, >> > > >> > >>>>> args) >> > > >> > >>>>> .getRemainingArgs(); >> > > >> > >>>>> if (otherArgs.length != 3) { >> > > >> > >>>>> System.err >> > > >> > >>>>> .println("Wrong number of >> > > >> > >>>>> arguments: " + otherArgs.length); >> > > >> > >>>>> System.err.println("Usage: " + >> > > >> > >>>>> "<tablename> <colfam> <qualifier>"); >> > > >> > >>>>> System.exit(-1); >> > > >> > >>>>> } >> > > >> > >>>>> >> > > >> > >>>>> try { >> > > >> > >>>>> testSumWithValidRange(conf, otherArgs); >> > > >> > >>>>> } catch (Throwable e) { >> > > >> > >>>>> e.printStackTrace(); >> > > >> > >>>>> } >> > > >> > >>>>> } >> > > >> > >>>>> >> > > >> > >>>>> /** >> > > >> > >>>>> * an infrastructure method to prepare rows for the >> > > testtable. >> > > >> > >>>>> * >> > > >> > >>>>> * @param base >> > > >> > >>>>> * @param n >> > > >> > >>>>> * @return >> > > >> > >>>>> */ >> > > >> > >>>>> private static byte[][] makeN(byte[] base, int n) { >> > > >> > >>>>> byte[][] ret = new byte[n][]; >> > > >> > >>>>> for (int i = 0; i < n; i++) { >> > > >> > >>>>> ret[i] = Bytes.add(base, >> > > >> > >>>>> Bytes.toBytes(i)); >> > > >> > >>>>> } >> > > >> > >>>>> return ret; >> > > >> > >>>>> } >> > > >> > >>>>> } >> > > >> > >>>>> >> > > >> > >>>>> Please can you suggest what might be causing and/or how >> > > >> > >>>>> we might fix this UnknownProtocolException? >> > > >> > >>>>> >> > > >> > >>>>> Also, does anyone have any working examples using the >> > > >> > >>>>> aggregation >> > > >> > >>>> protocol >> > > >> > >>>>> other than the test code? >> > > >> > >>>>> >> > > >> > >>>>> Thanks, >> > > >> > >>>>> Tom >> > > >> > >>>>> >> > > >> > >>>>> >> > > >> > >>>> >> > > >> > >> >> > > >> > >> >> > > >> > >> > > >> > >> > > >> >> > > >> > > >> > >> > >> >> >
