Hi Ted,
Here is the output. As you can see aClient is not nul:
AggregationClient aClient = new AggregationClient(conf);
System.err.println("aClient: "+aClient);
<<< aClient:
org.apache.hadoop.hbase.client.coprocessor.AggregationClient@28787c16
It will take us a little while to add log code to LCI... we have to edit the
source, rebuild 0.92, redistribute round our cluster, restart ;)
We'll get back to you when this is done.
Royston
-----Original Message-----
From: Ted Yu [mailto:[email protected]]
Sent: 03 January 2012 17:10
To: [email protected]
Subject: Re: AggregateProtocol Help
Royston:
Thanks for your effort trying to hunt down the problem.
Can you add a log after this line to see if aClient is null ?
AggregationClient aClient = new AggregationClient(conf);
I was looking at LongColumnInterpreter.add() which is called by
aClient.sum()
Can you add a few log statements in LongColumnInterpreter.add() to see what
parameters are passed to it ?
Cheers
On Tue, Jan 3, 2012 at 8:32 AM, Royston Sellman <
[email protected]> wrote:
> Hi Ted, Himanshu, Gary,
>
> Thanks again for your attention. I experimented with a shorter table
> and it looks like the timeout error was spurious...
>
> With the shorter table I now get an NPE when I call
> AggregationClient.sum().
> Here's the code snippet:
>
> // Test the table
> HTable table = new HTable(EDRP_TABLE);
> Get get = new Get(Bytes.toBytes("row-aa"));
> get.addColumn(Bytes.toBytes("EDRP"),
> Bytes.toBytes("advanceKWh"));
> Result result = table.get(get);
> byte [] val = result.getValue(Bytes.toBytes("EDRP"),
> Bytes.toBytes("advanceKWh"));
> System.out.println("Row aa = " + Bytes.toString(val));
>
> AggregationClient aClient = new AggregationClient(conf);
> Scan scan = new Scan();
> scan.addColumn(EDRP_FAMILY, EDRP_QUALIFIER);
> scan.setStartRow(Bytes.toBytes("row-ab"));
> scan.setStopRow(Bytes.toBytes("row-az"));
> System.out.println(Bytes.toString(EDRP_FAMILY) + ":" +
> Bytes.toString(EDRP_QUALIFIER));
> final ColumnInterpreter<Long, Long> ci = new
> LongColumnInterpreter();
> long sum=-1;
> try {
> sum = aClient.sum(EDRP_TABLE, ci, scan);
> } catch (Throwable e) {
> // TODO Auto-generated catch block
> e.printStackTrace();
> }
> System.out.println(sum);
>
> The first part is just to check that my table is OK. It prints the
> correct value for row aa. Then I check CF:CQ is correct. Then I get
> the -1 that sum was defined as. Then the NPE. Here is the run output:
>
> Row aa = 3.0
> EDRP:advanceKWh
> -1
> java.lang.NullPointerException
> at EDRPAggregator.testSumWithValidRange(EDRPAggregator.java:66)
> at EDRPAggregator.main(EDRPAggregator.java:96)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j
> ava:39
> )
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
> orImpl
> .java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
> Line 66 is the line:
> sum = aClient.sum(EDRP_TABLE, ci, scan);
>
> But I'm not sure that is really the line generating the NPE because I
> get the -1 output (same result when line 66 is not wrapped in try-catch
BTW).
> Anyway, -1 is clearly not the correct value for the sum.
>
> [Note: I'm setting a start and end row on the scan because I saw a
> comment in AggregationClient.java saying they must be set. However, I
> think this comment is obsolete as the test code in
> TestAggregateProtocol.java sometimes does not include start & end
> rows. The run is exactly the same without the start & stop rows ]
>
> We are struggling with Aggregations not least because
> TestAggregateProtocol.java is the only source of example code we could
> find and it is not ideal because it is designed to work with
> MiniCluster rather than a real cluster like ours. Is there any other
> example code in existence?
> I think that would really help us.
>
> Meanwhile, thanks for your patience with our questions. We are really
> impressed with the AggregationProtocol architecture/functionality and
> very keen to use it.
>
> Royston (and Tom)
>
> -----Original Message-----
> From: Gary Helmling [mailto:[email protected]]
> Sent: 02 January 2012 06:23
> To: [email protected]
> Subject: Re: AggregateProtocol Help
>
> Hi Royston,
>
> Try increasing the value set for hbase.rpc.timeout (by default 60
seconds).
> Add something like this to hbase-site.xml:
>
> <property>
> <name>hbase.rpc.timeout</name>
> <value>180000</value>
> </property>
>
> This would increase the timeout value to 3 minutes, for example.
>
> But as Ted and Himanshu mention, 557k rows is not a large table and
> should not be taking very long to process, unless these happen to be
> very, very wide rows. It sounds like there could be something else
> going on. Does debug level logging show anything else in the region
server log?
>
> --gh
>
> On Sun, Jan 1, 2012 at 5:53 PM, Ted Yu <[email protected]> wrote:
> > Thanks for the reminder Himanshu.
> >
> > Royston:
> > From this blog you can get some history on this subject:
> > http://zhihongyu.blogspot.com/2011/03/genericizing-endpointcoprocess
> > or
> > .html
> >
> >
> > On Sun, Jan 1, 2012 at 5:18 PM, Himanshu Vashishtha
> > <[email protected]
> >> wrote:
> >
> >> Hello Royston,
> >>
> >> Sorry to hear that you are getting trouble while using Aggregation
> >> functionalities.
> >>
> >> 557k rows seems to be a small table and a SocketTimeout does not
> >> seem to be an ok response.
> >> It will be good to know the region distribution as such. (how many
> regions?
> >> Is it a full table scan?)
> >>
> >> You are using the sum function; how are you using the
ColumnInterpreter.
> >> Can you enable the log level to debug to see why the RS is taking
> >> that long to respond (more than 113 sec).
> >> The 0 return value is the default result.
> >>
> >> Thanks for trying this out.
> >>
> >> Thanks,
> >> Himanshu
> >>
> >> On Sun, Jan 1, 2012 at 12:26 PM, Royston Sellman <
> >> [email protected]> wrote:
> >>
> >> > Hi Ted,
> >> >
> >> > I think 0 is the only value we ever see (I'll check tomorrow: the
> >> > server is down right now). Our table has 557,000 rows. I'll try a
> >> > much shorter table tomorrow.
> >> >
> >> > Yes, we have RS running on the NN, but it's a test cluster and we
> >> > are
> >> used
> >> > to it :)
> >> >
> >> > Do you think using AggregationProtocol is the best strategy for
> >> > the case where we want to use basic SQL-style functions like SUM,
> >> > AVG, STD, MIN, MAX? Do you think there is a better strategy?
> >> >
> >> > Many thanks,
> >> > Royston
> >> >
> >> >
> >> > On 1 Jan 2012, at 17:58, Ted Yu wrote:
> >> >
> >> > > Royston:
> >> > > Happy New Year to you too.
> >> > >
> >> > >>> java.net.SocketTimeoutException: Call to
> >> > >>> namenode/10.0.0.235:60020
> >> failed
> >> > on
> >> > >
> >> > > It seems the namenode above actually refers to a region server.
> >> > > This
> >> is a
> >> > > little bit confusing :-)
> >> > >
> >> > > The sum value below is 0.
> >> > > Have you ever seen a value greater than 0 ?
> >> > >
> >> > > How many rows are there in this CF:CQ ?
> >> > > The timeout was reported earlier by other people where there're
> >> > > many
> >> rows
> >> > > in the table.
> >> > >
> >> > > There is a JIRA to provide streaming support for coprocessor
> >> > > but the development there has stalled.
> >> > >
> >> > > Cheers
> >> > >
> >> > > On Sun, Jan 1, 2012 at 9:35 AM, Royston Sellman <
> >> > > [email protected]> wrote:
> >> > >
> >> > >> Hi Gary and Ted,
> >> > >>
> >> > >> Royston (Tom's colleague) here. Back onto this after the
> >> > >> Christmas/New
> >> > Year
> >> > >> break.
> >> > >>
> >> > >> Many thanks for your help so far. We enabled our database via
> >> > >> your hbase-site.xml mod and were able to move on. to other
> >> > >> errors. But I
> >> > think
> >> > >> we
> >> > >> are now actually getting an aggregation partially calculated
> >> > >> on our
> >> > table
> >> > >> (this feels like progress). The details:
> >> > >>
> >> > >> On running our client we now get this exception:
> >> > >> 11/12/31 17:51:09 WARN
> >> > >> client.HConnectionManager$HConnectionImplementation: Error
> >> > >> executing
> >> for
> >> > >> row
> >> > >>
> >> > >> java.util.concurrent.ExecutionException:
> >> > >> org.apache.hadoop.hbase.client.RetriesExhaustedException:
> >> > >> Failed after attempts=10, exceptions:
> >> > >> Sat Dec 31 17:41:30 GMT 2011,
> >> > >> org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@1fc4f0f8,
> >> > >> java.net.SocketTimeoutException: Call to
> >> > >> namenode/10.0.0.235:60020
> >> failed
> >> > >> on
> >> > >> socket timeout exception: java.net.SocketTimeoutException:
> >> > >> 60000
> >> millis
> >> > >> timeout while waiting for channel to be ready for read. ch :
> >> > >> java.nio.channels.SocketChannel[connected
> >> > >> local=/10.0.0.235:59999 remote=namenode/10.0.0.235:60020]
> >> > >> (8 more of these, making for 10 tries) Sat Dec 31 17:51:09 GMT
> >> > >> 2011, org.apache.hadoop.hbase.ipc.ExecRPCInvoker$1@1fc4f0f8,
> >> > >> java.net.SocketTimeoutException: Call to
> >> > >> namenode/10.0.0.235:60020
> >> failed
> >> > >> on
> >> > >> socket timeout exception: java.net.SocketTimeoutException:
> >> > >> 60000
> >> millis
> >> > >> timeout while waiting for channel to be ready for read. ch :
> >> > >> java.nio.channels.SocketChannel[connected
> >> > >> local=/10.0.0.235:59364 remote=namenode/10.0.0.235:60020]
> >> > >>
> >> > >> at
> >> > >> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:
> >> > >> 22
> >> > >> 2)
> >> > >> at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> >> > >> at
> >> > >>
> >> > >>
> >> >
> >>
>
>
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.
> >> > >> processExecs(HConnectionManager.java:1465)
> >> > >> at
> >> > >>
> >> org.apache.hadoop.hbase.client.HTable.coprocessorExec(HTable.java:1
> >> 55
> >> 5)
> >> > >> at
> >> > >>
> >> > >>
> >> >
> >> org.apache.hadoop.hbase.client.coprocessor.AggregationClient.sum(Ag
> >> gr
> >> egation
> >> > >> Client.java:229)
> >> > >> at
> >> > >> EDRPAggregator.testSumWithValidRange(EDRPAggregator.java:51)
> >> > >> at EDRPAggregator.main(EDRPAggregator.java:77)
> >> > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> > >> Method)
> >> > >> at
> >> > >>
> >> > >>
> >> >
> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
> >> java:39
> >> > >> )
> >> > >> at
> >> > >>
> >> > >>
> >> >
> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcc
> >> es
> >> sorImpl
> >> > >> .java:25)
> >> > >> at java.lang.reflect.Method.invoke(Method.java:597)
> >> > >> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> >> > >>
> >> > >>
> >> > >> Looking at the log (.regionserver-namenode.log) I see this
> >> > >> debug
> >> > message:
> >> > >>
> >> > >> 2011-12-31 17:42:23,472 DEBUG
> >> > >> org.apache.hadoop.hbase.coprocessor.AggregateImplementation:
> >> > >> Sum from
> >> > this
> >> > >> region is
> >> EDRPTestTbl,,1324485124322.7b9ee0d113db9b24ea9fdde90702d006.:
> >> > 0
> >> > >>
> >> > >> Where the sum value looks reasonable which makes me think the
> >> > >> sum of a CF:CQ worked. But I never see this value on stdout.
> >> > >> Then I see this warning:
> >> > >>
> >> > >> 2011-12-31 17:42:23,476 WARN org.apache.hadoop.ipc.HBaseServer:
> >> > >> (responseTooSlow):
> >> > >> {"processingtimems":113146,"call":"execCoprocess$
> >> > >> 2011-12-31 17:42:23,511 WARN org.apache.hadoop.ipc.HBaseServer:
> >> > >> IPC
> >> > Server
> >> > >> Responder, call execCoprocessor([B@4b22fad6, getSum(org.$
> >> > >> 2011-12-31 17:42:23,515 WARN org.apache.hadoop.ipc.HBaseServer:
> >> > >> IPC
> >> > Server
> >> > >> handler 1 on 60020 caught: java.nio.channels.ClosedChann$
> >> > >> at
> >> > >>
> >> sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java
> >> :1
> >> 33)
> >> > >> at
> >> sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
> >> > >> at
> >> > >>
> >> >
> >> org.apache.hadoop.hbase.ipc.HBaseServer.channelWrite(HBaseServer.ja
> >> va
> >> :1651)
> >> > >> at
> >> > >>
> >> > >>
> >> >
> >> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.processResponse(H
> >> Ba
> >> seServe
> >> > >> r.java:924)
> >> > >> at
> >> > >>
> >> > >>
> >> >
> >> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.doRespond(HBaseSe
> >> rv
> >> er.java
> >> > >> :1003)
> >> > >> at
> >> > >>
> >> > >>
> >> >
> >> org.apache.hadoop.hbase.ipc.HBaseServer$Call.sendResponseIfReady(HB
> >> as
> >> eServer
> >> > >> .java:409)
> >> > >> at
> >> > >>
> >> >
> >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:
> >> 1345)
> >> > >>
> >> > >> Have we missed out some step in the HBase/RegionServerconfig?
> >> > >> Or is
> >> our
> >> > >> client code still deficient?
> >> > >>
> >> > >> Can you offer any suggestions? Is there any example code for
> >> > >> the new Aggregations stuff.
> >> > >>
> >> > >> Thanks and Happy New Year to you guys,
> >> > >>
> >> > >> Royston (and Tom).
> >> > >>
> >> > >> (HBase 0.92, Hadoop 1.0)
> >> > >>
> >> > >>
> >> > >> -----Original Message-----
> >> > >> From: Gary Helmling [mailto:[email protected]]
> >> > >> Sent: 23 December 2011 18:06
> >> > >> To: [email protected]
> >> > >> Subject: Re: AggregateProtocol Help
> >> > >>
> >> > >> Hi Tom,
> >> > >>
> >> > >> The test code is not really the best guide for configuration.
> >> > >>
> >> > >> To enable the AggregateProtocol on all of your tables, add
> >> > >> this to the hbase-site.xml for the servers in your cluster:
> >> > >>
> >> > >> <property>
> >> > >> <name>hbase.coprocessor.user.region.classes</name>
> >> > >>
> >> > >>
> >> >
> >> <value>org.apache.hadoop.hbase.coprocessor.AggregateImplementation<
> >> /v
> >> alue>
> >> > >> </property>
> >> > >>
> >> > >> If you only want to use the aggregate functions on a specific
> >> > >> table
> >> (or
> >> > >> tables), then you can enable that individually for the table
> >> > >> from the
> >> > >> shell:
> >> > >>
> >> > >> 1) disable the table
> >> > >> hbase> disable 'EDRP7'
> >> > >>
> >> > >> 2) add the coprocessor
> >> > >> hbase> alter 'EDRP7', METHOD => 'table_att',
> >> > >>
> >> > >>
> >> > >>
> >> >
> >> 'coprocessor'=>'|org.apache.hadoop.hbase.coprocessor.AggregateImple
> >> me
> >> ntation
> >> > >> ||'
> >> > >>
> >> > >> (Note that the pipes in the value string are required)
> >> > >>
> >> > >> 3) re-enable the table
> >> > >> hbase> enable 'EDRP7'
> >> > >>
> >> > >>
> >> > >> Either way should work. With the second approach you will see
> >> > >> the coprocessor listed when you describe the table from the
> >> > >> shell, as Ted mentioned. With the first approach you will
> >> > >> not, but it should be
> >> > loaded
> >> > >> all the same.
> >> > >>
> >> > >> --gh
> >> > >>
> >> > >>
> >> > >> On Fri, Dec 23, 2011 at 7:04 AM, Ted Yu <[email protected]>
> wrote:
> >> > >>> I don't know why you chose HBaseTestingUtility to create the
> table.
> >> > >>> I guess you followed test code example.
> >> > >>>
> >> > >>> At least you should pass the conf to this ctor:
> >> > >>> public HBaseTestingUtility(Configuration conf) {
> >> > >>>
> >> > >>> If coprocessor was installed correctly, you should see
> >> > >>> something like(from
> >> > >>> HBASE-5070):
> >> > >>> coprocessor$1 =>
> >> > >>>
> '|org.apache.hadoop.hbase.constraint.ConstraintProcessor|1073741823|'
> >> > >>>
> >> > >>> Cheers
> >> > >>>
> >> > >>> On Fri, Dec 23, 2011 at 3:02 AM, Tom Wilcox
> >> > >>> <[email protected]>
> >> > >> wrote:
> >> > >>>
> >> > >>>> Hi,
> >> > >>>>
> >> > >>>> I am not sure how we load the AggregateImplementation into
> >> > >>>> the
> >> table.
> >> > >>>> When we are creating a table, we use the same functions as
> >> > >>>> the test
> >> as
> >> > >> follows...
> >> > >>>>
> >> > >>>> ...
> >> > >>>>>
> >> > >>>>> conf.set(CoprocessorHost.REGION_COPROCESSOR_CONF_KEY,
> >> > >>>>>
> >> > >>>>> "org.apache.hadoop.hbase.coprocessor.AggregateImplementation"
> >> > >>>>> );
> >> > >>>>>
> >> > >>>>> // Utility.CreateHBaseTable(conf, otherArgs[1],
> >> > >>>> otherArgs[2],
> >> > >>>>> true);
> >> > >>>>>
> >> > >>>>> HBaseTestingUtility util = new
> >> > >>>>> HBaseTestingUtility();
> >> > >>>>> HTable table = util.createTable(EDRP_TABLE,
> >> > >>>>> EDRP_FAMILY);
> >> > >>>>>
> >> > >>>>> AggregationClient aClient = new
> >> > >>>>> AggregationClient(conf);
> >> > >>>> ...
> >> > >>>>
> >> > >>>> Running DESCRIBE on a table produced shows the following output:
> >> > >>>>
> >> > >>>> hbase(main):002:0> describe 'EDRP7'
> >> > >>>> DESCRIPTION
> >> > >>>> ENABLED {NAME =>
> >> > >>>> 'EDRP7', FAMILIES => [{NAME => 'advanceKWh', BLOOMFILTER =>
> >> > >>>> 'NONE', REPLICATION_SCOPE => '0', VERSIONS => true '3',
> >> > >>>> COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL =>
> >> > >>>> '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', B
> >> > >>>> LOCKCACHE => 'true'}]}
> >> > >>>>
> >> > >>>> We are using the tip of 0.92 (cloned from the Git repo). See
> >> > >>>> the version string below:
> >> > >>>>
> >> > >>>> hbase(main):005:0> version
> >> > >>>> 0.92.0, r1208286, Thu Dec 15 13:16:03 GMT 2011
> >> > >>>>
> >> > >>>> We would really appreciate an example of how to create a
> >> > >>>> table that is enabled to handle Aggregation).
> >> > >>>>
> >> > >>>> Thanks
> >> > >>>>
> >> > >>>>
> >> > >>>> ________________________________________
> >> > >>>> From: Ted Yu [[email protected]]
> >> > >>>> Sent: 22 December 2011 17:03
> >> > >>>> To: [email protected]
> >> > >>>> Subject: Re: AggregateProtocol Help
> >> > >>>>
> >> > >>>> Have you loaded AggregateImplementation into your table ?
> >> > >>>> Can you show us the contents of the following command in
> >> > >>>> hbase
> >> shell:
> >> > >>>> describe 'your-table'
> >> > >>>>
> >> > >>>> BTW are you using the tip of 0.92 ?
> >> > >>>> HBASE-4946 would be of help for dynamically loaded
> >> > >>>> coprocessors
> >> which
> >> > >>>> you might use in the future.
> >> > >>>>
> >> > >>>> Cheers
> >> > >>>>
> >> > >>>> On Thu, Dec 22, 2011 at 8:09 AM, Tom Wilcox
> >> > >>>> <[email protected]>
> >> > >> wrote:
> >> > >>>>
> >> > >>>>> Hi,
> >> > >>>>>
> >> > >>>>> We are trying to use the aggregation functionality in HBase
> >> > >>>>> 0.92 and we have managed to get the test code working using
> >> > >>>>> the
> >> following
> >> > >> command:
> >> > >>>>>
> >> > >>>>> java -classpath junit-4.10.jar:build/*:$HBASELIBS/*
> >> > >>>>> org.junit.runner.JUnitCore
> >> > >>>>> org.apache.hadoop.hbase.coprocessor.TestAggregateProtocol
> >> > >>>>>
> >> > >>>>> Closer inspection of this test class has revealed that it
> >> > >>>>> uses a mini DFS cluster to populate and run the tests.
> >> > >>>>> These tests return
> >> > >> successfully.
> >> > >>>>>
> >> > >>>>> However, when we attempt to run similar code on our
> >> > >>>>> development HDFS cluster we experience the following error:
> >> > >>>>>
> >> > >>>>> 11/12/22 15:46:28 WARN
> >> > >>>>> client.HConnectionManager$HConnectionImplementation: Error
> >> > >>>>> executing for
> >> > >>>> row
> >> > >>>>> java.util.concurrent.ExecutionException:
> >> > >>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException:
> >> > >>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolExcepti
> >> > >>>>> on
> >> > >>>>> : No
> >> > >>>> matching
> >> > >>>>> handler for protocol
> >> > >>>> org.apache.hadoop.hbase.coprocessor.AggregateProtocol
> >> > >>>>> in region
> >> > EDRPTestTbl,,1324485124322.7b9ee0d113db9b24ea9fdde90702d006.
> >> > >>>>> at
> >> > >>>>> org.apache.hadoop.hbase.regionserver.HRegion.exec(HRegion.j
> >> > >>>>> av
> >> > >>>>> a:4010
> >> > >>>>> )
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> org.apache.hadoop.hbase.regionserver.HRegionServer.execCoprocessor(
> >> HR
> >> > >>>> egionServer.java:3040)
> >> > >>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> > >>>> Method)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
> >> > >>>> java:39)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcc
> >> es
> >> > >>>> sorImpl.java:25)
> >> > >>>>> at java.lang.reflect.Method.invoke(Method.java:597)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableR
> >> pc
> >> > >>>> Engine.java:364)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:
> >> > >>>> 1325)
> >> > >>>>> [sshexec]
> >> > >>>>> at
> >> > >>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.ja
> >> > >>>>> va
> >> > >>>>> :222)
> >> > >>>>> at
> >> > >>>>> java.util.concurrent.FutureTask.get(FutureTask.java:83)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplem
> >> en
> >> > >>>> tation.processExecs(HConnectionManager.java:1465)
> >> > >>>>> at
> >> > >>>>> org.apache.hadoop.hbase.client.HTable.coprocessorExec(HTable.
> >> > >>>>> java:1
> >> > >>>>> 555)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> org.apache.hadoop.hbase.client.coprocessor.AggregationClient.sum(Ag
> >> gr
> >> > >>>> egationClient.java:229)
> >> > >>>>> at
> >> > >>>>> EDRPAggregator.testSumWithValidRange(EDRPAggregator.java:51)
> >> > >>>>> at EDRPAggregator.main(EDRPAggregator.java:77)
> >> > >>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> > >>>> Method)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
> >> > >>>> java:39)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcc
> >> es
> >> > >>>> sorImpl.java:25)
> >> > >>>>> at java.lang.reflect.Method.invoke(Method.java:597)
> >> > >>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> >> > >>>>> Caused by:
> >> > >>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException:
> >> > >>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolExcepti
> >> > >>>>> on
> >> > >>>>> : No
> >> > >>>> matching
> >> > >>>>> handler for protocol
> >> > >>>> org.apache.hadoop.hbase.coprocessor.AggregateProtocol
> >> > >>>>> in region
> >> > EDRPTestTbl,,1324485124322.7b9ee0d113db9b24ea9fdde90702d006.
> >> > >>>>> at
> >> > >>>>> org.apache.hadoop.hbase.regionserver.HRegion.exec(HRegion.j
> >> > >>>>> av
> >> > >>>>> a:4010
> >> > >>>>> )
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> org.apache.hadoop.hbase.regionserver.HRegionServer.execCoprocessor(
> >> HR
> >> > >>>> egionServer.java:3040)
> >> > >>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> > >>>> Method)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
> >> > >>>> java:39)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcc
> >> es
> >> > >>>> sorImpl.java:25)
> >> > >>>>> at java.lang.reflect.Method.invoke(Method.java:597)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableR
> >> pc
> >> > >>>> Engine.java:364)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:
> >> > >>>> 1325)
> >> > >>>>> [sshexec]
> >> > >>>>> at
> >> > >>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Nati
> >> > >>>>> ve
> >> > >>>>> Method)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstru
> >> ct
> >> > >>>> orAccessorImpl.java:39)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Delegatin
> >> gC
> >> > >>>> onstructorAccessorImpl.java:27)
> >> > >>>>> at
> >> > >>>>> java.lang.reflect.Constructor.newInstance(Constructor.java:
> >> > >>>>> 51
> >> > >>>>> 3)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteExceptio
> >> n(
> >> > >>>> RemoteExceptionHandler.java:96)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplem
> >> en
> >> > >>>> tation.translateException(HConnectionManager.java:1651)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplem
> >> en
> >> > >>>> tation.getRegionServerWithRetries(HConnectionManager.java:13
> >> > >>>> 27
> >> > >>>> )
> >> > >>>>> at
> >> > >>>>> org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(ExecRPCIn
> >> > >>>>> vo
> >> > >>>>> ker.ja
> >> > >>>>> va:79)
> >> > >>>>> at $Proxy3.getSum(Unknown Source)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.call
> >> (A
> >> > >>>> ggregationClient.java:233)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.call
> >> (A
> >> > >>>> ggregationClient.java:230)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplem
> >> en
> >> > >>>> tation$4.call(HConnectionManager.java:1453)
> >> > >>>>> at
> >> > >>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.ja
> >> > >>>>> va
> >> > >>>>> :303)
> >> > >>>>> at
> >> > >>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolEx
> >> ec
> >> > >>>> utor.java:886)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut
> >> or
> >> > >>>> .java:908)
> >> > >>>>> at java.lang.Thread.run(Thread.java:662)
> >> > >>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolException:
> >> > >>>>> org.apache.hadoop.hbase.ipc.HBaseRPC$UnknownProtocolExcepti
> >> > >>>>> on
> >> > >>>>> : No
> >> > >>>> matching
> >> > >>>>> handler for protocol
> >> > >>>> org.apache.hadoop.hbase.coprocessor.AggregateProtocol
> >> > >>>>> in region
> >> > EDRPTestTbl,,1324485124322.7b9ee0d113db9b24ea9fdde90702d006.
> >> > >>>>> at
> >> > >>>>> org.apache.hadoop.hbase.regionserver.HRegion.exec(HRegion.j
> >> > >>>>> av
> >> > >>>>> a:4010
> >> > >>>>> )
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> org.apache.hadoop.hbase.regionserver.HRegionServer.execCoprocessor(
> >> HR
> >> > >>>> egionServer.java:3040)
> >> > >>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> > >>>> Method)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
> >> > >>>> java:39)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcc
> >> es
> >> > >>>> sorImpl.java:25)
> >> > >>>>> at java.lang.reflect.Method.invoke(Method.java:597)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableR
> >> pc
> >> > >>>> Engine.java:364)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:
> >> > >>>> 1325)
> >> > >>>>> [sshexec]
> >> > >>>>> at
> >> > >>>>> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Nati
> >> > >>>>> ve
> >> > >>>>> Method)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstru
> >> ct
> >> > >>>> orAccessorImpl.java:39)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Delegatin
> >> gC
> >> > >>>> onstructorAccessorImpl.java:27)
> >> > >>>>> at
> >> > >>>>> java.lang.reflect.Constructor.newInstance(Constructor.java:
> >> > >>>>> 51
> >> > >>>>> 3)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteExceptio
> >> n(
> >> > >>>> RemoteExceptionHandler.java:96)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplem
> >> en
> >> > >>>> tation.translateException(HConnectionManager.java:1651)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplem
> >> en
> >> > >>>> tation.getRegionServerWithRetries(HConnectionManager.java:13
> >> > >>>> 27
> >> > >>>> )
> >> > >>>>> at
> >> > >>>>> org.apache.hadoop.hbase.ipc.ExecRPCInvoker.invoke(ExecRPCIn
> >> > >>>>> vo
> >> > >>>>> ker.ja
> >> > >>>>> va:79)
> >> > >>>>> at $Proxy3.getSum(Unknown Source)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.call
> >> (A
> >> > >>>> ggregationClient.java:233)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> org.apache.hadoop.hbase.client.coprocessor.AggregationClient$4.call
> >> (A
> >> > >>>> ggregationClient.java:230)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplem
> >> en
> >> > >>>> tation$4.call(HConnectionManager.java:1453)
> >> > >>>>> at
> >> > >>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.ja
> >> > >>>>> va
> >> > >>>>> :303)
> >> > >>>>> at
> >> > >>>>> java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolEx
> >> ec
> >> > >>>> utor.java:886)
> >> > >>>>> at
> >> > >>>>>
> >> > >>>>
> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut
> >> or
> >> > >>>> .java:908)
> >> > >>>>> at java.lang.Thread.run(Thread.java:662)
> >> > >>>>>
> >> > >>>>> The source for our class is:
> >> > >>>>>
> >> > >>>>> import static org.junit.Assert.assertEquals;
> >> > >>>>>
> >> > >>>>> import java.io.IOException;
> >> > >>>>>
> >> > >>>>> import org.apache.hadoop.conf.Configuration;
> >> > >>>>> import org.apache.hadoop.hbase.HBaseConfiguration;
> >> > >>>>> import org.apache.hadoop.hbase.HBaseTestingUtility;
> >> > >>>>> import org.apache.hadoop.hbase.HConstants;
> >> > >>>>> import org.apache.hadoop.hbase.HTableDescriptor;
> >> > >>>>> import org.apache.hadoop.hbase.client.HTable;
> >> > >>>>> import org.apache.hadoop.hbase.client.Put;
> >> > >>>>> import org.apache.hadoop.hbase.client.Scan;
> >> > >>>>> import
> >> > >>>>> org.apache.hadoop.hbase.client.coprocessor.AggregationClien
> >> > >>>>> t;
> >> > >>>>> import
> >> > >>>>> org.apache.hadoop.hbase.client.coprocessor.LongColumnInterp
> >> > >>>>> re ter; import org.apache.hadoop.hbase.util.Bytes;
> >> > >>>>> import org.apache.hadoop.util.GenericOptionsParser;
> >> > >>>>> import
> >> > >>>>> org.apache.hadoop.hbase.coprocessor.ColumnInterpreter;
> >> > >>>>> import org.apache.hadoop.hbase.coprocessor.CoprocessorHost;
> >> > >>>>> import org.junit.Test;
> >> > >>>>>
> >> > >>>>> public class EDRPAggregator {
> >> > >>>>>
> >> > >>>>> // private static final byte[] EDRP_FAMILY =
> >> > >>>>> Bytes.toBytes("EDRP");
> >> > >>>>> // private static final byte[] EDRP_QUALIFIER =
> >> > >>>>> Bytes.toBytes("advanceKWh");
> >> > >>>>>
> >> > >>>>> private static byte[] ROW = Bytes.toBytes("testRow");
> >> > >>>>> private static final int ROWSIZE = 20;
> >> > >>>>> private static byte[][] ROWS = makeN(ROW, ROWSIZE);
> >> > >>>>> private static final byte[] TEST_QUALIFIER =
> >> > >>>>> Bytes.toBytes("TestQualifier");
> >> > >>>>> private static final byte[] TEST_MULTI_CQ =
> >> > >>>>> Bytes.toBytes("TestMultiCQ");
> >> > >>>>> private static final int rowSeperator1 = 5;
> >> > >>>>> private static final int rowSeperator2 = 12;
> >> > >>>>>
> >> > >>>>> public static void testSumWithValidRange(Configuration
> >> > >>>>> conf,
> >> > >>>>> String[] otherArgs) throws Throwable {
> >> > >>>>> byte[] EDRP_TABLE = Bytes.toBytes(otherArgs[1]);
> >> > >>>>> byte[] EDRP_FAMILY =
> >> > >>>>> Bytes.toBytes(otherArgs[2]);
> >> > >>>>>
> >> > >>>>>
> >> > >>>>> conf.set(CoprocessorHost.REGION_COPROCESSOR_CONF_KEY,
> >> > >>>>>
> >> > >>>>> "org.apache.hadoop.hbase.coprocessor.AggregateImplementation"
> >> > >>>>> );
> >> > >>>>>
> >> > >>>>> // Utility.CreateHBaseTable(conf, otherArgs[1],
> >> > >>>> otherArgs[2],
> >> > >>>>> true);
> >> > >>>>>
> >> > >>>>> HBaseTestingUtility util = new
> >> > >>>>> HBaseTestingUtility();
> >> > >>>>> HTable table = util.createTable(EDRP_TABLE,
> >> > >>>>> EDRP_FAMILY);
> >> > >>>>>
> >> > >>>>> AggregationClient aClient = new
> >> > >>>>> AggregationClient(conf);
> >> > >>>>> Scan scan = new Scan();
> >> > >>>>> scan.addColumn(EDRP_TABLE, EDRP_FAMILY);
> >> > >>>>> final ColumnInterpreter<Long, Long> ci = new
> >> > >>>>> LongColumnInterpreter();
> >> > >>>>> long sum =
> >> > >>>>> aClient.sum(Bytes.toBytes(otherArgs[0]),
> >> > >>>>> ci,
> >> > >>>> scan);
> >> > >>>>> System.out.println(sum);
> >> > >>>>> }
> >> > >>>>>
> >> > >>>>> /**
> >> > >>>>> * Main entry point.
> >> > >>>>> *
> >> > >>>>> * @param argsThe
> >> > >>>>> * command line parameters.
> >> > >>>>> * @throws Exception
> >> > >>>>> * When running the job fails.
> >> > >>>>> */
> >> > >>>>> public static void main(String[] args) throws
> >> > >>>>> Exception {
> >> > >>>>> Configuration conf =
> >> > >>>>> HBaseConfiguration.create();
> >> > >>>>>
> >> > >>>>> String[] otherArgs = new
> >> > >>>>> GenericOptionsParser(conf,
> >> > >>>>> args)
> >> > >>>>> .getRemainingArgs();
> >> > >>>>> if (otherArgs.length != 3) {
> >> > >>>>> System.err
> >> > >>>>> .println("Wrong number of
> >> > >>>>> arguments: " + otherArgs.length);
> >> > >>>>> System.err.println("Usage: " +
> >> > >>>>> "<tablename> <colfam> <qualifier>");
> >> > >>>>> System.exit(-1);
> >> > >>>>> }
> >> > >>>>>
> >> > >>>>> try {
> >> > >>>>> testSumWithValidRange(conf, otherArgs);
> >> > >>>>> } catch (Throwable e) {
> >> > >>>>> e.printStackTrace();
> >> > >>>>> }
> >> > >>>>> }
> >> > >>>>>
> >> > >>>>> /**
> >> > >>>>> * an infrastructure method to prepare rows for the
> testtable.
> >> > >>>>> *
> >> > >>>>> * @param base
> >> > >>>>> * @param n
> >> > >>>>> * @return
> >> > >>>>> */
> >> > >>>>> private static byte[][] makeN(byte[] base, int n) {
> >> > >>>>> byte[][] ret = new byte[n][];
> >> > >>>>> for (int i = 0; i < n; i++) {
> >> > >>>>> ret[i] = Bytes.add(base,
> >> > >>>>> Bytes.toBytes(i));
> >> > >>>>> }
> >> > >>>>> return ret;
> >> > >>>>> }
> >> > >>>>> }
> >> > >>>>>
> >> > >>>>> Please can you suggest what might be causing and/or how we
> >> > >>>>> might fix this UnknownProtocolException?
> >> > >>>>>
> >> > >>>>> Also, does anyone have any working examples using the
> >> > >>>>> aggregation
> >> > >>>> protocol
> >> > >>>>> other than the test code?
> >> > >>>>>
> >> > >>>>> Thanks,
> >> > >>>>> Tom
> >> > >>>>>
> >> > >>>>>
> >> > >>>>
> >> > >>
> >> > >>
> >> >
> >> >
> >>
>
>
