.679495.n3.nabble.com/Endpoint-coprocessor-performance-issue-tp4083614.html
Sent from the HBase User mailing list archive at Nabble.com.
/TestAggregateProtocol.java
Cheers
On Thu, May 7, 2015 at 11:27 PM, Gupta, Kunal IN BLR STS <
kunal.gu...@siemens.com> wrote:
> Hi,
>
> I want to implement EndPoint Coprocessor. From the link (
> http://www.3pillarglobal.com/insights/hbase-coprocessors) I created my
> own files as
Hi,
I want to implement EndPoint Coprocessor. From the link
(http://www.3pillarglobal.com/insights/hbase-coprocessors) I created my own
files as specified in the link. In the link they worked on Sum but I want to
have Average along with Filters so I did according to what they have specified
t; 'cf'
Thanks,
Chandra
-Original Message-
From: Ted Yu [mailto:yuzhih...@gmail.com]
Sent: Thursday, April 10, 2014 5:36 PM
To: user@hbase.apache.org
Cc: user@hbase.apache.org
Subject: Re: endpoint coprocessor
Here is a reference implementation for aggregation :
h
an think of Endpoint Coprocessor as stored procedures in SQL DBs.
> In order to use an Endpoint Coprocessor you need to build and install it on
> the server side and then it invoke through HBase RPC.
>
> You can make use of
> https://blogs.apache.org/hbase/entry/coprocessor_introduction
You can think of Endpoint Coprocessor as stored procedures in SQL DBs.
In order to use an Endpoint Coprocessor you need to build and install it on
the server side and then it invoke through HBase RPC.
You can make use of
https://blogs.apache.org/hbase/entry/coprocessor_introduction.
2014-04-14
se classes and calling mechanism not available from hbase
> shell. Then planning to use java client code to invoke coprocessor.
>
> Any pointers to java client to invoke aggregation<
> http://search-hadoop.com/c/HBase:hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/Aggr
from hbase shell.
Then planning to use java client code to invoke coprocessor.
Any pointers to java client to invoke
aggregation<http://search-hadoop.com/c/HBase:hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/AggregateImplementation.java||Hbase+aggregation+endpoint>
coprocessor will
12:10 PM
> To: user@hbase.apache.org
> Subject: Re: endpoint coprocessor
>
> Bear in mind each region will return its top n, then you will have to run
> another top n in your client code. This introduce a numerical error : top
> on top.
>
> On Thursday, April 10, 2014, Bogala,
, April 11, 2014 12:10 PM
To: user@hbase.apache.org
Subject: Re: endpoint coprocessor
Bear in mind each region will return its top n, then you will have to run
another top n in your client code. This introduce a numerical error : top on
top.
On Thursday, April 10, 2014, Bogala, Chandra Reddy
Bear in mind each region will return its top n, then you will have to run
another top n in your client code. This introduce a numerical error : top
on top.
On Thursday, April 10, 2014, Bogala, Chandra Reddy
wrote:
> Hi,
> I am planning to write endpoint coprocessor to calculate TOP N r
ogala, Chandra Reddy"
wrote:
> Hi,
> I am planning to write endpoint coprocessor to calculate TOP N results for my
> usecase. I got confused with old apis and new apis.
> I followed below links and try to implement. But looks like api's changed a
> lot. I don't see many
Hi,
I am planning to write endpoint coprocessor to calculate TOP N results for my
usecase. I got confused with old apis and new apis.
I followed below links and try to implement. But looks like api's changed a
lot. I don't see many of these classes in hbase jars. We are using Hbase
On Thu, Oct 17, 2013 at 12:18 PM, Asaf Mesika wrote:
> Hi,
>
> I'm writing an Endpoint Coprocessor for HBase v0.94.6 (cdh4.3.1).
>
> I'm trying to understand how the CoprocessorProtocol and
> CoprocessorEndPoint implementation versioning works, both backwards and
> fo
Hi,
I'm writing an Endpoint Coprocessor for HBase v0.94.6 (cdh4.3.1).
I'm trying to understand how the CoprocessorProtocol and
CoprocessorEndPoint implementation versioning works, both backwards and
forward.
For instance, I have my protocol as:
public interface MyQueryProtoc
> So should we close HBASE-5492 as a dup?
>
> Yes, that would make sense. Done.
> In any case, it looks like the same issue described in HBASE-6870, so if
we can get that in, it should solve your problem.
So should we close HBASE-5492 as a dup?
On Fri, Mar 8, 2013 at 9:34 AM, Gary Helmling wrote:
> > I profiled it and getStartKeysInRange is taking all the time. Recall I'm
> I profiled it and getStartKeysInRange is taking all the time. Recall I'm
> running 0.92.1. I think these factors are consistent with
> https://issues.apache.org/jira/browse/HBASE-5492, which was fixed in
> 0.92.3.
>
> We'll be upgrading soon, so I'll be able to verify the perf issue is gone.
>
U
with more testing and will
> > report back. I'm definitely seeing significant load on .META. but want to
> > see what I can determine about the root cause
> >
> >
> > Sent from my Samsung smartphone on AT&T
> >
> >
> > Original message --
t cause
>
>
> Sent from my Samsung smartphone on AT&T
>
>
> ---- Original message
> Subject: RE: endpoint coprocessor performance
> From: Anoop Sam John
> To: "user@hbase.apache.org"
> CC:
>
>
> Yes agree with Andrew here... I checked
ng
up time in your case?
HBASE-6877 also I am seeing.
-Anoop-
From: Andrew Purtell [apurt...@apache.org]
Sent: Wednesday, March 06, 2013 7:28 AM
To: user@hbase.apache.org
Subject: Re: endpoint coprocessor performance
> In current logic, HTable#coprocesso
-
From: Andrew Purtell [apurt...@apache.org]
Sent: Wednesday, March 06, 2013 7:28 AM
To: user@hbase.apache.org
Subject: Re: endpoint coprocessor performance
> In current logic, HTable#coprocessorExec always scan the whole table, its
efficiency is low
No, I don't think that is correct.
In its
> In current logic, HTable#coprocessorExec always scan the whole table, its
efficiency is low
No, I don't think that is correct.
In its current logic, coprocessorExec always scans the META table for all
regions of the target table, to find the up to date locations, and then
dispatches the exec in
Thanks so much! This describes exactly what I'm seeing. I did notice
extremely heavy load on the region server carrying .META., as described in
HBASE-6870:
In current logic, HTable#coprocessorExec always scan the whole table,
its efficiency
is low and will affect the Regionserver carrying .META. u
great question from Kim and follow-up/answers.
2013/3/4 Gary Helmling
> I see this is HBASE-6870. I thought that sounded familiar.
>
>
> On Mon, Mar 4, 2013 at 6:23 PM, Gary Helmling wrote:
>
> >
> > Check your logs for whether your end-point coprocessor is hitting
> >> zookeeper on every inv
I see this is HBASE-6870. I thought that sounded familiar.
On Mon, Mar 4, 2013 at 6:23 PM, Gary Helmling wrote:
>
> Check your logs for whether your end-point coprocessor is hitting
>> zookeeper on every invocation to figure out the region start key.
>> Unfortunately (at least last time I chec
> Check your logs for whether your end-point coprocessor is hitting
> zookeeper on every invocation to figure out the region start key.
> Unfortunately (at least last time I checked), the default way of invoking
> an end point coprocessor doesn't use the meta cache. You can go through a
> combinati
ct values. I ran performance tests
>> with two distinct value implementations: one using endpoint coprocessors,
>> and one using just scans (computing distinct values client side only). I
>> noticed that the endpoint coprocessor implementation averaged 80 ms slower
>> than
to use coprocessors. One
interesting scenario is computing distinct values. I ran performance tests
with two distinct value implementations: one using endpoint coprocessors,
and one using just scans (computing distinct values client side only). I
noticed that the endpoint coprocessor implementation avera
de only). I
> noticed that the endpoint coprocessor implementation averaged 80 ms slower
> than the scan implementation. Details of that are below for anyone
> interested.
>
> To drill into the performance, I instrumented the code and ultimately
> deployed a no-op endpoint coprocessor, to l
t values client side only). I
> noticed that the endpoint coprocessor implementation averaged 80 ms slower
> than the scan implementation. Details of that are below for anyone
> interested.
>
> To drill into the performance, I instrumented the code and ultimately
> deployed a no-o
Forget about it, my bad :)
On Wed, Jan 16, 2013 at 2:48 PM, Amit Sela wrote:
> Hi all,
>
> It seems like I can't load Endpoint coprocessor from shell but I have no
> problem loading RegionObserver from shell.
> In both cases I pack a jar file, copy it to HDFS and l
Hi all,
It seems like I can't load Endpoint coprocessor from shell but I have no
problem loading RegionObserver from shell.
In both cases I pack a jar file, copy it to HDFS and load from shell using
table_att but only the RegionObserver is loaded (I can see it in the
webapp).
Is it suppos
Fei DIng,
I think you're making the solution harder than it should be.
To start with, the only think you need to do is use co-processors to keep the
indexes in sync with the underlying table.
The code called from the co-processor will depend on the type of action and the
type of index you ar
Hi Michel,
On Fri, May 18, 2012 at 1:39 AM, Michael Segel wrote:
> > You should not let just any user run coprocessors on the server. That's
> madness.
> >
> > Best regards,
> >
> >- Andy
>
> Fei Ding,
>
> I'm a little confused.
> Are you trying to solve the problem of querying data efficient
> You should not let just any user run coprocessors on the server. That's
> madness.
>
> Best regards,
>
>- Andy
Fei Ding,
I'm a little confused.
Are you trying to solve the problem of querying data efficiently from a table,
or are you trying to find an example of where and when to us
On Wed, May 16, 2012 at 6:43 PM, fding hbase wrote:
>> Not coprocessors in general. The client side support for Endpoints
>> (Exec, etc.) gives the developer the fiction of addressing the cluster
>> as a range of rows, and will parallelize per-region Endpoint
>> invocations, and collect the respon
On Thu, May 17, 2012 at 6:28 AM, Andrew Purtell wrote:
> Are HBase coprocessors explicitly wrong for this use case if the app
> logic
> needs to access multiple regions in a single call?
Not coprocessors in general. The client side support for Endpoints
> (Exec, etc.) gives the developer the
On Wed, May 16, 2012 at 2:40 PM, Dave Revell wrote:
> Many people will probably try to use coprocessors as a way of implementing
> app logic on top of HBase without the headaches of writing a daemon.
> Sometimes client-side approaches are inadvisable; for example, there may be
> several client lan
David,
Its not a question of a daemon, its a question of the problem you are trying to
solve.
Using this as an example.. you are not always going to select data from a given
table always using the same query. So you will not always want to use the index
on column A and then the index on column
Many people will probably try to use coprocessors as a way of implementing
app logic on top of HBase without the headaches of writing a daemon.
Sometimes client-side approaches are inadvisable; for example, there may be
several client languages/runtimes and the app logic should not be
reimplemented
I think we need to look at the base problem that is trying to be solved.
I mean the discussion on the RPC mechanism. but the problem that the OP is
trying to solve is how to use multiple indexes in a 'query'.
Note: I put ' ' around query because its a m/r job or a single thread where the
us
> On May 16, 2012, at 1:12 AM, fding hbase wrote:
>> But sadly, HBase ipc doesn't allow coprocessor chaining mechanism...
>> Someone mentioned on
>> http://grokbase.com/t/hbase/user/116hrhhf8m/coprocessor-failure-question-and-examples
>> :
>>
>> If a RegionObserver issues RPC to another table from
ean you wanted to do multi-key indexes where the key is a
>>>> multi-key part.
>>>>
>>>> Or did you mean that you really wanted to use multiple indexes at the
>> same
>>>> time on a single query?
>>>>
>>>> If its the latter, no
ort order?
> >>
> >> What happens when the results from the indexes exceed the amount of
> >> allocated memory?
> >>
> >> What I am suggesting you to do is to set aside the underpinnings of
> HBase
> >> and look at the problem you are trying to sol
; easy one...
>>
>>
>>
>> Sent from a remote device. Please excuse any typos...
>>
>> Mike Segel
>>
>> On May 14, 2012, at 4:35 AM, fding hbase wrote:
>>
>>> Hi all,
>>>
>>> Is it possible to use table scanner (diff
t; >
> > Is it possible to use table scanner (different from the host table
> region)
> > or
> > execute coprocessor of another table, in the endpoint coprocessor?
> > It looks like chaining coprocessors. But I found a possible deadlock!
> > Can anyone help me with this?
&
> Is it possible to use table scanner (different from the host table region)
> or
> execute coprocessor of another table, in the endpoint coprocessor?
> It looks like chaining coprocessors. But I found a possible deadlock!
> Can anyone help me with this?
>
> In my testing
Hi all,
Is it possible to use table scanner (different from the host table region)
or
execute coprocessor of another table, in the endpoint coprocessor?
It looks like chaining coprocessors. But I found a possible deadlock!
Can anyone help me with this?
In my testing environment I deployed the
49 matches
Mail list logo
