Re: scope of RegionCoprocessorEnvironment sharedData

[Veerraju Tadimeti]

yeah, you are right.  I am getting scanner in postScannerOpen().  I tested
with preScannerOpen() earlier ,which is null scanner.  Thanks alot.

Thanks,
Raju,
(972)273-0155.

On Thu, Jul 13, 2017 at 11:50 PM, Anoop John  wrote:

> No co processors can be configured either globally at cluster level
> (RS configuration) or at table level.
>
> You dont need preScannerOpen.  I checked the code and we are passing
> the actual scanner been created to postScannerOpen() method.  This
> being null in ur case been strange!  Pls make sure u r implementing
> post hook not pre.
>
> -Anoop-
>
> On Wed, Jul 12, 2017 at 3:01 AM, Veerraju Tadimeti 
> wrote:
> > Can I load coprocessor dynamically for a scan operation, it should not
> be loaded for another scan operation if not intended
> >  btw I invoke scan from hive
> >
> > Sent from my iPhone
> >
> >> On Jul 11, 2017, at 4:15 PM, Veerraju Tadimeti 
> wrote:
> >>
> >> hi,
> >>
> >> Hi John,
> >>
> >> Thanks for the reply.
> >>
> >> I implemented #2 (another way) in ur above post:
> >>
> >>
> >>
> >> i debug the logs  : in PostScannerOpen() , regionScanner method
> parameter object is null
> >>
> >> Also, in preScannerOpen() , i returned return super.preScannerOpen(e,
> scan, new DelegateRegionScanner(s));
> >> in postScannerNext() , internalScanner object is
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl
> >>
> >> #1 way (Put scanner in local map) - may not be possible, cos if two
> different scan operation with and without attribute hits at the same time,
> how can we differentiate in postScannerNext.
> >>
> >>
> >> Thanks,
> >> Raju,
> >> (972)273-0155.
> >>
> >>> On Tue, Jul 11, 2017 at 8:05 AM, Anoop John 
> wrote:
> >>> Ya. It is the same RegionScanner impl in use only being passed.  Ya
> >>> the param type should have been RegionScanner  I guess. We made that
> >>> mistake!
> >>> -Anoop-
> >>>
> >>> On Mon, Jul 10, 2017 at 8:37 PM, Ted Yu  wrote:
> >>> > The tricky part is that postScannerNext() passes InternalScanner
> parameter
> >>> > instead of RegionScanner.
> >>> >
> >>> > FYI
> >>> >
> >>> > On Sun, Jul 9, 2017 at 10:57 PM, Anoop John 
> wrote:
> >>> >
> >>> >> Ya as Ted said, u are not getting Scan object in the postScannerNext
> >>> >> and so can not make use of the attribute in Scan within this hook.
> >>> >> Just setting the sharedData variable will cause issue with
> concurrent
> >>> >> scans. (As u imagine)
> >>> >>
> >>> >> So I can think of solving this in 2 possible ways. (May be more ways
> >>> >> possible)
> >>> >>
> >>> >> 1.  U keep a Map within ur CP impl.  You implement postScannerOpen
> >>> >> where u will get the ref to Scanner been created as well as the
> Scan.
> >>> >> If the Scan is having attribute, keep that scanner within ur Map.
> >>> >> During postScannerNext  check if the coming in scanner is there in
> ur
> >>> >> Map. If so that means this is the one where u can do the action.
> >>> >> Also dont forget to implement postScannerClose and remove that
> scanner
> >>> >> from the Map.   Here u might have some perf penalty as u have to add
> >>> >> and get from Map which has to be a concurrent map too.
> >>> >>
> >>> >> Another way
> >>> >>
> >>> >> 2. Create a custom scanner implementing RegionScanner.   The new one
> >>> >> has to take an original Region Scanner and just delegate the calls.
> On
> >>> >> postScannerOpen, u will get the original scanner been created and u
> >>> >> can just wrap it with ur new scanner object. ( If the Scan object is
> >>> >> having required attribute)..  In postScannerNext() u can check for
> ur
> >>> >> own RegionScanner type and if so u can do action.
> >>> >>
> >>> >>
> >>> >> -Anoop-
> >>> >>
> >>> >>
> >>> >> On Sat, Jul 8, 2017 at 9:13 PM, Ted Yu  wrote:
> >>> >> > if (canUseGetOperation(e)) {
> >>> >> >//logic goes here
> >>> >> >
> >>> >> > Does your Get target the same region being scanned ?
> >>> >> > If not, issuing the Get is not advised since the other region may
> be
> >>> >> hosted
> >>> >> > on different region server.
> >>> >> >
> >>> >> > Cheers
> >>> >> >
> >>> >> > On Thu, Jul 6, 2017 at 7:14 AM, Veerraju Tadimeti <
> tvvr...@gmail.com>
> >>> >> wrote:
> >>> >> >
> >>> >> >> hi,
> >>> >> >>
> >>> >> >> I have few questions regarding scope of
> *RegionCoprocessorEnvironment*
> >>> >> >>  sharedData.
> >>> >> >>
> >>> >> >>
> >>> >> >>
> >>> >> >>- *Is sharedData map is shared accross all instances
> simultaneously
> >>> >> ?*
> >>> >> >>   -  I am putting a variable in sharedData in
> preScannerOpen()
> >>> >> based on
> >>> >> >>   scan attribute,
> >>> >> >>   - check that variable exists in postScannerNext() then
> apply
> >>> >> logic,
> >>> >> >>   - remove the variable postScannerClose().
> >>> >> >>   - If data is in multiple regions, when one coprocessor
> 

Re: scope of RegionCoprocessorEnvironment sharedData

[Anoop John]

No co processors can be configured either globally at cluster level
(RS configuration) or at table level.

You dont need preScannerOpen.  I checked the code and we are passing
the actual scanner been created to postScannerOpen() method.  This
being null in ur case been strange!  Pls make sure u r implementing
post hook not pre.

-Anoop-

On Wed, Jul 12, 2017 at 3:01 AM, Veerraju Tadimeti  wrote:
> Can I load coprocessor dynamically for a scan operation, it should not be 
> loaded for another scan operation if not intended
>  btw I invoke scan from hive
>
> Sent from my iPhone
>
>> On Jul 11, 2017, at 4:15 PM, Veerraju Tadimeti  wrote:
>>
>> hi,
>>
>> Hi John,
>>
>> Thanks for the reply.
>>
>> I implemented #2 (another way) in ur above post:
>>
>>
>>
>> i debug the logs  : in PostScannerOpen() , regionScanner method parameter 
>> object is null
>>
>> Also, in preScannerOpen() , i returned return super.preScannerOpen(e, scan, 
>> new DelegateRegionScanner(s));
>> in postScannerNext() , internalScanner object is 
>> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl
>>
>> #1 way (Put scanner in local map) - may not be possible, cos if two 
>> different scan operation with and without attribute hits at the same time, 
>> how can we differentiate in postScannerNext.
>>
>>
>> Thanks,
>> Raju,
>> (972)273-0155.
>>
>>> On Tue, Jul 11, 2017 at 8:05 AM, Anoop John  wrote:
>>> Ya. It is the same RegionScanner impl in use only being passed.  Ya
>>> the param type should have been RegionScanner  I guess. We made that
>>> mistake!
>>> -Anoop-
>>>
>>> On Mon, Jul 10, 2017 at 8:37 PM, Ted Yu  wrote:
>>> > The tricky part is that postScannerNext() passes InternalScanner parameter
>>> > instead of RegionScanner.
>>> >
>>> > FYI
>>> >
>>> > On Sun, Jul 9, 2017 at 10:57 PM, Anoop John  wrote:
>>> >
>>> >> Ya as Ted said, u are not getting Scan object in the postScannerNext
>>> >> and so can not make use of the attribute in Scan within this hook.
>>> >> Just setting the sharedData variable will cause issue with concurrent
>>> >> scans. (As u imagine)
>>> >>
>>> >> So I can think of solving this in 2 possible ways. (May be more ways
>>> >> possible)
>>> >>
>>> >> 1.  U keep a Map within ur CP impl.  You implement postScannerOpen
>>> >> where u will get the ref to Scanner been created as well as the Scan.
>>> >> If the Scan is having attribute, keep that scanner within ur Map.
>>> >> During postScannerNext  check if the coming in scanner is there in ur
>>> >> Map. If so that means this is the one where u can do the action.
>>> >> Also dont forget to implement postScannerClose and remove that scanner
>>> >> from the Map.   Here u might have some perf penalty as u have to add
>>> >> and get from Map which has to be a concurrent map too.
>>> >>
>>> >> Another way
>>> >>
>>> >> 2. Create a custom scanner implementing RegionScanner.   The new one
>>> >> has to take an original Region Scanner and just delegate the calls. On
>>> >> postScannerOpen, u will get the original scanner been created and u
>>> >> can just wrap it with ur new scanner object. ( If the Scan object is
>>> >> having required attribute)..  In postScannerNext() u can check for ur
>>> >> own RegionScanner type and if so u can do action.
>>> >>
>>> >>
>>> >> -Anoop-
>>> >>
>>> >>
>>> >> On Sat, Jul 8, 2017 at 9:13 PM, Ted Yu  wrote:
>>> >> > if (canUseGetOperation(e)) {
>>> >> >//logic goes here
>>> >> >
>>> >> > Does your Get target the same region being scanned ?
>>> >> > If not, issuing the Get is not advised since the other region may be
>>> >> hosted
>>> >> > on different region server.
>>> >> >
>>> >> > Cheers
>>> >> >
>>> >> > On Thu, Jul 6, 2017 at 7:14 AM, Veerraju Tadimeti 
>>> >> wrote:
>>> >> >
>>> >> >> hi,
>>> >> >>
>>> >> >> I have few questions regarding scope of *RegionCoprocessorEnvironment*
>>> >> >>  sharedData.
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >>- *Is sharedData map is shared accross all instances simultaneously
>>> >> ?*
>>> >> >>   -  I am putting a variable in sharedData in preScannerOpen()
>>> >> based on
>>> >> >>   scan attribute,
>>> >> >>   - check that variable exists in postScannerNext() then apply
>>> >> logic,
>>> >> >>   - remove the variable postScannerClose().
>>> >> >>   - If data is in multiple regions, when one coprocessor removes
>>> >> >>   variable in postScannerClose(), will the variable is NULL for
>>> >> another
>>> >> >>   region coprocessor in postScannerNext() ?
>>> >> >>
>>> >> >>
>>> >> >>- *Is sharedData map is shared across all the client request
>>> >> >>operations ?*
>>> >> >>
>>> >> >> If a variable is set in sharedData for one client operation(say SCAN),
>>> >> will
>>> >> >> the variable is available for another client operation(new SCAN) ?
>>> >> >>
>>> >> >>
>>> >> >>-  *Will 

Re: scope of RegionCoprocessorEnvironment sharedData

[Veerraju Tadimeti]

Can I load coprocessor dynamically for a scan operation, it should not be 
loaded for another scan operation if not intended
 btw I invoke scan from hive

Sent from my iPhone

> On Jul 11, 2017, at 4:15 PM, Veerraju Tadimeti  wrote:
> 
> hi,
> 
> Hi John,
> 
> Thanks for the reply.
> 
> I implemented #2 (another way) in ur above post:
> 
> 
> 
> i debug the logs  : in PostScannerOpen() , regionScanner method parameter 
> object is null
> 
> Also, in preScannerOpen() , i returned return super.preScannerOpen(e, scan, 
> new DelegateRegionScanner(s)); 
> in postScannerNext() , internalScanner object is 
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl
> 
> #1 way (Put scanner in local map) - may not be possible, cos if two different 
> scan operation with and without attribute hits at the same time, how can we 
> differentiate in postScannerNext. 
> 
> 
> Thanks,
> Raju,
> (972)273-0155.
> 
>> On Tue, Jul 11, 2017 at 8:05 AM, Anoop John  wrote:
>> Ya. It is the same RegionScanner impl in use only being passed.  Ya
>> the param type should have been RegionScanner  I guess. We made that
>> mistake!
>> -Anoop-
>> 
>> On Mon, Jul 10, 2017 at 8:37 PM, Ted Yu  wrote:
>> > The tricky part is that postScannerNext() passes InternalScanner parameter
>> > instead of RegionScanner.
>> >
>> > FYI
>> >
>> > On Sun, Jul 9, 2017 at 10:57 PM, Anoop John  wrote:
>> >
>> >> Ya as Ted said, u are not getting Scan object in the postScannerNext
>> >> and so can not make use of the attribute in Scan within this hook.
>> >> Just setting the sharedData variable will cause issue with concurrent
>> >> scans. (As u imagine)
>> >>
>> >> So I can think of solving this in 2 possible ways. (May be more ways
>> >> possible)
>> >>
>> >> 1.  U keep a Map within ur CP impl.  You implement postScannerOpen
>> >> where u will get the ref to Scanner been created as well as the Scan.
>> >> If the Scan is having attribute, keep that scanner within ur Map.
>> >> During postScannerNext  check if the coming in scanner is there in ur
>> >> Map. If so that means this is the one where u can do the action.
>> >> Also dont forget to implement postScannerClose and remove that scanner
>> >> from the Map.   Here u might have some perf penalty as u have to add
>> >> and get from Map which has to be a concurrent map too.
>> >>
>> >> Another way
>> >>
>> >> 2. Create a custom scanner implementing RegionScanner.   The new one
>> >> has to take an original Region Scanner and just delegate the calls. On
>> >> postScannerOpen, u will get the original scanner been created and u
>> >> can just wrap it with ur new scanner object. ( If the Scan object is
>> >> having required attribute)..  In postScannerNext() u can check for ur
>> >> own RegionScanner type and if so u can do action.
>> >>
>> >>
>> >> -Anoop-
>> >>
>> >>
>> >> On Sat, Jul 8, 2017 at 9:13 PM, Ted Yu  wrote:
>> >> > if (canUseGetOperation(e)) {
>> >> >//logic goes here
>> >> >
>> >> > Does your Get target the same region being scanned ?
>> >> > If not, issuing the Get is not advised since the other region may be
>> >> hosted
>> >> > on different region server.
>> >> >
>> >> > Cheers
>> >> >
>> >> > On Thu, Jul 6, 2017 at 7:14 AM, Veerraju Tadimeti 
>> >> wrote:
>> >> >
>> >> >> hi,
>> >> >>
>> >> >> I have few questions regarding scope of *RegionCoprocessorEnvironment*
>> >> >>  sharedData.
>> >> >>
>> >> >>
>> >> >>
>> >> >>- *Is sharedData map is shared accross all instances simultaneously
>> >> ?*
>> >> >>   -  I am putting a variable in sharedData in preScannerOpen()
>> >> based on
>> >> >>   scan attribute,
>> >> >>   - check that variable exists in postScannerNext() then apply
>> >> logic,
>> >> >>   - remove the variable postScannerClose().
>> >> >>   - If data is in multiple regions, when one coprocessor removes
>> >> >>   variable in postScannerClose(), will the variable is NULL for
>> >> another
>> >> >>   region coprocessor in postScannerNext() ?
>> >> >>
>> >> >>
>> >> >>- *Is sharedData map is shared across all the client request
>> >> >>operations ?*
>> >> >>
>> >> >> If a variable is set in sharedData for one client operation(say SCAN),
>> >> will
>> >> >> the variable is available for another client operation(new SCAN) ?
>> >> >>
>> >> >>
>> >> >>-  *Will the variables be garbage collected even if we dont 
>> >> >> implement
>> >> >>(removed variables in sharedData) postScannerClose() method*
>> >> >>
>> >> >>
>> >> >> Please find below the logic that I am using currently
>> >> >> *CODE: *
>> >> >>
>> >> >> public RegionScanner
>> >> >> *preScannerOpen*(ObserverContext
>> >> >> e, Scan scan, RegionScanner s) throws IOException {
>> >> >> byte[] useGetInPostScannerNext = scan.getAttribute(USE_GET_
>> >> >> OPERATION_IN_POST_SCANNER_NEXT);
>> >> >> String 

Re: scope of RegionCoprocessorEnvironment sharedData

[Veerraju Tadimeti]

hi,

Hi John,

Thanks for the reply.

I implemented #2 (*another way*) in ur above post:



i debug the logs  : in PostScannerOpen() , regionScanner method parameter
object is null

Also, in preScannerOpen() , i returned *return super.preScannerOpen(e,
scan, new DelegateRegionScanner(s)); *
in postScannerNext() , internalScanner object is
*org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl*

#1 way (Put scanner in local map) - may not be possible, cos if two
different scan operation with and without attribute hits at the same time,
how can we differentiate in postScannerNext.


Thanks,
Raju,
(972)273-0155.

On Tue, Jul 11, 2017 at 8:05 AM, Anoop John  wrote:

> Ya. It is the same RegionScanner impl in use only being passed.  Ya
> the param type should have been RegionScanner  I guess. We made that
> mistake!
> -Anoop-
>
> On Mon, Jul 10, 2017 at 8:37 PM, Ted Yu  wrote:
> > The tricky part is that postScannerNext() passes InternalScanner
> parameter
> > instead of RegionScanner.
> >
> > FYI
> >
> > On Sun, Jul 9, 2017 at 10:57 PM, Anoop John 
> wrote:
> >
> >> Ya as Ted said, u are not getting Scan object in the postScannerNext
> >> and so can not make use of the attribute in Scan within this hook.
> >> Just setting the sharedData variable will cause issue with concurrent
> >> scans. (As u imagine)
> >>
> >> So I can think of solving this in 2 possible ways. (May be more ways
> >> possible)
> >>
> >> 1.  U keep a Map within ur CP impl.  You implement postScannerOpen
> >> where u will get the ref to Scanner been created as well as the Scan.
> >> If the Scan is having attribute, keep that scanner within ur Map.
> >> During postScannerNext  check if the coming in scanner is there in ur
> >> Map. If so that means this is the one where u can do the action.
> >> Also dont forget to implement postScannerClose and remove that scanner
> >> from the Map.   Here u might have some perf penalty as u have to add
> >> and get from Map which has to be a concurrent map too.
> >>
> >> Another way
> >>
> >> 2. Create a custom scanner implementing RegionScanner.   The new one
> >> has to take an original Region Scanner and just delegate the calls. On
> >> postScannerOpen, u will get the original scanner been created and u
> >> can just wrap it with ur new scanner object. ( If the Scan object is
> >> having required attribute)..  In postScannerNext() u can check for ur
> >> own RegionScanner type and if so u can do action.
> >>
> >>
> >> -Anoop-
> >>
> >>
> >> On Sat, Jul 8, 2017 at 9:13 PM, Ted Yu  wrote:
> >> > if (canUseGetOperation(e)) {
> >> >//logic goes here
> >> >
> >> > Does your Get target the same region being scanned ?
> >> > If not, issuing the Get is not advised since the other region may be
> >> hosted
> >> > on different region server.
> >> >
> >> > Cheers
> >> >
> >> > On Thu, Jul 6, 2017 at 7:14 AM, Veerraju Tadimeti 
> >> wrote:
> >> >
> >> >> hi,
> >> >>
> >> >> I have few questions regarding scope of
> *RegionCoprocessorEnvironment*
> >> >>  sharedData.
> >> >>
> >> >>
> >> >>
> >> >>- *Is sharedData map is shared accross all instances
> simultaneously
> >> ?*
> >> >>   -  I am putting a variable in sharedData in preScannerOpen()
> >> based on
> >> >>   scan attribute,
> >> >>   - check that variable exists in postScannerNext() then apply
> >> logic,
> >> >>   - remove the variable postScannerClose().
> >> >>   - If data is in multiple regions, when one coprocessor removes
> >> >>   variable in postScannerClose(), will the variable is NULL for
> >> another
> >> >>   region coprocessor in postScannerNext() ?
> >> >>
> >> >>
> >> >>- *Is sharedData map is shared across all the client request
> >> >>operations ?*
> >> >>
> >> >> If a variable is set in sharedData for one client operation(say
> SCAN),
> >> will
> >> >> the variable is available for another client operation(new SCAN) ?
> >> >>
> >> >>
> >> >>-  *Will the variables be garbage collected even if we dont
> implement
> >> >>(removed variables in sharedData) postScannerClose() method*
> >> >>
> >> >>
> >> >> Please find below the logic that I am using currently
> >> >> *CODE: *
> >> >>
> >> >> public RegionScanner
> >> >> *preScannerOpen*(ObserverContext
> >> >> e, Scan scan, RegionScanner s) throws IOException {
> >> >> byte[] useGetInPostScannerNext = scan.getAttribute(USE_GET_
> >> >> OPERATION_IN_POST_SCANNER_NEXT);
> >> >> String useGetInPostScannerNextStr = Bytes.toString(
> >> >> useGetInPostScannerNext);
> >> >> if (Boolean.parseBoolean(useGetInPostScannerNextStr)) {
> >> >> e.getEnvironment().getSharedData().put(USE_GET_
> >> >> OPERATION_IN_POST_SCANNER_NEXT, useGetInPostScannerNextStr);
> >> >> }
> >> >> return super.preScannerOpen(e, scan, s);
> >> >> }
> >> >>
> >> >> @Override
> >> 

Re: scope of RegionCoprocessorEnvironment sharedData

[Anoop John]

Ya. It is the same RegionScanner impl in use only being passed.  Ya
the param type should have been RegionScanner  I guess. We made that
mistake!
-Anoop-

On Mon, Jul 10, 2017 at 8:37 PM, Ted Yu  wrote:
> The tricky part is that postScannerNext() passes InternalScanner parameter
> instead of RegionScanner.
>
> FYI
>
> On Sun, Jul 9, 2017 at 10:57 PM, Anoop John  wrote:
>
>> Ya as Ted said, u are not getting Scan object in the postScannerNext
>> and so can not make use of the attribute in Scan within this hook.
>> Just setting the sharedData variable will cause issue with concurrent
>> scans. (As u imagine)
>>
>> So I can think of solving this in 2 possible ways. (May be more ways
>> possible)
>>
>> 1.  U keep a Map within ur CP impl.  You implement postScannerOpen
>> where u will get the ref to Scanner been created as well as the Scan.
>> If the Scan is having attribute, keep that scanner within ur Map.
>> During postScannerNext  check if the coming in scanner is there in ur
>> Map. If so that means this is the one where u can do the action.
>> Also dont forget to implement postScannerClose and remove that scanner
>> from the Map.   Here u might have some perf penalty as u have to add
>> and get from Map which has to be a concurrent map too.
>>
>> Another way
>>
>> 2. Create a custom scanner implementing RegionScanner.   The new one
>> has to take an original Region Scanner and just delegate the calls. On
>> postScannerOpen, u will get the original scanner been created and u
>> can just wrap it with ur new scanner object. ( If the Scan object is
>> having required attribute)..  In postScannerNext() u can check for ur
>> own RegionScanner type and if so u can do action.
>>
>>
>> -Anoop-
>>
>>
>> On Sat, Jul 8, 2017 at 9:13 PM, Ted Yu  wrote:
>> > if (canUseGetOperation(e)) {
>> >//logic goes here
>> >
>> > Does your Get target the same region being scanned ?
>> > If not, issuing the Get is not advised since the other region may be
>> hosted
>> > on different region server.
>> >
>> > Cheers
>> >
>> > On Thu, Jul 6, 2017 at 7:14 AM, Veerraju Tadimeti 
>> wrote:
>> >
>> >> hi,
>> >>
>> >> I have few questions regarding scope of *RegionCoprocessorEnvironment*
>> >>  sharedData.
>> >>
>> >>
>> >>
>> >>- *Is sharedData map is shared accross all instances simultaneously
>> ?*
>> >>   -  I am putting a variable in sharedData in preScannerOpen()
>> based on
>> >>   scan attribute,
>> >>   - check that variable exists in postScannerNext() then apply
>> logic,
>> >>   - remove the variable postScannerClose().
>> >>   - If data is in multiple regions, when one coprocessor removes
>> >>   variable in postScannerClose(), will the variable is NULL for
>> another
>> >>   region coprocessor in postScannerNext() ?
>> >>
>> >>
>> >>- *Is sharedData map is shared across all the client request
>> >>operations ?*
>> >>
>> >> If a variable is set in sharedData for one client operation(say SCAN),
>> will
>> >> the variable is available for another client operation(new SCAN) ?
>> >>
>> >>
>> >>-  *Will the variables be garbage collected even if we dont implement
>> >>(removed variables in sharedData) postScannerClose() method*
>> >>
>> >>
>> >> Please find below the logic that I am using currently
>> >> *CODE: *
>> >>
>> >> public RegionScanner
>> >> *preScannerOpen*(ObserverContext
>> >> e, Scan scan, RegionScanner s) throws IOException {
>> >> byte[] useGetInPostScannerNext = scan.getAttribute(USE_GET_
>> >> OPERATION_IN_POST_SCANNER_NEXT);
>> >> String useGetInPostScannerNextStr = Bytes.toString(
>> >> useGetInPostScannerNext);
>> >> if (Boolean.parseBoolean(useGetInPostScannerNextStr)) {
>> >> e.getEnvironment().getSharedData().put(USE_GET_
>> >> OPERATION_IN_POST_SCANNER_NEXT, useGetInPostScannerNextStr);
>> >> }
>> >> return super.preScannerOpen(e, scan, s);
>> >> }
>> >>
>> >> @Override
>> >> public boolean *postScannerNext*(final
>> >> ObserverContext
>> >> e,
>> >> final InternalScanner s, final List results, final
>> int
>> >> limit,
>> >> final boolean hasMore) throws IOException {
>> >> try {
>> >>
>> >> if (canUseGetOperation(e)) {
>> >>
>> >>//logic goes here
>> >> }
>> >> } catch (Exception ex) {
>> >> logger.error("Exception in postScannerNext ", ex);
>> >> throw new IOException(ex);
>> >> }
>> >> return hasMore;
>> >> }
>> >>
>> >> @Override
>> >> public void
>> >> *postScannerClose*(ObserverContext
>> >> e, InternalScanner s) throws IOException {
>> >> if (canUseGetOperation(e)) {
>> >> e.getEnvironment().getSharedData().remove(USE_
>> >> GET_OPERATION_IN_POST_SCANNER_NEXT);
>> >> }
>> >> super.postScannerClose(e, s);
>> >>  

Re: scope of RegionCoprocessorEnvironment sharedData

[Ted Yu]

The tricky part is that postScannerNext() passes InternalScanner parameter
instead of RegionScanner.

FYI

On Sun, Jul 9, 2017 at 10:57 PM, Anoop John  wrote:

> Ya as Ted said, u are not getting Scan object in the postScannerNext
> and so can not make use of the attribute in Scan within this hook.
> Just setting the sharedData variable will cause issue with concurrent
> scans. (As u imagine)
>
> So I can think of solving this in 2 possible ways. (May be more ways
> possible)
>
> 1.  U keep a Map within ur CP impl.  You implement postScannerOpen
> where u will get the ref to Scanner been created as well as the Scan.
> If the Scan is having attribute, keep that scanner within ur Map.
> During postScannerNext  check if the coming in scanner is there in ur
> Map. If so that means this is the one where u can do the action.
> Also dont forget to implement postScannerClose and remove that scanner
> from the Map.   Here u might have some perf penalty as u have to add
> and get from Map which has to be a concurrent map too.
>
> Another way
>
> 2. Create a custom scanner implementing RegionScanner.   The new one
> has to take an original Region Scanner and just delegate the calls. On
> postScannerOpen, u will get the original scanner been created and u
> can just wrap it with ur new scanner object. ( If the Scan object is
> having required attribute)..  In postScannerNext() u can check for ur
> own RegionScanner type and if so u can do action.
>
>
> -Anoop-
>
>
> On Sat, Jul 8, 2017 at 9:13 PM, Ted Yu  wrote:
> > if (canUseGetOperation(e)) {
> >//logic goes here
> >
> > Does your Get target the same region being scanned ?
> > If not, issuing the Get is not advised since the other region may be
> hosted
> > on different region server.
> >
> > Cheers
> >
> > On Thu, Jul 6, 2017 at 7:14 AM, Veerraju Tadimeti 
> wrote:
> >
> >> hi,
> >>
> >> I have few questions regarding scope of *RegionCoprocessorEnvironment*
> >>  sharedData.
> >>
> >>
> >>
> >>- *Is sharedData map is shared accross all instances simultaneously
> ?*
> >>   -  I am putting a variable in sharedData in preScannerOpen()
> based on
> >>   scan attribute,
> >>   - check that variable exists in postScannerNext() then apply
> logic,
> >>   - remove the variable postScannerClose().
> >>   - If data is in multiple regions, when one coprocessor removes
> >>   variable in postScannerClose(), will the variable is NULL for
> another
> >>   region coprocessor in postScannerNext() ?
> >>
> >>
> >>- *Is sharedData map is shared across all the client request
> >>operations ?*
> >>
> >> If a variable is set in sharedData for one client operation(say SCAN),
> will
> >> the variable is available for another client operation(new SCAN) ?
> >>
> >>
> >>-  *Will the variables be garbage collected even if we dont implement
> >>(removed variables in sharedData) postScannerClose() method*
> >>
> >>
> >> Please find below the logic that I am using currently
> >> *CODE: *
> >>
> >> public RegionScanner
> >> *preScannerOpen*(ObserverContext
> >> e, Scan scan, RegionScanner s) throws IOException {
> >> byte[] useGetInPostScannerNext = scan.getAttribute(USE_GET_
> >> OPERATION_IN_POST_SCANNER_NEXT);
> >> String useGetInPostScannerNextStr = Bytes.toString(
> >> useGetInPostScannerNext);
> >> if (Boolean.parseBoolean(useGetInPostScannerNextStr)) {
> >> e.getEnvironment().getSharedData().put(USE_GET_
> >> OPERATION_IN_POST_SCANNER_NEXT, useGetInPostScannerNextStr);
> >> }
> >> return super.preScannerOpen(e, scan, s);
> >> }
> >>
> >> @Override
> >> public boolean *postScannerNext*(final
> >> ObserverContext
> >> e,
> >> final InternalScanner s, final List results, final
> int
> >> limit,
> >> final boolean hasMore) throws IOException {
> >> try {
> >>
> >> if (canUseGetOperation(e)) {
> >>
> >>//logic goes here
> >> }
> >> } catch (Exception ex) {
> >> logger.error("Exception in postScannerNext ", ex);
> >> throw new IOException(ex);
> >> }
> >> return hasMore;
> >> }
> >>
> >> @Override
> >> public void
> >> *postScannerClose*(ObserverContext
> >> e, InternalScanner s) throws IOException {
> >> if (canUseGetOperation(e)) {
> >> e.getEnvironment().getSharedData().remove(USE_
> >> GET_OPERATION_IN_POST_SCANNER_NEXT);
> >> }
> >> super.postScannerClose(e, s);
> >> }
> >>
> >> private boolean *canUseGetOperation*(final
> >> ObserverContext
> >> e) {
> >> String useGetOperationInPostScannerNext = (String)
> >> e.getEnvironment().getSharedData().get(USE_GET_
> OPERATION_IN_POST_SCANNER_
> >> NEXT);
> >> return Boolean.parseBoolean(useGetOperationInPostScannerNext);
> >> }
> >>
> >> Thanks,
> 

Re: scope of RegionCoprocessorEnvironment sharedData

[Anoop John]

Ya as Ted said, u are not getting Scan object in the postScannerNext
and so can not make use of the attribute in Scan within this hook.
Just setting the sharedData variable will cause issue with concurrent
scans. (As u imagine)

So I can think of solving this in 2 possible ways. (May be more ways possible)

1.  U keep a Map within ur CP impl.  You implement postScannerOpen
where u will get the ref to Scanner been created as well as the Scan.
If the Scan is having attribute, keep that scanner within ur Map.
During postScannerNext  check if the coming in scanner is there in ur
Map. If so that means this is the one where u can do the action.
Also dont forget to implement postScannerClose and remove that scanner
from the Map.   Here u might have some perf penalty as u have to add
and get from Map which has to be a concurrent map too.

Another way

2. Create a custom scanner implementing RegionScanner.   The new one
has to take an original Region Scanner and just delegate the calls. On
postScannerOpen, u will get the original scanner been created and u
can just wrap it with ur new scanner object. ( If the Scan object is
having required attribute)..  In postScannerNext() u can check for ur
own RegionScanner type and if so u can do action.


-Anoop-


On Sat, Jul 8, 2017 at 9:13 PM, Ted Yu  wrote:
> if (canUseGetOperation(e)) {
>//logic goes here
>
> Does your Get target the same region being scanned ?
> If not, issuing the Get is not advised since the other region may be hosted
> on different region server.
>
> Cheers
>
> On Thu, Jul 6, 2017 at 7:14 AM, Veerraju Tadimeti  wrote:
>
>> hi,
>>
>> I have few questions regarding scope of *RegionCoprocessorEnvironment*
>>  sharedData.
>>
>>
>>
>>- *Is sharedData map is shared accross all instances simultaneously ?*
>>   -  I am putting a variable in sharedData in preScannerOpen() based on
>>   scan attribute,
>>   - check that variable exists in postScannerNext() then apply logic,
>>   - remove the variable postScannerClose().
>>   - If data is in multiple regions, when one coprocessor removes
>>   variable in postScannerClose(), will the variable is NULL for another
>>   region coprocessor in postScannerNext() ?
>>
>>
>>- *Is sharedData map is shared across all the client request
>>operations ?*
>>
>> If a variable is set in sharedData for one client operation(say SCAN), will
>> the variable is available for another client operation(new SCAN) ?
>>
>>
>>-  *Will the variables be garbage collected even if we dont implement
>>(removed variables in sharedData) postScannerClose() method*
>>
>>
>> Please find below the logic that I am using currently
>> *CODE: *
>>
>> public RegionScanner
>> *preScannerOpen*(ObserverContext
>> e, Scan scan, RegionScanner s) throws IOException {
>> byte[] useGetInPostScannerNext = scan.getAttribute(USE_GET_
>> OPERATION_IN_POST_SCANNER_NEXT);
>> String useGetInPostScannerNextStr = Bytes.toString(
>> useGetInPostScannerNext);
>> if (Boolean.parseBoolean(useGetInPostScannerNextStr)) {
>> e.getEnvironment().getSharedData().put(USE_GET_
>> OPERATION_IN_POST_SCANNER_NEXT, useGetInPostScannerNextStr);
>> }
>> return super.preScannerOpen(e, scan, s);
>> }
>>
>> @Override
>> public boolean *postScannerNext*(final
>> ObserverContext
>> e,
>> final InternalScanner s, final List results, final int
>> limit,
>> final boolean hasMore) throws IOException {
>> try {
>>
>> if (canUseGetOperation(e)) {
>>
>>//logic goes here
>> }
>> } catch (Exception ex) {
>> logger.error("Exception in postScannerNext ", ex);
>> throw new IOException(ex);
>> }
>> return hasMore;
>> }
>>
>> @Override
>> public void
>> *postScannerClose*(ObserverContext
>> e, InternalScanner s) throws IOException {
>> if (canUseGetOperation(e)) {
>> e.getEnvironment().getSharedData().remove(USE_
>> GET_OPERATION_IN_POST_SCANNER_NEXT);
>> }
>> super.postScannerClose(e, s);
>> }
>>
>> private boolean *canUseGetOperation*(final
>> ObserverContext
>> e) {
>> String useGetOperationInPostScannerNext = (String)
>> e.getEnvironment().getSharedData().get(USE_GET_OPERATION_IN_POST_SCANNER_
>> NEXT);
>> return Boolean.parseBoolean(useGetOperationInPostScannerNext);
>> }
>>
>> Thanks,
>> Raju,
>> (972)273-0155 <(972)%20273-0155>.
>>

Re: scope of RegionCoprocessorEnvironment sharedData

[Ted Yu]

if (canUseGetOperation(e)) {
   //logic goes here

Does your Get target the same region being scanned ?
If not, issuing the Get is not advised since the other region may be hosted
on different region server.

Cheers

On Thu, Jul 6, 2017 at 7:14 AM, Veerraju Tadimeti  wrote:

> hi,
>
> I have few questions regarding scope of *RegionCoprocessorEnvironment*
>  sharedData.
>
>
>
>- *Is sharedData map is shared accross all instances simultaneously ?*
>   -  I am putting a variable in sharedData in preScannerOpen() based on
>   scan attribute,
>   - check that variable exists in postScannerNext() then apply logic,
>   - remove the variable postScannerClose().
>   - If data is in multiple regions, when one coprocessor removes
>   variable in postScannerClose(), will the variable is NULL for another
>   region coprocessor in postScannerNext() ?
>
>
>- *Is sharedData map is shared across all the client request
>operations ?*
>
> If a variable is set in sharedData for one client operation(say SCAN), will
> the variable is available for another client operation(new SCAN) ?
>
>
>-  *Will the variables be garbage collected even if we dont implement
>(removed variables in sharedData) postScannerClose() method*
>
>
> Please find below the logic that I am using currently
> *CODE: *
>
> public RegionScanner
> *preScannerOpen*(ObserverContext
> e, Scan scan, RegionScanner s) throws IOException {
> byte[] useGetInPostScannerNext = scan.getAttribute(USE_GET_
> OPERATION_IN_POST_SCANNER_NEXT);
> String useGetInPostScannerNextStr = Bytes.toString(
> useGetInPostScannerNext);
> if (Boolean.parseBoolean(useGetInPostScannerNextStr)) {
> e.getEnvironment().getSharedData().put(USE_GET_
> OPERATION_IN_POST_SCANNER_NEXT, useGetInPostScannerNextStr);
> }
> return super.preScannerOpen(e, scan, s);
> }
>
> @Override
> public boolean *postScannerNext*(final
> ObserverContext
> e,
> final InternalScanner s, final List results, final int
> limit,
> final boolean hasMore) throws IOException {
> try {
>
> if (canUseGetOperation(e)) {
>
>//logic goes here
> }
> } catch (Exception ex) {
> logger.error("Exception in postScannerNext ", ex);
> throw new IOException(ex);
> }
> return hasMore;
> }
>
> @Override
> public void
> *postScannerClose*(ObserverContext
> e, InternalScanner s) throws IOException {
> if (canUseGetOperation(e)) {
> e.getEnvironment().getSharedData().remove(USE_
> GET_OPERATION_IN_POST_SCANNER_NEXT);
> }
> super.postScannerClose(e, s);
> }
>
> private boolean *canUseGetOperation*(final
> ObserverContext
> e) {
> String useGetOperationInPostScannerNext = (String)
> e.getEnvironment().getSharedData().get(USE_GET_OPERATION_IN_POST_SCANNER_
> NEXT);
> return Boolean.parseBoolean(useGetOperationInPostScannerNext);
> }
>
> Thanks,
> Raju,
> (972)273-0155 <(972)%20273-0155>.
>

Re: scope of RegionCoprocessorEnvironment sharedData

[Ted Yu]

The tricky part about correlating preScannerOpen() with postScannerNext()
call is that:

Scan object is not passed to postScannerNext() (therefore the attribute
passed from client is absent)
The thread handling postScannerNext() may be different from the thread
handling preScannerOpen()

There is RegionScannerHolder class but it is private to RSRpcServices.

FYI

On Thu, Jul 6, 2017 at 1:59 PM, Veerraju Tadimeti  wrote:

> I would like to invoke postScannerNext() based on a scan attribute. When
> two different scans are invoked simultaneously, one with attribute and one
> without attribute ,  how can I invoke postScannerNext() based on the scan
> attribute.
>
> Currently i am doing like this
>
>- checking the scan attribute in preScannerOpen(),
>   - if exists , set that attribute in sharedData,
>- check that variable in postScannerNext()
>- execute logic only if variable exists in sharedData
>
> If first scan operation with attribute is requested, we set that in
> sharedData in preScannerOpen(), meanwhile if second scan operation without
> attribute is requested, I think , according to your reply,  the
> postScannerNext() will get invoked for both the operations .
>
> is this correct? if Yes, what is the alternative.
>
> Thanks in Advance.
>
>
> Thanks,
> Raju,
> (972)273-0155.
>
> On Thu, Jul 6, 2017 at 1:39 PM, Ted Yu  wrote:
>
> > If the two different scan operations are for the same region, same
> instance
> > of coprocessor is involved.
> >
> > Otherwise, different instances of coprocessor are involved.
> >
> > On Thu, Jul 6, 2017 at 11:22 AM, Veerraju Tadimeti 
> > wrote:
> >
> > > for two different scan operations , will same instance of coprocessor
> > > executes?
> > >
> > > Thanks,
> > > Raju,
> > > (972)273-0155.
> > >
> > > On Thu, Jul 6, 2017 at 11:53 AM, Ted Yu  wrote:
> > >
> > > > sharedData is created when coprocessor is loaded (per region).
> > > >
> > > > So as long as the coprocessor instance doesn't change, the data you
> put
> > > in
> > > > should be there.
> > > >
> > > > On Thu, Jul 6, 2017 at 9:20 AM, Veerraju Tadimeti  >
> > > > wrote:
> > > >
> > > > > Is sharedData per Operation request ?
> > > > >
> > > > > I mean for a scan operation, if we put variables in sharedData in
> > > > > coprocessor, with a value,   will the same variables exists(with
> the
> > > same
> > > > > value) in sharedData in coprocessor for new scan operation ?
> > > > >
> > > > > Thanks,
> > > > > Raju,
> > > > > (972)273-0155.
> > > > >
> > > > > On Thu, Jul 6, 2017 at 10:38 AM, Ted Yu 
> wrote:
> > > > >
> > > > > > sharedData is in RegionEnvironment which is created
> > > > > > by RegionCoprocessorHost.
> > > > > > RegionCoprocessorHost is per region.
> > > > > >
> > > > > > When variable is removed in postScannerClose(), it wouldn't
> affect
> > > > > > sharedData
> > > > > > in another region.
> > > > > >
> > > > > > FYI
> > > > > >
> > > > > > On Thu, Jul 6, 2017 at 7:14 AM, Veerraju Tadimeti <
> > tvvr...@gmail.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > hi,
> > > > > > >
> > > > > > > I have few questions regarding scope of
> > > > *RegionCoprocessorEnvironment*
> > > > > > >  sharedData.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >- *Is sharedData map is shared accross all instances
> > > > simultaneously
> > > > > ?*
> > > > > > >   -  I am putting a variable in sharedData in
> > preScannerOpen()
> > > > > based
> > > > > > on
> > > > > > >   scan attribute,
> > > > > > >   - check that variable exists in postScannerNext() then
> > apply
> > > > > logic,
> > > > > > >   - remove the variable postScannerClose().
> > > > > > >   - If data is in multiple regions, when one coprocessor
> > > removes
> > > > > > >   variable in postScannerClose(), will the variable is NULL
> > for
> > > > > > another
> > > > > > >   region coprocessor in postScannerNext() ?
> > > > > > >
> > > > > > >
> > > > > > >- *Is sharedData map is shared across all the client
> > request
> > > > > > >operations ?*
> > > > > > >
> > > > > > > If a variable is set in sharedData for one client operation(say
> > > > SCAN),
> > > > > > will
> > > > > > > the variable is available for another client operation(new
> SCAN)
> > ?
> > > > > > >
> > > > > > >
> > > > > > >-  *Will the variables be garbage collected even if we dont
> > > > > implement
> > > > > > >(removed variables in sharedData) postScannerClose() method*
> > > > > > >
> > > > > > >
> > > > > > > Please find below the logic that I am using currently
> > > > > > > *CODE: *
> > > > > > >
> > > > > > > public RegionScanner
> > > > > > > *preScannerOpen*(ObserverContext
> > > > > > > e, Scan scan, RegionScanner s) throws IOException {
> > > > > > > byte[] useGetInPostScannerNext =
> > scan.getAttribute(USE_GET_
> > > > > > > 

Re: scope of RegionCoprocessorEnvironment sharedData

[Veerraju Tadimeti]

I would like to invoke postScannerNext() based on a scan attribute. When
two different scans are invoked simultaneously, one with attribute and one
without attribute ,  how can I invoke postScannerNext() based on the scan
attribute.

Currently i am doing like this

   - checking the scan attribute in preScannerOpen(),
  - if exists , set that attribute in sharedData,
   - check that variable in postScannerNext()
   - execute logic only if variable exists in sharedData

If first scan operation with attribute is requested, we set that in
sharedData in preScannerOpen(), meanwhile if second scan operation without
attribute is requested, I think , according to your reply,  the
postScannerNext() will get invoked for both the operations .

is this correct? if Yes, what is the alternative.

Thanks in Advance.


Thanks,
Raju,
(972)273-0155.

On Thu, Jul 6, 2017 at 1:39 PM, Ted Yu  wrote:

> If the two different scan operations are for the same region, same instance
> of coprocessor is involved.
>
> Otherwise, different instances of coprocessor are involved.
>
> On Thu, Jul 6, 2017 at 11:22 AM, Veerraju Tadimeti 
> wrote:
>
> > for two different scan operations , will same instance of coprocessor
> > executes?
> >
> > Thanks,
> > Raju,
> > (972)273-0155.
> >
> > On Thu, Jul 6, 2017 at 11:53 AM, Ted Yu  wrote:
> >
> > > sharedData is created when coprocessor is loaded (per region).
> > >
> > > So as long as the coprocessor instance doesn't change, the data you put
> > in
> > > should be there.
> > >
> > > On Thu, Jul 6, 2017 at 9:20 AM, Veerraju Tadimeti 
> > > wrote:
> > >
> > > > Is sharedData per Operation request ?
> > > >
> > > > I mean for a scan operation, if we put variables in sharedData in
> > > > coprocessor, with a value,   will the same variables exists(with the
> > same
> > > > value) in sharedData in coprocessor for new scan operation ?
> > > >
> > > > Thanks,
> > > > Raju,
> > > > (972)273-0155.
> > > >
> > > > On Thu, Jul 6, 2017 at 10:38 AM, Ted Yu  wrote:
> > > >
> > > > > sharedData is in RegionEnvironment which is created
> > > > > by RegionCoprocessorHost.
> > > > > RegionCoprocessorHost is per region.
> > > > >
> > > > > When variable is removed in postScannerClose(), it wouldn't affect
> > > > > sharedData
> > > > > in another region.
> > > > >
> > > > > FYI
> > > > >
> > > > > On Thu, Jul 6, 2017 at 7:14 AM, Veerraju Tadimeti <
> tvvr...@gmail.com
> > >
> > > > > wrote:
> > > > >
> > > > > > hi,
> > > > > >
> > > > > > I have few questions regarding scope of
> > > *RegionCoprocessorEnvironment*
> > > > > >  sharedData.
> > > > > >
> > > > > >
> > > > > >
> > > > > >- *Is sharedData map is shared accross all instances
> > > simultaneously
> > > > ?*
> > > > > >   -  I am putting a variable in sharedData in
> preScannerOpen()
> > > > based
> > > > > on
> > > > > >   scan attribute,
> > > > > >   - check that variable exists in postScannerNext() then
> apply
> > > > logic,
> > > > > >   - remove the variable postScannerClose().
> > > > > >   - If data is in multiple regions, when one coprocessor
> > removes
> > > > > >   variable in postScannerClose(), will the variable is NULL
> for
> > > > > another
> > > > > >   region coprocessor in postScannerNext() ?
> > > > > >
> > > > > >
> > > > > >- *Is sharedData map is shared across all the client
> request
> > > > > >operations ?*
> > > > > >
> > > > > > If a variable is set in sharedData for one client operation(say
> > > SCAN),
> > > > > will
> > > > > > the variable is available for another client operation(new SCAN)
> ?
> > > > > >
> > > > > >
> > > > > >-  *Will the variables be garbage collected even if we dont
> > > > implement
> > > > > >(removed variables in sharedData) postScannerClose() method*
> > > > > >
> > > > > >
> > > > > > Please find below the logic that I am using currently
> > > > > > *CODE: *
> > > > > >
> > > > > > public RegionScanner
> > > > > > *preScannerOpen*(ObserverContext
> > > > > > e, Scan scan, RegionScanner s) throws IOException {
> > > > > > byte[] useGetInPostScannerNext =
> scan.getAttribute(USE_GET_
> > > > > > OPERATION_IN_POST_SCANNER_NEXT);
> > > > > > String useGetInPostScannerNextStr = Bytes.toString(
> > > > > > useGetInPostScannerNext);
> > > > > > if (Boolean.parseBoolean(useGetInPostScannerNextStr)) {
> > > > > > e.getEnvironment().getSharedData().put(USE_GET_
> > > > > > OPERATION_IN_POST_SCANNER_NEXT, useGetInPostScannerNextStr);
> > > > > > }
> > > > > > return super.preScannerOpen(e, scan, s);
> > > > > > }
> > > > > >
> > > > > > @Override
> > > > > > public boolean *postScannerNext*(final
> > > > > > ObserverContext
> > > > > > e,
> > > > > > final InternalScanner s, final List results,
> > > final
> > > > > int
> > > > > > limit,
> > > > > > final 

Re: scope of RegionCoprocessorEnvironment sharedData

[Ted Yu]

If the two different scan operations are for the same region, same instance
of coprocessor is involved.

Otherwise, different instances of coprocessor are involved.

On Thu, Jul 6, 2017 at 11:22 AM, Veerraju Tadimeti 
wrote:

> for two different scan operations , will same instance of coprocessor
> executes?
>
> Thanks,
> Raju,
> (972)273-0155.
>
> On Thu, Jul 6, 2017 at 11:53 AM, Ted Yu  wrote:
>
> > sharedData is created when coprocessor is loaded (per region).
> >
> > So as long as the coprocessor instance doesn't change, the data you put
> in
> > should be there.
> >
> > On Thu, Jul 6, 2017 at 9:20 AM, Veerraju Tadimeti 
> > wrote:
> >
> > > Is sharedData per Operation request ?
> > >
> > > I mean for a scan operation, if we put variables in sharedData in
> > > coprocessor, with a value,   will the same variables exists(with the
> same
> > > value) in sharedData in coprocessor for new scan operation ?
> > >
> > > Thanks,
> > > Raju,
> > > (972)273-0155.
> > >
> > > On Thu, Jul 6, 2017 at 10:38 AM, Ted Yu  wrote:
> > >
> > > > sharedData is in RegionEnvironment which is created
> > > > by RegionCoprocessorHost.
> > > > RegionCoprocessorHost is per region.
> > > >
> > > > When variable is removed in postScannerClose(), it wouldn't affect
> > > > sharedData
> > > > in another region.
> > > >
> > > > FYI
> > > >
> > > > On Thu, Jul 6, 2017 at 7:14 AM, Veerraju Tadimeti  >
> > > > wrote:
> > > >
> > > > > hi,
> > > > >
> > > > > I have few questions regarding scope of
> > *RegionCoprocessorEnvironment*
> > > > >  sharedData.
> > > > >
> > > > >
> > > > >
> > > > >- *Is sharedData map is shared accross all instances
> > simultaneously
> > > ?*
> > > > >   -  I am putting a variable in sharedData in preScannerOpen()
> > > based
> > > > on
> > > > >   scan attribute,
> > > > >   - check that variable exists in postScannerNext() then apply
> > > logic,
> > > > >   - remove the variable postScannerClose().
> > > > >   - If data is in multiple regions, when one coprocessor
> removes
> > > > >   variable in postScannerClose(), will the variable is NULL for
> > > > another
> > > > >   region coprocessor in postScannerNext() ?
> > > > >
> > > > >
> > > > >- *Is sharedData map is shared across all the client request
> > > > >operations ?*
> > > > >
> > > > > If a variable is set in sharedData for one client operation(say
> > SCAN),
> > > > will
> > > > > the variable is available for another client operation(new SCAN) ?
> > > > >
> > > > >
> > > > >-  *Will the variables be garbage collected even if we dont
> > > implement
> > > > >(removed variables in sharedData) postScannerClose() method*
> > > > >
> > > > >
> > > > > Please find below the logic that I am using currently
> > > > > *CODE: *
> > > > >
> > > > > public RegionScanner
> > > > > *preScannerOpen*(ObserverContext
> > > > > e, Scan scan, RegionScanner s) throws IOException {
> > > > > byte[] useGetInPostScannerNext = scan.getAttribute(USE_GET_
> > > > > OPERATION_IN_POST_SCANNER_NEXT);
> > > > > String useGetInPostScannerNextStr = Bytes.toString(
> > > > > useGetInPostScannerNext);
> > > > > if (Boolean.parseBoolean(useGetInPostScannerNextStr)) {
> > > > > e.getEnvironment().getSharedData().put(USE_GET_
> > > > > OPERATION_IN_POST_SCANNER_NEXT, useGetInPostScannerNextStr);
> > > > > }
> > > > > return super.preScannerOpen(e, scan, s);
> > > > > }
> > > > >
> > > > > @Override
> > > > > public boolean *postScannerNext*(final
> > > > > ObserverContext
> > > > > e,
> > > > > final InternalScanner s, final List results,
> > final
> > > > int
> > > > > limit,
> > > > > final boolean hasMore) throws IOException {
> > > > > try {
> > > > >
> > > > > if (canUseGetOperation(e)) {
> > > > >
> > > > >//logic goes here
> > > > > }
> > > > > } catch (Exception ex) {
> > > > > logger.error("Exception in postScannerNext ", ex);
> > > > > throw new IOException(ex);
> > > > > }
> > > > > return hasMore;
> > > > > }
> > > > >
> > > > > @Override
> > > > > public void
> > > > > *postScannerClose*(ObserverContext
> > > > > e, InternalScanner s) throws IOException {
> > > > > if (canUseGetOperation(e)) {
> > > > > e.getEnvironment().getSharedData().remove(USE_
> > > > > GET_OPERATION_IN_POST_SCANNER_NEXT);
> > > > > }
> > > > > super.postScannerClose(e, s);
> > > > > }
> > > > >
> > > > > private boolean *canUseGetOperation*(final
> > > > > ObserverContext
> > > > > e) {
> > > > > String useGetOperationInPostScannerNext = (String)
> > > > > e.getEnvironment().getSharedData().get(USE_GET_
> > > > OPERATION_IN_POST_SCANNER_
> > > > > NEXT);
> > > > > return 

Re: scope of RegionCoprocessorEnvironment sharedData

[Veerraju Tadimeti]

for two different scan operations , will same instance of coprocessor
executes?

Thanks,
Raju,
(972)273-0155.

On Thu, Jul 6, 2017 at 11:53 AM, Ted Yu  wrote:

> sharedData is created when coprocessor is loaded (per region).
>
> So as long as the coprocessor instance doesn't change, the data you put in
> should be there.
>
> On Thu, Jul 6, 2017 at 9:20 AM, Veerraju Tadimeti 
> wrote:
>
> > Is sharedData per Operation request ?
> >
> > I mean for a scan operation, if we put variables in sharedData in
> > coprocessor, with a value,   will the same variables exists(with the same
> > value) in sharedData in coprocessor for new scan operation ?
> >
> > Thanks,
> > Raju,
> > (972)273-0155.
> >
> > On Thu, Jul 6, 2017 at 10:38 AM, Ted Yu  wrote:
> >
> > > sharedData is in RegionEnvironment which is created
> > > by RegionCoprocessorHost.
> > > RegionCoprocessorHost is per region.
> > >
> > > When variable is removed in postScannerClose(), it wouldn't affect
> > > sharedData
> > > in another region.
> > >
> > > FYI
> > >
> > > On Thu, Jul 6, 2017 at 7:14 AM, Veerraju Tadimeti 
> > > wrote:
> > >
> > > > hi,
> > > >
> > > > I have few questions regarding scope of
> *RegionCoprocessorEnvironment*
> > > >  sharedData.
> > > >
> > > >
> > > >
> > > >- *Is sharedData map is shared accross all instances
> simultaneously
> > ?*
> > > >   -  I am putting a variable in sharedData in preScannerOpen()
> > based
> > > on
> > > >   scan attribute,
> > > >   - check that variable exists in postScannerNext() then apply
> > logic,
> > > >   - remove the variable postScannerClose().
> > > >   - If data is in multiple regions, when one coprocessor removes
> > > >   variable in postScannerClose(), will the variable is NULL for
> > > another
> > > >   region coprocessor in postScannerNext() ?
> > > >
> > > >
> > > >- *Is sharedData map is shared across all the client request
> > > >operations ?*
> > > >
> > > > If a variable is set in sharedData for one client operation(say
> SCAN),
> > > will
> > > > the variable is available for another client operation(new SCAN) ?
> > > >
> > > >
> > > >-  *Will the variables be garbage collected even if we dont
> > implement
> > > >(removed variables in sharedData) postScannerClose() method*
> > > >
> > > >
> > > > Please find below the logic that I am using currently
> > > > *CODE: *
> > > >
> > > > public RegionScanner
> > > > *preScannerOpen*(ObserverContext
> > > > e, Scan scan, RegionScanner s) throws IOException {
> > > > byte[] useGetInPostScannerNext = scan.getAttribute(USE_GET_
> > > > OPERATION_IN_POST_SCANNER_NEXT);
> > > > String useGetInPostScannerNextStr = Bytes.toString(
> > > > useGetInPostScannerNext);
> > > > if (Boolean.parseBoolean(useGetInPostScannerNextStr)) {
> > > > e.getEnvironment().getSharedData().put(USE_GET_
> > > > OPERATION_IN_POST_SCANNER_NEXT, useGetInPostScannerNextStr);
> > > > }
> > > > return super.preScannerOpen(e, scan, s);
> > > > }
> > > >
> > > > @Override
> > > > public boolean *postScannerNext*(final
> > > > ObserverContext
> > > > e,
> > > > final InternalScanner s, final List results,
> final
> > > int
> > > > limit,
> > > > final boolean hasMore) throws IOException {
> > > > try {
> > > >
> > > > if (canUseGetOperation(e)) {
> > > >
> > > >//logic goes here
> > > > }
> > > > } catch (Exception ex) {
> > > > logger.error("Exception in postScannerNext ", ex);
> > > > throw new IOException(ex);
> > > > }
> > > > return hasMore;
> > > > }
> > > >
> > > > @Override
> > > > public void
> > > > *postScannerClose*(ObserverContext
> > > > e, InternalScanner s) throws IOException {
> > > > if (canUseGetOperation(e)) {
> > > > e.getEnvironment().getSharedData().remove(USE_
> > > > GET_OPERATION_IN_POST_SCANNER_NEXT);
> > > > }
> > > > super.postScannerClose(e, s);
> > > > }
> > > >
> > > > private boolean *canUseGetOperation*(final
> > > > ObserverContext
> > > > e) {
> > > > String useGetOperationInPostScannerNext = (String)
> > > > e.getEnvironment().getSharedData().get(USE_GET_
> > > OPERATION_IN_POST_SCANNER_
> > > > NEXT);
> > > > return Boolean.parseBoolean(useGetOperationInPostScannerNe
> xt);
> > > > }
> > > >
> > > > Thanks,
> > > > Raju,
> > > > (972)273-0155 <(972)%20273-0155>.
> > > >
> > >
> >
>

Re: scope of RegionCoprocessorEnvironment sharedData

[Ted Yu]

sharedData is created when coprocessor is loaded (per region).

So as long as the coprocessor instance doesn't change, the data you put in
should be there.

On Thu, Jul 6, 2017 at 9:20 AM, Veerraju Tadimeti  wrote:

> Is sharedData per Operation request ?
>
> I mean for a scan operation, if we put variables in sharedData in
> coprocessor, with a value,   will the same variables exists(with the same
> value) in sharedData in coprocessor for new scan operation ?
>
> Thanks,
> Raju,
> (972)273-0155.
>
> On Thu, Jul 6, 2017 at 10:38 AM, Ted Yu  wrote:
>
> > sharedData is in RegionEnvironment which is created
> > by RegionCoprocessorHost.
> > RegionCoprocessorHost is per region.
> >
> > When variable is removed in postScannerClose(), it wouldn't affect
> > sharedData
> > in another region.
> >
> > FYI
> >
> > On Thu, Jul 6, 2017 at 7:14 AM, Veerraju Tadimeti 
> > wrote:
> >
> > > hi,
> > >
> > > I have few questions regarding scope of *RegionCoprocessorEnvironment*
> > >  sharedData.
> > >
> > >
> > >
> > >- *Is sharedData map is shared accross all instances simultaneously
> ?*
> > >   -  I am putting a variable in sharedData in preScannerOpen()
> based
> > on
> > >   scan attribute,
> > >   - check that variable exists in postScannerNext() then apply
> logic,
> > >   - remove the variable postScannerClose().
> > >   - If data is in multiple regions, when one coprocessor removes
> > >   variable in postScannerClose(), will the variable is NULL for
> > another
> > >   region coprocessor in postScannerNext() ?
> > >
> > >
> > >- *Is sharedData map is shared across all the client request
> > >operations ?*
> > >
> > > If a variable is set in sharedData for one client operation(say SCAN),
> > will
> > > the variable is available for another client operation(new SCAN) ?
> > >
> > >
> > >-  *Will the variables be garbage collected even if we dont
> implement
> > >(removed variables in sharedData) postScannerClose() method*
> > >
> > >
> > > Please find below the logic that I am using currently
> > > *CODE: *
> > >
> > > public RegionScanner
> > > *preScannerOpen*(ObserverContext
> > > e, Scan scan, RegionScanner s) throws IOException {
> > > byte[] useGetInPostScannerNext = scan.getAttribute(USE_GET_
> > > OPERATION_IN_POST_SCANNER_NEXT);
> > > String useGetInPostScannerNextStr = Bytes.toString(
> > > useGetInPostScannerNext);
> > > if (Boolean.parseBoolean(useGetInPostScannerNextStr)) {
> > > e.getEnvironment().getSharedData().put(USE_GET_
> > > OPERATION_IN_POST_SCANNER_NEXT, useGetInPostScannerNextStr);
> > > }
> > > return super.preScannerOpen(e, scan, s);
> > > }
> > >
> > > @Override
> > > public boolean *postScannerNext*(final
> > > ObserverContext
> > > e,
> > > final InternalScanner s, final List results, final
> > int
> > > limit,
> > > final boolean hasMore) throws IOException {
> > > try {
> > >
> > > if (canUseGetOperation(e)) {
> > >
> > >//logic goes here
> > > }
> > > } catch (Exception ex) {
> > > logger.error("Exception in postScannerNext ", ex);
> > > throw new IOException(ex);
> > > }
> > > return hasMore;
> > > }
> > >
> > > @Override
> > > public void
> > > *postScannerClose*(ObserverContext
> > > e, InternalScanner s) throws IOException {
> > > if (canUseGetOperation(e)) {
> > > e.getEnvironment().getSharedData().remove(USE_
> > > GET_OPERATION_IN_POST_SCANNER_NEXT);
> > > }
> > > super.postScannerClose(e, s);
> > > }
> > >
> > > private boolean *canUseGetOperation*(final
> > > ObserverContext
> > > e) {
> > > String useGetOperationInPostScannerNext = (String)
> > > e.getEnvironment().getSharedData().get(USE_GET_
> > OPERATION_IN_POST_SCANNER_
> > > NEXT);
> > > return Boolean.parseBoolean(useGetOperationInPostScannerNext);
> > > }
> > >
> > > Thanks,
> > > Raju,
> > > (972)273-0155 <(972)%20273-0155>.
> > >
> >
>

Re: scope of RegionCoprocessorEnvironment sharedData

[Veerraju Tadimeti]

Is sharedData per Operation request ?

I mean for a scan operation, if we put variables in sharedData in
coprocessor, with a value,   will the same variables exists(with the same
value) in sharedData in coprocessor for new scan operation ?

Thanks,
Raju,
(972)273-0155.

On Thu, Jul 6, 2017 at 10:38 AM, Ted Yu  wrote:

> sharedData is in RegionEnvironment which is created
> by RegionCoprocessorHost.
> RegionCoprocessorHost is per region.
>
> When variable is removed in postScannerClose(), it wouldn't affect
> sharedData
> in another region.
>
> FYI
>
> On Thu, Jul 6, 2017 at 7:14 AM, Veerraju Tadimeti 
> wrote:
>
> > hi,
> >
> > I have few questions regarding scope of *RegionCoprocessorEnvironment*
> >  sharedData.
> >
> >
> >
> >- *Is sharedData map is shared accross all instances simultaneously ?*
> >   -  I am putting a variable in sharedData in preScannerOpen() based
> on
> >   scan attribute,
> >   - check that variable exists in postScannerNext() then apply logic,
> >   - remove the variable postScannerClose().
> >   - If data is in multiple regions, when one coprocessor removes
> >   variable in postScannerClose(), will the variable is NULL for
> another
> >   region coprocessor in postScannerNext() ?
> >
> >
> >- *Is sharedData map is shared across all the client request
> >operations ?*
> >
> > If a variable is set in sharedData for one client operation(say SCAN),
> will
> > the variable is available for another client operation(new SCAN) ?
> >
> >
> >-  *Will the variables be garbage collected even if we dont implement
> >(removed variables in sharedData) postScannerClose() method*
> >
> >
> > Please find below the logic that I am using currently
> > *CODE: *
> >
> > public RegionScanner
> > *preScannerOpen*(ObserverContext
> > e, Scan scan, RegionScanner s) throws IOException {
> > byte[] useGetInPostScannerNext = scan.getAttribute(USE_GET_
> > OPERATION_IN_POST_SCANNER_NEXT);
> > String useGetInPostScannerNextStr = Bytes.toString(
> > useGetInPostScannerNext);
> > if (Boolean.parseBoolean(useGetInPostScannerNextStr)) {
> > e.getEnvironment().getSharedData().put(USE_GET_
> > OPERATION_IN_POST_SCANNER_NEXT, useGetInPostScannerNextStr);
> > }
> > return super.preScannerOpen(e, scan, s);
> > }
> >
> > @Override
> > public boolean *postScannerNext*(final
> > ObserverContext
> > e,
> > final InternalScanner s, final List results, final
> int
> > limit,
> > final boolean hasMore) throws IOException {
> > try {
> >
> > if (canUseGetOperation(e)) {
> >
> >//logic goes here
> > }
> > } catch (Exception ex) {
> > logger.error("Exception in postScannerNext ", ex);
> > throw new IOException(ex);
> > }
> > return hasMore;
> > }
> >
> > @Override
> > public void
> > *postScannerClose*(ObserverContext
> > e, InternalScanner s) throws IOException {
> > if (canUseGetOperation(e)) {
> > e.getEnvironment().getSharedData().remove(USE_
> > GET_OPERATION_IN_POST_SCANNER_NEXT);
> > }
> > super.postScannerClose(e, s);
> > }
> >
> > private boolean *canUseGetOperation*(final
> > ObserverContext
> > e) {
> > String useGetOperationInPostScannerNext = (String)
> > e.getEnvironment().getSharedData().get(USE_GET_
> OPERATION_IN_POST_SCANNER_
> > NEXT);
> > return Boolean.parseBoolean(useGetOperationInPostScannerNext);
> > }
> >
> > Thanks,
> > Raju,
> > (972)273-0155 <(972)%20273-0155>.
> >
>

Re: scope of RegionCoprocessorEnvironment sharedData

[Ted Yu]

sharedData is in RegionEnvironment which is created
by RegionCoprocessorHost.
RegionCoprocessorHost is per region.

When variable is removed in postScannerClose(), it wouldn't affect sharedData
in another region.

FYI

On Thu, Jul 6, 2017 at 7:14 AM, Veerraju Tadimeti  wrote:

> hi,
>
> I have few questions regarding scope of *RegionCoprocessorEnvironment*
>  sharedData.
>
>
>
>- *Is sharedData map is shared accross all instances simultaneously ?*
>   -  I am putting a variable in sharedData in preScannerOpen() based on
>   scan attribute,
>   - check that variable exists in postScannerNext() then apply logic,
>   - remove the variable postScannerClose().
>   - If data is in multiple regions, when one coprocessor removes
>   variable in postScannerClose(), will the variable is NULL for another
>   region coprocessor in postScannerNext() ?
>
>
>- *Is sharedData map is shared across all the client request
>operations ?*
>
> If a variable is set in sharedData for one client operation(say SCAN), will
> the variable is available for another client operation(new SCAN) ?
>
>
>-  *Will the variables be garbage collected even if we dont implement
>(removed variables in sharedData) postScannerClose() method*
>
>
> Please find below the logic that I am using currently
> *CODE: *
>
> public RegionScanner
> *preScannerOpen*(ObserverContext
> e, Scan scan, RegionScanner s) throws IOException {
> byte[] useGetInPostScannerNext = scan.getAttribute(USE_GET_
> OPERATION_IN_POST_SCANNER_NEXT);
> String useGetInPostScannerNextStr = Bytes.toString(
> useGetInPostScannerNext);
> if (Boolean.parseBoolean(useGetInPostScannerNextStr)) {
> e.getEnvironment().getSharedData().put(USE_GET_
> OPERATION_IN_POST_SCANNER_NEXT, useGetInPostScannerNextStr);
> }
> return super.preScannerOpen(e, scan, s);
> }
>
> @Override
> public boolean *postScannerNext*(final
> ObserverContext
> e,
> final InternalScanner s, final List results, final int
> limit,
> final boolean hasMore) throws IOException {
> try {
>
> if (canUseGetOperation(e)) {
>
>//logic goes here
> }
> } catch (Exception ex) {
> logger.error("Exception in postScannerNext ", ex);
> throw new IOException(ex);
> }
> return hasMore;
> }
>
> @Override
> public void
> *postScannerClose*(ObserverContext
> e, InternalScanner s) throws IOException {
> if (canUseGetOperation(e)) {
> e.getEnvironment().getSharedData().remove(USE_
> GET_OPERATION_IN_POST_SCANNER_NEXT);
> }
> super.postScannerClose(e, s);
> }
>
> private boolean *canUseGetOperation*(final
> ObserverContext
> e) {
> String useGetOperationInPostScannerNext = (String)
> e.getEnvironment().getSharedData().get(USE_GET_OPERATION_IN_POST_SCANNER_
> NEXT);
> return Boolean.parseBoolean(useGetOperationInPostScannerNext);
> }
>
> Thanks,
> Raju,
> (972)273-0155 <(972)%20273-0155>.
>