yeah, you are right. I am getting scanner in postScannerOpen(). I tested with preScannerOpen() earlier ,which is null scanner. Thanks alot.
Thanks, Raju, (972)273-0155. On Thu, Jul 13, 2017 at 11:50 PM, Anoop John <[email protected]> wrote: > No co processors can be configured either globally at cluster level > (RS configuration) or at table level. > > You dont need preScannerOpen. I checked the code and we are passing > the actual scanner been created to postScannerOpen() method. This > being null in ur case been strange! Pls make sure u r implementing > post hook not pre. > > -Anoop- > > On Wed, Jul 12, 2017 at 3:01 AM, Veerraju Tadimeti <[email protected]> > wrote: > > Can I load coprocessor dynamically for a scan operation, it should not > be loaded for another scan operation if not intended > > btw I invoke scan from hive > > > > Sent from my iPhone > > > >> On Jul 11, 2017, at 4:15 PM, Veerraju Tadimeti <[email protected]> > wrote: > >> > >> hi, > >> > >> Hi John, > >> > >> Thanks for the reply. > >> > >> I implemented #2 (another way) in ur above post: > >> > >> > >> > >> i debug the logs : in PostScannerOpen() , regionScanner method > parameter object is null > >> > >> Also, in preScannerOpen() , i returned return super.preScannerOpen(e, > scan, new DelegateRegionScanner(s)); > >> in postScannerNext() , internalScanner object is > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl > >> > >> #1 way (Put scanner in local map) - may not be possible, cos if two > different scan operation with and without attribute hits at the same time, > how can we differentiate in postScannerNext. > >> > >> > >> Thanks, > >> Raju, > >> (972)273-0155. > >> > >>> On Tue, Jul 11, 2017 at 8:05 AM, Anoop John <[email protected]> > wrote: > >>> Ya. It is the same RegionScanner impl in use only being passed. Ya > >>> the param type should have been RegionScanner I guess. We made that > >>> mistake! > >>> -Anoop- > >>> > >>> On Mon, Jul 10, 2017 at 8:37 PM, Ted Yu <[email protected]> wrote: > >>> > The tricky part is that postScannerNext() passes InternalScanner > parameter > >>> > instead of RegionScanner. > >>> > > >>> > FYI > >>> > > >>> > On Sun, Jul 9, 2017 at 10:57 PM, Anoop John <[email protected]> > wrote: > >>> > > >>> >> Ya as Ted said, u are not getting Scan object in the postScannerNext > >>> >> and so can not make use of the attribute in Scan within this hook. > >>> >> Just setting the sharedData variable will cause issue with > concurrent > >>> >> scans. (As u imagine) > >>> >> > >>> >> So I can think of solving this in 2 possible ways. (May be more ways > >>> >> possible) > >>> >> > >>> >> 1. U keep a Map within ur CP impl. You implement postScannerOpen > >>> >> where u will get the ref to Scanner been created as well as the > Scan. > >>> >> If the Scan is having attribute, keep that scanner within ur Map. > >>> >> During postScannerNext check if the coming in scanner is there in > ur > >>> >> Map. If so that means this is the one where u can do the action. > >>> >> Also dont forget to implement postScannerClose and remove that > scanner > >>> >> from the Map. Here u might have some perf penalty as u have to add > >>> >> and get from Map which has to be a concurrent map too. > >>> >> > >>> >> Another way > >>> >> > >>> >> 2. Create a custom scanner implementing RegionScanner. The new one > >>> >> has to take an original Region Scanner and just delegate the calls. > On > >>> >> postScannerOpen, u will get the original scanner been created and u > >>> >> can just wrap it with ur new scanner object. ( If the Scan object is > >>> >> having required attribute).. In postScannerNext() u can check for > ur > >>> >> own RegionScanner type and if so u can do action. > >>> >> > >>> >> > >>> >> -Anoop- > >>> >> > >>> >> > >>> >> On Sat, Jul 8, 2017 at 9:13 PM, Ted Yu <[email protected]> wrote: > >>> >> > if (canUseGetOperation(e)) { > >>> >> > //logic goes here > >>> >> > > >>> >> > Does your Get target the same region being scanned ? > >>> >> > If not, issuing the Get is not advised since the other region may > be > >>> >> hosted > >>> >> > on different region server. > >>> >> > > >>> >> > Cheers > >>> >> > > >>> >> > On Thu, Jul 6, 2017 at 7:14 AM, Veerraju Tadimeti < > [email protected]> > >>> >> wrote: > >>> >> > > >>> >> >> hi, > >>> >> >> > >>> >> >> I have few questions regarding scope of > *RegionCoprocessorEnvironment* > >>> >> >> sharedData. > >>> >> >> > >>> >> >> > >>> >> >> > >>> >> >> - *Is sharedData map is shared accross all instances > simultaneously > >>> >> ?* > >>> >> >> - I am putting a variable in sharedData in > preScannerOpen() > >>> >> based on > >>> >> >> scan attribute, > >>> >> >> - check that variable exists in postScannerNext() then > apply > >>> >> logic, > >>> >> >> - remove the variable postScannerClose(). > >>> >> >> - If data is in multiple regions, when one coprocessor > removes > >>> >> >> variable in postScannerClose(), will the variable is NULL > for > >>> >> another > >>> >> >> region coprocessor in postScannerNext() ? > >>> >> >> > >>> >> >> > >>> >> >> - * Is sharedData map is shared across all the client > request > >>> >> >> operations ?* > >>> >> >> > >>> >> >> If a variable is set in sharedData for one client operation(say > SCAN), > >>> >> will > >>> >> >> the variable is available for another client operation(new SCAN) > ? > >>> >> >> > >>> >> >> > >>> >> >> - *Will the variables be garbage collected even if we dont > implement > >>> >> >> (removed variables in sharedData) postScannerClose() method* > >>> >> >> > >>> >> >> > >>> >> >> Please find below the logic that I am using currently > >>> >> >> *CODE: * > >>> >> >> > >>> >> >> public RegionScanner > >>> >> >> *preScannerOpen*(ObserverContext<RegionCoprocessorEnvironment> > >>> >> >> e, Scan scan, RegionScanner s) throws IOException { > >>> >> >> byte[] useGetInPostScannerNext = > scan.getAttribute(USE_GET_ > >>> >> >> OPERATION_IN_POST_SCANNER_NEXT); > >>> >> >> String useGetInPostScannerNextStr = Bytes.toString( > >>> >> >> useGetInPostScannerNext); > >>> >> >> if (Boolean.parseBoolean(useGetInPostScannerNextStr)) { > >>> >> >> e.getEnvironment().getSharedData().put(USE_GET_ > >>> >> >> OPERATION_IN_POST_SCANNER_NEXT, useGetInPostScannerNextStr); > >>> >> >> } > >>> >> >> return super.preScannerOpen(e, scan, s); > >>> >> >> } > >>> >> >> > >>> >> >> @Override > >>> >> >> public boolean *postScannerNext*(final > >>> >> >> ObserverContext<RegionCoprocessorEnvironment> > >>> >> >> e, > >>> >> >> final InternalScanner s, final List<Result> results, > final > >>> >> int > >>> >> >> limit, > >>> >> >> final boolean hasMore) throws IOException { > >>> >> >> try { > >>> >> >> > >>> >> >> if (canUseGetOperation(e)) { > >>> >> >> > >>> >> >> //logic goes here > >>> >> >> } > >>> >> >> } catch (Exception ex) { > >>> >> >> logger.error("Exception in postScannerNext ", ex); > >>> >> >> throw new IOException(ex); > >>> >> >> } > >>> >> >> return hasMore; > >>> >> >> } > >>> >> >> > >>> >> >> @Override > >>> >> >> public void > >>> >> >> *postScannerClose*(ObserverContext<RegionCoprocessorEnvironment> > >>> >> >> e, InternalScanner s) throws IOException { > >>> >> >> if (canUseGetOperation(e)) { > >>> >> >> e.getEnvironment().getSharedData().remove(USE_ > >>> >> >> GET_OPERATION_IN_POST_SCANNER_NEXT); > >>> >> >> } > >>> >> >> super.postScannerClose(e, s); > >>> >> >> } > >>> >> >> > >>> >> >> private boolean *canUseGetOperation*(final > >>> >> >> ObserverContext<RegionCoprocessorEnvironment> > >>> >> >> e) { > >>> >> >> String useGetOperationInPostScannerNext = (String) > >>> >> >> e.getEnvironment().getSharedData().get(USE_GET_ > >>> >> OPERATION_IN_POST_SCANNER_ > >>> >> >> NEXT); > >>> >> >> return Boolean.parseBoolean( > useGetOperationInPostScannerNext); > >>> >> >> } > >>> >> >> > >>> >> >> Thanks, > >>> >> >> Raju, > >>> >> >> (972)273-0155 <(972)%20273-0155>. > >>> >> >> > >>> >> > >> >
