No co processors can be configured either globally at cluster level (RS configuration) or at table level.
You dont need preScannerOpen. I checked the code and we are passing the actual scanner been created to postScannerOpen() method. This being null in ur case been strange! Pls make sure u r implementing post hook not pre. -Anoop- On Wed, Jul 12, 2017 at 3:01 AM, Veerraju Tadimeti <[email protected]> wrote: > Can I load coprocessor dynamically for a scan operation, it should not be > loaded for another scan operation if not intended > btw I invoke scan from hive > > Sent from my iPhone > >> On Jul 11, 2017, at 4:15 PM, Veerraju Tadimeti <[email protected]> wrote: >> >> hi, >> >> Hi John, >> >> Thanks for the reply. >> >> I implemented #2 (another way) in ur above post: >> >> >> >> i debug the logs : in PostScannerOpen() , regionScanner method parameter >> object is null >> >> Also, in preScannerOpen() , i returned return super.preScannerOpen(e, scan, >> new DelegateRegionScanner(s)); >> in postScannerNext() , internalScanner object is >> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl >> >> #1 way (Put scanner in local map) - may not be possible, cos if two >> different scan operation with and without attribute hits at the same time, >> how can we differentiate in postScannerNext. >> >> >> Thanks, >> Raju, >> (972)273-0155. >> >>> On Tue, Jul 11, 2017 at 8:05 AM, Anoop John <[email protected]> wrote: >>> Ya. It is the same RegionScanner impl in use only being passed. Ya >>> the param type should have been RegionScanner I guess. We made that >>> mistake! >>> -Anoop- >>> >>> On Mon, Jul 10, 2017 at 8:37 PM, Ted Yu <[email protected]> wrote: >>> > The tricky part is that postScannerNext() passes InternalScanner parameter >>> > instead of RegionScanner. >>> > >>> > FYI >>> > >>> > On Sun, Jul 9, 2017 at 10:57 PM, Anoop John <[email protected]> wrote: >>> > >>> >> Ya as Ted said, u are not getting Scan object in the postScannerNext >>> >> and so can not make use of the attribute in Scan within this hook. >>> >> Just setting the sharedData variable will cause issue with concurrent >>> >> scans. (As u imagine) >>> >> >>> >> So I can think of solving this in 2 possible ways. (May be more ways >>> >> possible) >>> >> >>> >> 1. U keep a Map within ur CP impl. You implement postScannerOpen >>> >> where u will get the ref to Scanner been created as well as the Scan. >>> >> If the Scan is having attribute, keep that scanner within ur Map. >>> >> During postScannerNext check if the coming in scanner is there in ur >>> >> Map. If so that means this is the one where u can do the action. >>> >> Also dont forget to implement postScannerClose and remove that scanner >>> >> from the Map. Here u might have some perf penalty as u have to add >>> >> and get from Map which has to be a concurrent map too. >>> >> >>> >> Another way >>> >> >>> >> 2. Create a custom scanner implementing RegionScanner. The new one >>> >> has to take an original Region Scanner and just delegate the calls. On >>> >> postScannerOpen, u will get the original scanner been created and u >>> >> can just wrap it with ur new scanner object. ( If the Scan object is >>> >> having required attribute).. In postScannerNext() u can check for ur >>> >> own RegionScanner type and if so u can do action. >>> >> >>> >> >>> >> -Anoop- >>> >> >>> >> >>> >> On Sat, Jul 8, 2017 at 9:13 PM, Ted Yu <[email protected]> wrote: >>> >> > if (canUseGetOperation(e)) { >>> >> > //logic goes here >>> >> > >>> >> > Does your Get target the same region being scanned ? >>> >> > If not, issuing the Get is not advised since the other region may be >>> >> hosted >>> >> > on different region server. >>> >> > >>> >> > Cheers >>> >> > >>> >> > On Thu, Jul 6, 2017 at 7:14 AM, Veerraju Tadimeti <[email protected]> >>> >> wrote: >>> >> > >>> >> >> hi, >>> >> >> >>> >> >> I have few questions regarding scope of *RegionCoprocessorEnvironment* >>> >> >> sharedData. >>> >> >> >>> >> >> >>> >> >> >>> >> >> - *Is sharedData map is shared accross all instances simultaneously >>> >> ?* >>> >> >> - I am putting a variable in sharedData in preScannerOpen() >>> >> based on >>> >> >> scan attribute, >>> >> >> - check that variable exists in postScannerNext() then apply >>> >> logic, >>> >> >> - remove the variable postScannerClose(). >>> >> >> - If data is in multiple regions, when one coprocessor removes >>> >> >> variable in postScannerClose(), will the variable is NULL for >>> >> another >>> >> >> region coprocessor in postScannerNext() ? >>> >> >> >>> >> >> >>> >> >> - * Is sharedData map is shared across all the client request >>> >> >> operations ?* >>> >> >> >>> >> >> If a variable is set in sharedData for one client operation(say SCAN), >>> >> will >>> >> >> the variable is available for another client operation(new SCAN) ? >>> >> >> >>> >> >> >>> >> >> - *Will the variables be garbage collected even if we dont >>> >> >> implement >>> >> >> (removed variables in sharedData) postScannerClose() method* >>> >> >> >>> >> >> >>> >> >> Please find below the logic that I am using currently >>> >> >> *CODE: * >>> >> >> >>> >> >> public RegionScanner >>> >> >> *preScannerOpen*(ObserverContext<RegionCoprocessorEnvironment> >>> >> >> e, Scan scan, RegionScanner s) throws IOException { >>> >> >> byte[] useGetInPostScannerNext = scan.getAttribute(USE_GET_ >>> >> >> OPERATION_IN_POST_SCANNER_NEXT); >>> >> >> String useGetInPostScannerNextStr = Bytes.toString( >>> >> >> useGetInPostScannerNext); >>> >> >> if (Boolean.parseBoolean(useGetInPostScannerNextStr)) { >>> >> >> e.getEnvironment().getSharedData().put(USE_GET_ >>> >> >> OPERATION_IN_POST_SCANNER_NEXT, useGetInPostScannerNextStr); >>> >> >> } >>> >> >> return super.preScannerOpen(e, scan, s); >>> >> >> } >>> >> >> >>> >> >> @Override >>> >> >> public boolean *postScannerNext*(final >>> >> >> ObserverContext<RegionCoprocessorEnvironment> >>> >> >> e, >>> >> >> final InternalScanner s, final List<Result> results, final >>> >> int >>> >> >> limit, >>> >> >> final boolean hasMore) throws IOException { >>> >> >> try { >>> >> >> >>> >> >> if (canUseGetOperation(e)) { >>> >> >> >>> >> >> //logic goes here >>> >> >> } >>> >> >> } catch (Exception ex) { >>> >> >> logger.error("Exception in postScannerNext ", ex); >>> >> >> throw new IOException(ex); >>> >> >> } >>> >> >> return hasMore; >>> >> >> } >>> >> >> >>> >> >> @Override >>> >> >> public void >>> >> >> *postScannerClose*(ObserverContext<RegionCoprocessorEnvironment> >>> >> >> e, InternalScanner s) throws IOException { >>> >> >> if (canUseGetOperation(e)) { >>> >> >> e.getEnvironment().getSharedData().remove(USE_ >>> >> >> GET_OPERATION_IN_POST_SCANNER_NEXT); >>> >> >> } >>> >> >> super.postScannerClose(e, s); >>> >> >> } >>> >> >> >>> >> >> private boolean *canUseGetOperation*(final >>> >> >> ObserverContext<RegionCoprocessorEnvironment> >>> >> >> e) { >>> >> >> String useGetOperationInPostScannerNext = (String) >>> >> >> e.getEnvironment().getSharedData().get(USE_GET_ >>> >> OPERATION_IN_POST_SCANNER_ >>> >> >> NEXT); >>> >> >> return Boolean.parseBoolean(useGetOperationInPostScannerNext); >>> >> >> } >>> >> >> >>> >> >> Thanks, >>> >> >> Raju, >>> >> >> (972)273-0155 <(972)%20273-0155>. >>> >> >> >>> >> >>
