Can I load coprocessor dynamically for a scan operation, it should not be loaded for another scan operation if not intended btw I invoke scan from hive
Sent from my iPhone > On Jul 11, 2017, at 4:15 PM, Veerraju Tadimeti <[email protected]> wrote: > > hi, > > Hi John, > > Thanks for the reply. > > I implemented #2 (another way) in ur above post: > > > > i debug the logs : in PostScannerOpen() , regionScanner method parameter > object is null > > Also, in preScannerOpen() , i returned return super.preScannerOpen(e, scan, > new DelegateRegionScanner(s)); > in postScannerNext() , internalScanner object is > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl > > #1 way (Put scanner in local map) - may not be possible, cos if two different > scan operation with and without attribute hits at the same time, how can we > differentiate in postScannerNext. > > > Thanks, > Raju, > (972)273-0155. > >> On Tue, Jul 11, 2017 at 8:05 AM, Anoop John <[email protected]> wrote: >> Ya. It is the same RegionScanner impl in use only being passed. Ya >> the param type should have been RegionScanner I guess. We made that >> mistake! >> -Anoop- >> >> On Mon, Jul 10, 2017 at 8:37 PM, Ted Yu <[email protected]> wrote: >> > The tricky part is that postScannerNext() passes InternalScanner parameter >> > instead of RegionScanner. >> > >> > FYI >> > >> > On Sun, Jul 9, 2017 at 10:57 PM, Anoop John <[email protected]> wrote: >> > >> >> Ya as Ted said, u are not getting Scan object in the postScannerNext >> >> and so can not make use of the attribute in Scan within this hook. >> >> Just setting the sharedData variable will cause issue with concurrent >> >> scans. (As u imagine) >> >> >> >> So I can think of solving this in 2 possible ways. (May be more ways >> >> possible) >> >> >> >> 1. U keep a Map within ur CP impl. You implement postScannerOpen >> >> where u will get the ref to Scanner been created as well as the Scan. >> >> If the Scan is having attribute, keep that scanner within ur Map. >> >> During postScannerNext check if the coming in scanner is there in ur >> >> Map. If so that means this is the one where u can do the action. >> >> Also dont forget to implement postScannerClose and remove that scanner >> >> from the Map. Here u might have some perf penalty as u have to add >> >> and get from Map which has to be a concurrent map too. >> >> >> >> Another way >> >> >> >> 2. Create a custom scanner implementing RegionScanner. The new one >> >> has to take an original Region Scanner and just delegate the calls. On >> >> postScannerOpen, u will get the original scanner been created and u >> >> can just wrap it with ur new scanner object. ( If the Scan object is >> >> having required attribute).. In postScannerNext() u can check for ur >> >> own RegionScanner type and if so u can do action. >> >> >> >> >> >> -Anoop- >> >> >> >> >> >> On Sat, Jul 8, 2017 at 9:13 PM, Ted Yu <[email protected]> wrote: >> >> > if (canUseGetOperation(e)) { >> >> > //logic goes here >> >> > >> >> > Does your Get target the same region being scanned ? >> >> > If not, issuing the Get is not advised since the other region may be >> >> hosted >> >> > on different region server. >> >> > >> >> > Cheers >> >> > >> >> > On Thu, Jul 6, 2017 at 7:14 AM, Veerraju Tadimeti <[email protected]> >> >> wrote: >> >> > >> >> >> hi, >> >> >> >> >> >> I have few questions regarding scope of *RegionCoprocessorEnvironment* >> >> >> sharedData. >> >> >> >> >> >> >> >> >> >> >> >> - *Is sharedData map is shared accross all instances simultaneously >> >> ?* >> >> >> - I am putting a variable in sharedData in preScannerOpen() >> >> based on >> >> >> scan attribute, >> >> >> - check that variable exists in postScannerNext() then apply >> >> logic, >> >> >> - remove the variable postScannerClose(). >> >> >> - If data is in multiple regions, when one coprocessor removes >> >> >> variable in postScannerClose(), will the variable is NULL for >> >> another >> >> >> region coprocessor in postScannerNext() ? >> >> >> >> >> >> >> >> >> - * Is sharedData map is shared across all the client request >> >> >> operations ?* >> >> >> >> >> >> If a variable is set in sharedData for one client operation(say SCAN), >> >> will >> >> >> the variable is available for another client operation(new SCAN) ? >> >> >> >> >> >> >> >> >> - *Will the variables be garbage collected even if we dont >> >> >> implement >> >> >> (removed variables in sharedData) postScannerClose() method* >> >> >> >> >> >> >> >> >> Please find below the logic that I am using currently >> >> >> *CODE: * >> >> >> >> >> >> public RegionScanner >> >> >> *preScannerOpen*(ObserverContext<RegionCoprocessorEnvironment> >> >> >> e, Scan scan, RegionScanner s) throws IOException { >> >> >> byte[] useGetInPostScannerNext = scan.getAttribute(USE_GET_ >> >> >> OPERATION_IN_POST_SCANNER_NEXT); >> >> >> String useGetInPostScannerNextStr = Bytes.toString( >> >> >> useGetInPostScannerNext); >> >> >> if (Boolean.parseBoolean(useGetInPostScannerNextStr)) { >> >> >> e.getEnvironment().getSharedData().put(USE_GET_ >> >> >> OPERATION_IN_POST_SCANNER_NEXT, useGetInPostScannerNextStr); >> >> >> } >> >> >> return super.preScannerOpen(e, scan, s); >> >> >> } >> >> >> >> >> >> @Override >> >> >> public boolean *postScannerNext*(final >> >> >> ObserverContext<RegionCoprocessorEnvironment> >> >> >> e, >> >> >> final InternalScanner s, final List<Result> results, final >> >> int >> >> >> limit, >> >> >> final boolean hasMore) throws IOException { >> >> >> try { >> >> >> >> >> >> if (canUseGetOperation(e)) { >> >> >> >> >> >> //logic goes here >> >> >> } >> >> >> } catch (Exception ex) { >> >> >> logger.error("Exception in postScannerNext ", ex); >> >> >> throw new IOException(ex); >> >> >> } >> >> >> return hasMore; >> >> >> } >> >> >> >> >> >> @Override >> >> >> public void >> >> >> *postScannerClose*(ObserverContext<RegionCoprocessorEnvironment> >> >> >> e, InternalScanner s) throws IOException { >> >> >> if (canUseGetOperation(e)) { >> >> >> e.getEnvironment().getSharedData().remove(USE_ >> >> >> GET_OPERATION_IN_POST_SCANNER_NEXT); >> >> >> } >> >> >> super.postScannerClose(e, s); >> >> >> } >> >> >> >> >> >> private boolean *canUseGetOperation*(final >> >> >> ObserverContext<RegionCoprocessorEnvironment> >> >> >> e) { >> >> >> String useGetOperationInPostScannerNext = (String) >> >> >> e.getEnvironment().getSharedData().get(USE_GET_ >> >> OPERATION_IN_POST_SCANNER_ >> >> >> NEXT); >> >> >> return Boolean.parseBoolean(useGetOperationInPostScannerNext); >> >> >> } >> >> >> >> >> >> Thanks, >> >> >> Raju, >> >> >> (972)273-0155 <(972)%20273-0155>. >> >> >> >> >> >
