On Wed, Oct 11, 2017 at 11:57 AM, Andrew Purtell <apurt...@apache.org> wrote:
> > What about the Duo suggestion? Purge bypass flag and replace it w/ preXXX > in a few select methods returning a boolean on whether bypass? Would that > work? (Would have to figure metrics still). > > That would work. > > Let me take a look at what'd be involved then. Thanks, S > > On Wed, Oct 11, 2017 at 11:56 AM, Stack <st...@duboce.net> wrote: > > > The YARN Timeline Server has the FlowRunCoprocessor. It does bypass when > > user does a Get returning instead the result of its own (Flow) Scan > result. > > Not sure how we'd do alternative here; Timeline Server is keeping Tags > > internally. > > > > > > On Wed, Oct 11, 2017 at 10:59 AM, Andrew Purtell <apurt...@apache.org> > > wrote: > > > > > Rather than continue to support a weird bypass() which works in some > > places > > > and not in others, perhaps we can substitute it with an exception? So > if > > > the coprocessor throws this exception in the pre hook then where it is > > > allowed we catch it and do the right thing, and where it is not allowed > > we > > > don't catch it and the server aborts. This will at least improve the > > silent > > > bypass() failure problem. I also don't like, in retrospect, that > calling > > > this environment method has magic side effects. Everyone understands > how > > > exceptions work, so it will be clearer. > > > > > > > > We could do that though throw and catch of exceptions would be costly. > > > > What about the Duo suggestion? Purge bypass flag and replace it w/ preXXX > > in a few select methods returning a boolean on whether bypass? Would that > > work? (Would have to figure metrics still). > > > > > > > > > In any case we should try to address the Tephra and Phoenix cases > brought > > > up in this discussion. They look like we can find alternatives. Shall I > > > file JIRAs to follow up? > > > > > > > > > > > On Phoenix Increment by-pass, an ornery item is that Phoenix wants to use > > its long encoding writing Increments. Not sure how we'd do that, > > selectively. > > > > St.Ack > > > > > > > > > On Wed, Oct 11, 2017 at 6:00 AM, 张铎(Duo Zhang) <palomino...@gmail.com> > > > wrote: > > > > > > > These examples are great. > > > > > > > > And I think for normal region operations such as get, put, delete, > > > > checkAndXXX, increment, it is OK to bypass the real operation after > > > preXXX > > > > as the semantic is clear enough. Instead of calling env.bypass, maybe > > > just > > > > let these preXXX methods return a boolean is enough to tell the HBase > > > > framework that we have already done the real operation so just give > up > > > and > > > > return? > > > > > > > > Thanks. > > > > > > > > 2017-10-11 3:19 GMT+08:00 Gary Helmling <ghelml...@gmail.com>: > > > > > > > > > The Tephra TransactionProcessor CP makes use of bypass() in > > preDelete() > > > > to > > > > > override handling of delete tombstones in a transactional way: > > > > > https://github.com/apache/incubator-tephra/blob/master/ > > > > > tephra-hbase-compat-1.3/src/main/java/org/apache/tephra/ > > > > hbase/coprocessor/ > > > > > TransactionProcessor.java#L244 > > > > > > > > > > The CDAP IncrementHandler CP also makes use of bypass() in > preGetOp() > > > and > > > > > preIncrementAfterRRowLock() to provide a transaction implementation > > of > > > > > readless increments: > > > > > https://github.com/caskdata/cdap/blob/develop/cdap-hbase- > > > > > compat-1.1/src/main/java/co/cask/cdap/data2/increment/ > > > > > hbase11/IncrementHandler.java#L121 > > > > > > > > > > What would be the alternate approach for these applications? In > both > > > > cases > > > > > they need to impose their own semantics on the underlying KeyValue > > > > > storage. Is there a different way this can be done? > > > > > > > > > > > > > > > On Tue, Oct 10, 2017 at 11:58 AM Anoop John <anoop.hb...@gmail.com > > > > > > wrote: > > > > > > > > > > > Wrap core scanners is different right? That can be done in post > > > > > > hooks. I have seen many use cases for this.. Its the question > abt > > > > > > the pre hooks where we have not yet created the core object (like > > > > > > scanner). The CP pre code itself doing the work of object > creation > > > > > > and so the core code is been bypassed. Well the wrapping thing > > can > > > > > > be done in pre hook also. First create the core object by CP code > > > > > > itself and then do the wrapped object and return.. I have seen in > > one > > > > > > jira issue where the usage was this way.. The wrapping can be > > done > > > > > > in post also in such cases I believe. > > > > > > > > > > > > -Anoop- > > > > > > > > > > > > On Wed, Oct 11, 2017 at 12:23 AM, Andrew Purtell < > > > apurt...@apache.org> > > > > > > wrote: > > > > > > > I think we should continue to support overriding function by > > object > > > > > > > inheritance. I didn't mention this and am not proposing more > than > > > > > > removing > > > > > > > the bypass() sematic. No more no less. Phoenix absolutely > depends > > > on > > > > > > being > > > > > > > able to wrap core scanners and return the wrappers. > > > > > > > > > > > > > > > > > > > > > On Tue, Oct 10, 2017 at 11:50 AM, Anoop John < > > > anoop.hb...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > >> When we say bypass the core code, it can be done today not > only > > by > > > > > > >> calling bypass but by returning a not null object for some of > > the > > > > pre > > > > > > >> hooks. Like preScannerOpen() if it return a scanner object, > we > > > will > > > > > > >> avoid the remaining core code execution for creation of the > > > > > > >> scanner(s). So this proposal include this aspect also and > > remove > > > > any > > > > > > >> possible way of bypassing the core code by the CP hook code > > > > execution > > > > > > >> ? Am +1. > > > > > > >> > > > > > > >> -Anoop- > > > > > > >> > > > > > > >> On Tue, Oct 10, 2017 at 11:40 PM, Andrew Purtell < > > > > apurt...@apache.org > > > > > > > > > > > > >> wrote: > > > > > > >> > The coprocessor API provides an environment method, > bypass(), > > > that > > > > > > when > > > > > > >> > called from a preXXX hook will cause the core code to skip > all > > > > > > remaining > > > > > > >> > processing. This capability was introduced on HBASE-3348. > > Since > > > > this > > > > > > >> time I > > > > > > >> > think we are more enlightened about the complications of > this > > > > > feature. > > > > > > >> (Or, > > > > > > >> > anyway, speaking for myself:) > > > > > > >> > > > > > > > >> > Not all hooks provide the bypass semantic. Where this is the > > > case > > > > > the > > > > > > >> > javadoc for the hook says so, but it can be missed. If you > > call > > > > > > bypass() > > > > > > >> in > > > > > > >> > a hook where it is not supported it is a no-op. This can > lead > > > to a > > > > > > poor > > > > > > >> > developer experience. > > > > > > >> > > > > > > > >> > Where bypass is supported what is being bypassed is all of > the > > > > core > > > > > > code > > > > > > >> > implementing the remainder of the operation. In order to > > > > understand > > > > > > what > > > > > > >> > calling bypass() will skip, a coprocessor implementer should > > > read > > > > > and > > > > > > >> > understand all of the remaining code and its nuances. > > Although I > > > > > think > > > > > > >> this > > > > > > >> > is good practice for coprocessor developers in general, it > > > > demands a > > > > > > >> lot. I > > > > > > >> > think it would provide a much better developer experience if > > we > > > > > didn't > > > > > > >> > allow bypass, even though it means - in theory - a > coprocessor > > > > would > > > > > > be a > > > > > > >> > lot more limited in some ways than before. What is skipped > is > > > > > > extremely > > > > > > >> > version dependent. That core code will vary, perhaps > > > > significantly, > > > > > > even > > > > > > >> > between point releases. We do not provide the promise of > > > > consistent > > > > > > >> > behavior even between point releases for the bypass > semantic. > > To > > > > > > achieve > > > > > > >> > that we could not change any code between hook points. > > Therefore > > > > the > > > > > > >> > coprocessor implementer becomes an HBase core developer in > > > > practice > > > > > as > > > > > > >> soon > > > > > > >> > as they rely on bypass(). Every release of HBase may break > the > > > > > > assumption > > > > > > >> > that the replacement for the bypassed code takes care of all > > > > > necessary > > > > > > >> > skipped concerns. Because those concerns can change at any > > > point, > > > > > > such an > > > > > > >> > assumption is never safe. > > > > > > >> > > > > > > > >> > I say "in theory" because I would be surprised if anyone is > > > > relying > > > > > on > > > > > > >> the > > > > > > >> > bypass for the above reason. I seem to recall that Phoenix > > might > > > > use > > > > > > it > > > > > > >> in > > > > > > >> > one place to promote a normal mutation into an atomic > > operation, > > > > by > > > > > > >> > substituting one for the other, but if so that objective > could > > > be > > > > > > >> > reimplemented using their new locking manager. > > > > > > >> > > > > > > > >> > -- > > > > > > >> > Best regards, > > > > > > >> > Andrew > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Best regards, > > > > > > > Andrew > > > > > > > > > > > > > > Words like orphans lost among the crosstalk, meaning torn from > > > > truth's > > > > > > > decrepit hands > > > > > > > - A23, Crosstalk > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Best regards, > > > Andrew > > > > > > Words like orphans lost among the crosstalk, meaning torn from truth's > > > decrepit hands > > > - A23, Crosstalk > > > > > > > > > -- > Best regards, > Andrew > > Words like orphans lost among the crosstalk, meaning torn from truth's > decrepit hands > - A23, Crosstalk >
