On Fri, Aug 9, 2019 at 3:57 PM Geoffrey Jacoby <[email protected]> wrote:
>
> You can't say "a mutable WALEdit is too dangerous" without making
> assumptions about the developer it's too dangerous _for_. Ideally these
> assumptions should be explicit and agreed upon by the community. I think
> differing audience assumptions are where at least some of the disagreements
> are coming from.
>
All sounds good but I have a bit of trouble with your using WALEdit in your
illustration above.
To be explicit, the reluctance exposing WAL{Edit,Key} is because when the
system goes awry, from experience, debug is consuming. Errors in WAL*
accounting are particularly hard to fathom because the problem shows up
usually at a displacement in time and location (replay or on other end of a
replication). Often the issue triggers only when large amounts of data. All
WAL*s flow together whatever the source out the one firehose. The
*assumption* is that a CP writer randomly mangling a WAL* that happens to
break the cluster's recovery may not be around or game for the fun debug
sessions required figuring root cause.
<joke>We could add a Geoffrey Jacoby annotation on CP APIs?</joke>
Thanks,
S
Geoffrey
>
>
> On Fri, Aug 9, 2019 at 8:55 AM Andrew Purtell <[email protected]>
> wrote:
>
> > The future of the coprocessor API is an interesting topic. I think we
> have
> > a range of opinions in the community and I would like to hear more of
> them.
> >
> > Because of the compatibility headaches sometimes I’d like to rip them
> out.
> > Sometimes they are essential for accomplishing something in Phoenix or
> our
> > in house backup solution, for example. For me my opinion is very mixed.
> >
> > When first conceived the first use case was security and we took
> > inspiration from OS kernel examples in Linux and TrustedBSD (now merged
> > with FreeBSD) where upcall interfaces were made available where
> > authoritative access control decisions are made. The set of hooks has
> > expanded over time as users have requested extensions. This style of
> > interface is powerful in that it enables a mixin approach to composition
> of
> > functionality and extension. However in retrospect the maintenance
> burdens
> > were not fully appreciated, at least by me. I had assumed we would be
> > allowed more freedom to change but my experiences with Phoenix educated
> me
> > on the kind of downstream headaches that result when we make those
> changes.
> >
> > If I were to do it again I would attempt an abstract and fluent interface
> > where extensions would register intents and receive callbacks in a much
> > more granular way, like:
> >
> > onRegion().onRPC().onGet().then(...)
> >
> > I suppose this looks kind of like Mockito. There would be no overlarge
> > interfaces full of upcall methods that break source compatibility on
> every
> > change (in branch-1). Although we could not avoid the complexity of
> > ensuring the right callbacks are invoked at the right places on the right
> > code paths the extension interface and its types would be decoupled from
> > internals and the kind of compatibility headaches we impose on
> > downstreamers (and ourselves) would mostly disappear. This has been
> > proposed on an old JIRA somewhere...
> >
> > Of course a redo like this would be a very complex and time consuming
> > project, and a port of say something like Phoenix would be a reboot of
> > multi man year efforts, and as far as I know nobody working in the code
> > today has that kind of sustained available time and attention. It’s too
> > late. We have to make the best of the legacy of past engineering choices.
> > If that is not correct then it would be a very pleasant surprise indeed.
> >
> > Given that we have the current model and we have downstreamers like
> > Phoenix depending on them I think we are limited in the kind of clean ups
> > we might like to do and need to be tolerant of the requirement to
> maintain
> > these interfaces and their functionally across major versions.
> Deprecation
> > is fine but if it is done without the input of the known consumer are we
> > really being fair? Unclear.
> >
> >
> > > On Aug 8, 2019, at 6:39 PM, 张铎(Duo Zhang) <[email protected]>
> wrote:
> > >
> > > When releasing 2.0.0 we faced a lot of problems because we exposes so
> > many
> > > internal classes to CPs. It is really hard to both consider the
> > > compatibility and development on HBase. And then we have done lots of
> > works
> > > to abstract interfaces for CPs to use and hide the actual
> implementation
> > > classes to be HBase only.
> > >
> > > The work is not fully done, so we still left some methods which exposes
> > > internal classes there with a deprecated annotation, I think most of
> them
> > > are for Phoenix. This is a trade off and I think it is also acceptable.
> > > Phoenix could still use the deprecated methods, and we will not remove
> > them
> > > unless we find an alternate solution. And I think if anyone wants to
> > remove
> > > them without replacment you will first jump out and give a -1 :)
> > >
> > > Specific to HBASE-22623, I still think we should add the deprecated
> > > annotation to keep the API consistent, otherwise users will be confused
> > > that whether they can use the WALEdit. And on the abstraction of
> > WALEdit, I
> > > think we used to rely on a high level abstraction in HBASE-20952. But
> now
> > > since there is little progress there, I think we can start the work of
> > > abstracting WALEdit only.
> > >
> > > Thanks.
> > >
> > > Andrew Purtell <[email protected]> 于2019年8月9日周五 上午3:21写道：
> > >
> > >> Please let me direct your attention to the tail of HBASE-22623 for a
> > larger
> > >> discussion. I tried to sum it up as follows:
> > >>
> > >> An opinion that we should have more and more coprocessor interfaces to
> > >> address new use cases is valid. An opinion that coprocessors are too
> > >> invasive and should be 'cleaned up' is also valid. An opinion that the
> > >> compatibility headaches of coprocessor interfaces are annoying is
> > valid. An
> > >> opinion that Phoenix can be considered as a valid use case when
> > considering
> > >> interface changes is valid. An opinion that only HBase level concerns
> > >> should motivate API changes is valid. These opinions are strawmen. I
> > think
> > >> they approach actual positions in the community but I do not imply any
> > >> specific person has one of them. These strawmen are at least partially
> > >> contradictory. It is going to be an ongoing process to sort them out
> > into
> > >> something that makes sense and can get consensus.
> > >>
> > >> So while as committer I am moving forward on HBASE-22623 because I
> don't
> > >> see a veto but instead a disagreement on the margins (deprecation or
> > not)
> > >> motivated on larger principles, I also want to raise the visibility of
> > the
> > >> disagreement because I think it impacts our relationship with another
> > >> project at Apache at a minimum, but also future technical directions
> of
> > an
> > >> important subset of interfaces.
> > >>
> > >> For your consideration.
> > >>
> > >> --
> > >> Best regards,
> > >> Andrew
> > >>
> > >> Words like orphans lost among the crosstalk, meaning torn from truth's
> > >> decrepit hands
> > >> - A23, Crosstalk
> > >>
> >
>
