For coprocessors in general, I think Andrew sums the issues up well. I just want to add that it's really important to have a mental model of who the audience for a coprocessor is when designing. I remember when coprocessors were first introduced, I read articles comparing them to triggers and stored procedures, and the docs still describe them that way in section 112.1. But they're really not like triggers and SPs, because a user-written stored procedure shouldn't be able to crash your database. :-) Andrew's comparison to kernel extensions is apt I think.
Writing a Linux kernel extension doesn't require you to be Linus or one of his lieutenants, but it does require more sophistication that just writing a user-space Linux program does. Their entire purpose is for people who aren't at the level of Linus to affect the kernel behavior in limited ways. Likewise, with HBase coprocessors. They should provide safe enough abstractions that they don't require an HBase committer to implement safely, but I think it's fine to require more HBase knowledge than what's required to use the client API. You can't say "a mutable WALEdit is too dangerous" without making assumptions about the developer it's too dangerous _for_. Ideally these assumptions should be explicit and agreed upon by the community. I think differing audience assumptions are where at least some of the disagreements are coming from. Geoffrey On Fri, Aug 9, 2019 at 8:55 AM Andrew Purtell <[email protected]> wrote: > The future of the coprocessor API is an interesting topic. I think we have > a range of opinions in the community and I would like to hear more of them. > > Because of the compatibility headaches sometimes I’d like to rip them out. > Sometimes they are essential for accomplishing something in Phoenix or our > in house backup solution, for example. For me my opinion is very mixed. > > When first conceived the first use case was security and we took > inspiration from OS kernel examples in Linux and TrustedBSD (now merged > with FreeBSD) where upcall interfaces were made available where > authoritative access control decisions are made. The set of hooks has > expanded over time as users have requested extensions. This style of > interface is powerful in that it enables a mixin approach to composition of > functionality and extension. However in retrospect the maintenance burdens > were not fully appreciated, at least by me. I had assumed we would be > allowed more freedom to change but my experiences with Phoenix educated me > on the kind of downstream headaches that result when we make those changes. > > If I were to do it again I would attempt an abstract and fluent interface > where extensions would register intents and receive callbacks in a much > more granular way, like: > > onRegion().onRPC().onGet().then(...) > > I suppose this looks kind of like Mockito. There would be no overlarge > interfaces full of upcall methods that break source compatibility on every > change (in branch-1). Although we could not avoid the complexity of > ensuring the right callbacks are invoked at the right places on the right > code paths the extension interface and its types would be decoupled from > internals and the kind of compatibility headaches we impose on > downstreamers (and ourselves) would mostly disappear. This has been > proposed on an old JIRA somewhere... > > Of course a redo like this would be a very complex and time consuming > project, and a port of say something like Phoenix would be a reboot of > multi man year efforts, and as far as I know nobody working in the code > today has that kind of sustained available time and attention. It’s too > late. We have to make the best of the legacy of past engineering choices. > If that is not correct then it would be a very pleasant surprise indeed. > > Given that we have the current model and we have downstreamers like > Phoenix depending on them I think we are limited in the kind of clean ups > we might like to do and need to be tolerant of the requirement to maintain > these interfaces and their functionally across major versions. Deprecation > is fine but if it is done without the input of the known consumer are we > really being fair? Unclear. > > > > On Aug 8, 2019, at 6:39 PM, 张铎(Duo Zhang) <[email protected]> wrote: > > > > When releasing 2.0.0 we faced a lot of problems because we exposes so > many > > internal classes to CPs. It is really hard to both consider the > > compatibility and development on HBase. And then we have done lots of > works > > to abstract interfaces for CPs to use and hide the actual implementation > > classes to be HBase only. > > > > The work is not fully done, so we still left some methods which exposes > > internal classes there with a deprecated annotation, I think most of them > > are for Phoenix. This is a trade off and I think it is also acceptable. > > Phoenix could still use the deprecated methods, and we will not remove > them > > unless we find an alternate solution. And I think if anyone wants to > remove > > them without replacment you will first jump out and give a -1 :) > > > > Specific to HBASE-22623, I still think we should add the deprecated > > annotation to keep the API consistent, otherwise users will be confused > > that whether they can use the WALEdit. And on the abstraction of > WALEdit, I > > think we used to rely on a high level abstraction in HBASE-20952. But now > > since there is little progress there, I think we can start the work of > > abstracting WALEdit only. > > > > Thanks. > > > > Andrew Purtell <[email protected]> 于2019年8月9日周五 上午3:21写道： > > > >> Please let me direct your attention to the tail of HBASE-22623 for a > larger > >> discussion. I tried to sum it up as follows: > >> > >> An opinion that we should have more and more coprocessor interfaces to > >> address new use cases is valid. An opinion that coprocessors are too > >> invasive and should be 'cleaned up' is also valid. An opinion that the > >> compatibility headaches of coprocessor interfaces are annoying is > valid. An > >> opinion that Phoenix can be considered as a valid use case when > considering > >> interface changes is valid. An opinion that only HBase level concerns > >> should motivate API changes is valid. These opinions are strawmen. I > think > >> they approach actual positions in the community but I do not imply any > >> specific person has one of them. These strawmen are at least partially > >> contradictory. It is going to be an ongoing process to sort them out > into > >> something that makes sense and can get consensus. > >> > >> So while as committer I am moving forward on HBASE-22623 because I don't > >> see a veto but instead a disagreement on the margins (deprecation or > not) > >> motivated on larger principles, I also want to raise the visibility of > the > >> disagreement because I think it impacts our relationship with another > >> project at Apache at a minimum, but also future technical directions of > an > >> important subset of interfaces. > >> > >> For your consideration. > >> > >> -- > >> Best regards, > >> Andrew > >> > >> Words like orphans lost among the crosstalk, meaning torn from truth's > >> decrepit hands > >> - A23, Crosstalk > >> >
