+1 Difficult work. Glad to see it landing.
> On Sep 30, 2016, at 4:30 PM, Stack <[email protected]> wrote: > > I intend to do a mass commit late this weekend that moves us on to a shaded > protobuf-3.1.0, either Sunday night or Monday morning. > > If objection, please speak up or if need more time for > consideration/review, just shout. > > I want to merge the branch HBASE-16264 into master (it is running here up > on jenkins https://builds.apache.org/view/H-L/view/HBase/job/HBASE-16264/). > The branch at HBASE-16264 has three significant bodies-of-work that > unfortunately are tangled and can only go in of a piece. > > * HBASE-16264 <https://issues.apache.org/jira/browse/HBASE-16264> The > shading of our protobuf usage so we can upgrade and/or run with a patched > protobuf WITHOUT breaking REST, Spark, and in particular, Coprocessor > Endpoints. > * HBASE-16567 <https://issues.apache.org/jira/browse/HBASE-16567> A move > up on to (shaded) protobuf-3.1.0 > * HBASE-16741 <https://issues.apache.org/jira/browse/HBASE-16741> An > amendment of our generate protobufs step to include shading and a bundling > of protobuf src (with a means of calling a patch srcs hook) > > Together we're talking about 40MB of change mostly made of the movement of > generated files or the application of a pattern that alters where we get > imports from. When done, you should notice no difference and should be able > to go about your business as per usual. Upside is that we will be able to > avoid coming onheap doing protobuf marshalling/unmarshalling as protobuf > 2.5.0 requires. Downside is that we repeat a good portion of our internal > protos, once non-shaded so Coprocessor Endpoints can keep working and then > again as shaded for internal use. > > I provide some more overview below on the changes. See the shading doc > here: > https://docs.google.com/document/d/1H4NgLXQ9Y9KejwobddCqaVMEDCGbyDcXtdF5iAfDIEk/edit# > for more detail (Patches are up on review board -- except the latest > HBASE-16264 which is too big for JIRA and RB). I am currently working on a > devs chapter for the book on protobuf going forward that will go in as part > of this patch. > > Thanks, > St.Ack > > Items of note: > > * Two new modules; one named hbase-protocol-shaded that is used by hbase > core. It has in it a shaded (and later patched) protobuf. The other new > module is hbase-endpoint which goes after hbase-server and has those > bundled endpoints that I was able to break out of core (there are a few > that are hopelessly entangled that need to be undone as CPEPs but > fortunately belong in core: Auth, Access, MultiRow). > * I've tested running a branch-1 CPEP against a master with these patches > in place and stuff like ACL (A CPEP) run from the branch-1 shell work > against the branch-2 server. > > > > > >> On Mon, Aug 22, 2016 at 5:20 PM, Stack <[email protected]> wrote: >> >> This project goes on. I updated HBASE-1563 "Shade protobuf" with some doc >> on a final approach. We need to be able to refer to both shaded and >> non-shaded protobuf so we can support sending HDFS old-school pb Messages >> but also so Coprocessor Endpoints keep working though internally protobufs >> have been relocated. Funny you should ask, but yes, there are some >> downsides (as predicted by contributors on the JIRA). I'd be interested to >> hear if they are too burdensome. In particular, your IDE experience gets a >> little convoluted as you will need to add to your build path, a jar with >> the relocated pbs. A pain. >> >> Thanks, >> St.Ack >> >> >>> On Wed, Apr 13, 2016 at 6:09 AM, Stack <[email protected]> wrote: >>> >>>> On Tue, Apr 12, 2016 at 9:26 PM, Sean Busbey <[email protected]> wrote: >>>> >>>>> On Tue, Apr 12, 2016 at 6:17 PM, Stack <[email protected]> wrote: >>>>> >>>>> >>>>> On an initial pass, the only difficult part seems to be interaction >>>> with >>>>> HDFS in asyncwal (might just pull in the HDFS messages). >>>>> >>>>> >>>> >>>> I have some idea how we can make this work either by pushing asyncwal >>>> upstream to HDFS or through some maven tricks, depending on how much >>>> time we have. >>>> >>> >>> Maven tricks? Tell us more. Here or drop a note up in the issue. >>> Thanks Sean, >>> St.Ack >>> >> >>
