I have 2.5 and 3.0 installed as /opt/protobuf-<version>, and have bash scripts that add the appropriate version's bin directory to the path. Not particularly onerous as I also have to switch between required JDK versions, so the scripts also set JAVA_HOME at and add JDK bin to the path for the required JDK for the build.
Unlike with the scala compiler, which is after all JVM bytecode at a fundamental level, I don't think maven automation for automatic download and execution is possible. protoc is a native binary. > On Oct 1, 2016, at 11:30 PM, 张铎 <[email protected]> wrote: > > Do we need to install protoc 3.0 manully before building HBase or the maven > protobuf plugin will automatically download the protoc compiler? Maybe we > need to install protoc 3.0 in the precommit docker file. > > 2016-10-02 14:20 GMT+08:00 张铎 <[email protected]>: > >> >> 2016-10-02 0:50 GMT+08:00 Stack <[email protected]>: >> >>>> On Sat, Oct 1, 2016 at 7:20 AM, 张铎 <[email protected]> wrote: >>>> >>>> Can we setup a compatibility checker job in jenkins? Start a >>> minicluster in >>>> one process, and use a client in another process to communicate with it. >>>> The version of the client should be >= 0.98 and <= the version of the >>>> minicluster. Of course we need to design the testing code carefully to >>> make >>>> sure that we have tested all the cases. >>>> >>>> >>> +1. We need this up and running before we put out an hbase-2.0.0. I know >>> Matteo does this test manually on a regular basis but a formalization >>> would >>> help. I can add an exercise of Coprocessor Endpoints. I believe this is on >>> Dima's list of TODOs but will let him speak for himself. >>> >>> >>>> And also I think we should make sure that no proto3 only feature is >>>> introduced in our proto files until branch-1 is dead. Maybe a precommit >>>> check? >>>> >>>> >>> I think you mean wire-format breaking changes? Agree. We have our PB3 set >>> to 2.5 compat mode and yes, we can't move on from this until we are in a >>> place where we can say no to 2.5 clients. >>> >> Yes, for example, pb2.5 does not support map so we should not use map in >> our proto files. >> >>> >>> Making use of PB3isms cannot be avoided. PB3.1 adds a native replacement >>> for our HBaseZeroCopyByteString/ByteStringer hack. It also adds 'unsafe' >>> methods that we need to exploit if we are to keep our read/write paths >>> offheap. >>> >>> St.Ack >>> >>> >>> >>> >>> >>>> Thanks. >>>> >>>> 2016-10-01 11:55 GMT+08:00 Sean Busbey <[email protected]>: >>>> >>>>> have we experimentally confirmed that wire compatibility is >>>>> maintained? I saw one mention of expecting wire compatibility to be >>>>> fine, but nothing with someone using e.g. the clusterdock work or >>>>> something to mix servers / clients or do replication. >>>>> >>>>>> On Fri, Sep 30, 2016 at 6:30 PM, Stack <[email protected]> wrote: >>>>>> I intend to do a mass commit late this weekend that moves us on to a >>>>> shaded >>>>>> protobuf-3.1.0, either Sunday night or Monday morning. >>>>>> >>>>>> If objection, please speak up or if need more time for >>>>>> consideration/review, just shout. >>>>>> >>>>>> I want to merge the branch HBASE-16264 into master (it is running >>> here >>>> up >>>>>> on jenkins https://builds.apache.org/view/H-L/view/HBase/job/HBASE- >>>>> 16264/). >>>>>> The branch at HBASE-16264 has three significant bodies-of-work that >>>>>> unfortunately are tangled and can only go in of a piece. >>>>>> >>>>>> * HBASE-16264 <https://issues.apache.org/jira/browse/HBASE-16264> >>> The >>>>>> shading of our protobuf usage so we can upgrade and/or run with a >>>> patched >>>>>> protobuf WITHOUT breaking REST, Spark, and in particular, >>> Coprocessor >>>>>> Endpoints. >>>>>> * HBASE-16567 <https://issues.apache.org/jira/browse/HBASE-16567> >>> A >>>>> move >>>>>> up on to (shaded) protobuf-3.1.0 >>>>>> * HBASE-16741 <https://issues.apache.org/jira/browse/HBASE-16741> >>> An >>>>>> amendment of our generate protobufs step to include shading and a >>>>> bundling >>>>>> of protobuf src (with a means of calling a patch srcs hook) >>>>>> >>>>>> Together we're talking about 40MB of change mostly made of the >>> movement >>>>> of >>>>>> generated files or the application of a pattern that alters where we >>>> get >>>>>> imports from. When done, you should notice no difference and should >>> be >>>>> able >>>>>> to go about your business as per usual. Upside is that we will be >>> able >>>> to >>>>>> avoid coming onheap doing protobuf marshalling/unmarshalling as >>>> protobuf >>>>>> 2.5.0 requires. Downside is that we repeat a good portion of our >>>> internal >>>>>> protos, once non-shaded so Coprocessor Endpoints can keep working >>> and >>>>> then >>>>>> again as shaded for internal use. >>>>>> >>>>>> I provide some more overview below on the changes. See the shading >>> doc >>>>>> here: >>>>>> https://docs.google.com/document/d/1H4NgLXQ9Y9KejwobddCqaVMEDCGby >>>>> DcXtdF5iAfDIEk/edit# >>>>>> for more detail (Patches are up on review board -- except the latest >>>>>> HBASE-16264 which is too big for JIRA and RB). I am currently >>> working >>>> on >>>>> a >>>>>> devs chapter for the book on protobuf going forward that will go in >>> as >>>>> part >>>>>> of this patch. >>>>>> >>>>>> Thanks, >>>>>> St.Ack >>>>>> >>>>>> Items of note: >>>>>> >>>>>> * Two new modules; one named hbase-protocol-shaded that is used by >>>> hbase >>>>>> core. It has in it a shaded (and later patched) protobuf. The other >>> new >>>>>> module is hbase-endpoint which goes after hbase-server and has those >>>>>> bundled endpoints that I was able to break out of core (there are a >>> few >>>>>> that are hopelessly entangled that need to be undone as CPEPs but >>>>>> fortunately belong in core: Auth, Access, MultiRow). >>>>>> * I've tested running a branch-1 CPEP against a master with these >>>>> patches >>>>>> in place and stuff like ACL (A CPEP) run from the branch-1 shell >>> work >>>>>> against the branch-2 server. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> On Mon, Aug 22, 2016 at 5:20 PM, Stack <[email protected]> wrote: >>>>>>> >>>>>>> This project goes on. I updated HBASE-1563 "Shade protobuf" with >>> some >>>>> doc >>>>>>> on a final approach. We need to be able to refer to both shaded and >>>>>>> non-shaded protobuf so we can support sending HDFS old-school pb >>>>> Messages >>>>>>> but also so Coprocessor Endpoints keep working though internally >>>>> protobufs >>>>>>> have been relocated. Funny you should ask, but yes, there are some >>>>>>> downsides (as predicted by contributors on the JIRA). I'd be >>>> interested >>>>> to >>>>>>> hear if they are too burdensome. In particular, your IDE experience >>>>> gets a >>>>>>> little convoluted as you will need to add to your build path, a jar >>>> with >>>>>>> the relocated pbs. A pain. >>>>>>> >>>>>>> Thanks, >>>>>>> St.Ack >>>>>>> >>>>>>> >>>>>>>> On Wed, Apr 13, 2016 at 6:09 AM, Stack <[email protected]> wrote: >>>>>>>> >>>>>>>> On Tue, Apr 12, 2016 at 9:26 PM, Sean Busbey <[email protected]> >>>>> wrote: >>>>>>>> >>>>>>>>>> On Tue, Apr 12, 2016 at 6:17 PM, Stack <[email protected]> wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On an initial pass, the only difficult part seems to be >>>> interaction >>>>>>>>> with >>>>>>>>>> HDFS in asyncwal (might just pull in the HDFS messages). >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> I have some idea how we can make this work either by pushing >>>> asyncwal >>>>>>>>> upstream to HDFS or through some maven tricks, depending on how >>> much >>>>>>>>> time we have. >>>>>>>>> >>>>>>>> >>>>>>>> Maven tricks? Tell us more. Here or drop a note up in the issue. >>>>>>>> Thanks Sean, >>>>>>>> St.Ack >>>>>>>> >>>>>>> >>>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> busbey >>>>> >>>> >>> >> >>
