The protoc generated files (such as MasterProtos) are checked into source repo, right ?
Do we need proto3 on the precommit image(s) ? On Mon, Oct 3, 2016 at 5:18 AM, 张铎 <[email protected]> wrote: > Then I think we need to file an issue to change the protoc version > installed in the precommit docker file to 3.x before the merge. Otherwise > the precommit build for protoc check maybe broken after the merge... > > 2016-10-03 1:18 GMT+08:00 Andrew Purtell <[email protected]>: > > > I have 2.5 and 3.0 installed as /opt/protobuf-<version>, and have bash > > scripts that add the appropriate version's bin directory to the path. Not > > particularly onerous as I also have to switch between required JDK > > versions, so the scripts also set JAVA_HOME at and add JDK bin to the > path > > for the required JDK for the build. > > > > Unlike with the scala compiler, which is after all JVM bytecode at a > > fundamental level, I don't think maven automation for automatic download > > and execution is possible. protoc is a native binary. > > > > > On Oct 1, 2016, at 11:30 PM, 张铎 <[email protected]> wrote: > > > > > > Do we need to install protoc 3.0 manully before building HBase or the > > maven > > > protobuf plugin will automatically download the protoc compiler? Maybe > we > > > need to install protoc 3.0 in the precommit docker file. > > > > > > 2016-10-02 14:20 GMT+08:00 张铎 <[email protected]>: > > > > > >> > > >> 2016-10-02 0:50 GMT+08:00 Stack <[email protected]>: > > >> > > >>>> On Sat, Oct 1, 2016 at 7:20 AM, 张铎 <[email protected]> wrote: > > >>>> > > >>>> Can we setup a compatibility checker job in jenkins? Start a > > >>> minicluster in > > >>>> one process, and use a client in another process to communicate with > > it. > > >>>> The version of the client should be >= 0.98 and <= the version of > the > > >>>> minicluster. Of course we need to design the testing code carefully > to > > >>> make > > >>>> sure that we have tested all the cases. > > >>>> > > >>>> > > >>> +1. We need this up and running before we put out an hbase-2.0.0. I > > know > > >>> Matteo does this test manually on a regular basis but a formalization > > >>> would > > >>> help. I can add an exercise of Coprocessor Endpoints. I believe this > > is on > > >>> Dima's list of TODOs but will let him speak for himself. > > >>> > > >>> > > >>>> And also I think we should make sure that no proto3 only feature is > > >>>> introduced in our proto files until branch-1 is dead. Maybe a > > precommit > > >>>> check? > > >>>> > > >>>> > > >>> I think you mean wire-format breaking changes? Agree. We have our > PB3 > > set > > >>> to 2.5 compat mode and yes, we can't move on from this until we are > in > > a > > >>> place where we can say no to 2.5 clients. > > >>> > > >> Yes, for example, pb2.5 does not support map so we should not use map > in > > >> our proto files. > > >> > > >>> > > >>> Making use of PB3isms cannot be avoided. PB3.1 adds a native > > replacement > > >>> for our HBaseZeroCopyByteString/ByteStringer hack. It also adds > > 'unsafe' > > >>> methods that we need to exploit if we are to keep our read/write > paths > > >>> offheap. > > >>> > > >>> St.Ack > > >>> > > >>> > > >>> > > >>> > > >>> > > >>>> Thanks. > > >>>> > > >>>> 2016-10-01 11:55 GMT+08:00 Sean Busbey <[email protected]>: > > >>>> > > >>>>> have we experimentally confirmed that wire compatibility is > > >>>>> maintained? I saw one mention of expecting wire compatibility to be > > >>>>> fine, but nothing with someone using e.g. the clusterdock work or > > >>>>> something to mix servers / clients or do replication. > > >>>>> > > >>>>>> On Fri, Sep 30, 2016 at 6:30 PM, Stack <[email protected]> wrote: > > >>>>>> I intend to do a mass commit late this weekend that moves us on > to a > > >>>>> shaded > > >>>>>> protobuf-3.1.0, either Sunday night or Monday morning. > > >>>>>> > > >>>>>> If objection, please speak up or if need more time for > > >>>>>> consideration/review, just shout. > > >>>>>> > > >>>>>> I want to merge the branch HBASE-16264 into master (it is running > > >>> here > > >>>> up > > >>>>>> on jenkins https://builds.apache.org/ > view/H-L/view/HBase/job/HBASE- > > >>>>> 16264/). > > >>>>>> The branch at HBASE-16264 has three significant bodies-of-work > that > > >>>>>> unfortunately are tangled and can only go in of a piece. > > >>>>>> > > >>>>>> * HBASE-16264 <https://issues.apache.org/jira/browse/HBASE-16264> > > >>> The > > >>>>>> shading of our protobuf usage so we can upgrade and/or run with a > > >>>> patched > > >>>>>> protobuf WITHOUT breaking REST, Spark, and in particular, > > >>> Coprocessor > > >>>>>> Endpoints. > > >>>>>> * HBASE-16567 <https://issues.apache.org/jira/browse/HBASE-16567> > > >>> A > > >>>>> move > > >>>>>> up on to (shaded) protobuf-3.1.0 > > >>>>>> * HBASE-16741 <https://issues.apache.org/jira/browse/HBASE-16741> > > >>> An > > >>>>>> amendment of our generate protobufs step to include shading and a > > >>>>> bundling > > >>>>>> of protobuf src (with a means of calling a patch srcs hook) > > >>>>>> > > >>>>>> Together we're talking about 40MB of change mostly made of the > > >>> movement > > >>>>> of > > >>>>>> generated files or the application of a pattern that alters where > we > > >>>> get > > >>>>>> imports from. When done, you should notice no difference and > should > > >>> be > > >>>>> able > > >>>>>> to go about your business as per usual. Upside is that we will be > > >>> able > > >>>> to > > >>>>>> avoid coming onheap doing protobuf marshalling/unmarshalling as > > >>>> protobuf > > >>>>>> 2.5.0 requires. Downside is that we repeat a good portion of our > > >>>> internal > > >>>>>> protos, once non-shaded so Coprocessor Endpoints can keep working > > >>> and > > >>>>> then > > >>>>>> again as shaded for internal use. > > >>>>>> > > >>>>>> I provide some more overview below on the changes. See the shading > > >>> doc > > >>>>>> here: > > >>>>>> https://docs.google.com/document/d/1H4NgLXQ9Y9KejwobddCqaVMEDCGby > > >>>>> DcXtdF5iAfDIEk/edit# > > >>>>>> for more detail (Patches are up on review board -- except the > latest > > >>>>>> HBASE-16264 which is too big for JIRA and RB). I am currently > > >>> working > > >>>> on > > >>>>> a > > >>>>>> devs chapter for the book on protobuf going forward that will go > in > > >>> as > > >>>>> part > > >>>>>> of this patch. > > >>>>>> > > >>>>>> Thanks, > > >>>>>> St.Ack > > >>>>>> > > >>>>>> Items of note: > > >>>>>> > > >>>>>> * Two new modules; one named hbase-protocol-shaded that is used by > > >>>> hbase > > >>>>>> core. It has in it a shaded (and later patched) protobuf. The > other > > >>> new > > >>>>>> module is hbase-endpoint which goes after hbase-server and has > those > > >>>>>> bundled endpoints that I was able to break out of core (there are > a > > >>> few > > >>>>>> that are hopelessly entangled that need to be undone as CPEPs but > > >>>>>> fortunately belong in core: Auth, Access, MultiRow). > > >>>>>> * I've tested running a branch-1 CPEP against a master with these > > >>>>> patches > > >>>>>> in place and stuff like ACL (A CPEP) run from the branch-1 shell > > >>> work > > >>>>>> against the branch-2 server. > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>>> On Mon, Aug 22, 2016 at 5:20 PM, Stack <[email protected]> wrote: > > >>>>>>> > > >>>>>>> This project goes on. I updated HBASE-1563 "Shade protobuf" with > > >>> some > > >>>>> doc > > >>>>>>> on a final approach. We need to be able to refer to both shaded > and > > >>>>>>> non-shaded protobuf so we can support sending HDFS old-school pb > > >>>>> Messages > > >>>>>>> but also so Coprocessor Endpoints keep working though internally > > >>>>> protobufs > > >>>>>>> have been relocated. Funny you should ask, but yes, there are > some > > >>>>>>> downsides (as predicted by contributors on the JIRA). I'd be > > >>>> interested > > >>>>> to > > >>>>>>> hear if they are too burdensome. In particular, your IDE > experience > > >>>>> gets a > > >>>>>>> little convoluted as you will need to add to your build path, a > jar > > >>>> with > > >>>>>>> the relocated pbs. A pain. > > >>>>>>> > > >>>>>>> Thanks, > > >>>>>>> St.Ack > > >>>>>>> > > >>>>>>> > > >>>>>>>> On Wed, Apr 13, 2016 at 6:09 AM, Stack <[email protected]> > wrote: > > >>>>>>>> > > >>>>>>>> On Tue, Apr 12, 2016 at 9:26 PM, Sean Busbey <[email protected] > > > > >>>>> wrote: > > >>>>>>>> > > >>>>>>>>>> On Tue, Apr 12, 2016 at 6:17 PM, Stack <[email protected]> > > wrote: > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> On an initial pass, the only difficult part seems to be > > >>>> interaction > > >>>>>>>>> with > > >>>>>>>>>> HDFS in asyncwal (might just pull in the HDFS messages). > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> I have some idea how we can make this work either by pushing > > >>>> asyncwal > > >>>>>>>>> upstream to HDFS or through some maven tricks, depending on how > > >>> much > > >>>>>>>>> time we have. > > >>>>>>>>> > > >>>>>>>> > > >>>>>>>> Maven tricks? Tell us more. Here or drop a note up in the issue. > > >>>>>>>> Thanks Sean, > > >>>>>>>> St.Ack > > >>>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> -- > > >>>>> busbey > > >>>>> > > >>>> > > >>> > > >> > > >> > > >
