Yes we are not changing the proto files as such.. We use the old ones but regenerate the java files using PB 3 compiler. But within our code we will be using PB 3 specific features/APIs . Eg: As Stack mentioned, the way to wrap a byte[] into PB ByteString.
-Anoop- On Mon, Oct 3, 2016 at 10:46 AM, ramkrishna vasudevan <[email protected]> wrote: > +1 on merge. Have been following up the JIRA from Stack and have discussed > on the upsides and downsides. +1 to do the merge. > Wire compatibility should be fine as per our testing. But still +1 to test > more in case we don't miss any case. > > Regards > Ram > > On Sun, Oct 2, 2016 at 11:41 PM, Andrew Purtell <[email protected]> > wrote: > >> I don't know why you are responding to my email with this, but if is >> confusion as to how to install multiple versions of protoc safely, let me >> point you to the the '--prefix=' option of the ./configure script. >> >> <check out 2.5> >> ./configure --prefix=/opt/protobuf-2.5 && make && make install >> >> <check out 3.0> >> ./configure --prefix=/opt/protobuf-3.0 && make && make install >> >> >> > On Oct 2, 2016, at 10:41 AM, Ted Yu <[email protected]> wrote: >> > >> > I just installed protobuf 3 on my Mac. >> > >> > In case anyone doesn't have protobuf-2.5.0 source locally, please >> remember >> > to backup protoc executable before running 'sudo make install' under the >> > protobuf 3 dir. >> > >> > Cheers >> > >> > On Sun, Oct 2, 2016 at 10:18 AM, Andrew Purtell < >> [email protected]> >> > wrote: >> > >> >> I have 2.5 and 3.0 installed as /opt/protobuf-<version>, and have bash >> >> scripts that add the appropriate version's bin directory to the path. >> Not >> >> particularly onerous as I also have to switch between required JDK >> >> versions, so the scripts also set JAVA_HOME at and add JDK bin to the >> path >> >> for the required JDK for the build. >> >> >> >> Unlike with the scala compiler, which is after all JVM bytecode at a >> >> fundamental level, I don't think maven automation for automatic download >> >> and execution is possible. protoc is a native binary. >> >> >> >>> On Oct 1, 2016, at 11:30 PM, 张铎 <[email protected]> wrote: >> >>> >> >>> Do we need to install protoc 3.0 manully before building HBase or the >> >> maven >> >>> protobuf plugin will automatically download the protoc compiler? Maybe >> we >> >>> need to install protoc 3.0 in the precommit docker file. >> >>> >> >>> 2016-10-02 14:20 GMT+08:00 张铎 <[email protected]>: >> >>> >> >>>> >> >>>> 2016-10-02 0:50 GMT+08:00 Stack <[email protected]>: >> >>>> >> >>>>>> On Sat, Oct 1, 2016 at 7:20 AM, 张铎 <[email protected]> wrote: >> >>>>>> >> >>>>>> Can we setup a compatibility checker job in jenkins? Start a >> >>>>> minicluster in >> >>>>>> one process, and use a client in another process to communicate with >> >> it. >> >>>>>> The version of the client should be >= 0.98 and <= the version of >> the >> >>>>>> minicluster. Of course we need to design the testing code carefully >> to >> >>>>> make >> >>>>>> sure that we have tested all the cases. >> >>>>>> >> >>>>>> >> >>>>> +1. We need this up and running before we put out an hbase-2.0.0. I >> >> know >> >>>>> Matteo does this test manually on a regular basis but a formalization >> >>>>> would >> >>>>> help. I can add an exercise of Coprocessor Endpoints. I believe this >> >> is on >> >>>>> Dima's list of TODOs but will let him speak for himself. >> >>>>> >> >>>>> >> >>>>>> And also I think we should make sure that no proto3 only feature is >> >>>>>> introduced in our proto files until branch-1 is dead. Maybe a >> >> precommit >> >>>>>> check? >> >>>>>> >> >>>>>> >> >>>>> I think you mean wire-format breaking changes? Agree. We have our >> PB3 >> >> set >> >>>>> to 2.5 compat mode and yes, we can't move on from this until we are >> in >> >> a >> >>>>> place where we can say no to 2.5 clients. >> >>>>> >> >>>> Yes, for example, pb2.5 does not support map so we should not use map >> in >> >>>> our proto files. >> >>>> >> >>>>> >> >>>>> Making use of PB3isms cannot be avoided. PB3.1 adds a native >> >> replacement >> >>>>> for our HBaseZeroCopyByteString/ByteStringer hack. It also adds >> >> 'unsafe' >> >>>>> methods that we need to exploit if we are to keep our read/write >> paths >> >>>>> offheap. >> >>>>> >> >>>>> St.Ack >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>>> Thanks. >> >>>>>> >> >>>>>> 2016-10-01 11:55 GMT+08:00 Sean Busbey <[email protected]>: >> >>>>>> >> >>>>>>> have we experimentally confirmed that wire compatibility is >> >>>>>>> maintained? I saw one mention of expecting wire compatibility to be >> >>>>>>> fine, but nothing with someone using e.g. the clusterdock work or >> >>>>>>> something to mix servers / clients or do replication. >> >>>>>>> >> >>>>>>>> On Fri, Sep 30, 2016 at 6:30 PM, Stack <[email protected]> wrote: >> >>>>>>>> I intend to do a mass commit late this weekend that moves us on >> to a >> >>>>>>> shaded >> >>>>>>>> protobuf-3.1.0, either Sunday night or Monday morning. >> >>>>>>>> >> >>>>>>>> If objection, please speak up or if need more time for >> >>>>>>>> consideration/review, just shout. >> >>>>>>>> >> >>>>>>>> I want to merge the branch HBASE-16264 into master (it is running >> >>>>> here >> >>>>>> up >> >>>>>>>> on jenkins https://builds.apache.org/ >> view/H-L/view/HBase/job/HBASE- >> >>>>>>> 16264/). >> >>>>>>>> The branch at HBASE-16264 has three significant bodies-of-work >> that >> >>>>>>>> unfortunately are tangled and can only go in of a piece. >> >>>>>>>> >> >>>>>>>> * HBASE-16264 <https://issues.apache.org/jira/browse/HBASE-16264> >> >>>>> The >> >>>>>>>> shading of our protobuf usage so we can upgrade and/or run with a >> >>>>>> patched >> >>>>>>>> protobuf WITHOUT breaking REST, Spark, and in particular, >> >>>>> Coprocessor >> >>>>>>>> Endpoints. >> >>>>>>>> * HBASE-16567 <https://issues.apache.org/jira/browse/HBASE-16567> >> >>>>> A >> >>>>>>> move >> >>>>>>>> up on to (shaded) protobuf-3.1.0 >> >>>>>>>> * HBASE-16741 <https://issues.apache.org/jira/browse/HBASE-16741> >> >>>>> An >> >>>>>>>> amendment of our generate protobufs step to include shading and a >> >>>>>>> bundling >> >>>>>>>> of protobuf src (with a means of calling a patch srcs hook) >> >>>>>>>> >> >>>>>>>> Together we're talking about 40MB of change mostly made of the >> >>>>> movement >> >>>>>>> of >> >>>>>>>> generated files or the application of a pattern that alters where >> we >> >>>>>> get >> >>>>>>>> imports from. When done, you should notice no difference and >> should >> >>>>> be >> >>>>>>> able >> >>>>>>>> to go about your business as per usual. Upside is that we will be >> >>>>> able >> >>>>>> to >> >>>>>>>> avoid coming onheap doing protobuf marshalling/unmarshalling as >> >>>>>> protobuf >> >>>>>>>> 2.5.0 requires. Downside is that we repeat a good portion of our >> >>>>>> internal >> >>>>>>>> protos, once non-shaded so Coprocessor Endpoints can keep working >> >>>>> and >> >>>>>>> then >> >>>>>>>> again as shaded for internal use. >> >>>>>>>> >> >>>>>>>> I provide some more overview below on the changes. See the shading >> >>>>> doc >> >>>>>>>> here: >> >>>>>>>> https://docs.google.com/document/d/1H4NgLXQ9Y9KejwobddCqaVMEDCGby >> >>>>>>> DcXtdF5iAfDIEk/edit# >> >>>>>>>> for more detail (Patches are up on review board -- except the >> latest >> >>>>>>>> HBASE-16264 which is too big for JIRA and RB). I am currently >> >>>>> working >> >>>>>> on >> >>>>>>> a >> >>>>>>>> devs chapter for the book on protobuf going forward that will go >> in >> >>>>> as >> >>>>>>> part >> >>>>>>>> of this patch. >> >>>>>>>> >> >>>>>>>> Thanks, >> >>>>>>>> St.Ack >> >>>>>>>> >> >>>>>>>> Items of note: >> >>>>>>>> >> >>>>>>>> * Two new modules; one named hbase-protocol-shaded that is used by >> >>>>>> hbase >> >>>>>>>> core. It has in it a shaded (and later patched) protobuf. The >> other >> >>>>> new >> >>>>>>>> module is hbase-endpoint which goes after hbase-server and has >> those >> >>>>>>>> bundled endpoints that I was able to break out of core (there are >> a >> >>>>> few >> >>>>>>>> that are hopelessly entangled that need to be undone as CPEPs but >> >>>>>>>> fortunately belong in core: Auth, Access, MultiRow). >> >>>>>>>> * I've tested running a branch-1 CPEP against a master with these >> >>>>>>> patches >> >>>>>>>> in place and stuff like ACL (A CPEP) run from the branch-1 shell >> >>>>> work >> >>>>>>>> against the branch-2 server. >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>>> On Mon, Aug 22, 2016 at 5:20 PM, Stack <[email protected]> wrote: >> >>>>>>>>> >> >>>>>>>>> This project goes on. I updated HBASE-1563 "Shade protobuf" with >> >>>>> some >> >>>>>>> doc >> >>>>>>>>> on a final approach. We need to be able to refer to both shaded >> and >> >>>>>>>>> non-shaded protobuf so we can support sending HDFS old-school pb >> >>>>>>> Messages >> >>>>>>>>> but also so Coprocessor Endpoints keep working though internally >> >>>>>>> protobufs >> >>>>>>>>> have been relocated. Funny you should ask, but yes, there are >> some >> >>>>>>>>> downsides (as predicted by contributors on the JIRA). I'd be >> >>>>>> interested >> >>>>>>> to >> >>>>>>>>> hear if they are too burdensome. In particular, your IDE >> experience >> >>>>>>> gets a >> >>>>>>>>> little convoluted as you will need to add to your build path, a >> jar >> >>>>>> with >> >>>>>>>>> the relocated pbs. A pain. >> >>>>>>>>> >> >>>>>>>>> Thanks, >> >>>>>>>>> St.Ack >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>>> On Wed, Apr 13, 2016 at 6:09 AM, Stack <[email protected]> >> wrote: >> >>>>>>>>>> >> >>>>>>>>>> On Tue, Apr 12, 2016 at 9:26 PM, Sean Busbey <[email protected] >> > >> >>>>>>> wrote: >> >>>>>>>>>> >> >>>>>>>>>>>> On Tue, Apr 12, 2016 at 6:17 PM, Stack <[email protected]> >> >> wrote: >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> On an initial pass, the only difficult part seems to be >> >>>>>> interaction >> >>>>>>>>>>> with >> >>>>>>>>>>>> HDFS in asyncwal (might just pull in the HDFS messages). >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> I have some idea how we can make this work either by pushing >> >>>>>> asyncwal >> >>>>>>>>>>> upstream to HDFS or through some maven tricks, depending on how >> >>>>> much >> >>>>>>>>>>> time we have. >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> Maven tricks? Tell us more. Here or drop a note up in the issue. >> >>>>>>>>>> Thanks Sean, >> >>>>>>>>>> St.Ack >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> -- >> >>>>>>> busbey >> >>>>>>> >> >>>>>> >> >>>>> >> >>>> >> >>>> >> >> >>
