2016-10-02 0:50 GMT+08:00 Stack <[email protected]>: > On Sat, Oct 1, 2016 at 7:20 AM, 张铎 <[email protected]> wrote: > > > Can we setup a compatibility checker job in jenkins? Start a minicluster > in > > one process, and use a client in another process to communicate with it. > > The version of the client should be >= 0.98 and <= the version of the > > minicluster. Of course we need to design the testing code carefully to > make > > sure that we have tested all the cases. > > > > > +1. We need this up and running before we put out an hbase-2.0.0. I know > Matteo does this test manually on a regular basis but a formalization would > help. I can add an exercise of Coprocessor Endpoints. I believe this is on > Dima's list of TODOs but will let him speak for himself. > > > > And also I think we should make sure that no proto3 only feature is > > introduced in our proto files until branch-1 is dead. Maybe a precommit > > check? > > > > > I think you mean wire-format breaking changes? Agree. We have our PB3 set > to 2.5 compat mode and yes, we can't move on from this until we are in a > place where we can say no to 2.5 clients. > Yes, for example, pb2.5 does not support map so we should not use map in our proto files.
> > Making use of PB3isms cannot be avoided. PB3.1 adds a native replacement > for our HBaseZeroCopyByteString/ByteStringer hack. It also adds 'unsafe' > methods that we need to exploit if we are to keep our read/write paths > offheap. > > St.Ack > > > > > > > Thanks. > > > > 2016-10-01 11:55 GMT+08:00 Sean Busbey <[email protected]>: > > > > > have we experimentally confirmed that wire compatibility is > > > maintained? I saw one mention of expecting wire compatibility to be > > > fine, but nothing with someone using e.g. the clusterdock work or > > > something to mix servers / clients or do replication. > > > > > > On Fri, Sep 30, 2016 at 6:30 PM, Stack <[email protected]> wrote: > > > > I intend to do a mass commit late this weekend that moves us on to a > > > shaded > > > > protobuf-3.1.0, either Sunday night or Monday morning. > > > > > > > > If objection, please speak up or if need more time for > > > > consideration/review, just shout. > > > > > > > > I want to merge the branch HBASE-16264 into master (it is running > here > > up > > > > on jenkins https://builds.apache.org/view/H-L/view/HBase/job/HBASE- > > > 16264/). > > > > The branch at HBASE-16264 has three significant bodies-of-work that > > > > unfortunately are tangled and can only go in of a piece. > > > > > > > > * HBASE-16264 <https://issues.apache.org/jira/browse/HBASE-16264> > The > > > > shading of our protobuf usage so we can upgrade and/or run with a > > patched > > > > protobuf WITHOUT breaking REST, Spark, and in particular, Coprocessor > > > > Endpoints. > > > > * HBASE-16567 <https://issues.apache.org/jira/browse/HBASE-16567> A > > > move > > > > up on to (shaded) protobuf-3.1.0 > > > > * HBASE-16741 <https://issues.apache.org/jira/browse/HBASE-16741> > An > > > > amendment of our generate protobufs step to include shading and a > > > bundling > > > > of protobuf src (with a means of calling a patch srcs hook) > > > > > > > > Together we're talking about 40MB of change mostly made of the > movement > > > of > > > > generated files or the application of a pattern that alters where we > > get > > > > imports from. When done, you should notice no difference and should > be > > > able > > > > to go about your business as per usual. Upside is that we will be > able > > to > > > > avoid coming onheap doing protobuf marshalling/unmarshalling as > > protobuf > > > > 2.5.0 requires. Downside is that we repeat a good portion of our > > internal > > > > protos, once non-shaded so Coprocessor Endpoints can keep working and > > > then > > > > again as shaded for internal use. > > > > > > > > I provide some more overview below on the changes. See the shading > doc > > > > here: > > > > https://docs.google.com/document/d/1H4NgLXQ9Y9KejwobddCqaVMEDCGby > > > DcXtdF5iAfDIEk/edit# > > > > for more detail (Patches are up on review board -- except the latest > > > > HBASE-16264 which is too big for JIRA and RB). I am currently working > > on > > > a > > > > devs chapter for the book on protobuf going forward that will go in > as > > > part > > > > of this patch. > > > > > > > > Thanks, > > > > St.Ack > > > > > > > > Items of note: > > > > > > > > * Two new modules; one named hbase-protocol-shaded that is used by > > hbase > > > > core. It has in it a shaded (and later patched) protobuf. The other > new > > > > module is hbase-endpoint which goes after hbase-server and has those > > > > bundled endpoints that I was able to break out of core (there are a > few > > > > that are hopelessly entangled that need to be undone as CPEPs but > > > > fortunately belong in core: Auth, Access, MultiRow). > > > > * I've tested running a branch-1 CPEP against a master with these > > > patches > > > > in place and stuff like ACL (A CPEP) run from the branch-1 shell work > > > > against the branch-2 server. > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Aug 22, 2016 at 5:20 PM, Stack <[email protected]> wrote: > > > > > > > >> This project goes on. I updated HBASE-1563 "Shade protobuf" with > some > > > doc > > > >> on a final approach. We need to be able to refer to both shaded and > > > >> non-shaded protobuf so we can support sending HDFS old-school pb > > > Messages > > > >> but also so Coprocessor Endpoints keep working though internally > > > protobufs > > > >> have been relocated. Funny you should ask, but yes, there are some > > > >> downsides (as predicted by contributors on the JIRA). I'd be > > interested > > > to > > > >> hear if they are too burdensome. In particular, your IDE experience > > > gets a > > > >> little convoluted as you will need to add to your build path, a jar > > with > > > >> the relocated pbs. A pain. > > > >> > > > >> Thanks, > > > >> St.Ack > > > >> > > > >> > > > >> On Wed, Apr 13, 2016 at 6:09 AM, Stack <[email protected]> wrote: > > > >> > > > >>> On Tue, Apr 12, 2016 at 9:26 PM, Sean Busbey <[email protected]> > > > wrote: > > > >>> > > > >>>> On Tue, Apr 12, 2016 at 6:17 PM, Stack <[email protected]> wrote: > > > >>>> > > > > >>>> > > > > >>>> > On an initial pass, the only difficult part seems to be > > interaction > > > >>>> with > > > >>>> > HDFS in asyncwal (might just pull in the HDFS messages). > > > >>>> > > > > >>>> > > > > >>>> > > > >>>> I have some idea how we can make this work either by pushing > > asyncwal > > > >>>> upstream to HDFS or through some maven tricks, depending on how > much > > > >>>> time we have. > > > >>>> > > > >>> > > > >>> Maven tricks? Tell us more. Here or drop a note up in the issue. > > > >>> Thanks Sean, > > > >>> St.Ack > > > >>> > > > >> > > > >> > > > > > > > > > > > > -- > > > busbey > > > > > >
