Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

> There are precompile binaries for protoc on maven. See here > https://repo.maven.apache.org/maven2/com/google/protobuf/protoc/3.1.0/ > I have tested it before. I do not have protoc3 installed but I can compile this project. I stand corrected. Good to know. I didn't think Maven hosted native

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

On Tue, Oct 4, 2016 at 5:19 AM, 张铎 wrote: > Got it sir. > > And for the new module for CPEP, I think we could put all the classes > annotated as InterfaceAudience.COPROC in it. Currently they are annotated Private. Do you mean supporting classes? There are none in

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

2016-10-03 1:18 GMT+08:00 Andrew Purtell : > I have 2.5 and 3.0 installed as /opt/protobuf-, and have bash > scripts that add the appropriate version's bin directory to the path. Not > particularly onerous as I also have to switch between required JDK > versions, so the

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

Got it sir. And for the new module for CPEP, I think we could put all the classes annotated as InterfaceAudience.COPROC in it. Ideally, the CPEP should only depend on interfaces and data structures. Of course, there is a long way to go. Thanks. 2016-10-04 11:31 GMT+08:00 Stack

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

I just pushed the below set of patches. I'll be keeping an eye out on the build. Hopefully not too many teething issues. St.Ack On Mon, Oct 3, 2016 at 8:31 PM, Stack wrote: > On Mon, Oct 3, 2016 at 5:40 PM, 张铎 wrote: > >> OK, this means we will still

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

On Mon, Oct 3, 2016 at 5:40 PM, 张铎 wrote: > OK, this means we will still use pb2.5 for hbase-protocol? And > hbase-protocol-shaded is maintained manually? I still think the precommit > job should have the ability to check the hbase-protocol-shaded... > > Yes. For now. Let

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

On Mon, Oct 3, 2016 at 7:01 PM, Enis Söztutar wrote: > > > > > > If hadoop shaded its artifacts so no leakage whether an HDFS or YARN > > context (or Spark -- Spark2 is pb2.5.0) and downstreamers made use of our > > shaded artifacts everywhere then we could avoid this effort.

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

> > > If hadoop shaded its artifacts so no leakage whether an HDFS or YARN > context (or Spark -- Spark2 is pb2.5.0) and downstreamers made use of our > shaded artifacts everywhere then we could avoid this effort. > > Some form of isolation should show up in hadoop3 before it ships. My guess > is

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

On Mon, Oct 3, 2016 at 4:28 PM, Enis Söztutar wrote: > On Mon, Oct 3, 2016 at 2:55 PM, Andrew Purtell > wrote: > > > I would not expect Hadoop to shade PB in HDFS in any near time frame. I > > don't think we have a good chance of impressing upon them a

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

OK, this means we will still use pb2.5 for hbase-protocol? And hbase-protocol-shaded is maintained manually? I still think the precommit job should have the ability to check the hbase-protocol-shaded... I skimmed the branch, the proto files are placed both in hbase-protocol and

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

On Mon, Oct 3, 2016 at 2:55 PM, Andrew Purtell wrote: > I would not expect Hadoop to shade PB in HDFS in any near time frame. I > don't think we have a good chance of impressing upon them a sense of > urgency based on their spotty track of listening to downstreamer concerns.

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

I would not expect Hadoop to shade PB in HDFS in any near time frame. I don't think we have a good chance of impressing upon them a sense of urgency based on their spotty track of listening to downstreamer concerns. Even if we could and they deliver, what about all of the already deployed 2.x? Or

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

Is PB only coming via HDFS client or there are other usages that pollute our classpath? We are doing all this gymnastics just because upstream does not do shading is frustrating. Any idea for how likely we can push shaded hdfs to Hadoop? Should we focus on that instead rather than working around

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

On Mon, Oct 3, 2016 at 9:41 AM, Ted Yu wrote: > The precommit job uses compile-protobuf profile for verification. The > absence of compile-protobuf profile in hbase-protocol-shaded module means > precommit job would only invoke the existing compile-protobuf profile > in

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

The precommit job uses compile-protobuf profile for verification. The absence of compile-protobuf profile in hbase-protocol-shaded module means precommit job would only invoke the existing compile-protobuf profile in hbase-protocol module. w.r.t. path, when two protoc executables (corresponding

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

On Mon, Oct 3, 2016 at 7:16 AM, Ted Yu wrote: > Looks like compile-protobuf profile is not in hbase-protocol-shaded/pom.xml > (in HBASE-16264 branch) > > Sorry. I don't get what you are saying here (The target in the new module is generate-sources. See the included README.

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

Looks like compile-protobuf profile is not in hbase-protocol-shaded/pom.xml (in HBASE-16264 branch) Seems precommit jobs should pass with the current formation. In the future, if we add another profile for compiling proto3 files, we need to specify the path to proto3 compiler. Please correct me

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

The protoc generated files (such as MasterProtos) are checked into source repo, right ? Do we need proto3 on the precommit image(s) ? On Mon, Oct 3, 2016 at 5:18 AM, 张铎 wrote: > Then I think we need to file an issue to change the protoc version > installed in the

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

Then I think we need to file an issue to change the protoc version installed in the precommit docker file to 3.x before the merge. Otherwise the precommit build for protoc check maybe broken after the merge... 2016-10-03 1:18 GMT+08:00 Andrew Purtell : > I have 2.5 and

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

I don't know why you are responding to my email with this, but if is confusion as to how to install multiple versions of protoc safely, let me point you to the the '--prefix=' option of the ./configure script. ./configure --prefix=/opt/protobuf-2.5 && make && make install ./configure

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

I just installed protobuf 3 on my Mac. In case anyone doesn't have protobuf-2.5.0 source locally, please remember to backup protoc executable before running 'sudo make install' under the protobuf 3 dir. Cheers On Sun, Oct 2, 2016 at 10:18 AM, Andrew Purtell wrote: >

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

I have 2.5 and 3.0 installed as /opt/protobuf-, and have bash scripts that add the appropriate version's bin directory to the path. Not particularly onerous as I also have to switch between required JDK versions, so the scripts also set JAVA_HOME at and add JDK bin to the path for the required

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

Do we need to install protoc 3.0 manully before building HBase or the maven protobuf plugin will automatically download the protoc compiler? Maybe we need to install protoc 3.0 in the precommit docker file. 2016-10-02 14:20 GMT+08:00 张铎 : > > 2016-10-02 0:50 GMT+08:00

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

2016-10-02 0:50 GMT+08:00 Stack : > On Sat, Oct 1, 2016 at 7:20 AM, 张铎 wrote: > > > Can we setup a compatibility checker job in jenkins? Start a minicluster > in > > one process, and use a client in another process to communicate with it. > > The version

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

+1 for the merge. I had reviewed the big patch which adds new modules and replace refs to shaded pb path. Will just see the others also. Thanks for this great work Stack. Anoop On Sunday, October 2, 2016, Stack wrote: > On Sat, Oct 1, 2016 at 11:44 AM, Andrew Purtell

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

On Sat, Oct 1, 2016 at 11:44 AM, Andrew Purtell wrote: Slower? Hmm. For what it's worth after the merge (Monday?) I will take the > result through the YCSB workload set with security active and pull out JFR > trace files and GC logs. A Phoenix perf test would also be

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

+1 to pushing on this +1 to pushing further on the off heaping work as a result Slower? Hmm. For what it's worth after the merge (Monday?) I will take the result through the YCSB workload set with security active and pull out JFR trace files and GC logs. A Phoenix perf test would also be

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

On Sat, Oct 1, 2016 at 9:41 AM, Stack wrote: > > > On Fri, Sep 30, 2016 at 8:55 PM, Sean Busbey wrote: > >> have we experimentally confirmed that wire compatibility is >> maintained? I saw one mention of expecting wire compatibility to be >> fine, but

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

On Sat, Oct 1, 2016 at 7:20 AM, 张铎 wrote: > Can we setup a compatibility checker job in jenkins? Start a minicluster in > one process, and use a client in another process to communicate with it. > The version of the client should be >= 0.98 and <= the version of the >

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

On Fri, Sep 30, 2016 at 8:55 PM, Sean Busbey wrote: > have we experimentally confirmed that wire compatibility is > maintained? I saw one mention of expecting wire compatibility to be > fine, but nothing with someone using e.g. the clusterdock work or > something to mix

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

Can we setup a compatibility checker job in jenkins? Start a minicluster in one process, and use a client in another process to communicate with it. The version of the client should be >= 0.98 and <= the version of the minicluster. Of course we need to design the testing code carefully to make

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

have we experimentally confirmed that wire compatibility is maintained? I saw one mention of expecting wire compatibility to be fine, but nothing with someone using e.g. the clusterdock work or something to mix servers / clients or do replication. On Fri, Sep 30, 2016 at 6:30 PM, Stack

Re: [NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

+1 Difficult work. Glad to see it landing. > On Sep 30, 2016, at 4:30 PM, Stack wrote: > > I intend to do a mass commit late this weekend that moves us on to a shaded > protobuf-3.1.0, either Sunday night or Monday morning. > > If objection, please speak up or if need more

[NOTICE] Merge of shaded protobuf 3.1.0 (WAS => [DISCUSS] Shade protobuf so we can move to a newer version)

I intend to do a mass commit late this weekend that moves us on to a shaded protobuf-3.1.0, either Sunday night or Monday morning. If objection, please speak up or if need more time for consideration/review, just shout. I want to merge the branch HBASE-16264 into master (it is running here up on

Re: [DISCUSS] Shade protobuf so we can move to a newer version

This project goes on. I updated HBASE-1563 "Shade protobuf" with some doc on a final approach. We need to be able to refer to both shaded and non-shaded protobuf so we can support sending HDFS old-school pb Messages but also so Coprocessor Endpoints keep working though internally protobufs have

Re: [DISCUSS] Shade protobuf so we can move to a newer version

On Tue, Apr 12, 2016 at 9:26 PM, Sean Busbey wrote: > On Tue, Apr 12, 2016 at 6:17 PM, Stack wrote: > > > > > > On an initial pass, the only difficult part seems to be interaction with > > HDFS in asyncwal (might just pull in the HDFS messages). > > > > > >

Re: [DISCUSS] Shade protobuf so we can move to a newer version

I think we could do it with some maven tricks. Just do not relocate the protobuf things in asyncwal? Move the related classes to a new sub project? Thanks. 2016-04-13 12:26 GMT+08:00 Sean Busbey : > On Tue, Apr 12, 2016 at 6:17 PM, Stack wrote: > > > > > >

Re: [DISCUSS] Shade protobuf so we can move to a newer version

On Tue, Apr 12, 2016 at 6:17 PM, Stack wrote: > > > On an initial pass, the only difficult part seems to be interaction with > HDFS in asyncwal (might just pull in the HDFS messages). > > I have some idea how we can make this work either by pushing asyncwal upstream to HDFS or

Re: [DISCUSS] Shade protobuf so we can move to a newer version

Yeah I think it's a good idea however we need to remove PB from all our user facing api's first. There are some brave souls working on it here: https://issues.apache.org/jira/browse/HBASE-15174 On Tue, Apr 12, 2016 at 4:17 PM, Stack wrote: > I opened HBASE-15638 Shade protobuf

[DISCUSS] Shade protobuf so we can move to a newer version

I opened HBASE-15638 Shade protobuf but should probably raise the larger intent here. You fellows might have an opinion (smile). We need to shade PB so we can move to a different version. We need to move to a different version because we want to save on buffer copies -- newer versions of PB have