On Thu, Jul 6, 2017 at 11:52 AM, Stack <[email protected]> wrote: > FYI: > > hbase-thirdparty has had its first release. Yesterday I > committed HBASE-17056 "Remove checked in PB generated files" which apart > from purging all checked-in generated files (30MB), it moves our hbase core > (master and branch-2) to start using the thirdparty jar. > > Things might be interesting over next few days so shout if you run into > issues. > > FYI, I reverted HBASE-17056 for the moment. Build seems unstable. OOMEs and other interesting issues along w/ some awkwardness w/ dependencies. Let me spend some more time on it. Will try again later.
St.Ack > One issue is the need for mvn install where before mvn compile might have > been enough (see below for example of the issue you'd see now if you did > mvn clean compile only). We need the mvn install because we need to shade > the generated files so they use the relocated protobuf; shade happens after > we've made a module jar. > > IDEs will complain too if they pick up generated src from target dirs > since they'll be unsatisfied references to protobuf3 objects -- unless you > point at the shaded jar. Let me see if I can do something about the latter. > > Let me know if HBASE-17056 is too much for devs to bear. Can always revert > (though nice having the protobuf generation in-line w/ the build). It is > sort of a side-benefit on the general shading project so could back it out > and still have the shading of netty, guava, protobuf, etc. > > Thanks, > > St.Ack > > [ERROR] Failed to execute goal org.apache.maven.plugins: > maven-compiler-plugin:3.6.1:compile (default-compile) on project > hbase-procedure: Compilation failure: Compilation failure: > [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/ > apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[105,11] > cannot access com.google.protobuf.GeneratedMessageV3 > [ERROR] class file for com.google.protobuf.GeneratedMessageV3 not found > [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/ > apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[129,7] > cannot access com.google.protobuf.GeneratedMessageV3.Builder > [ERROR] class file for com.google.protobuf.GeneratedMessageV3$Builder not > found > [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/ > apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[133,22] > cannot find symbol > [ERROR] symbol: method writeDelimitedTo(org.apache. > hadoop.fs.FSDataOutputStream) > [ERROR] location: class org.apache.hadoop.hbase.shaded.protobuf.generated. > ProcedureProtos.ProcedureStoreTracker > [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/ > apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[217,20] > cannot find symbol > [ERROR] symbol: method writeDelimitedTo(org.apache. > hadoop.hbase.procedure2.util.ByteSlot) > [ERROR] location: class org.apache.hadoop.hbase.shaded.protobuf.generated. > ProcedureProtos.ProcedureWALEntry > [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/ > apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[240,20] > cannot find symbol > [ERROR] symbol: method writeDelimitedTo(org.apache. > hadoop.hbase.procedure2.util.ByteSlot) > [ERROR] location: class org.apache.hadoop.hbase.shaded.protobuf.generated. > ProcedureProtos.ProcedureWALEntry > [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/ > apache/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[254,20] > cannot find symbol > [ERROR] symbol: method writeDelimitedTo(org.apache. > hadoop.hbase.procedure2.util.ByteSlot) > [ERROR] location: class org.apache.hadoop.hbase.shaded.protobuf.generated. > ProcedureProtos.ProcedureWALEntry > [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/ > apache/hadoop/hbase/procedure2/RemoteProcedureException.java:[98,30] > cannot find symbol > [ERROR] symbol: method toByteArray() > [ERROR] location: class org.apache.hadoop.hbase.shaded.protobuf.generated. > ErrorHandlingProtos.ForeignExceptionMessage > [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/ > apache/hadoop/hbase/procedure2/StateMachineProcedure.java:[267,17] cannot > find symbol > [ERROR] symbol: method writeDelimitedTo(java.io.OutputStream) > [ERROR] location: class org.apache.hadoop.hbase.shaded.protobuf.generated. > ProcedureProtos.StateMachineProcedureData > [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/ > apache/hadoop/hbase/procedure2/ProcedureUtil.java:[130,56] incompatible > types: org.apache.hadoop.hbase.shaded.com.google.protobuf.ByteString > cannot be converted to com.google.protobuf.ByteString > [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/ > apache/hadoop/hbase/procedure2/ProcedureUtil.java:[137,54] incompatible > types: org.apache.hadoop.hbase.shaded.com.google.protobuf.ByteString > cannot be converted to com.google.protobuf.ByteString > [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/ > apache/hadoop/hbase/procedure2/ProcedureUtil.java:[237,56] incompatible > types: org.apache.hadoop.hbase.shaded.com.google.protobuf.ByteString > cannot be converted to com.google.protobuf.ByteString > [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/ > apache/hadoop/hbase/procedure2/SequentialProcedure.java:[75,17] cannot > find symbol > [ERROR] symbol: method writeDelimitedTo(java.io.OutputStream) > [ERROR] location: class org.apache.hadoop.hbase.shaded.protobuf.generated. > ProcedureProtos.SequentialProcedureData > [ERROR] -> [Help 1] > [ERROR] > [ERROR] To see the full stack trace of the errors, re-run Maven with the > -e switch. > [ERROR] Re-run Maven using the -X switch to enable full debug logging. > [ERROR] > [ERROR] For more information about the errors and possible solutions, > please read the following articles: > [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ > MojoFailureException > [ERROR] > [ERROR] After correcting the problems, you can resume the build with the > command > [ERROR] mvn <goals> -rf :hbase-procedure > > On Fri, Jun 30, 2017 at 3:09 PM, Stack <[email protected]> wrote: > >> I just started a VOTE on hbase-thirdparty and the first RC made from it. >> Thanks, >> St.Ack >> >> On Tue, Jun 27, 2017 at 3:02 PM, Stack <[email protected]> wrote: >> >>> Bit of an update. >>> >>> I'd suggest we go ahead w/ the hbase-thirdparty project [2]. It took a >>> while but in its current form -- a few poms that package a few jars [1]-- >>> it at least enables the below: >>> >>> + Allows us to skip checking in protobuf generated files (25MB!); they >>> can be generated inline w/ the build because the hackery patching protobuf >>> has been moved out to hbase-thirdparty. There is a patch up on HBASE-17056. >>> + Update our guava from 12.0 to 22.0 w/o clashing w/ the guava of >>> others. There is a patch at HBASE-17908. It is taking a bit of wrangling >>> getting it to land because I pared back transitive includes from hadoop and >>> it takes a while to work through the failures. >>> >>> Other benefits are the protobuf-util lib is on the classpath now -- its >>> in hbase-thirdparty relocated; depends on pb and guava -- so we have >>> facility to goat "HBASE-18106 Redo ProcedureInfo and LockInfo" and shading >>> netty is almost done so we can do with netty as we wilt independent of >>> hadoop and downstreamers (the hard part -- relocation of the .so -- should >>> be done). >>> >>> Let me figure how to run a vote for a couple of poms..... >>> >>> St.Ack >>> >>> 1. https://repository.apache.org/content/groups/snapshots/or >>> g/apache/hbase/thirdparty/ (see hbase-shaded-thirdparty and >>> hbase-shaded-protobuf) >>> 2. https://git-wip-us.apache.org/repos/asf/hbase-thirdparty >>> >>> >>> On Tue, Jun 20, 2017 at 11:04 AM, Josh Elser <[email protected]> >>> wrote: >>> >>>> On 6/20/17 1:28 AM, Stack wrote: >>>> >>>>> On Thu, Apr 13, 2017 at 4:46 PM, Josh Elser<[email protected]> wrote: >>>>> >>>>> ... >>>>>> >>>>>> I think pushing this part forward with some code is the next logical >>>>>> step. >>>>>> Seems to be consensus about taking our known internal dependencies and >>>>>> performing this shade magic. >>>>>> >>>>>> >>>>>> I opened HBASE-18240 "Add hbase-auxillary, a project with hbase >>>>> utility >>>>> including an hbase-shaded-thirdparty module with guava, netty, etc." >>>>> >>>>> It has a tarball attached that bundles the outline of an >>>>> hbase-auxillary >>>>> project (groupId:org.apache.hbase.auxillary). This project is >>>>> intended to >>>>> be standalone, in its own repository, publishing its own artifacts >>>>> under >>>>> the aegis of this project's PMC. >>>>> >>>>> It includes the first instance of an auxillary utility, a module named >>>>> hbase-thirdparty-shaded (artifactId:hbase-thirdparty-shaded). Herein >>>>> we'll >>>>> pull down 3rd party libs and republish at an offset; e.g. >>>>> com.google.common.* from guava will be at >>>>> org.apache.hbase.thirdparty.shaded.com.google.common.*. Currently it >>>>> builds >>>>> a jar that includes a relocated guava 22.0. >>>>> >>>>> I then messed around making hbase-common use it (You have to build the >>>>> hbase-auxillary into your local repo). I put up a patch on the issue. >>>>> Mostly its mass find-and-replace w/ some clean up of transitive >>>>> includes of >>>>> guava from hadoop-common and some small fixup of methods renamed >>>>> between >>>>> guava 12.0 and 22.0. >>>>> >>>>> Unless objection, I was going to press on. Sean offered to help set up >>>>> new >>>>> repo. We can always undo and delete it if this project fails. >>>>> >>>>> When done, the hope is we are on a modern version of guava and our >>>>> netty >>>>> and protobuf 3 will be be relocated, 'hidden' from downstream (and >>>>> won't >>>>> clash w/ upstream). I hope to also purge the pre-build we have in our >>>>> modules that do protobuf moving this hackery out and under >>>>> hbase-thirdparty-shaded. >>>>> >>>>> St.Ack >>>>> >>>> >>>> Kudos on the JFDI approach :). I think having something concrete to >>>> show is the best way to judge success of it. >>>> >>>> Will keep an eye on HBASE-18240. >>>> >>>> >>> >> >
