On Thu, Jul 6, 2017 at 11:17 PM, Stack <[email protected]> wrote: > On Thu, Jul 6, 2017 at 11:52 AM, Stack <[email protected]> wrote: > >> FYI: >> >> hbase-thirdparty has had its first release. Yesterday I >> committed HBASE-17056 "Remove checked in PB generated files" which apart >> from purging all checked-in generated files (30MB), it moves our hbase core >> (master and branch-2) to start using the thirdparty jar. >> >> Things might be interesting over next few days so shout if you run into >> issues. >> >> > FYI, I reverted HBASE-17056 for the moment. Build seems unstable. OOMEs > and other interesting issues along w/ some awkwardness w/ dependencies. Let > me spend some more time on it. Will try again later. > >
Just a heads-up. With Chia-Ping Tsai and Guanghao Zhang's help, I was able to figure a few odd issues in previous push. I just pushed again. Kick me if you see OOMEs or weirdness.... I'll be keeping an eye out myself too. Thanks for your patience. St.Ack > St.Ack > > > >> One issue is the need for mvn install where before mvn compile might have >> been enough (see below for example of the issue you'd see now if you did >> mvn clean compile only). We need the mvn install because we need to shade >> the generated files so they use the relocated protobuf; shade happens after >> we've made a module jar. >> >> IDEs will complain too if they pick up generated src from target dirs >> since they'll be unsatisfied references to protobuf3 objects -- unless you >> point at the shaded jar. Let me see if I can do something about the latter. >> >> Let me know if HBASE-17056 is too much for devs to bear. Can always >> revert (though nice having the protobuf generation in-line w/ the build). >> It is sort of a side-benefit on the general shading project so could back >> it out and still have the shading of netty, guava, protobuf, etc. >> >> Thanks, >> >> St.Ack >> >> [ERROR] Failed to execute goal >> org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile >> (default-compile) on project hbase-procedure: Compilation failure: >> Compilation failure: >> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac >> he/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[105,11] >> cannot access com.google.protobuf.GeneratedMessageV3 >> [ERROR] class file for com.google.protobuf.GeneratedMessageV3 not found >> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac >> he/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[129,7] >> cannot access com.google.protobuf.GeneratedMessageV3.Builder >> [ERROR] class file for com.google.protobuf.GeneratedMessageV3$Builder >> not found >> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac >> he/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[133,22] >> cannot find symbol >> [ERROR] symbol: method writeDelimitedTo(org.apache.ha >> doop.fs.FSDataOutputStream) >> [ERROR] location: class org.apache.hadoop.hbase.shaded >> .protobuf.generated.ProcedureProtos.ProcedureStoreTracker >> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac >> he/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[217,20] >> cannot find symbol >> [ERROR] symbol: method writeDelimitedTo(org.apache.ha >> doop.hbase.procedure2.util.ByteSlot) >> [ERROR] location: class org.apache.hadoop.hbase.shaded >> .protobuf.generated.ProcedureProtos.ProcedureWALEntry >> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac >> he/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[240,20] >> cannot find symbol >> [ERROR] symbol: method writeDelimitedTo(org.apache.ha >> doop.hbase.procedure2.util.ByteSlot) >> [ERROR] location: class org.apache.hadoop.hbase.shaded >> .protobuf.generated.ProcedureProtos.ProcedureWALEntry >> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac >> he/hadoop/hbase/procedure2/store/wal/ProcedureWALFormat.java:[254,20] >> cannot find symbol >> [ERROR] symbol: method writeDelimitedTo(org.apache.ha >> doop.hbase.procedure2.util.ByteSlot) >> [ERROR] location: class org.apache.hadoop.hbase.shaded >> .protobuf.generated.ProcedureProtos.ProcedureWALEntry >> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac >> he/hadoop/hbase/procedure2/RemoteProcedureException.java:[98,30] cannot >> find symbol >> [ERROR] symbol: method toByteArray() >> [ERROR] location: class org.apache.hadoop.hbase.shaded >> .protobuf.generated.ErrorHandlingProtos.ForeignExceptionMessage >> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac >> he/hadoop/hbase/procedure2/StateMachineProcedure.java:[267,17] cannot >> find symbol >> [ERROR] symbol: method writeDelimitedTo(java.io.OutputStream) >> [ERROR] location: class org.apache.hadoop.hbase.shaded >> .protobuf.generated.ProcedureProtos.StateMachineProcedureData >> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac >> he/hadoop/hbase/procedure2/ProcedureUtil.java:[130,56] incompatible >> types: org.apache.hadoop.hbase.shaded.com.google.protobuf.ByteString >> cannot be converted to com.google.protobuf.ByteString >> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac >> he/hadoop/hbase/procedure2/ProcedureUtil.java:[137,54] incompatible >> types: org.apache.hadoop.hbase.shaded.com.google.protobuf.ByteString >> cannot be converted to com.google.protobuf.ByteString >> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac >> he/hadoop/hbase/procedure2/ProcedureUtil.java:[237,56] incompatible >> types: org.apache.hadoop.hbase.shaded.com.google.protobuf.ByteString >> cannot be converted to com.google.protobuf.ByteString >> [ERROR] /home/stack/hbase.git/hbase-procedure/src/main/java/org/apac >> he/hadoop/hbase/procedure2/SequentialProcedure.java:[75,17] cannot find >> symbol >> [ERROR] symbol: method writeDelimitedTo(java.io.OutputStream) >> [ERROR] location: class org.apache.hadoop.hbase.shaded >> .protobuf.generated.ProcedureProtos.SequentialProcedureData >> [ERROR] -> [Help 1] >> [ERROR] >> [ERROR] To see the full stack trace of the errors, re-run Maven with the >> -e switch. >> [ERROR] Re-run Maven using the -X switch to enable full debug logging. >> [ERROR] >> [ERROR] For more information about the errors and possible solutions, >> please read the following articles: >> [ERROR] [Help 1] http://cwiki.apache.org/conflu >> ence/display/MAVEN/MojoFailureException >> [ERROR] >> [ERROR] After correcting the problems, you can resume the build with the >> command >> [ERROR] mvn <goals> -rf :hbase-procedure >> >> On Fri, Jun 30, 2017 at 3:09 PM, Stack <[email protected]> wrote: >> >>> I just started a VOTE on hbase-thirdparty and the first RC made from it. >>> Thanks, >>> St.Ack >>> >>> On Tue, Jun 27, 2017 at 3:02 PM, Stack <[email protected]> wrote: >>> >>>> Bit of an update. >>>> >>>> I'd suggest we go ahead w/ the hbase-thirdparty project [2]. It took a >>>> while but in its current form -- a few poms that package a few jars [1]-- >>>> it at least enables the below: >>>> >>>> + Allows us to skip checking in protobuf generated files (25MB!); they >>>> can be generated inline w/ the build because the hackery patching protobuf >>>> has been moved out to hbase-thirdparty. There is a patch up on HBASE-17056. >>>> + Update our guava from 12.0 to 22.0 w/o clashing w/ the guava of >>>> others. There is a patch at HBASE-17908. It is taking a bit of wrangling >>>> getting it to land because I pared back transitive includes from hadoop and >>>> it takes a while to work through the failures. >>>> >>>> Other benefits are the protobuf-util lib is on the classpath now -- its >>>> in hbase-thirdparty relocated; depends on pb and guava -- so we have >>>> facility to goat "HBASE-18106 Redo ProcedureInfo and LockInfo" and shading >>>> netty is almost done so we can do with netty as we wilt independent of >>>> hadoop and downstreamers (the hard part -- relocation of the .so -- should >>>> be done). >>>> >>>> Let me figure how to run a vote for a couple of poms..... >>>> >>>> St.Ack >>>> >>>> 1. https://repository.apache.org/content/groups/snapshots/or >>>> g/apache/hbase/thirdparty/ (see hbase-shaded-thirdparty and >>>> hbase-shaded-protobuf) >>>> 2. https://git-wip-us.apache.org/repos/asf/hbase-thirdparty >>>> >>>> >>>> On Tue, Jun 20, 2017 at 11:04 AM, Josh Elser <[email protected]> >>>> wrote: >>>> >>>>> On 6/20/17 1:28 AM, Stack wrote: >>>>> >>>>>> On Thu, Apr 13, 2017 at 4:46 PM, Josh Elser<[email protected]> >>>>>> wrote: >>>>>> >>>>>> ... >>>>>>> >>>>>>> I think pushing this part forward with some code is the next logical >>>>>>> step. >>>>>>> Seems to be consensus about taking our known internal dependencies >>>>>>> and >>>>>>> performing this shade magic. >>>>>>> >>>>>>> >>>>>>> I opened HBASE-18240 "Add hbase-auxillary, a project with hbase >>>>>> utility >>>>>> including an hbase-shaded-thirdparty module with guava, netty, etc." >>>>>> >>>>>> It has a tarball attached that bundles the outline of an >>>>>> hbase-auxillary >>>>>> project (groupId:org.apache.hbase.auxillary). This project is >>>>>> intended to >>>>>> be standalone, in its own repository, publishing its own artifacts >>>>>> under >>>>>> the aegis of this project's PMC. >>>>>> >>>>>> It includes the first instance of an auxillary utility, a module named >>>>>> hbase-thirdparty-shaded (artifactId:hbase-thirdparty-shaded). Herein >>>>>> we'll >>>>>> pull down 3rd party libs and republish at an offset; e.g. >>>>>> com.google.common.* from guava will be at >>>>>> org.apache.hbase.thirdparty.shaded.com.google.common.*. Currently it >>>>>> builds >>>>>> a jar that includes a relocated guava 22.0. >>>>>> >>>>>> I then messed around making hbase-common use it (You have to build the >>>>>> hbase-auxillary into your local repo). I put up a patch on the issue. >>>>>> Mostly its mass find-and-replace w/ some clean up of transitive >>>>>> includes of >>>>>> guava from hadoop-common and some small fixup of methods renamed >>>>>> between >>>>>> guava 12.0 and 22.0. >>>>>> >>>>>> Unless objection, I was going to press on. Sean offered to help set >>>>>> up new >>>>>> repo. We can always undo and delete it if this project fails. >>>>>> >>>>>> When done, the hope is we are on a modern version of guava and our >>>>>> netty >>>>>> and protobuf 3 will be be relocated, 'hidden' from downstream (and >>>>>> won't >>>>>> clash w/ upstream). I hope to also purge the pre-build we have in our >>>>>> modules that do protobuf moving this hackery out and under >>>>>> hbase-thirdparty-shaded. >>>>>> >>>>>> St.Ack >>>>>> >>>>> >>>>> Kudos on the JFDI approach :). I think having something concrete to >>>>> show is the best way to judge success of it. >>>>> >>>>> Will keep an eye on HBASE-18240. >>>>> >>>>> >>>> >>> >> >
