For coprocessors, our compatibility matrix says that we can break everything, so I think dependency is not the first thing they need to consider when upgrading between major versions?
2017-02-08 11:48 GMT+08:00 Ted Yu <yuzhih...@gmail.com>: > bq. Better to start using the new API's available in Java 8 > > +1 to the above. > If no new Guava construct is introduced and we replace current Guava usage > with Java 8 counterpart(s), in the future we can get rid of Guava > dependency. > > On Tue, Feb 7, 2017 at 6:50 PM, Nick Dimiduk <ndimi...@gmail.com> wrote: > > > For the client: I'm a fan of shaded client modules by default and > > minimizing the exposure of that surface area of 3rd party libs (none, if > > possible). For example, Elastic Search has a similar set of challenges, > the > > solve it by advocating users shade from step 1. It's addressed first > thing > > in the docs for their client libs. We could take it a step further by > > making the shaded client the default client (o.a.hbase:hbase-client) > > artifact and internally consume an hbase-client-unshaded. Turns the whole > > thing on it's head in a way that's better for the naive user. > > > > For MR/Spark/etc connectors: We're probably stuck as it is until > necessary > > classes can be extracted from hbase-server. I haven't looked into this > > lately, so I hesitate to give a prescription. > > > > For coprocessors: They forfeit their right to 3rd party library > dependency > > stability by entering our process space. Maybe in 3.0 or 4.0 we can > rebuild > > on jigsaw or OSGi, but for today I think the best we should do is provide > > relatively stable internal APIs. I also find it unlikely that we'd want > to > > spend loads of cycles optimizing for this usecase. There's other, bigger > > fish, IMHO. > > > > For size/compile time: I think these ultimately matter less than user > > experience. Let's find a solution that sucks less for downstreamers and > > work backward on reducing bloat. > > > > On the point of leaning heavily on Guava: their pace is traditionally too > > fast for us to expose in any public API. Maybe that's changing, in which > > case we could reconsider for 3.0. Better to start using the new API's > > available in Java 8... > > > > Thanks for taking this up, Stack. > > -n > > > > On Tue, Feb 7, 2017 at 12:22 PM Stack <st...@duboce.net> wrote: > > > > > Here's an old thorny issue that won't go away. I'd like to hear what > > folks > > > are thinking these times. > > > > > > My immediate need is that I want to upgrade Guava [1]. I want to move > us > > to > > > guava 21.0, the latest release [2]. We currently depend on guava 12.0. > > > Hadoop's guava -- 11.0 -- is also on our CLASSPATH (three times). We > > could > > > just do it in an hbase-2.0.0, a major version release, but then > > > downstreamers and coprocessors that may have been a little lazy and > that > > > have transitively come to depend on our versions of libs will break > [3]. > > > Then there is the murky area around the running of YARN/MR/Spark jobs > > where > > > the ordering of libs on the CLASSPATH gets interesting where fat-jaring > > or > > > command-line antics can get you over (most) problems if you persevere. > > > > > > Multiply the above by netty, jackson, and a few other favorites. > > > > > > Our proffered solution to the above is the shaded hbase artifact > project; > > > have applications and tasks refer to the shaded hbase client instead. > > > Because we've not done the work to narrow the surface area we expose to > > > downstreamers, most consumers of our API -- certainly in a spark/MR > > context > > > since our MR utility is buried in hbase-server module still -- need > both > > > the shaded hbase client and server on their CLASSPATH (i.e. near all of > > > hbase). > > > > > > Leaving aside for the moment that our shaded client and server need > > > untangling, getting folks up on the shaded artifacts takes effort > > > evangelizing. We also need to be doing work to make sure our shading > > > doesn't leak dependencies, that it works for all deploy scenarios, and > > that > > > this route forward is well doc'd, and so on. > > > > > > I don't see much evidence of our pushing the shaded artifacts route nor > > of > > > their being used. What is the perception of others? > > > > > > I played with adding a new module to host shaded 3rd party libs[4]. The > > > downsides are a couple; would have to internally, refer to the offset > > > version of the lib and we bulk up our tarball by a bunch of megs (Build > > > gets a few seconds longer, not much). Upside is that we can float over > a > > > variety of hadoop/spark versions using whatever guava or netty we want; > > > downstreamers and general users should have an easier time of it too > > > because they'll be less likely to run into library clashes. is this > > project > > > worth finishing? > > > > > > WDYT? > > > St.Ack > > > > > > 1. I wanted to make use of the protobuf to-json tool. It is in the > > > extra-jar, protobuf-util. It requires a guava 16.0. > > > 2. Guava is a quality lib that should be at the core of all our dev but > > we > > > are gun shy around using it because it semver's with gusto at a rate > that > > > is orders of magnitude in advance of the Hadoop/HBase cadence. > > > 3. We are trying to minimize breakage when we go to hbase-2.0.0. > > > 4. HBASE-15749 suggested this but was shutdown because it made no case > > for > > > why we'd want to do it. > > > > > >
