When did we retire 0.98 (grin) Seriously, let's not have Hadoop version specific builds beyond 0.98. Every other week or so we find and fix breakage in the Hadoop 1 build because nobody develops with Hadoop 1 or runs it in production. The combinatorics will be problematic starting with only two branches based on this experience.
I don't see how coprocessors can be isolated from our dependencies unless we implement classloader/environment support for that, which would be a lot of work and yet not isolate them from other changes they'll face due to proximity to other low level details. On Wednesday, March 11, 2015, Sean Busbey <bus...@cloudera.com> wrote: > ugh, yes please let's not return to the days of hadoop-version-specific > HBase builds. > > I'm just spinning up on isolating third-party dependencies over in Hadoop. > I'd be happy to apply whatever works there within HBase. This only really > helps us for HBase 2.y though, right? > > Without sidetracking this thread too much, before I file a jira for > isolating third-party deps what scope are we looking for? hbase-client and > the MR tooling only? i.e I'd prefer to state that folks making use of > LimitedPrivate features like coprocessors don't get isolation from our > dependencies. > > > On Wed, Mar 11, 2015 at 5:44 PM, Nick Dimiduk <ndimi...@gmail.com > <javascript:;>> wrote: > > > The advantage of shading/JarJaring our dependencies is we *don't* get > stuck > > with hbase-A.B-hadoop-Z.Y. Those releases are a hassle for users, and a > > hassle for release managers. The disadvantage being runtime bloat in the > > form of an excessive number of classes to load. > > > > I think there's a middle ground where we shade only those with a bad > > track-record. Enis has a good handle on this list. > > > > On Wed, Mar 11, 2015 at 3:38 PM, Enis Söztutar <enis....@gmail.com > <javascript:;>> wrote: > > > > > > > > > > > > > > Can we do something even more generic like shade [1] our dependencies > > for > > > > some of the common sources of dependency pain like guava, jackson, > and > > > > netty? This way we decouple hbase's pain from hadoop's and allow > > clients > > > > to choose their own versions of libs to use. We'd likely still cause > > > some > > > > inconvenience for 1.0 ->1.1 but this should eliminate this problem > from > > > > biting us again if we are on hadopo 3.x->3.x+1 and hbase 2.y and > hbase > > > > 2.y+1. > > > > > > > > > > I think shading some of the dependencies is a good idea. Obvious stuff > > are > > > protobuf, netty, guava, jackson, etc. > > > We should also convince Hadoop to do it as well. We should decouple our > > > guava and > > > protobuf version from that of Hadoop's. I've read that elastic search > > uses > > > maven shade > > > plugin, we can look at what they have done. > > > > > > > > > > > > > > Jon. > > > > > > > > > > > > [1] http://maven.apache.org/plugins/maven-shade-plugin/ > > > > > > > > On Wed, Mar 11, 2015 at 2:04 PM, Enis Söztutar <e...@apache.org > <javascript:;>> > > wrote: > > > > > > > > > Over at https://issues.apache.org/jira/browse/HBASE-13149, there > is > > > some > > > > > discussion about changing the jackson version from 1.8 to 1.9 that > is > > > > worth > > > > > bringing here because of the wider audience. The discussion is > about > > > > > jackson version specific in this issue, but we should also consider > > > > future > > > > > changes to libs. > > > > > > > > > > The question is, whether we can change the dep version of jackson > > from > > > > 1.8 > > > > > to 1.9 in HBase-1.1. According to our guidelines, we said that we > > will > > > > not > > > > > do breaking changes to our deps in minor versions. From the > analysis > > of > > > > > jackson compat attached at jira, it seems that jackson have some > > > changes > > > > > breaking BC. > > > > > > > > > > So, from a purist perspective, if we follow our guidelines > literally, > > > we > > > > > should not make this change. However, I think we should make that > > > change > > > > in > > > > > 1.1 (and make similar changes in the future as well), because: > > > > > > > > > > 1. Hadoop upgraded its version of jackson in 2.5, and HBase MR jobs > > > fail > > > > > flat out without corresponding change in HBase (with Hadoop-2.5+). > We > > > do > > > > > not have any control over Hadoop or similar core dependencies. We > > > cannot > > > > be > > > > > realistically expected to guarantee more than what they do. I would > > > > rather > > > > > do the change in 1.1 and not inconvenience the users, than not do > the > > > > > change, and have to explain how to replace the jars in the docs. > This > > > > > argument should be extended beyond the jackson dependency. > > > > > > > > > > 2. HBase is not at the maturity level for following the dependency > > > compat > > > > > guide literally without reducing the dep footprint, better > isolation > > of > > > > MR > > > > > / Mini cluster out of hbase-server and possible participation from > > > > hadoop. > > > > > We cannot easily bump up the major version for these as well, since > > > major > > > > > version implies other things. > > > > > > > > > > So, my proposal is: > > > > > - Commit HBASE-13149 to master and 1.1 > > > > > - Either change the dependency compat story for minor versions to > > > false, > > > > > or add a footnote saying that there may be exceptions because of > the > > > > > reasons listed above. > > > > > > > > > > Let me know what you guys think. > > > > > Enis > > > > > > > > > > > > > > > > > > > > > -- > > > > // Jonathan Hsieh (shay) > > > > // HBase Tech Lead, Software Engineer, Cloudera > > > > // j...@cloudera.com <javascript:;> // @jmhsieh > > > > > > > > > > > > > -- > Sean > -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
