On Wed, Jul 31, 2013 at 11:24 AM, Konstantin Boudnik <[email protected]> wrote:
> Dmitriy, > > lib_managed/ is controlled by sbt build - Maven uses ~/.m2/repository > directly. Hence, earlier Matei point stands on the need to fix it in sbt > build. > > I am not sure why would Cloudera solve the issue in the Spark build? :) > I wasn't saying Сloudera should. I just meant to say i saw other people stranded out there on that very issue right now while searching for the cause. This incompatibility bug is not at all obvious to track. Ok i guess it should be ok for sbt build to declare HBASE_VERSION and use it. I also discovered that cdh hbase counterpart does not seem to be published to any repository spark uses (unlike hadoop client jars). So to resolve, i also had to add a direct cloudera resolver. it is probably more than you want to do for spark, but any person using hbase and cdh would probably run into this surprisingly hard to track issue. > > Cos > > On Tue, Jul 30, 2013 at 11:13PM, Dmitriy Lyubimov wrote: > > FWIW it does get landed in lib_managed... > > > > i beleive there's a post in CDH group with the same very problem i had , > > still unresolved/unadvised by Cloudera. > > > > > > On Tue, Jul 30, 2013 at 10:07 PM, Konstantin Boudnik <[email protected]> > wrote: > > > > > Ok, that makes sense, although I thought examples might be a good part > of > > > the > > > documentation for the beginners' references, etc. But I have problems > > > modifying > > > the assembly accordingly. Will do it the first thing in the morning. > > > > > > Thanks for the input. > > > Cos > > > > > > On Tue, Jul 30, 2013 at 09:57PM, Matei Zaharia wrote: > > > > Basically the way to think of the assembly is that it should have > > > libraries > > > > that users' client programs need to run. These are core, repl > (needed if > > > > they use the shell), and likely bagel and streaming and mllib, though > > > > originally we'd opted to leave those out. We are still deciding on > that > > > -- > > > > could go either way. > > > > > > > > Matei > > > > > > > > On Jul 30, 2013, at 9:56 PM, Matei Zaharia <[email protected]> > > > wrote: > > > > > > > > > Yeah, that is true. But the assembly shouldn't include the examples > > > project at all IMO -- if it does now, we should remove it. > > > > > > > > > > Matei > > > > > > > > > > On Jul 30, 2013, at 9:47 PM, Konstantin Boudnik <[email protected]> > > > wrote: > > > > > > > > > >> Matei, > > > > >> > > > > >> Hbase dependencies aren't actually included into the Maven > assembly > > > as of this > > > > >> moment, because scope of hbase dependency in examples' module is > > > "compile"; but > > > > >> the assembly is only includes those with "runtime". Hence it is > > > automatically > > > > >> excluded. > > > > >> > > > > >> I believe, hbase is needed for examples during the execution time, > > > and if so - > > > > >> it would have to be fixed in the module. This will lead to need to > > > exclude it > > > > >> from the assembly, in turn. > > > > >> > > > > >> And of course... :) > > > > >> > > > > >> s/putt/pull/ > > > > >> > > > > >> Cos > > > > >> > > > > >> On Tue, Jul 30, 2013 at 09:08PM, Matei Zaharia wrote: > > > > >>> Yeah, and maybe we will want to change to Maven as the > recommended > > > tool for > > > > >>> assembly building. I want to look into this more for the 0.8 > release. > > > > >>> > > > > >>> Matei > > > > >>> > > > > >>> On Jul 30, 2013, at 9:04 PM, Konstantin Boudnik <[email protected]> > > > wrote: > > > > >>> > > > > >>>> On Tue, Jul 30, 2013 at 08:44PM, Matei Zaharia wrote: > > > > >>>>> Let's at the very least make it configurable, but an even > better > > > thing will > > > > >>>>> be to make sbt assembly not include it. I think the only thing > > > that depends > > > > >>>>> on HBase is the examples project, but unfortunately SBT puts > all > > > its JARs in > > > > >>>>> the lib_managed folder and just stupidly creates an assembly by > > > grouping > > > > >>>>> those. The Maven build, for example, should not do that. > > > > >>>> > > > > >>>> It is very easy to exclude dependencies in Maven assembly, like > it > > > is done for > > > > >>>> Hadoop. Lemme send out a putt request - a good finding indeed, > > > Dmitriy, thank > > > > >>>> you! > > > > >>>> > > > > >>>> Cos > > > > >>>> > > > > >>>>> Matei > > > > >>>>> > > > > >>>>> On Jul 30, 2013, at 7:40 PM, Dmitriy Lyubimov < > [email protected]> > > > wrote: > > > > >>>>> > > > > >>>>>> Hello, > > > > >>>>>> > > > > >>>>>> after couple of days(!) of trying to understand where i get > the > > > > >>>>>> "NoSuchMethod" error, i traced it down to the fact that 0.8 > now > > > includes > > > > >>>>>> hbase. > > > > >>>>>> > > > > >>>>>> While it is assumed that hadoop version is specified, hbase > > > version is > > > > >>>>>> fixed. This seem to create problem if hbase is used with a > > > particular > > > > >>>>>> version of CDH hadoop client in the backend. (there's a known > > > compatibility > > > > >>>>>> bug). > > > > >>>>>> > > > > >>>>>> wouldn't it make sense in this case to allow to declare hbase > > > version as > > > > >>>>>> well, perhaps even tie it to the CDH version? > > > > >>>>>> > > > > >>>>>> At the very least i think it deserves a specific mention in > the > > > header > > > > >>>>>> section to provide opportunity to override, just like hadoop > > > version does? > > > > >>>>>> > > > > >>>>>> Thanks. > > > > >>>>>> -D > > > > >>>>> > > > > >>> > > > > > > > > > > > > >
