Dmitriy, lib_managed/ is controlled by sbt build - Maven uses ~/.m2/repository directly. Hence, earlier Matei point stands on the need to fix it in sbt build.
I am not sure why would Cloudera solve the issue in the Spark build? :) Cos On Tue, Jul 30, 2013 at 11:13PM, Dmitriy Lyubimov wrote: > FWIW it does get landed in lib_managed... > > i beleive there's a post in CDH group with the same very problem i had , > still unresolved/unadvised by Cloudera. > > > On Tue, Jul 30, 2013 at 10:07 PM, Konstantin Boudnik <[email protected]> wrote: > > > Ok, that makes sense, although I thought examples might be a good part of > > the > > documentation for the beginners' references, etc. But I have problems > > modifying > > the assembly accordingly. Will do it the first thing in the morning. > > > > Thanks for the input. > > Cos > > > > On Tue, Jul 30, 2013 at 09:57PM, Matei Zaharia wrote: > > > Basically the way to think of the assembly is that it should have > > libraries > > > that users' client programs need to run. These are core, repl (needed if > > > they use the shell), and likely bagel and streaming and mllib, though > > > originally we'd opted to leave those out. We are still deciding on that > > -- > > > could go either way. > > > > > > Matei > > > > > > On Jul 30, 2013, at 9:56 PM, Matei Zaharia <[email protected]> > > wrote: > > > > > > > Yeah, that is true. But the assembly shouldn't include the examples > > project at all IMO -- if it does now, we should remove it. > > > > > > > > Matei > > > > > > > > On Jul 30, 2013, at 9:47 PM, Konstantin Boudnik <[email protected]> > > wrote: > > > > > > > >> Matei, > > > >> > > > >> Hbase dependencies aren't actually included into the Maven assembly > > as of this > > > >> moment, because scope of hbase dependency in examples' module is > > "compile"; but > > > >> the assembly is only includes those with "runtime". Hence it is > > automatically > > > >> excluded. > > > >> > > > >> I believe, hbase is needed for examples during the execution time, > > and if so - > > > >> it would have to be fixed in the module. This will lead to need to > > exclude it > > > >> from the assembly, in turn. > > > >> > > > >> And of course... :) > > > >> > > > >> s/putt/pull/ > > > >> > > > >> Cos > > > >> > > > >> On Tue, Jul 30, 2013 at 09:08PM, Matei Zaharia wrote: > > > >>> Yeah, and maybe we will want to change to Maven as the recommended > > tool for > > > >>> assembly building. I want to look into this more for the 0.8 release. > > > >>> > > > >>> Matei > > > >>> > > > >>> On Jul 30, 2013, at 9:04 PM, Konstantin Boudnik <[email protected]> > > wrote: > > > >>> > > > >>>> On Tue, Jul 30, 2013 at 08:44PM, Matei Zaharia wrote: > > > >>>>> Let's at the very least make it configurable, but an even better > > thing will > > > >>>>> be to make sbt assembly not include it. I think the only thing > > that depends > > > >>>>> on HBase is the examples project, but unfortunately SBT puts all > > its JARs in > > > >>>>> the lib_managed folder and just stupidly creates an assembly by > > grouping > > > >>>>> those. The Maven build, for example, should not do that. > > > >>>> > > > >>>> It is very easy to exclude dependencies in Maven assembly, like it > > is done for > > > >>>> Hadoop. Lemme send out a putt request - a good finding indeed, > > Dmitriy, thank > > > >>>> you! > > > >>>> > > > >>>> Cos > > > >>>> > > > >>>>> Matei > > > >>>>> > > > >>>>> On Jul 30, 2013, at 7:40 PM, Dmitriy Lyubimov <[email protected]> > > wrote: > > > >>>>> > > > >>>>>> Hello, > > > >>>>>> > > > >>>>>> after couple of days(!) of trying to understand where i get the > > > >>>>>> "NoSuchMethod" error, i traced it down to the fact that 0.8 now > > includes > > > >>>>>> hbase. > > > >>>>>> > > > >>>>>> While it is assumed that hadoop version is specified, hbase > > version is > > > >>>>>> fixed. This seem to create problem if hbase is used with a > > particular > > > >>>>>> version of CDH hadoop client in the backend. (there's a known > > compatibility > > > >>>>>> bug). > > > >>>>>> > > > >>>>>> wouldn't it make sense in this case to allow to declare hbase > > version as > > > >>>>>> well, perhaps even tie it to the CDH version? > > > >>>>>> > > > >>>>>> At the very least i think it deserves a specific mention in the > > header > > > >>>>>> section to provide opportunity to override, just like hadoop > > version does? > > > >>>>>> > > > >>>>>> Thanks. > > > >>>>>> -D > > > >>>>> > > > >>> > > > > > > > > >
