And I have just submitted https://github.com/mesos/spark/pull/763
that depends on yours for mllib support. Cos On Wed, Jul 31, 2013 at 02:27PM, Shivaram Venkataraman wrote: > Actually I sent a pull request for this yesterday > https://github.com/mesos/spark/pull/753 > > Shivaram > > > On Wed, Jul 31, 2013 at 2:22 PM, Konstantin Boudnik <[email protected]> wrote: > > > Interestingly, mllib isn't even getting built by maven... I will address > > it as > > a separate patch then. > > > > On Tue, Jul 30, 2013 at 09:57PM, Matei Zaharia wrote: > > > Basically the way to think of the assembly is that it should have > > libraries > > > that users' client programs need to run. These are core, repl (needed if > > > they use the shell), and likely bagel and streaming and mllib, though > > > originally we'd opted to leave those out. We are still deciding on that > > -- > > > could go either way. > > > > > > Matei > > > > > > On Jul 30, 2013, at 9:56 PM, Matei Zaharia <[email protected]> > > wrote: > > > > > > > Yeah, that is true. But the assembly shouldn't include the examples > > project at all IMO -- if it does now, we should remove it. > > > > > > > > Matei > > > > > > > > On Jul 30, 2013, at 9:47 PM, Konstantin Boudnik <[email protected]> > > wrote: > > > > > > > >> Matei, > > > >> > > > >> Hbase dependencies aren't actually included into the Maven assembly > > as of this > > > >> moment, because scope of hbase dependency in examples' module is > > "compile"; but > > > >> the assembly is only includes those with "runtime". Hence it is > > automatically > > > >> excluded. > > > >> > > > >> I believe, hbase is needed for examples during the execution time, > > and if so - > > > >> it would have to be fixed in the module. This will lead to need to > > exclude it > > > >> from the assembly, in turn. > > > >> > > > >> And of course... :) > > > >> > > > >> s/putt/pull/ > > > >> > > > >> Cos > > > >> > > > >> On Tue, Jul 30, 2013 at 09:08PM, Matei Zaharia wrote: > > > >>> Yeah, and maybe we will want to change to Maven as the recommended > > tool for > > > >>> assembly building. I want to look into this more for the 0.8 release. > > > >>> > > > >>> Matei > > > >>> > > > >>> On Jul 30, 2013, at 9:04 PM, Konstantin Boudnik <[email protected]> > > wrote: > > > >>> > > > >>>> On Tue, Jul 30, 2013 at 08:44PM, Matei Zaharia wrote: > > > >>>>> Let's at the very least make it configurable, but an even better > > thing will > > > >>>>> be to make sbt assembly not include it. I think the only thing > > that depends > > > >>>>> on HBase is the examples project, but unfortunately SBT puts all > > its JARs in > > > >>>>> the lib_managed folder and just stupidly creates an assembly by > > grouping > > > >>>>> those. The Maven build, for example, should not do that. > > > >>>> > > > >>>> It is very easy to exclude dependencies in Maven assembly, like it > > is done for > > > >>>> Hadoop. Lemme send out a putt request - a good finding indeed, > > Dmitriy, thank > > > >>>> you! > > > >>>> > > > >>>> Cos > > > >>>> > > > >>>>> Matei > > > >>>>> > > > >>>>> On Jul 30, 2013, at 7:40 PM, Dmitriy Lyubimov <[email protected]> > > wrote: > > > >>>>> > > > >>>>>> Hello, > > > >>>>>> > > > >>>>>> after couple of days(!) of trying to understand where i get the > > > >>>>>> "NoSuchMethod" error, i traced it down to the fact that 0.8 now > > includes > > > >>>>>> hbase. > > > >>>>>> > > > >>>>>> While it is assumed that hadoop version is specified, hbase > > version is > > > >>>>>> fixed. This seem to create problem if hbase is used with a > > particular > > > >>>>>> version of CDH hadoop client in the backend. (there's a known > > compatibility > > > >>>>>> bug). > > > >>>>>> > > > >>>>>> wouldn't it make sense in this case to allow to declare hbase > > version as > > > >>>>>> well, perhaps even tie it to the CDH version? > > > >>>>>> > > > >>>>>> At the very least i think it deserves a specific mention in the > > header > > > >>>>>> section to provide opportunity to override, just like hadoop > > version does? > > > >>>>>> > > > >>>>>> Thanks. > > > >>>>>> -D > > > >>>>> > > > >>> > > > > > > > > >
