Can we use kryo v 2.21 (instead of 2.24) and fasterxml 2.4.4 instead of 2.7.2 (this is what worries me)
I am also working on fishing the mahout jars directly out of MAHOUT_HOME=..../ ... instead of including in the pom, so let's not get to carried away with pruning dependencies. At this point its more of a, "can anyone one think of a specific reason this shouldn't work". Mahout com.esotericsoftware.kryo:kryo:jar:2.24:compile com.fasterxml.jackson.core:jackson-core:jar:2.7.2:compile Spark (1.6) com.esotericsoftware.kryo:kryo:jar:2.21:compile com.fasterxml.jackson.core:jackson-core:jar:2.4.4:compile Trevor Grant Data Scientist https://github.com/rawkintrevo http://stackexchange.com/users/3002022/rawkintrevo http://trevorgrant.org *"Fortunate is he, who is able to know the causes of things." -Virgil* On Tue, May 31, 2016 at 7:55 PM, Andrew Palumbo <[email protected]> wrote: > which dependencies need to be removed? > > I saw the Kryo version on the PR was conflicting also, that may be a flink > thing. I think the version at one point was being enforced in the flink > module at least. > > ________________________________________ > From: Trevor Grant <[email protected]> > Sent: Tuesday, May 31, 2016 8:41:15 PM > To: [email protected] > Subject: Re: Zeppelin Integration PR > > As a follow up to this- it would be nice to remove the dependencies from > the pom.xml... > > All we REALLY need to do is make sure we can get to the required jars and > load them. By including them in the pom we are ensuring they are > available, but there is surely some other way to get ahold of them. Since > we have assumed that Mahout is installed on the system and MAHOUT_HOME=... > we can probably leverage that... > > > > Trevor Grant > Data Scientist > https://github.com/rawkintrevo > http://stackexchange.com/users/3002022/rawkintrevo > http://trevorgrant.org > > *"Fortunate is he, who is able to know the causes of things." -Virgil* > > > On Tue, May 31, 2016 at 7:31 PM, Trevor Grant <[email protected]> > wrote: > > > For what it is worth, simply removing the dependencies from pom.xml > breaks > > the Mahout interpreter. > > > > Upon a little further testing in cluster mode, so long as the > dependencies > > are included in pom.xml, the appropriate Mahout jars are shipped off to > the > > cluster and everything works swimmingly (in Zeppelin there is a local > Spark > > Interpretter internal to Zeppelin and then the 'real' cluster that > > everything gets shipped off to. Sometimes you can make things work in > local > > mode that won't work in cluster mode) > > > > The moral of this story is that the patch DOES in fact work in local and > > cluster mode, so we just need to work out the dependencies and the > > licensing (and a couple of fail safes to make sure the user is running > > Spark version > 1.5.2) and we should be good to go. > > > > > > Trevor Grant > > Data Scientist > > https://github.com/rawkintrevo > > http://stackexchange.com/users/3002022/rawkintrevo > > http://trevorgrant.org > > > > *"Fortunate is he, who is able to know the causes of things." -Virgil* > > > > > > On Tue, May 31, 2016 at 4:22 PM, Trevor Grant <[email protected]> > > wrote: > > > >> Hey folks, > >> > >> looks like we're making progress on the Mahout-Zeppelin integration. > >> > >> Any who are interested check out: > >> https://github.com/apache/incubator-zeppelin/pull/928 > >> > >> Regarding Moon's last comments: > >> Does anyone know off hand if anything will break if we roll back the > >> conflicting packages to the Spark 1.6 version? > >> > >> Also regarding the pom.xml and: > >> "Packaging > >> If mahout requires to be loaded in spark executor's classpath, then > >> adding mahout dependency in pom.xml will not be enough to work with > Spark > >> cluster. Could you clarify if mahout need to be loaded in spark > executor?" > >> > >> All we need to do is load the jars appropriate Mahout jars, I'm not > >> familiar enough with the Spark Interpreter or Spark or Java to know > exactly > >> what would happen, any thoughts on this? > >> > >> Tonight I might just try removing mahout dependencies from pom.xml and > >> seeing what happens? that would solve all of these problems I think. As > >> long as user has 'mvn install'ed Mahout, should be gtg? > >> > >> Trevor Grant > >> Data Scientist > >> https://github.com/rawkintrevo > >> http://stackexchange.com/users/3002022/rawkintrevo > >> http://trevorgrant.org > >> > >> *"Fortunate is he, who is able to know the causes of things." -Virgil* > >> > >> > > >
