Would you mind having a look at https://github.com/apache/incubator-zeppelin/pull/928/files to see if I'm missing anything critical.
The idea is the user specifies a directory containing the necessary (to be covered in the setup documentation), and the jars are loaded from there. Also adds some configuration settings (mainly Kyro) when 'spark.mahout' is true. Finally imports the mahout and sets up the sdc from the already declared sc. Based on my testing that works in local and cluster mode. Thanks, tg Trevor Grant Data Scientist https://github.com/rawkintrevo http://stackexchange.com/users/3002022/rawkintrevo http://trevorgrant.org *"Fortunate is he, who is able to know the causes of things." -Virgil* On Wed, Jun 1, 2016 at 12:48 PM, Dmitriy Lyubimov <[email protected]> wrote: > On Wed, Jun 1, 2016 at 10:46 AM, Dmitriy Lyubimov <[email protected]> > wrote: > > > > > > > On Wed, Jun 1, 2016 at 7:47 AM, Trevor Grant <[email protected]> > > wrote: > > > >> > >> Other approaches? > >> > >> For background, Zeppelin starts a Spark Shell and we need to make sure > all > >> of the required Mahout jars get loaded in the class path when spark > >> starts. > >> The question is where do all of these JARs relatively live. > >> > > > > How does zeppelin copes with extra dependencies for other interpreters > > (even spark itself)? I guess we should follow the same practice there. > > > > Release independence of location algorithm largely depends on jar filters > > (again, see filters in the spark binding package). It is possible that > > artifacts required may change but not very likely (i don't think they > ever > > changed since 0.10). so it should be possible to build (mahout) > > release-independent logic to locate, filter and assert the necessary > jars. > > > > PS this may change soon though if/when custom javacpp code is built, we > may probably want to keep all native things as separate release artifacts, > as they are basically treated as optionally available accellerators and may > or may not be properly loaded in all situations. hence they may warrant a > seaprate jar vehicle. > > > > > > >> > >> Thanks for any feedback, > >> tg > >> > >> > >> > >> > >> > >> > >> Trevor Grant > >> Data Scientist > >> https://github.com/rawkintrevo > >> http://stackexchange.com/users/3002022/rawkintrevo > >> http://trevorgrant.org > >> > >> *"Fortunate is he, who is able to know the causes of things." -Virgil* > >> > > > > >
