We have the job jar in hadoop, which includes all dependencies. The same construct works for Spark if the *job.jar files get included in the classpath and they currently are.
I created a job.jar for spark but Dmitriy wants to remove it. That would be ok because the dependency got move into math-scala but the issue has only moved. I don’t see how we can leave it to the user to configure their environment to include the right version of the right classes. As it stands now I can’t run Mahout without that math-scala job.jar. I realize there are other ways to solve this but don’t know of a better way. If I don’t hear back with some other option my next commit will put a job.jar in math-scala and take it out of spark. Dmitriy would like to remove it from both places but without some other solution I don’t see how this can work—obviously we can call it something like managedlibs.jar but I have at least one dependency that does not get included due to other mahout job jars.
