When I wrote about this in another thread, I was experiencing issues with the Spark-YARN interpreter expecting different version of some JARs (specifically netty) than the YARN I wished to integrate. As the interpreters run in stand alone JVMs, it seemed to make sense to me to have their classpaths distinct from the Zeppelin application.
While I agree it would be easier for deployment to have a unified classpath, I don't think it's realistic in the Hadoop universe. There are too many different components and distributions to tie then to one classpath. Even between versions of Hadoop, there is only so much that can be accomplished with profiles in the pom to cover all the permutations. Even with the profiles provided, it was not clear to me which combination (spark, yarn, hadoop) would provide a clean running application. paul On Tue, May 19, 2015 at 1:41 PM, James Carman <[email protected]> wrote: > On Mon, May 18, 2015 at 11:46 PM Sharad Agarwal <[email protected]> wrote: > > > > > Interpreters should work more like a plugin. All interpreters compile and > > runtime dependency should be fully isolated regardless of their maturity > > level. This would keep the core minimal and writing new interpreter > easier. > > > > +1, if you're wanting to support true "plugins" and you're not isolating > their classloaders in some way, then you're asking for trouble. > -- *Paul Curtis* Senior Sales Engineer *O: *+1 203-660-0015 *M:* +1 203-539-9705 <http://mapr.com> Now Available - Free Hadoop On-Demand Training <http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available>
