I think I'd like to volunteer to implement this feature. My the perspective is: we solve 2 immediate problems and at the end have a maturing enough interpreter API to be able so add pyspark support.
Immediate problem we solve are: - A multiple interpreters running right now mix stdout/err - in case of JVMs there also is a Classloader collision problem, which does not allow SparkSQL to work with spark 1.2 Suggested solution: To separate each interpreter to a it's own process. This means bringing to the codebase things like: - API for managing the runtime state of that process - then IPC implementation itself (thrift?) - basic ClassLoading for JVM based interpreters Please, let me know if there is something I have missed here! -- Kind regards, Alexander > On 13 Jan 2015, at 20:26, moon soo Lee <[email protected]> wrote: > > Hi guys, > > I'm bringing an issue https://github.com/NFLabs/zeppelin/issues/278 to this > mailing list for discussion. > > Zeppelin creates interpreter instance with each separate classloader to > avoid interfere(dependency conflictions, singletons, static members) with > other interpreter instance. It was working well until now but i can see > some limitations. > > a) When multiple interpreter instances are running concurrently, they can > not avoid interfere of their stdin/stdout/stderr. > b) When interpreter's one dependency is designed(== hardcoded) to use > Application classloader, it won't work within Zeppelin because Zeppelin > loads interpreter's dependency jars in it's threadcontext classloader, not > Application classloader. > > Run interpreter in separate process is the solution i can think. > In detail, because of interpreter is abstracted by it's public methods, > everything will be simply done if we can call those method remotely by some > sort of RPC mechanism. > > Therefore > > a) Main entry point and run script to run interpreter in separate process > b) RPC mechanism between Zeppelin and separate interpreter process > c) Option to enabling/disabling this capability. > > are major tasks i'm thinking. > > What do you guys think? Please share if there're some idea. > > Best, > moon
