Dear all, We have a spark 0.8.1 cluster on mesos 0.15. Some of my colleagues are familiar with python, but some of features are developed under java. I am looking for a way to integrate java and python on spark.
I notice that the initialization of pyspark does not include a field to distribute jar files to slaves. After exploring the source code and do some hacking, I could control the java sparkcontext object through py4j, but the jar files are not delivered to slaves. Moreover, it seems that the spark lauch the process through the spark home on pyspark but through the spark.executor.uri on scala. Is there a recommended way to develop spark application with both java/scala and python? Should I suggest my team to unify the language? Thanks!