I have implemented to run interpreters in yarn cluster and succeed to launches SparkInterpreter with local mode in yarn cluster. BTW, I've tested it with yarn-client but it needs to set SPARK_HOME.
I'm not sure if it was possible before, but I want to do it with embedded spark. Thanks for testing it. On Wed, May 10, 2017 at 2:16 AM, Jeff Zhang <zjf...@gmail.com> wrote: > yarn-client mode doesn't work for embeded spark. But does it work before ? > I think embeded spark should only work with local mode > > > > > Jongyoul Lee <jongy...@gmail.com>于2017年5月9日周二 上午10:02写道： > > > Hi devs, > > > > I need you help to verify some mode of SparkInterpreter. I saw the > message > > below from spark website in yarn mode: > > > > ``` > > To make Spark runtime jars accessible from YARN side, you can specify > > spark.yarn.archive or spark.yarn.jars. For details please refer to Spark > > Properties > > <http://spark.apache.org/docs/latest/running-on-yarn.html# > spark-properties > > >. > > If neither spark.yarn.archive nor spark.yarn.jars is specified, Spark > will > > create a zip file with all jars under $SPARK_HOME/jarsand upload it to > the > > distributed cache. > > ``` > > > > It means if you use internal spark, you cannot run yarn mode in current > > master. Can anyone test it and let me know the result? > > > > Thanks in advance, > > Jongyoul > > > > -- > > 이종열, Jongyoul Lee, 李宗烈 > > http://madeng.net > > > -- 이종열, Jongyoul Lee, 李宗烈 http://madeng.net
