Re: Run spark 2.2 on yarn as usual java application

Hi Jörn, thanks for your reply Oozie starts ooze java action as single "long running" MapReduce Mapper. This mapper is responsible for calling main class. Main class belongs to user and this main class starts spark job. yarn-cluster is not an option for me. I have to do something special to

Re: Run spark 2.2 on yarn as usual java application

Maybe you should better run it in yarn cluster mode. Yarn client would start the driver on the oozie server. > On 19. Mar 2018, at 12:58, Serega Sheypak wrote: > > I'm trying to run it as Oozie java action and reduce env dependency. The only > thing I need is Hadoop

Re: Run spark 2.2 on yarn as usual java application

I'm trying to run it as Oozie java action and reduce env dependency. The only thing I need is Hadoop Configuration to talk to hdfs and yarn. Spark submit is a shell thing. Trying to do all from jvm. Oozie java action starts main class which inststiates SparkConf and session. It works well in local

Re: Run spark 2.2 on yarn as usual java application

Hi, What's the deployment process then (if not using spark-submit)? How is the AM deployed? Why would you want to skip spark-submit? Jacek On 19 Mar 2018 00:20, "Serega Sheypak" wrote: > Hi, Is it even possible to run spark on yarn as usual java application? > I've

Run spark 2.2 on yarn as usual java application

Hi, Is it even possible to run spark on yarn as usual java application? I've built jat using maven with spark-yarn dependency and I manually populate SparkConf with all hadoop properties. SparkContext fails to start with exception: 1. Caused by: java.lang.IllegalStateException: Library