Thanks for the reply. -- Jonathan (Bill) Sparks Software Architecture Cray
From: Tom Graves <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>>, Tom Graves <[email protected]<mailto:[email protected]>> Date: Friday, November 15, 2013 8:15 AM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: SPARK + YARN the general case Hey Bill, Currently the Spark on Yarn only supports batch mode where you submit your job via the yarn Client. Note that this will hook the spark UI up to the Yarn ResourceManager web UI. Is there something more you were looking for then just finding the spark web ui for various jobs? There is a pull request (101) to get spark shell working with YARN. Tom On Thursday, November 14, 2013 10:57 AM, Bill Sparks <[email protected]<mailto:[email protected]>> wrote: Sorry for the following question, but I just need a little clarity on expectations of Spark using YARN. Is it possible to use the spark-shell with YARN ? Or is the only way to submit a Spark job to YARN is by write a Java application and submit it via the yarn.Client application. Also is there a way of running the Spark master so that it can communicate with YARN so I can use the web UI for job tracking. Thanks, Bill
