Tom,
Can you just clarify that when you say "Spark on Yarn only supports
batch mode" that you are not excluding Spark Streaming from working with
Yarn? A quick scan of the Spark Streaming documentation makes no
mention of Yarn, but I thought that this should be possible.
Thanks,
Philip
On 11/15/2013 7:15 AM, Tom Graves wrote:
Hey Bill,
Currently the Spark on Yarn only supports batch mode where you submit
your job via the yarn Client. Note that this will hook the spark UI
up to the Yarn ResourceManager web UI. Is there something more you
were looking for then just finding the spark web ui for various jobs?
There is a pull request (101) to get spark shell working with YARN.
Tom
On Thursday, November 14, 2013 10:57 AM, Bill Sparks
<[email protected]> wrote:
Sorry for the following question, but I just need a little clarity on
expectations of Spark using YARN.
Is it possible to use the spark-shell with YARN ? Or is the only way
to submit a Spark job to YARN is by write a Java application and
submit it via the yarn.Client application.
Also is there a way of running the Spark master so that it can
communicate with YARN so I can use the web UI for job tracking.
Thanks,
Bill
