Hey Phillip, I haven't actually tried spark streaming on YARN at this point so I can't say for sure, but as you say just from what I've read I don't see anything that would prevent it from working.
Since I don't know of anyone that has tried it I wouldn't be surprised it atleast something small needs to be fixed to support it though. Tom On Friday, November 15, 2013 9:40 AM, Philip Ogren <[email protected]> wrote: Tom, Can you just clarify that when you say "Spark on Yarn only supports batch mode" that you are not excluding Spark Streaming from working with Yarn? A quick scan of the Spark Streaming documentation makes no mention of Yarn, but I thought that this should be possible. Thanks, Philip On 11/15/2013 7:15 AM, Tom Graves wrote: Hey Bill, > > >Currently the Spark on Yarn only supports batch mode where you submit your job >via the yarn Client. Note that this will hook the spark UI up to the Yarn >ResourceManager web UI. Is there something more you were looking for then >just finding the spark web ui for various jobs? > > >There is a pull request (101) to get spark shell working with YARN. > > >Tom > > > >On Thursday, November 14, 2013 10:57 AM, Bill Sparks <[email protected]> wrote: > >Sorry for the following question, but I just need a little clarity on >expectations of Spark using YARN. > > >Is it possible to use the spark-shell with YARN ? Or is the only way to submit >a Spark job to YARN is by write a Java application and submit it via the >yarn.Client application. > > >Also is there a way of running the Spark master so that it can communicate >with YARN so I can use the web UI for job tracking. > > >Thanks, > Bill > >
