Shark is not currently supported on yarn. There are 2 ways this could be done that come to mind. One would be to run shark as the application itself that gets started on the application master in the current yarn-standalone mode, the other is with using the yarn-client introduced in the spark-shell pull request. I saw some changes that went into Shark that were to support running it along with the yarn-client pull request (101), but I haven't had time to actually try these yet.
Tom On Friday, November 15, 2013 10:45 AM, Michael (Bach) Bui <[email protected]> wrote: Hi Tom, I have another question on SoY. Seems like the current implementation will not support interactive type of application like Shark, right? Thanks. On Nov 15, 2013, at 8:15 AM, Tom Graves <[email protected]> wrote: Hey Bill, > > >Currently the Spark on Yarn only supports batch mode where you submit your job >via the yarn Client. Note that this will hook the spark UI up to the Yarn >ResourceManager web UI. Is there something more you were looking for then >just finding the spark web ui for various jobs? > > >There is a pull request (101) to get spark shell working with YARN. > > >Tom > > > >On Thursday, November 14, 2013 10:57 AM, Bill Sparks <[email protected]> wrote: > >Sorry for the following question, but I just need a little clarity on >expectations of Spark using YARN. > > >Is it possible to use the spark-shell with YARN ? Or is the only way to submit >a Spark job to YARN is by write a Java application and submit it via the >yarn.Client application. > > >Also is there a way of running the Spark master so that it can communicate >with YARN so I can use the web UI for job tracking. > > >Thanks, > Bill > >
