Yes that is correct. It has a static set of nodes currently. We want to make that more dynamic in the future also.
Tom On Friday, November 15, 2013 2:16 PM, Michael (Bach) Bui <[email protected]> wrote: Tom, more on Shark type of applications on Yarn. In the current implementation, during the duration of a SparkContext execution, Yarn will give an unchanged set of nodes to the SparkContext, is that right? If that is the case, IMO, it may not be the best architecture for Shark, because users may load data from nodes that are not in the given set of nodes. Am I right? On Nov 15, 2013, at 12:51 PM, Tom Graves <[email protected]> wrote: Shark is not currently supported on yarn. There are 2 ways this could be done that come to mind. One would be to run shark as the application itself that gets started on the application master in the current yarn-standalone mode, the other is with using the yarn-client introduced in the spark-shell pull request. I saw some changes that went into Shark that were to support running it along with the yarn-client pull request (101), but I haven't had time to actually try these yet. > > >Tom > > > >On Friday, November 15, 2013 10:45 AM, Michael (Bach) Bui <[email protected]> >wrote: > >Hi Tom, > > >I have another question on SoY. Seems like the current implementation will not >support interactive type of application like Shark, right? >Thanks. > > > >On Nov 15, 2013, at 8:15 AM, Tom Graves <[email protected]> wrote: > >Hey Bill, >> >> >>Currently the Spark on Yarn only supports batch mode where you submit your >>job via the yarn Client. Note that this will hook the spark UI up to the >>Yarn ResourceManager web UI. Is there something more you were looking for >>then just finding the spark web ui for various jobs? >> >> >>There is a pull request (101) to get spark shell working with YARN. >> >> >>Tom >> >> >> >>On Thursday, November 14, 2013 10:57 AM, Bill Sparks <[email protected]> wrote: >> >>Sorry for the following question, but I just need a little clarity on >>expectations of Spark using YARN. >> >> >>Is it possible to use the spark-shell with YARN ? Or is the only way to >>submit a Spark job to YARN is by write a Java application and submit it via >>the yarn.Client application. >> >> >>Also is there a way of running the Spark master so that it can communicate >>with YARN so I can use the web UI for job tracking. >> >> >>Thanks, >> Bill >> >> > > >
