you can check from here http://spark.incubator.apache.org/docs/latest/cluster-overview.html
the driver is more like a per-application task scheduler whereas the master is a resource manager to allocate resources to applications and maintain workers with the command you use, the driver will run on the worker nodes Best, -- Nan Zhu On Tuesday, February 18, 2014 at 10:32 PM, David Thomas wrote: > I was using that exact command line (./bin/spark-class > org.apache.spark.deploy.Client launch) to launch the app within the cluster. > Now I was running this from the master node, but the Worker column shows just > one worker in the cluster. So what is the significance of that column? > Shouldn't the driver be running on the master node? > > > On Tue, Feb 18, 2014 at 8:31 PM, Nan Zhu <zhunanmcg...@gmail.com > (mailto:zhunanmcg...@gmail.com)> wrote: > > the driver is running on the machine where you run command like > > ./spark-shell > > > > but in 0.9, you can run in-cluster driver > > http://spark.incubator.apache.org/docs/latest/spark-standalone.html#launching-applications-inside-the-cluster > > > > > > Best, > > > > -- > > Nan Zhu > > > > > > On Tuesday, February 18, 2014 at 10:06 PM, David Thomas wrote: > > > > > On the spark web UI, there are two tables - Running Applications and > > > Running Drivers. What is the difference between these two? > > > > > > When I submit an application to run on the cluster, I can see the driver > > > table getting populated, but why is only one worker listed under Worker > > > column? What is the role of this single worker? Shouldn't the driver > > > program be running on the master node? > > >
