That's a very old page, try this instead:
http://spark.apache.org/docs/latest/running-on-mesos.html
When you run your Spark job on Mesos, tasks will be started on the slave
nodes as needed, since "fine-grained" mode is the default.
For a job like your example, very few tasks will be needed. Actually only
one would be enough, but the default number of partitions will be used. I
believe 8 is the default for Mesos. For local mode ("local[*]"), it's the
number of cores. You can also set the propoerty "spark.default.parallelism".
HTH,
Dean
Dean Wampler, Ph.D.
Author: Programming Scala, 2nd Edition
<http://shop.oreilly.com/product/0636920033073.do> (O'Reilly)
Typesafe <http://typesafe.com>
@deanwampler <http://twitter.com/deanwampler>
http://polyglotprogramming.com
On Mon, Mar 23, 2015 at 11:46 AM, Anirudha Jadhav <[email protected]> wrote:
> i have a mesos cluster, which i deploy spark to by using instructions on
> http://spark.apache.org/docs/0.7.2/running-on-mesos.html
>
> after that the spark shell starts up fine.
> then i try the following on the shell:
>
> val data = 1 to 10000
>
> val distData = sc.parallelize(data)
>
> distData.filter(_< 10).collect()
>
> open spark web ui at host:4040 and see an active job.
>
> NOW, how do i start workers or spark workers on mesos ? who completes my
> job?
> thanks,
>
> --
> Ani
>
