Hi Weide, The answer to your first question about local[2] can be found in the "Running the Examples and Shell" section of https://spark.apache.org/docs/latest/
Note that all of the sample programs take a <master> parameter specifying > the cluster URL to connect to. This can be a URL for a distributed > cluster<https://spark.apache.org/docs/latest/scala-programming-guide.html#master-urls>, > or local to run locally with one thread, or local[N] to run locally with > N threads. You should start by using local for testing. For the second, I'm not sure how the thread count affects the number of kafka consumers. My first guess would be that a thread is 1:1 with a kafka consumer, but I'm not super familiar with Spark streaming. Andrew On Sat, May 3, 2014 at 7:00 AM, Weide Zhang <[email protected]> wrote: > in spark kafka example, > > it says > > `./bin/run-example org.apache.spark.streaming.examples.KafkaWordCount > local[2] zoo01,zoo02,zoo03 my-consumer-group topic1,topic2 1` > > can any one tell me what does local[2] represent ? i thought master url > should be sth like spark://hostname:portname . > > also, the thread count is specified as 1 in kafka example, what will > happen thread count goes to more than 1 ? does that mean multiple kafka > consumer will be created on different spark workers ? I'm not sure how does > the code mapped to realtime worker thread allocation ? Is there any > documentation on that ? > > Weide >
