Here are my answers. But am just getting started with Spark Streaming - so please correct me if am wrong. 1) Yes 2) Receivers will run on executors. Its actually a job thats submitted where # of tasks equals # of receivers. An executor can actually run more than one task at the same time. Hence you could have more number of receivers than executors but its not recommended I think. 3) As said in 2, the executor where receiver task is running can be used for map/reduce tasks. In yarn-cluster mode, the driver program is actually run as application master (lives in the first container thats launched) and this is not an executor - hence its not used for other operations. 4) the driver runs in a separate container. I think the same executor can be used for receiver and the processing task also (this part am not very sure)
On Fri, Jul 11, 2014 at 12:29 AM, Yan Fang <yanfang...@gmail.com> wrote: > Hi all, > > I am working to improve the parallelism of the Spark Streaming > application. But I have problem in understanding how the executors are used > and the application is distributed. > > 1. In YARN, is one executor equal one container? > > 2. I saw the statement that a streaming receiver runs on one work machine ( > *"n**ote that each input DStream creates a single receiver (running on a > worker machine) that receives a single stream of data"*). Does the "work > machine" mean the executor or physical machine? If I have more receivers > than the executors, will it still work? > > 3. Is the executor that holds receiver also used for other operations, > such as map, reduce, or fully occupied by the receiver? Similarly, if I run > in yarn-cluster mode, is the executor running driver program used by other > operations too? > > 4. So if I have a driver program (cluster mode) and streaming receiver, do > I have to have at least 2 executors because the program and streaming > receiver have to be on different executors? > > Thank you. Sorry for having so many questions but I do want to understand > how the Spark Streaming distributes in order to assign reasonable > recourse.*_* Thank you again. > > Best, > > Fang, Yan > yanfang...@gmail.com > +1 (206) 849-4108 >
