Yes, and spark mesos scheduler already uses the same executor id per slave.
Tim On Sat, Jan 10, 2015 at 7:53 AM, Jongyoul Lee <[email protected]> wrote: > Hi Tim, > > I've read those emails from you, and I've found right answers on my > questions. Sorry for asking same questions again. If two tasks have same > executorId, those tasks can be running simultaneously, right? It helps > improve spark codes. > > Thx, > Jongyoul Lee. > > On Monday, January 5, 2015, Jongyoul Lee <[email protected]> wrote: > >> Hi Tim and dev, >> >> In case that spark on mesos is running with coarse grain mode, I've found >> that each task's executor has a different id. I don't think all tasks >> shares the same executor. If two different taskInfo, which have same >> executorInfo - which have same executorId -, is running under a same slave, >> Are those containerId same? And in generally, Is it possible running two >> tasks on the same executor with same containerId simultaneously? >> >> Thanks in advance, >> JL >> >> On Fri, Jan 2, 2015 at 5:39 PM, Jongyoul Lee <[email protected] >> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >> >>> Hi Tim, >>> >>> I'm using Fine grain mode? Is it also intended behaviour in that mode? >>> >>> Jongyoul >>> >>> On Fri, Jan 2, 2015 at 5:33 PM, Timothy Chen <[email protected] >>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>> >>>> Hi Jongyul, >>>> >>>> Are you using Coarse grain or Fine grain mode? >>>> >>>> In coarse grain mode all tasks are sharing the same spark and mesos >>>> executor, so the logs are all forwarded to the same stderr/stdout >>>> files. >>>> >>>> Tim >>>> >>>> On Fri, Jan 2, 2015 at 12:25 AM, Jongyoul Lee <[email protected] >>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>>> > Hi Tim, >>>> > >>>> > I've found a case that two different task runs under same containerId >>>> > simultaneously when I've run a heavy - that means that job use all >>>> > resources - spark job. Spark mesos scheduler creates ExecutorInfo with >>>> > slaveId when it launches task. I saw two different tasks write logs as >>>> same >>>> > file (in same directory including containerId) Is It possible? Sorry >>>> for an >>>> > ambiguous question. >>>> > >>>> > JL >>>> > >>>> > On Fri, Jan 2, 2015 at 5:05 PM, Timothy Chen <[email protected] >>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>>> > >>>> >> Hi Jongyoul, >>>> >> >>>> >> Strictly speaking, Mesos containerizer creates a container, and >>>> >> launches the executor inside of that container which launches the >>>> >> task. >>>> >> >>>> >> The code path you're looking at is in the slave side (slave.cpp), >>>> >> which calls containerizer->launch when executor is not found. >>>> >> >>>> >> Either way your observation is correct, that launchExecutor in the >>>> >> slave is only called when a executor is not found in the framework >>>> >> object internally in the slave object. >>>> >> >>>> >> Tim >>>> >> >>>> >> >>>> >> >>>> >> On Fri, Jan 2, 2015 at 12:00 AM, Jongyoul Lee <[email protected] >>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>>> >> > Thanks, Nishant and Tim, >>>> >> > >>>> >> > I'm using mesos_containerizer for running spark tasks. In case of >>>> >> > mesos_containerizer - or anyone else -, Could you please tell me >>>> when >>>> >> > executor creates a new container or give me more hint to find that? >>>> I've >>>> >> > found code that framework->launchExecutor is called when >>>> >> > framework->getExecutor returns null. >>>> >> > >>>> >> > JL >>>> >> > >>>> >> > On Fri, Jan 2, 2015 at 4:17 AM, Timothy Chen <[email protected] >>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >>>> >> > >>>> >> >> Hi Jongyoul, >>>> >> >> >>>> >> >> It depends on what you refer to as a "container". >>>> >> >> >>>> >> >> In Mesos actually there are several concepts in play, >>>> >> >> tasks that is either a custom executor or a command, executors that >>>> >> >> are responsible for launching tasks, and containerizers that are >>>> >> >> responsible for creating the containers to launch executors and/or >>>> >> >> tasks. >>>> >> >> >>>> >> >> So if you are referring to an actual container that in mesos is >>>> either >>>> >> >> cgroups or docker, then depending on the implementation of the >>>> >> >> particular containerizer (Mesos or Docker or External, etc), a >>>> >> >> container can have multiple tasks or executors although currently >>>> all >>>> >> >> of them launches one container per executor. Executor itself can >>>> >> >> potentially launch multiple tasks that are running simultaneously >>>> as >>>> >> >> long as the executor is registered and referred to the same id by >>>> >> >> multiple tasks. >>>> >> >> >>>> >> >> In the slave object itself, it only tracks frameworks and >>>> executors, >>>> >> >> and when it doens't find it registered in memory it will ask the >>>> >> >> containerizer to launch it. >>>> >> >> >>>> >> >> Tim >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> >> On Thu, Jan 1, 2015 at 11:01 AM, Nishant Suneja < >>>> >> [email protected] >>>> <javascript:_e(%7B%7D,'cvml','[email protected]');>> >>>> >> >> wrote: >>>> >> >> > Hi Jongyoul >>>> >> >> > >>>> >> >> > Take a look at the slave/state.hpp file, and go through the >>>> hierarchy >>>> >> of >>>> >> >> > the state information starting from slave to task. >>>> >> >> > You will see that an executor state has a map of run states, >>>> which is >>>> >> >> > indexed by containerId. >>>> >> >> > It also has "latest" field to denote the current containerId >>>> which its >>>> >> >> > using. >>>> >> >> > >>>> >> >> > So, to answer your question, at any point of time, an executor >>>> should >>>> >> >> have >>>> >> >> > just one active container (which means one process on the >>>> system), >>>> >> >> > and that process's working directory would be identified by the >>>> >> "latest" >>>> >> >> > containerId field in the ExecutorState. >>>> >> >> > But, it can have multiple directories from the previous runs, >>>> each >>>> >> >> > identified by a unique containerId. >>>> >> >> > These directories would eventually be garbage collected. >>>> >> >> > >>>> >> >> > Nishant >>>> >> >> > >>>> >> >> > >>>> >> >> > >>>> >> >> > >>>> >> >> > >>>> >> >> > On Tue, Dec 30, 2014 at 6:12 PM, Jongyoul Lee < >>>> [email protected] <javascript:_e(%7B%7D,'cvml','[email protected]');>> >>>> >> >> wrote: >>>> >> >> > >>>> >> >> >> Hi, dev >>>> >> >> >> >>>> >> >> >> I'm curious about the relationship between executor and >>>> container. >>>> >> Can >>>> >> >> any >>>> >> >> >> executor have multiple container? or have just one? In codes of >>>> >> >> slave.cpp, >>>> >> >> >> launchExecutor only can create container and is called only >>>> when the >>>> >> >> return >>>> >> >> >> value of getExecutor is null. Thus I think each executor can >>>> have one >>>> >> >> >> container only. Is it correct? If I'm wrong, please tell me >>>> correct >>>> >> >> >> procedure. >>>> >> >> >> >>>> >> >> >> Thanks in advance, >>>> >> >> >> JL >>>> >> >> >> >>>> >> >> >> -- >>>> >> >> >> 이종열, Jongyoul Lee, 李宗烈 >>>> >> >> >> http://madeng.net >>>> >> >> >> >>>> >> >> >>>> >> > >>>> >> > >>>> >> > >>>> >> > -- >>>> >> > 이종열, Jongyoul Lee, 李宗烈 >>>> >> > http://madeng.net >>>> >> >>>> > >>>> > >>>> > >>>> > -- >>>> > 이종열, Jongyoul Lee, 李宗烈 >>>> > http://madeng.net >>>> >>> >>> >>> >>> -- >>> 이종열, Jongyoul Lee, 李宗烈 >>> http://madeng.net >>> >> >> >> >> -- >> 이종열, Jongyoul Lee, 李宗烈 >> http://madeng.net >> > > > -- > 이종열, Jongyoul Lee, 李宗烈 > http://madeng.net
