+dev This is a great question for the mailing list. Things have changed a little bit since Vinod and I first wrote the Hadoop port, but my answers here should still be valid.
First, in using Mesos with Hadoop, why does every task in Mesos need to > launch a TaskTracker ? Why not just launch one TaskTracker in each executor? Mesos is task-centric, an executor cannot be launched without a task. In the case of the Hadoop framework, we designed it using a 1:1 executor:task model, so we are in fact launching 1 TaskTracker for each executor. We do not send multiple tasks to a single executor. We ended up having to write a custom Executor for hadoop, rather than relying on the built-in Command Executor. Ideally we would be able to send multiple 'map' and 'reduce' tasks to a single TaskTracker executor, but this level of control is not possible without some serious patching in Hadoop pre-2.0. Second, what do we have to do in the function launchtask(ExecutorDriver* > driver, const TaskInfo& task). Whether do we have to wait for the task > finished before exit launchtask() function or not? if not, how to update > the task status to master? What you do inside launchTask will depend on your application. If you have MapReduce computation framework, your tasks may be individual 'map' and 'reduce' operations. However, if your task is simply to run a command in the shell, you can omit the executor information and Mesos will use a built-in 'Command Executor' which will automatically run your shell command. You do not need to wait until the task completes inside launchTask. You can notify your scheduler at a later point about task completion via the driver call: ExecutorDriver::sendStatusUpdate. Third, I want to use Mesos with Torque(pbs). I want to launch only one > pbs_mom to run all the jobs that assigned to the slave. Can I launch just > one pbs_mom in TorqueExecutor's main() function and use the launchtask() > just to add resources to executor? I'm not familiar enough with Torque to answer this question. You could use launchTask only to add resources to the container running the Executor. Hope this helps! On Thu, Dec 19, 2013 at 6:42 AM, HUO Jing <[email protected]> wrote: > Hi, > Sorry to bother you，I have some questions about Mesos Executor. > First, in using Mesos with Hadoop, why does every task in Mesos need to > launch a TaskTracker ? Why not just launch one TaskTracker in each executor? > Second, what do we have to do in the function launchtask(ExecutorDriver* > driver, const TaskInfo& task). Whether do we have to wait for the task > finished before exit launchtask() function or not? if not, how to update > the task status to master? > Third, I want to use Mesos with Torque(pbs). I want to launch only one > pbs_mom to run all the jobs that assigned to the slave. Can I launch just > one pbs_mom in TorqueExecutor's main() function and use the launchtask() > just to add resources to executor? > > Thank you very much! > > Best wishes! > > Huojing from IHEP in China >
