Thanks for your response! Yes the executor (launchTask) only gets one task that it executes synchronously and finishes. Since launchTask is a callback, my intuition is the scheduler should launch these tasks in parallel (even within a single machine) after calculating the resources required. I can create a new thread in launchTask() callback and return immediately but that will cause a lost slave since the scheduler assumes it is finished but there is a zombie thread still around. Hence, I am not completely sure creating new threads will solve this issue.
I am using the C++ framework. Is there an example on how this is accomplished in current frameworks? I looked at Spark and it does not seem to be doing anything special for its callbacks to ensure that multiple tasks on a single machine execute in parallel. Thanks, Asim On Mon, Jun 30, 2014 at 4:48 PM, Sharma Podila <[email protected]> wrote: > A likely scenario is that your executor is running the task synchronously > inside the callback to launchTask(). If you make it instead run the task > asynchronously (e.g., in a separate thread), that should resolve it. > > > On Mon, Jun 30, 2014 at 12:48 PM, Asim <[email protected]> wrote: > >> Hi, >> >> I want to launch multiple tasks on multiple machines (t >> m) that can >> run simultaneously. Currently, I find that every machine processes the >> tasks in a serial fashion one after another. >> >> I have written a framework with a scheduler and a executor. The scheduler >> launches a task list on a bunch of machines (that show up as offers). When >> I send a task list to run with driver->launchTasks(offers[i].id(), >> tasks[i]) I find that every machine picks up one task at a time (and then >> goes to the next). This happens even though the offer can accommodate more >> than one task from this task list easily. >> >> Is there something that I am missing? >> >> Thanks, >> Asim >> >> >
