I’m running on Yarn.
> On Aug 13, 2015, at 10:58 PM, Philip Weaver <philip.wea...@gmail.com> wrote:
>
> Are you running on mesos, yarn or standalone? If you're on mesos, are you
> using coarse grain or fine grained mode?
>
> On Thu, Aug 13, 2015 at 10:13 PM, Ara Vartanian <arav...@cs.wisc.edu
> <mailto:arav...@cs.wisc.edu>> wrote:
> I’m observing an unusual situation where my step duration increases as I add
> further executors to my cluster. My algorithm is fully data parallelizable
> into a map phase, followed by a reduce step at the end that amounts to matrix
> addition. So I’ve kicked a cluster of, say, 100 executors with 4 cores per
> executor and before running the algorithm I’ve repartitioned the RDD into 400
> partitions. I can see in the Spark UI that each of the 400 (map) tasks takes
> about 2 seconds. However, the entire step is taking over a minute, and this
> is because the launch times of the tasks as reported in the Spark UI are
> staggered. For example, the first 100 might be launched in the same second,
> then another group 3 seconds later, and so forth (with the durations slowly
> expanding). With a task time of 2 seconds, this “launch lag” is dominating
> the computation time and only gets worse as I add nodes.
>
> Any insight on how I could go about diagnosing this would be greatly
> appreciated.
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> <mailto:user-unsubscr...@spark.apache.org>
> For additional commands, e-mail: user-h...@spark.apache.org
> <mailto:user-h...@spark.apache.org>
>
>
