I’m observing an unusual situation where my step duration increases as I add 
further executors to my cluster. My algorithm is fully data parallelizable into 
a map phase, followed by a reduce step at the end that amounts to matrix 
addition. So I’ve kicked a cluster of, say, 100 executors with 4 cores per 
executor and before running the algorithm I’ve repartitioned the RDD into 400 
partitions. I can see in the Spark UI that each of the 400 (map) tasks takes 
about 2 seconds. However, the entire step is taking over a minute, and this is 
because the launch times of the tasks as reported in the Spark UI are 
staggered. For example, the first 100 might be launched in the same second, 
then another group 3 seconds later, and so forth (with the durations slowly 
expanding). With a task time of 2 seconds, this “launch lag” is dominating the 
computation time and only gets worse as I add nodes.

Any insight on how I could go about diagnosing this would be greatly 
appreciated.



---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to