Github user tgravescs commented on the issue:
https://github.com/apache/spark/pull/21953
We have seen jobs overloading the first disk returned by Yarn.
Unfortunately the details of the job have long expired. Its in general a good
practice to distribute the load anyway.
I remember one of the jobs was python. I think it was the case if you
look in like EvalPythonExec.scala:
// The queue used to buffer input rows so we can drain it to
// combine input with output from Python.
val queue = HybridRowQueue(context.taskMemoryManager(),
new File(Utils.getLocalDir(SparkEnv.get.conf)), child.output.length)
That is always going to hit the disk yarn returns first for every container
on that node.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
