Github user JoshRosen commented on the issue:
https://github.com/apache/spark/pull/21390
Context for other reviewers: the issue addressed by this patch is actually
a real issue in practice, especially for long-lived Spark clusters; I have seen
this specific problem play a large contributing role to certain production
out-of-disk-space failures.
One thing I'd like to note: as implemented here, this patch only addresses
this problem for Spark's built-in "Standalone" cluster manager. @jiangxb1987,
could you mention that limitation in the PR title and description? My personal
preference is to proceed incrementally by merging this Standalone-only PR and
and deferring support for other cluster managers to future PRs (perhaps from
experts familiar with those other cluster managers).
I'll take a more detailed look tomorrow, but just wanted to provide
motivation for other reviewers who might leave comments before then.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
