Thanks. This is an important direction to explore and my apologies for the late reply.
One thing that is really hard about this is that with different layers of abstractions, we often use other libraries that might allocate large amount of memory (e.g. snappy library, Parquet itself), which makes it very difficult to track. That's where I see how most of the OOMs or crashes happen. How do you propose solving those? On Tue, Jun 20, 2017 at 4:15 PM, Jose Soltren <j...@cloudera.com> wrote: > https://issues.apache.org/jira/browse/SPARK-21157 > > Hi - often times, Spark applications are killed for overrunning available > memory by YARN, Mesos, or the OS. In SPARK-21157, I propose a design for > grabbing and reporting "total memory" usage for Spark executors - that is, > memory usage as visible from the OS, including on-heap and off-heap memory > used by Spark and third party libraries. This builds on many ideas from > SPARK-9103. > > I'd really welcome some review and some feedback of this design proposal. > I think this could be a helpful feature for Spark users who are trying to > triage memory usage issues. In the future I'd like to think about reporting > memory usage from third party libraries like Netty, as was originally > proposed in SPARK-9103. > > Cheers, > --José >
