Devs,

While investigating another issue, I came across this OOM error when using
the Console Sink with any source that can be larger than the available
driver memory. In my case, I was using the File source and I had a 14G file
in the monitored dir.

I traced back the issue to a `df.collect` in the Console Sink code.
I created a JIRA for it: https://issues.apache.org/jira/browse/SPARK-21710
and a PR is available: https://github.com/apache/spark/pull/18923

I hope a committer can check it out.

-kr, Gerard.

Reply via email to