Github user bersprockets commented on the issue:
https://github.com/apache/spark/pull/21899
> Is it possible to include the actual size of the in-memory table so far
in the msg as well?
Possibly. The state of the relation might be messy when I go to query its
size.
>Also, does catching the OOM and throwing our own mess with
HeapDumpOnOutOfMemoryError?
@squito From my tests, it seems the heap dump is taken before the exception
is caught.
<pre>
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid70644.hprof ...
Heap dump file created [842225516 bytes in 2.412 secs]
java.lang.OutOfMemoryError: Not enough memory to build and broadcast the
table to all worker nodes. As a workaround, you can either disable broadcast by
setting spark.sql.autoBroadcastJoinThreshold to -1 or increase the spark driver
memory by setting spark.driver.memory to a higher value
at
org.apache.spark.sql.execution.joins.LongToUnsafeRowMap.grow(HashedRelation.scala:628)
</pre>
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
