jin xing created SPARK-24379:
--------------------------------
Summary: BroadcastExchangeExec should catch SparkOutOfMemory and
re-throw SparkFatalException, which wraps SparkOutOfMemory inside.
Key: SPARK-24379
URL: https://issues.apache.org/jira/browse/SPARK-24379
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.3.0
Reporter: jin xing
After SPARK-22827, Spark won't fails the entire executor but only fails the
task suffering SparkOutOfMemoryError. In current BroadcastExchangeExec, it's
try-catch OutOfMemoryError. Think about below scenario:
# SparkOutOfMemoryError(subclass of OutOfMemoryError) is thrown in
scala.concurrent.Future;
# SparkOutOfMemoryError is caught and a OutOfMemoryError is wrapped in
SparkFatalException and re-thrown;
# ThreadUtils.awaitResult catches SparkFatalException and a OutOfMemoryError
is thrown;
# The OutOfMemoryError will go to
SparkUncaughtExceptionHandler.uncaughtException and Executor fails.
So it make more sense to catch SparkOutOfMemory and re-throw
SparkFatalException, which wraps SparkOutOfMemory inside.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
