Stephan Ewen created FLINK-16225:
------------------------------------
Summary: Metaspace Out Of Memory should be handled as Fatal Error
in TaskManager
Key: FLINK-16225
URL: https://issues.apache.org/jira/browse/FLINK-16225
Project: Flink
Issue Type: Improvement
Components: Runtime / Task
Reporter: Stephan Ewen
Fix For: 1.11.0
When an {{OutOfMemory (Metaspace)}} exception happens, there is usually no way
to recover. This is often the result of user code or libraries that have subtle
class loading leaks.
The one way to recover is to kill the TaskManagers and to let the resource
orchestrators (K8s, Yarn, Mesos) restart them. Flink's fault tolerance should
then be able to recover the job.
I would suggest to implement this the following way:
* The user code ClassLoader takes an "OOM Handler", which is called when class
loading causes an OOM exception.
* The handler wraps this into an Exception with a good error message (see
below) and invokes the TaskManager's {{FatalErrorHandler}}.
* The {{FatalErrorHandler}} in turn should attempt to cancel everything and
notify the JM before shutting down. That way, we get decent error reporting and
users can see what is going on.
The error message should describe the following:
* If user sees the error consistently on the first deploy, then the metaspace
is simply too small for their application, and they need to explicitly increase
it in the configuration
* If the user sees occasionally TaskManagers in a session cluster failing with
that exception when deploying new jobs, then some user code or library probably
has a class leak. The TM failure / restart is done in order to forcefully clean
up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
