GitHub user NicoK opened a pull request:
https://github.com/apache/flink/pull/4416
[FLINK-7279][minicluster] fix a deadlock between TM and cluster shutdown
## What is the purpose of the change
The `MiniCluster` can deadlock if the fatal error handler is called while
the `MiniCluster` shuts down. The reason is that the shut down happens under a
lock which is required by the fatal error handler as well. If now the
`MiniCluster` tries to shut down the underlying RPC service which waits for all
actors to terminate, it will never complete because one actor is still waiting
for the lock.
## Brief change log
- guard both shutdown methods by a new `ReentrantLock` and ignore the TM
shutdown in the `TerminatingFatalErrorHandler` if the cluster is already shut
down.
## Verifying this change
This change is already covered by existing tests, such as
`MiniClusterITCase` which was instable because of this bug (also see
[FLINK-7115]).
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (no)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (JavaDocs)
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/NicoK/flink flink-7279
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4416.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4416
----
commit 7815deb974316d58da0646bd26ce718bbd597ba7
Author: Nico Kruber <[email protected]>
Date: 2017-07-28T09:45:48Z
[FLINK-7279][minicluster] fix a deadlock between TM and cluster shutdown
The MiniCluster can deadlock if the fatal error handler is called while the
MiniCluster shuts down. The reason is that the shut down happens under a
lock
which is required by the fatal error handler as well. If now the MiniCluster
tries to shut down the underlying RPC service which waits for all actors to
terminate, it will never complete because one actor is still waiting for the
lock.
Solution: guard both shutdown methods by a new ReentrantLock and ignore the
TM
shutdown in the TerminatingFatalErrorHandler if the cluster is already shut
down.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
