Ben Sidhom created BEAM-5110:
--------------------------------
Summary: Reconile Flink JVM singleton management with deployment
Key: BEAM-5110
URL: https://issues.apache.org/jira/browse/BEAM-5110
Project: Beam
Issue Type: Bug
Components: runner-flink
Reporter: Ben Sidhom
[~angoenka] noticed through debugging that multiple instances of
BatchFlinkExecutableStageContext.BatchFactory are loaded for a given job when
executing in standalone cluster mode. This context factory is responsible for
maintaining singleton state across a TaskManager (JVM) in order to share SDK
Environments across workers in a given job. The multiple-loading breaks
singleton semantics and results in an indeterminate number of Environments
being created.
It turns out that the [Flink classloading
mechanism|https://ci.apache.org/projects/flink/flink-docs-release-1.5/monitoring/debugging_classloading.html]
is determined by deployment mode. Note that "user code" as referenced by this
link is actually the Flink job server jar. Actual end-user code lives inside of
the SDK Environment and uploaded artifacts.
In order to maintain singletons without resorting to IPC (for example, using
file locks and/or additional gRPC servers), we need to force non-dynamic
classloading. For example, this happens when jobs are submitted to YARN for
one-off deployments via `flink run`. However, connecting to an existing (Flink
standalone) deployment results in dynamic classloading.
We should investigate this behavior and either document (and attempt to
enforce) deployment modes that are consistent with our requirements, or (if
possible) create a custom classloader that enforces singleton loading.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
