[
https://issues.apache.org/jira/browse/BEAM-5110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16573688#comment-16573688
]
Ankur Goenka edited comment on BEAM-5110 at 8/8/18 6:43 PM:
------------------------------------------------------------
We can invert the class loading order using
[https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#classloader-resolve-order]
There are additional configs as well which can be used to load cerain classes
from the parent class loader first and then the dynamic class loader
was (Author: angoenka):
We can invert the class loading order using
https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html#classloader-resolve-order
> Reconile Flink JVM singleton management with deployment
> -------------------------------------------------------
>
> Key: BEAM-5110
> URL: https://issues.apache.org/jira/browse/BEAM-5110
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
> Reporter: Ben Sidhom
> Assignee: Ben Sidhom
> Priority: Major
>
> [~angoenka] noticed through debugging that multiple instances of
> BatchFlinkExecutableStageContext.BatchFactory are loaded for a given job when
> executing in standalone cluster mode. This context factory is responsible for
> maintaining singleton state across a TaskManager (JVM) in order to share SDK
> Environments across workers in a given job. The multiple-loading breaks
> singleton semantics and results in an indeterminate number of Environments
> being created.
> It turns out that the [Flink classloading
> mechanism|https://ci.apache.org/projects/flink/flink-docs-release-1.5/monitoring/debugging_classloading.html]
> is determined by deployment mode. Note that "user code" as referenced by
> this link is actually the Flink job server jar. Actual end-user code lives
> inside of the SDK Environment and uploaded artifacts.
> In order to maintain singletons without resorting to IPC (for example, using
> file locks and/or additional gRPC servers), we need to force non-dynamic
> classloading. For example, this happens when jobs are submitted to YARN for
> one-off deployments via `flink run`. However, connecting to an existing
> (Flink standalone) deployment results in dynamic classloading.
> We should investigate this behavior and either document (and attempt to
> enforce) deployment modes that are consistent with our requirements, or (if
> possible) create a custom classloader that enforces singleton loading.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
