[
https://issues.apache.org/jira/browse/BEAM-5110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16580554#comment-16580554
]
Ankur Goenka edited comment on BEAM-5110 at 8/14/18 11:47 PM:
--------------------------------------------------------------
With the worker_id introduction, fn_api is capable of supporting and managing
multiple SDKHarness on a single machine.
The challenge with python is that a single python process can only use 1 cpu
core.
Going forward there can be other cases where we might want to have more than 1
instance of the same SDKHarness environment.
But this seems to be a separate problem and should be handled separately from
the current issue.
was (Author: angoenka):
With the worker_id introduction, fn_api is capable of supporting and managing
multiple SDKHarness on a single machine.
The challenge with python is that a single python process and only use 1 cpu
core.
Going forward there can be other cases where we might want to have more than 1
instance of the same SDKHarness environment.
But this seems to be a separate problem and should be handled separately from
the current issue.
> Reconile Flink JVM singleton management with deployment
> -------------------------------------------------------
>
> Key: BEAM-5110
> URL: https://issues.apache.org/jira/browse/BEAM-5110
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
> Reporter: Ben Sidhom
> Assignee: Ben Sidhom
> Priority: Major
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> [~angoenka] noticed through debugging that multiple instances of
> BatchFlinkExecutableStageContext.BatchFactory are loaded for a given job when
> executing in standalone cluster mode. This context factory is responsible for
> maintaining singleton state across a TaskManager (JVM) in order to share SDK
> Environments across workers in a given job. The multiple-loading breaks
> singleton semantics and results in an indeterminate number of Environments
> being created.
> It turns out that the [Flink classloading
> mechanism|https://ci.apache.org/projects/flink/flink-docs-release-1.5/monitoring/debugging_classloading.html]
> is determined by deployment mode. Note that "user code" as referenced by
> this link is actually the Flink job server jar. Actual end-user code lives
> inside of the SDK Environment and uploaded artifacts.
> In order to maintain singletons without resorting to IPC (for example, using
> file locks and/or additional gRPC servers), we need to force non-dynamic
> classloading. For example, this happens when jobs are submitted to YARN for
> one-off deployments via `flink run`. However, connecting to an existing
> (Flink standalone) deployment results in dynamic classloading.
> We should investigate this behavior and either document (and attempt to
> enforce) deployment modes that are consistent with our requirements, or (if
> possible) create a custom classloader that enforces singleton loading.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
