[
https://issues.apache.org/jira/browse/BEAM-5308?focusedWorklogId=141542&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-141542
]
ASF GitHub Bot logged work on BEAM-5308:
----------------------------------------
Author: ASF GitHub Bot
Created on: 05/Sep/18 22:14
Start Date: 05/Sep/18 22:14
Worklog Time Spent: 10m
Work Description: tweise closed pull request #6335: [BEAM-5308]
Correct/cleanup DockerOnMac code in DockerJobBundleFactory
URL: https://github.com/apache/beam/pull/6335
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DockerJobBundleFactory.java
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DockerJobBundleFactory.java
index 3178a2e2b3b..fec29231a97 100644
---
a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DockerJobBundleFactory.java
+++
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DockerJobBundleFactory.java
@@ -43,11 +43,6 @@
public class DockerJobBundleFactory extends JobBundleFactoryBase {
private static final Logger LOG =
LoggerFactory.getLogger(DockerJobBundleFactory.class);
- // Port offset for MacOS since we don't have host networking and need to use
published ports
- private static final int MAC_PORT_START = 8100;
- private static final int MAC_PORT_END = 8200;
- private static final AtomicInteger MAC_PORT = new
AtomicInteger(MAC_PORT_START);
-
/** Factory that creates {@link JobBundleFactory} for the given {@link
JobInfo}. */
public interface JobBundleFactoryFactory {
JobBundleFactory create(JobInfo jobInfo) throws Exception;
@@ -63,10 +58,6 @@ public JobBundleFactory create(JobInfo jobInfo) throws
Exception {
}
});
- // TODO: This host name seems to change with every other Docker release. Do
we attempt to keep up
- // or attempt to document the supported Docker version(s)?
- private static final String DOCKER_FOR_MAC_HOST = "host.docker.internal";
-
public static JobBundleFactory create(JobInfo jobInfo) throws Exception {
return FACTORY.get().create(jobInfo);
}
@@ -113,21 +104,7 @@ protected ServerFactory getServerFactory() {
case LINUX:
return ServerFactory.createDefault();
case MAC:
- // NOTE: Deployment on Macs is intended for local development. As of
18.03, Docker-for-Mac
- // does not implement host networking (--networking=host is
effectively a no-op). Instead,
- // we use a special DNS entry that points to the host:
- //
https://docs.docker.com/docker-for-mac/networking/#use-cases-and-workarounds
- // The special hostname has historically changed between versions, so
this is subject to
- // breakages and will likely only support the latest version at any
time.
-
- // We need to use a fixed port range due to non-existing host
networking in Docker-for-Mac.
- // The port range needs to be published when bringing up the Docker
container, see
- // DockerEnvironmentFactory.
-
- return ServerFactory.createWithUrlFactoryAndPortSupplier(
- (host, port) -> HostAndPort.fromParts(DOCKER_FOR_MAC_HOST,
port).toString(),
- // We only use the published Docker ports 8100-8200 in a
round-robin fashion
- () -> MAC_PORT.getAndUpdate(val -> val == MAC_PORT_END ?
MAC_PORT_START : val + 1));
+ return DockerOnMac.getServerFactory();
default:
LOG.warn("Unknown Docker platform. Falling back to default server
factory");
return ServerFactory.createDefault();
@@ -140,7 +117,7 @@ private static Platform getPlatform() {
// The DOCKER_MAC_CONTAINER environment variable is necessary to detect
whether we run on
// a container on MacOs. MacOs internally uses a Linux VM which makes it
indistinguishable from Linux.
// We still need to apply port mapping due to missing host networking.
- if (osName.startsWith("mac") ||
"1".equals(System.getenv("DOCKER_MAC_CONTAINER"))) {
+ if (osName.startsWith("mac") || DockerOnMac.RUNNING_INSIDE_DOCKER_ON_MAC) {
return Platform.MAC;
} else if (osName.startsWith("linux")) {
return Platform.LINUX;
@@ -154,6 +131,44 @@ private static Platform getPlatform() {
OTHER,
}
+ /**
+ * NOTE: Deployment on Macs is intended for local development. As of 18.03,
Docker-for-Mac does
+ * not implement host networking (--networking=host is effectively a no-op).
Instead, we use a
+ * special DNS entry that points to the host:
+ *
https://docs.docker.com/docker-for-mac/networking/#use-cases-and-workarounds
The special
+ * hostname has historically changed between versions, so this is subject to
breakages and will
+ * likely only support the latest version at any time.
+ */
+ private static class DockerOnMac {
+ // TODO: This host name seems to change with every other Docker release.
Do we attempt to keep up
+ // or attempt to document the supported Docker version(s)?
+ private static final String DOCKER_FOR_MAC_HOST = "host.docker.internal";
+
+ // True if we're inside a container (i.e. job-server container) with MacOS
as the host system
+ private static final boolean RUNNING_INSIDE_DOCKER_ON_MAC =
+ "1".equals(System.getenv("DOCKER_MAC_CONTAINER"));
+ // Port offset for MacOS since we don't have host networking and need to
use published ports
+ private static final int MAC_PORT_START = 8100;
+ private static final int MAC_PORT_END = 8200;
+ private static final AtomicInteger MAC_PORT = new
AtomicInteger(MAC_PORT_START);
+
+ private static ServerFactory getServerFactory() {
+ ServerFactory.UrlFactory dockerUrlFactory =
+ (host, port) -> HostAndPort.fromParts(DOCKER_FOR_MAC_HOST,
port).toString();
+ if (RUNNING_INSIDE_DOCKER_ON_MAC) {
+ // If we're already running in a container, we need to use a fixed
port range due to
+ // non-existing host networking in Docker-for-Mac. The port range
needs to be published
+ // when bringing up the Docker container, see DockerEnvironmentFactory.
+ return ServerFactory.createWithUrlFactoryAndPortSupplier(
+ dockerUrlFactory,
+ // We only use the published Docker ports 8100-8200 in a
round-robin fashion
+ () -> MAC_PORT.getAndUpdate(val -> val == MAC_PORT_END ?
MAC_PORT_START : val + 1));
+ } else {
+ return ServerFactory.createWithUrlFactory(dockerUrlFactory);
+ }
+ }
+ }
+
/** Create {@link EnvironmentFactory} for the given services. */
@Override
protected EnvironmentFactory getEnvironmentFactory(
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 141542)
Time Spent: 0.5h (was: 20m)
> JobBundleFactory BindException with FlinkRunner and remote cluster
> ------------------------------------------------------------------
>
> Key: BEAM-5308
> URL: https://issues.apache.org/jira/browse/BEAM-5308
> Project: Beam
> Issue Type: Task
> Components: runner-flink
> Reporter: Thomas Weise
> Assignee: Maximilian Michels
> Priority: Major
> Labels: portability
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Repeated execution of the same job on remote Flink cluster (not embedded in
> job server) fails with bind exception. There seem to be 2 issues:
> * Multiple instances of job bundle factory cannot be created (port conflict)
> * Job bundle factory is not released after job completes (and Docker
> container keeps on running). That's not the case in embedded mode).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
