[
https://issues.apache.org/jira/browse/BEAM-3327?focusedWorklogId=78768&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-78768
]
ASF GitHub Bot logged work on BEAM-3327:
----------------------------------------
Author: ASF GitHub Bot
Created on: 09/Mar/18 02:18
Start Date: 09/Mar/18 02:18
Worklog Time Spent: 10m
Work Description: bsidhom commented on a change in pull request #4751:
[BEAM-3327] Implement simple Docker container manager
URL: https://github.com/apache/beam/pull/4751#discussion_r173350125
##########
File path:
runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/environment/SingletonDockerEnvironmentManager.java
##########
@@ -0,0 +1,107 @@
+package org.apache.beam.runners.fnexecution.environment;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Arrays;
+import java.util.List;
+import java.util.concurrent.TimeoutException;
+import javax.annotation.concurrent.GuardedBy;
+import org.apache.beam.model.pipeline.v1.RunnerApi.Environment;
+import org.apache.beam.runners.fnexecution.GrpcFnServer;
+import org.apache.beam.runners.fnexecution.artifact.ArtifactRetrievalService;
+import
org.apache.beam.runners.fnexecution.control.SdkHarnessClientControlService;
+import org.apache.beam.runners.fnexecution.logging.GrpcLoggingService;
+import
org.apache.beam.runners.fnexecution.provisioning.StaticGrpcProvisionService;
+
+/** An {@link EnvironmentManager} that manages a single docker container. */
+public class SingletonDockerEnvironmentManager implements EnvironmentManager {
+
+ public static SingletonDockerEnvironmentManager forServices(
+ DockerWrapper docker,
+ GrpcFnServer<SdkHarnessClientControlService> controlServiceServer,
+ GrpcFnServer<GrpcLoggingService> loggingServiceServer,
+ GrpcFnServer<ArtifactRetrievalService> retrievalServiceServer,
+ GrpcFnServer<StaticGrpcProvisionService> provisioningServiceServer) {
+ return new SingletonDockerEnvironmentManager(docker, controlServiceServer,
loggingServiceServer,
+ retrievalServiceServer, provisioningServiceServer);
+ }
+
+ private final Object lock = new Object();
+ private final DockerWrapper docker;
+ private final GrpcFnServer<SdkHarnessClientControlService>
controlServiceServer;
+ private final GrpcFnServer<GrpcLoggingService> loggingServiceServer;
+ private final GrpcFnServer<ArtifactRetrievalService> retrievalServiceServer;
+ private final GrpcFnServer<StaticGrpcProvisionService>
provisioningServiceServer;
+
+ @GuardedBy("lock")
+ private RemoteEnvironment dockerEnvironment = null;
+
+ private SingletonDockerEnvironmentManager(
+ DockerWrapper docker,
+ GrpcFnServer<SdkHarnessClientControlService> controlServiceServer,
+ GrpcFnServer<GrpcLoggingService> loggingServiceServer,
+ GrpcFnServer<ArtifactRetrievalService> retrievalServiceServer,
+ GrpcFnServer<StaticGrpcProvisionService> provisioningServiceServer) {
+ this.docker = docker;
+ this.controlServiceServer = controlServiceServer;
+ this.loggingServiceServer = loggingServiceServer;
+ this.retrievalServiceServer = retrievalServiceServer;
+ this.provisioningServiceServer = provisioningServiceServer;
+ }
+
+ /**
+ * Retrieve a handle for the given environment. The same environment must be
requested every time.
+ * The same remote handle is returned to every caller, so the environment
cannot be used once
+ * closed.
+ */
+ @Override
+ public RemoteEnvironment getEnvironment(Environment environment) throws
Exception {
+ synchronized (lock) {
+ if (dockerEnvironment == null) {
+ dockerEnvironment = createDockerEnv(environment);
+ } else {
+ checkArgument(
+
environment.getUrl().equals(dockerEnvironment.getEnvironment().getUrl()),
+ "A %s must only be queried for a single %s. Existing %s, Argument
%s",
+ SingletonDockerEnvironmentManager.class.getSimpleName(),
+ Environment.class.getSimpleName(),
+ dockerEnvironment.getEnvironment().getUrl(),
+ environment.getUrl());
+ }
+ return dockerEnvironment;
+ }
+ }
+
+ private DockerContainerEnvironment createDockerEnv(Environment environment)
+ throws IOException, TimeoutException, InterruptedException {
+ // TODO: Generate environment id correctly.
+ String environmentId = Long.toString(-123);
+ Path workerPersistentDirectory =
Files.createTempDirectory("worker_persistent_directory");
+ Path semiPersistentDirectory =
Files.createTempDirectory("semi_persistent_dir");
+ String containerImage = environment.getUrl();
+ // TODO: The default service address will not work for Docker for Mac.
+ String loggingEndpoint =
loggingServiceServer.getApiServiceDescriptor().getUrl();
+ String artifactEndpoint =
retrievalServiceServer.getApiServiceDescriptor().getUrl();
+ String provisionEndpoint =
provisioningServiceServer.getApiServiceDescriptor().getUrl();
+ String controlEndpoint =
controlServiceServer.getApiServiceDescriptor().getUrl();
+ List<String> args = Arrays.asList(
+ "-v",
+ String.format("%s:%S", workerPersistentDirectory,
semiPersistentDirectory),
+ // TODO: This needs to be special-cased for Mac.
+ "--network=host",
Review comment:
I've played around with it a bit and it looks like Docker for Mac accepts
the `--network=host` flag but silently ignores it.
It looks like the only change we actually need to make between Mac and Linux
is to add a restriction on the temporary directory. By default, Docker for Mac
only accepts temporary mounts under `/tmp` and requires explicit user
configuration to use the default Java generated directory `/var/...`. In the
interest of making things "just work" for local development, we should add this
change, but I'll do that in a follow-up PR where I also address the network
hosts for Mac.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 78768)
Time Spent: 4h 10m (was: 4h)
> Add abstractions to manage Environment Instance lifecycles.
> -----------------------------------------------------------
>
> Key: BEAM-3327
> URL: https://issues.apache.org/jira/browse/BEAM-3327
> Project: Beam
> Issue Type: New Feature
> Components: runner-core
> Reporter: Thomas Groh
> Assignee: Ben Sidhom
> Priority: Major
> Labels: portability
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> This permits remote stage execution for arbitrary environments
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
