[jira] [Commented] (FLINK-5091) Formalize the AppMaster environment for docker compability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723724#comment-15723724
 ] 

ASF GitHub Bot commented on FLINK-5091:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2915


> Formalize the AppMaster environment for docker compability
> --
>
> Key: FLINK-5091
> URL: https://issues.apache.org/jira/browse/FLINK-5091
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management, Mesos
>Reporter: Eron Wright 
>Assignee: Eron Wright 
> Fix For: 1.2.0
>
>
> For scenarios where the AppMaster is launched from a docker image, it would 
> be ideal to use the installed Flink rather than rely on a special file layout 
> in the sandbox directory.
> This is related to DCOS integration, which (in 1.2) will launch the AppMaster 
> via Marathon (as a top-level DCOS service).  The existing code assumed that 
> only the dispatcher (coming in 1.3) would launch the AppMaster.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5091) Formalize the AppMaster environment for docker compability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723027#comment-15723027
 ] 

ASF GitHub Bot commented on FLINK-5091:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2915
  
Looks really good!. Merging with some minor changes.


> Formalize the AppMaster environment for docker compability
> --
>
> Key: FLINK-5091
> URL: https://issues.apache.org/jira/browse/FLINK-5091
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management, Mesos
>Reporter: Eron Wright 
>Assignee: Eron Wright 
> Fix For: 1.2.0
>
>
> For scenarios where the AppMaster is launched from a docker image, it would 
> be ideal to use the installed Flink rather than rely on a special file layout 
> in the sandbox directory.
> This is related to DCOS integration, which (in 1.2) will launch the AppMaster 
> via Marathon (as a top-level DCOS service).  The existing code assumed that 
> only the dispatcher (coming in 1.3) would launch the AppMaster.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5091) Formalize the AppMaster environment for docker compability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715487#comment-15715487
 ] 

ASF GitHub Bot commented on FLINK-5091:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2915#discussion_r90454802
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/overlays/AbstractContainerOverlay.java
 ---
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.clusterframework.overlays;
+
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.clusterframework.ContainerSpecification;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.FileVisitResult;
+import java.nio.file.Files;
+import java.nio.file.SimpleFileVisitor;
+import java.nio.file.attribute.BasicFileAttributes;
+
+/**
+ * An abstract container overlay.
--- End diff --

Could you elaborate a bit here?


> Formalize the AppMaster environment for docker compability
> --
>
> Key: FLINK-5091
> URL: https://issues.apache.org/jira/browse/FLINK-5091
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management, Mesos
>Reporter: Eron Wright 
>Assignee: Eron Wright 
> Fix For: 1.2.0
>
>
> For scenarios where the AppMaster is launched from a docker image, it would 
> be ideal to use the installed Flink rather than rely on a special file layout 
> in the sandbox directory.
> This is related to DCOS integration, which (in 1.2) will launch the AppMaster 
> via Marathon (as a top-level DCOS service).  The existing code assumed that 
> only the dispatcher (coming in 1.3) would launch the AppMaster.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5091) Formalize the AppMaster environment for docker compability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715494#comment-15715494
 ] 

ASF GitHub Bot commented on FLINK-5091:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2915#discussion_r90454287
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/overlays/ContainerOverlay.java
 ---
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.clusterframework.overlays;
+
+import org.apache.flink.runtime.clusterframework.ContainerSpecification;
+
+import java.io.IOException;
+
+/**
+ * A container overlay to produce a container specification.
+ *
+ * An overlay applies configuration elements, environment variables,
+ * system properties, and artifacts to a container specification.
+ */
+public interface ContainerOverlay {
+
+   /**
+* Configure the given container specification.
+ */
--- End diff --

Indention is off here


> Formalize the AppMaster environment for docker compability
> --
>
> Key: FLINK-5091
> URL: https://issues.apache.org/jira/browse/FLINK-5091
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management, Mesos
>Reporter: Eron Wright 
>Assignee: Eron Wright 
> Fix For: 1.2.0
>
>
> For scenarios where the AppMaster is launched from a docker image, it would 
> be ideal to use the installed Flink rather than rely on a special file layout 
> in the sandbox directory.
> This is related to DCOS integration, which (in 1.2) will launch the AppMaster 
> via Marathon (as a top-level DCOS service).  The existing code assumed that 
> only the dispatcher (coming in 1.3) would launch the AppMaster.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5091) Formalize the AppMaster environment for docker compability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715486#comment-15715486
 ] 

ASF GitHub Bot commented on FLINK-5091:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2915#discussion_r90664693
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/overlays/FlinkDistributionOverlay.java
 ---
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.clusterframework.overlays;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.clusterframework.ContainerSpecification;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.Map;
+
+import static 
org.apache.flink.configuration.ConfigConstants.ENV_FLINK_BIN_DIR;
+import static 
org.apache.flink.configuration.ConfigConstants.ENV_FLINK_CONF_DIR;
+import static 
org.apache.flink.configuration.ConfigConstants.ENV_FLINK_HOME_DIR;
+import static 
org.apache.flink.configuration.ConfigConstants.ENV_FLINK_LIB_DIR;
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Overlays Flink into a container, based on supplied bin/conf/lib 
directories.
+ *
+ * The overlayed Flink is indistinguishable from (and interchangeable with)
+ * a normal installation of Flink.  For a docker image-based container, it 
should be
+ * possible to bypass this overlay and rely on the normal installation 
method.
+ *
+ * The following files are copied to the container:
+ *  - flink/bin/
+ *  - flink/conf/
+ *  - flink/lib/
+ */
+public class FlinkDistributionOverlay extends AbstractContainerOverlay {
+
+   private static final Logger LOG = 
LoggerFactory.getLogger(FlinkDistributionOverlay.class);
+
+   static final Path TARGET_ROOT = new Path("flink");
+
+   final File flinkBinPath;
+   final File flinkConfPath;
+   final File flinkLibPath;
+
+   public FlinkDistributionOverlay(File flinkBinPath, File flinkConfPath, 
File flinkLibPath) {
+   this.flinkBinPath = checkNotNull(flinkBinPath);
+   this.flinkConfPath = checkNotNull(flinkConfPath);
+   this.flinkLibPath = checkNotNull(flinkLibPath);
+   }
+
+   @Override
+   public void configure(ContainerSpecification container) throws 
IOException {
+
+   container.getEnvironmentVariables().put(ENV_FLINK_HOME_DIR, 
TARGET_ROOT.toString());
+
+   // add the paths to the container specification.
+   addPathRecursively(flinkBinPath, TARGET_ROOT, container);
+   addPathRecursively(flinkConfPath, TARGET_ROOT, container);
+   addPathRecursively(flinkLibPath, TARGET_ROOT, container);
+   }
+
+   public static Builder newBuilder() {
+   return new Builder();
+   }
+
+   /**
+* A builder for the {@link FlinkDistributionOverlay}.
+*/
+   public static class Builder {
+   File flinkBinPath;
+   File flinkConfPath;
+   File flinkLibPath;
+
+   /**
+* Configures the overlay using the current environment.
+*
+* Locates Flink using FLINK_???_DIR environment variables as 
provided to all Flink processes by config.sh.
+*
+* @param globalConfiguration the current configuration.
+ */
--- End diff --

indention is off here


> Formalize the AppMaster environment for docker compability
> --
>
> Key: FLINK-5091
> URL: https://issues.apache.org/jira/browse/FLINK-5091
> Project: Flink
>  Issue Type: Sub-task
>  Components: 

[jira] [Commented] (FLINK-5091) Formalize the AppMaster environment for docker compability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715492#comment-15715492
 ] 

ASF GitHub Bot commented on FLINK-5091:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2915#discussion_r90666707
  
--- Diff: 
flink-runtime/src/test/java/org/apache/flink/runtime/clusterframework/overlays/HadoopConfOverlayTest.java
 ---
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.clusterframework.overlays;
+
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.core.testutils.CommonTestUtils;
+import org.apache.flink.runtime.clusterframework.ContainerSpecification;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+
+import static 
org.apache.flink.runtime.clusterframework.overlays.HadoopConfOverlay.TARGET_CONF_DIR;
+
+public class HadoopConfOverlayTest extends ContainerOverlayTestBase {
+
+   @Rule
+   public TemporaryFolder tempFolder = new TemporaryFolder();
+
+   @Test
+   public void testConfigure() throws Exception {
+
+   File confDir = tempFolder.newFolder();
+   initConfDir(confDir);
+
+   HadoopConfOverlay overlay = new HadoopConfOverlay(confDir);
+
+   ContainerSpecification spec = new ContainerSpecification();
+   overlay.configure(spec);
+
+   assertEquals(TARGET_CONF_DIR.getPath(), 
spec.getEnvironmentVariables().get("HADOOP_CONF_DIR"));
+   assertEquals(TARGET_CONF_DIR.getPath(), 
spec.getDynamicConfiguration().getString(ConfigConstants.PATH_HADOOP_CONFIG, 
null));
+
+   checkArtifact(spec, new Path(TARGET_CONF_DIR, "core-site.xml"));
+   checkArtifact(spec, new Path(TARGET_CONF_DIR, "hdfs-site.xml"));
+   }
+
+   @Test
+   public void testNoConf() throws Exception {
+   HadoopConfOverlay overlay = new HadoopConfOverlay(null);
+
+   ContainerSpecification containerSpecification = new 
ContainerSpecification();
+   overlay.configure(containerSpecification);
+   }
+
+   @Test
+   public void testBuilderFromEnvironment() throws Exception {
+
+   // verify that the builder picks up various environment 
locations
+   HadoopConfOverlay.Builder builder;
+   Map env;
+
+   // fs.hdfs.hadoopconf
+   File confDir = tempFolder.newFolder();
+   initConfDir(confDir);
+   Configuration conf = new Configuration();
+   conf.setString(ConfigConstants.PATH_HADOOP_CONFIG, 
confDir.getAbsolutePath());
+   builder = HadoopConfOverlay.newBuilder().fromEnvironment(conf);
+   assertEquals(confDir, builder.hadoopConfDir);
+
+   // HADOOP_CONF_DIR
+   env = new HashMap(System.getenv());
+   env.remove("HADOOP_HOME");
+   env.put("HADOOP_CONF_DIR", confDir.getAbsolutePath());
+   CommonTestUtils.setEnv(env);
+   builder = HadoopConfOverlay.newBuilder().fromEnvironment(new 
Configuration());
+   assertEquals(confDir, builder.hadoopConfDir);
+
+   // HADOOP_HOME/conf
+   File homeDir = tempFolder.newFolder();
+   confDir = initConfDir(new File(homeDir, "conf"));
+   env = new HashMap(System.getenv());
+   env.remove("HADOOP_CONF_DIR");
+   env.put("HADOOP_HOME", homeDir.getAbsolutePath());
--- End diff --

Should we restore the original environment 

[jira] [Commented] (FLINK-5091) Formalize the AppMaster environment for docker compability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715491#comment-15715491
 ] 

ASF GitHub Bot commented on FLINK-5091:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2915#discussion_r90664938
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/overlays/SSLStoreOverlay.java
 ---
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.clusterframework.overlays;
+
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.clusterframework.ContainerSpecification;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+import java.io.File;
+import java.io.IOException;
+
+
+/**
+ * Overlays an SSL keystore/truststore into a container.
+ *
+ * The following files are placed into the container:
+ *  - keystore.jks
+ *  - truststore.jks
+ *
+ * The following Flink configuration entries are set:
+ *  - security.ssl.keystore
+ *  - security.ssl.truststore
+ */
+public class SSLStoreOverlay extends AbstractContainerOverlay {
+
+   private static final Logger LOG = 
LoggerFactory.getLogger(SSLStoreOverlay.class);
+
+   static final Path TARGET_KEYSTORE_PATH = new Path("keystore.jks");
+   static final Path TARGET_TRUSTSTORE_PATH = new Path("truststore.jks");
+
+   final Path keystore;
+   final Path truststore;
+
+   public SSLStoreOverlay(@Nullable File keystoreFile, @Nullable File 
truststoreFile) {
+   this.keystore = keystoreFile != null ? new 
Path(keystoreFile.toURI()) : null;
+   this.truststore = truststoreFile != null ? new 
Path(truststoreFile.toURI()) : null;
+   }
+
+   @Override
+   public void configure(ContainerSpecification container) throws 
IOException {
+   if(keystore != null) {
+   
container.getArtifacts().add(ContainerSpecification.Artifact.newBuilder()
+   .setSource(keystore)
+   .setDest(TARGET_KEYSTORE_PATH)
+   .setCachable(false)
+   .build());
+   
container.getDynamicConfiguration().setString(ConfigConstants.SECURITY_SSL_KEYSTORE,
 TARGET_KEYSTORE_PATH.getPath());
+   }
+   if(truststore != null) {
+   
container.getArtifacts().add(ContainerSpecification.Artifact.newBuilder()
+   .setSource(truststore)
+   .setDest(TARGET_TRUSTSTORE_PATH)
+   .setCachable(false)
+   .build());
+   
container.getDynamicConfiguration().setString(ConfigConstants.SECURITY_SSL_TRUSTSTORE,
 TARGET_TRUSTSTORE_PATH.getPath());
+   }
+   }
+
+   public static Builder newBuilder() {
+   return new Builder();
+   }
+
+   /**
+* A builder for the {@link Krb5ConfOverlay}.
+*/
+   public static class Builder {
+
+   File keystorePath;
+
+   File truststorePath;
+
+   /**
+* Configures the overlay using the current environment (and 
global configuration).
+*
+* The following Flink configuration settings are used to 
source the keystore and truststore:
+*  - security.ssl.keystore
+*  - security.ssl.truststore
+ */
--- End diff --

indention is off here


> Formalize the AppMaster environment for docker compability
> --
>
> Key: FLINK-5091
> URL: 

[jira] [Commented] (FLINK-5091) Formalize the AppMaster environment for docker compability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715493#comment-15715493
 ] 

ASF GitHub Bot commented on FLINK-5091:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2915#discussion_r90666953
  
--- Diff: 
flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/test/util/TestBaseUtils.java
 ---
@@ -542,42 +542,10 @@ protected static File asFile(String path) {
return configs;
}
 
-   // This code is taken from: http://stackoverflow.com/a/7201825/568695
-   // it changes the environment variables of this JVM. Use only for 
testing purposes!
-   @SuppressWarnings("unchecked")
public static void setEnv(Map newenv) {
--- End diff --

How about removing this method and redirecting the calls? Was there a 
dependency management reason you didn't want to do that?


> Formalize the AppMaster environment for docker compability
> --
>
> Key: FLINK-5091
> URL: https://issues.apache.org/jira/browse/FLINK-5091
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management, Mesos
>Reporter: Eron Wright 
>Assignee: Eron Wright 
> Fix For: 1.2.0
>
>
> For scenarios where the AppMaster is launched from a docker image, it would 
> be ideal to use the installed Flink rather than rely on a special file layout 
> in the sandbox directory.
> This is related to DCOS integration, which (in 1.2) will launch the AppMaster 
> via Marathon (as a top-level DCOS service).  The existing code assumed that 
> only the dispatcher (coming in 1.3) would launch the AppMaster.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5091) Formalize the AppMaster environment for docker compability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715490#comment-15715490
 ] 

ASF GitHub Bot commented on FLINK-5091:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2915#discussion_r90664782
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/overlays/KeytabOverlay.java
 ---
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.clusterframework.overlays;
+
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.clusterframework.ContainerSpecification;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+import java.io.File;
+import java.io.IOException;
+
+
+/**
+ * Overlays cluster-level Kerberos credentials (i.e. keytab) into a 
container.
+ *
+ * The folloowing Flink configuration entries are updated:
+ *  - security.keytab
+ */
+public class KeytabOverlay extends AbstractContainerOverlay {
+
+   private static final Logger LOG = 
LoggerFactory.getLogger(KeytabOverlay.class);
+
+   static final Path TARGET_PATH = new Path("krb5.keytab");
+
+   final Path keytab;
+
+   public KeytabOverlay(@Nullable File keytab) {
+   this.keytab = keytab != null ? new Path(keytab.toURI()) : null;
+   }
+
+   public KeytabOverlay(@Nullable Path keytab) {
+   this.keytab = keytab;
+   }
+
+   @Override
+   public void configure(ContainerSpecification container) throws 
IOException {
+   if(keytab != null) {
+   
container.getArtifacts().add(ContainerSpecification.Artifact.newBuilder()
+   .setSource(keytab)
+   .setDest(TARGET_PATH)
+   .setCachable(false)
+   .build());
+   
container.getDynamicConfiguration().setString(ConfigConstants.SECURITY_KEYTAB_KEY,
 TARGET_PATH.getPath());
+   }
+   }
+
+   public static Builder newBuilder() {
+   return new Builder();
+   }
+
+   /**
+* A builder for the {@link HadoopUserOverlay}.
+*/
+   public static class Builder {
+
+   File keytabPath;
+
+   /**
+* Configures the overlay using the current environment (and 
global configuration).
+*
+* The following Flink configuration settings are checked for a 
keytab:
+*  - security.keytab
+ */
--- End diff --

indention is off here


> Formalize the AppMaster environment for docker compability
> --
>
> Key: FLINK-5091
> URL: https://issues.apache.org/jira/browse/FLINK-5091
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management, Mesos
>Reporter: Eron Wright 
>Assignee: Eron Wright 
> Fix For: 1.2.0
>
>
> For scenarios where the AppMaster is launched from a docker image, it would 
> be ideal to use the installed Flink rather than rely on a special file layout 
> in the sandbox directory.
> This is related to DCOS integration, which (in 1.2) will launch the AppMaster 
> via Marathon (as a top-level DCOS service).  The existing code assumed that 
> only the dispatcher (coming in 1.3) would launch the AppMaster.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5091) Formalize the AppMaster environment for docker compability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715488#comment-15715488
 ] 

ASF GitHub Bot commented on FLINK-5091:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2915#discussion_r90665396
  
--- Diff: 
flink-runtime/src/test/java/org/apache/flink/runtime/clusterframework/overlays/HadoopConfOverlayTest.java
 ---
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.clusterframework.overlays;
+
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.core.testutils.CommonTestUtils;
+import org.apache.flink.runtime.clusterframework.ContainerSpecification;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+
+import static 
org.apache.flink.runtime.clusterframework.overlays.HadoopConfOverlay.TARGET_CONF_DIR;
+
+public class HadoopConfOverlayTest extends ContainerOverlayTestBase {
+
+   @Rule
+   public TemporaryFolder tempFolder = new TemporaryFolder();
+
+   @Test
+   public void testConfigure() throws Exception {
+
+   File confDir = tempFolder.newFolder();
+   initConfDir(confDir);
+
+   HadoopConfOverlay overlay = new HadoopConfOverlay(confDir);
+
+   ContainerSpecification spec = new ContainerSpecification();
+   overlay.configure(spec);
+
+   assertEquals(TARGET_CONF_DIR.getPath(), 
spec.getEnvironmentVariables().get("HADOOP_CONF_DIR"));
+   assertEquals(TARGET_CONF_DIR.getPath(), 
spec.getDynamicConfiguration().getString(ConfigConstants.PATH_HADOOP_CONFIG, 
null));
+
+   checkArtifact(spec, new Path(TARGET_CONF_DIR, "core-site.xml"));
+   checkArtifact(spec, new Path(TARGET_CONF_DIR, "hdfs-site.xml"));
+   }
+
+   @Test
+   public void testNoConf() throws Exception {
+   HadoopConfOverlay overlay = new HadoopConfOverlay(null);
+
+   ContainerSpecification containerSpecification = new 
ContainerSpecification();
+   overlay.configure(containerSpecification);
+   }
+
+   @Test
+   public void testBuilderFromEnvironment() throws Exception {
--- End diff --

indention is off here


> Formalize the AppMaster environment for docker compability
> --
>
> Key: FLINK-5091
> URL: https://issues.apache.org/jira/browse/FLINK-5091
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management, Mesos
>Reporter: Eron Wright 
>Assignee: Eron Wright 
> Fix For: 1.2.0
>
>
> For scenarios where the AppMaster is launched from a docker image, it would 
> be ideal to use the installed Flink rather than rely on a special file layout 
> in the sandbox directory.
> This is related to DCOS integration, which (in 1.2) will launch the AppMaster 
> via Marathon (as a top-level DCOS service).  The existing code assumed that 
> only the dispatcher (coming in 1.3) would launch the AppMaster.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5091) Formalize the AppMaster environment for docker compability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715495#comment-15715495
 ] 

ASF GitHub Bot commented on FLINK-5091:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2915#discussion_r90664795
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/overlays/Krb5ConfOverlay.java
 ---
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.clusterframework.overlays;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.clusterframework.ContainerSpecification;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+import java.io.File;
+import java.io.IOException;
+
+
+/**
+ * Overlays a Kerberos configuration file into a container.
+ *
+ * The following files are copied to the container:
+ *  - krb5.conf
+ *
+ * The following Java system properties are set in the container:
+ *  - java.security.krb5.conf
+ */
+public class Krb5ConfOverlay extends AbstractContainerOverlay {
+
+   private static final Logger LOG = 
LoggerFactory.getLogger(Krb5ConfOverlay.class);
+
+   static final String JAVA_SECURITY_KRB5_CONF = "java.security.krb5.conf";
+
+   static final Path TARGET_PATH = new Path("krb5.conf");
+   final Path krb5Conf;
+
+   public Krb5ConfOverlay(@Nullable File krb5Conf) {
+   this.krb5Conf = krb5Conf != null ? new Path(krb5Conf.toURI()) : 
null;
+   }
+
+   public Krb5ConfOverlay(@Nullable Path krb5Conf) {
+   this.krb5Conf = krb5Conf;
+   }
+
+   @Override
+   public void configure(ContainerSpecification container) throws 
IOException {
+   if(krb5Conf != null) {
+   
container.getArtifacts().add(ContainerSpecification.Artifact.newBuilder()
+   .setSource(krb5Conf)
+   .setDest(TARGET_PATH)
+   .setCachable(true)
+   .build());
+   
container.getSystemProperties().setString(JAVA_SECURITY_KRB5_CONF, 
TARGET_PATH.getPath());
+   }
+   }
+
+   public static Builder newBuilder() {
+   return new Builder();
+   }
+
+   /**
+* A builder for the {@link Krb5ConfOverlay}.
+*/
+   public static class Builder {
+
+   File krb5ConfPath;
+
+   /**
+* Configures the overlay using the current environment.
+*
+* Locates the krb5.conf configuration file as per
+* https://docs.oracle.com/javase/8/docs/technotes/guides/security/jgss/tutorials/KerberosReq.html;>Java
 documentation.
+* Note that the JRE doesn't support the KRB5_CONFIG 
environment variable (JDK-7045913).
+ */
--- End diff --

indention is off here


> Formalize the AppMaster environment for docker compability
> --
>
> Key: FLINK-5091
> URL: https://issues.apache.org/jira/browse/FLINK-5091
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management, Mesos
>Reporter: Eron Wright 
>Assignee: Eron Wright 
> Fix For: 1.2.0
>
>
> For scenarios where the AppMaster is launched from a docker image, it would 
> be ideal to use the installed Flink rather than rely on a special file layout 
> in the sandbox directory.
> This is related to DCOS integration, which (in 1.2) will launch the AppMaster 
> via Marathon (as a top-level DCOS service).  The existing code assumed that 
> only the dispatcher (coming in 1.3) would launch the AppMaster.  



--
This message was sent by 

[jira] [Commented] (FLINK-5091) Formalize the AppMaster environment for docker compability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15715489#comment-15715489
 ] 

ASF GitHub Bot commented on FLINK-5091:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2915#discussion_r90664759
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/overlays/HadoopUserOverlay.java
 ---
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.clusterframework.overlays;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.clusterframework.ContainerSpecification;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+
+/**
+ * Overlays a Hadoop user context into a container.
+ *
+ * The overlay essentially configures Hadoop's {@link 
UserGroupInformation} class,
+ * establishing the effective username for filesystem calls to HDFS in 
non-secure clusters.
+ *
+ * In secure clusters, the configured keytab establishes the effective 
user.
+ *
+ * The following environment variables are set in the container:
+ *  - HADOOP_USER_NAME
+ */
+public class HadoopUserOverlay implements ContainerOverlay {
+
+   private static final Logger LOG = 
LoggerFactory.getLogger(HadoopUserOverlay.class);
+
+   private final UserGroupInformation ugi;
+
+   public HadoopUserOverlay(@Nullable UserGroupInformation ugi) {
+   this.ugi = ugi;
+   }
+
+   @Override
+   public void configure(ContainerSpecification container) throws 
IOException {
+   if(ugi != null) {
+   // overlay the Hadoop user identity (w/ tokens)
+   
container.getEnvironmentVariables().put("HADOOP_USER_NAME", ugi.getUserName());
+   }
+   }
+
+   public static Builder newBuilder() {
+   return new Builder();
+   }
+
+   /**
+* A builder for the {@link HadoopUserOverlay}.
+*/
+   public static class Builder {
+
+   UserGroupInformation ugi;
+
+   /**
+* Configures the overlay using the current Hadoop user 
information (from {@link UserGroupInformation}).
+ */
--- End diff --

indention is off here


> Formalize the AppMaster environment for docker compability
> --
>
> Key: FLINK-5091
> URL: https://issues.apache.org/jira/browse/FLINK-5091
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management, Mesos
>Reporter: Eron Wright 
>Assignee: Eron Wright 
> Fix For: 1.2.0
>
>
> For scenarios where the AppMaster is launched from a docker image, it would 
> be ideal to use the installed Flink rather than rely on a special file layout 
> in the sandbox directory.
> This is related to DCOS integration, which (in 1.2) will launch the AppMaster 
> via Marathon (as a top-level DCOS service).  The existing code assumed that 
> only the dispatcher (coming in 1.3) would launch the AppMaster.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5091) Formalize the AppMaster environment for docker compability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15712570#comment-15712570
 ] 

ASF GitHub Bot commented on FLINK-5091:
---

Github user EronWright commented on a diff in the pull request:

https://github.com/apache/flink/pull/2915#discussion_r90498149
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java ---
@@ -1372,6 +1381,12 @@
/** The environment variable name which contains the location of the 
lib folder */
public static final String ENV_FLINK_LIB_DIR = "FLINK_LIB_DIR";
 
+   /** The environment variable name which contains the location of the 
bin directory */
+   public static final String ENV_FLINK_BIN_DIR = "FLINK_BIN_DIR";
+
+   /** The environment variable name which contains the Flink installation 
root directory */
+   public static final String ENV_FLINK_HOME_DIR = "FLINK_HOME";
+
--- End diff --

@Makman2  the standalone TM relies on the `FLINK_CONF_DIR` environment 
variable to find its configuration.  Now the Mesos TM does too, for 
consistency, rather than looking to the working directory.   

When the AM launches the TM as a Mesos task, it installs Flink to a 
subdirectory within the sandbox directory, and sets the `FLINK_HOME` 
environment variable accordingly.   The variable value is a relative path 
(based on the sandbox directory); it is assumed that the TM working directory 
is the sandbox directory.

Does this address your concern?


> Formalize the AppMaster environment for docker compability
> --
>
> Key: FLINK-5091
> URL: https://issues.apache.org/jira/browse/FLINK-5091
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management, Mesos
>Reporter: Eron Wright 
>Assignee: Eron Wright 
> Fix For: 1.2.0
>
>
> For scenarios where the AppMaster is launched from a docker image, it would 
> be ideal to use the installed Flink rather than rely on a special file layout 
> in the sandbox directory.
> This is related to DCOS integration, which (in 1.2) will launch the AppMaster 
> via Marathon (as a top-level DCOS service).  The existing code assumed that 
> only the dispatcher (coming in 1.3) would launch the AppMaster.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5091) Formalize the AppMaster environment for docker compability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711644#comment-15711644
 ] 

ASF GitHub Bot commented on FLINK-5091:
---

Github user Makman2 commented on a diff in the pull request:

https://github.com/apache/flink/pull/2915#discussion_r90423889
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java ---
@@ -1372,6 +1381,12 @@
/** The environment variable name which contains the location of the 
lib folder */
public static final String ENV_FLINK_LIB_DIR = "FLINK_LIB_DIR";
 
+   /** The environment variable name which contains the location of the 
bin directory */
+   public static final String ENV_FLINK_BIN_DIR = "FLINK_BIN_DIR";
+
+   /** The environment variable name which contains the Flink installation 
root directory */
+   public static final String ENV_FLINK_HOME_DIR = "FLINK_HOME";
+
--- End diff --

you sure these should be config values? What if someone wants to set these 
to `$MESOS_SANDBOX`, then we need to rely on shell expansion on an already 
started container^^ (Not sure this is a usecase)


> Formalize the AppMaster environment for docker compability
> --
>
> Key: FLINK-5091
> URL: https://issues.apache.org/jira/browse/FLINK-5091
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management, Mesos
>Reporter: Eron Wright 
>Assignee: Eron Wright 
> Fix For: 1.2.0
>
>
> For scenarios where the AppMaster is launched from a docker image, it would 
> be ideal to use the installed Flink rather than rely on a special file layout 
> in the sandbox directory.
> This is related to DCOS integration, which (in 1.2) will launch the AppMaster 
> via Marathon (as a top-level DCOS service).  The existing code assumed that 
> only the dispatcher (coming in 1.3) would launch the AppMaster.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5091) Formalize the AppMaster environment for docker compability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15711641#comment-15711641
 ] 

ASF GitHub Bot commented on FLINK-5091:
---

Github user Makman2 commented on a diff in the pull request:

https://github.com/apache/flink/pull/2915#discussion_r90423588
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/configuration/GlobalConfiguration.java
 ---
@@ -39,12 +39,31 @@
 
public static final String FLINK_CONF_FILENAME = "flink-conf.yaml";
 
+
--- End diff --

unrelated change :)


> Formalize the AppMaster environment for docker compability
> --
>
> Key: FLINK-5091
> URL: https://issues.apache.org/jira/browse/FLINK-5091
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management, Mesos
>Reporter: Eron Wright 
>Assignee: Eron Wright 
> Fix For: 1.2.0
>
>
> For scenarios where the AppMaster is launched from a docker image, it would 
> be ideal to use the installed Flink rather than rely on a special file layout 
> in the sandbox directory.
> This is related to DCOS integration, which (in 1.2) will launch the AppMaster 
> via Marathon (as a top-level DCOS service).  The existing code assumed that 
> only the dispatcher (coming in 1.3) would launch the AppMaster.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5091) Formalize the AppMaster environment for docker compability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710479#comment-15710479
 ] 

ASF GitHub Bot commented on FLINK-5091:
---

GitHub user EronWright opened a pull request:

https://github.com/apache/flink/pull/2915

[FLINK-5091] Formalize the Mesos AppMaster environment for docker 
compatibility

Fixes FLINK-5091.

- introduced ContainerSpecification.
- reworked how the TM container environment is constructed; eliminated
special-case environment variables, file layout.
- added dynamic configuration support to GlobalConfiguration.
- integrated the SecurityContext into AM/TM runners.
- added config setting for Mesos framework user.
- support DCOS side-channel authentication.
- set the FS default scheme.
- made the artifact server more generic (no assumption about existence
of dispatcher, Path-based).
- moved some test code related to overriding the JVM’s env.
- moved the Mesos containerizer config code to the 
MesosTaskManagerParameters.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/EronWright/flink feature-FLINK-5091

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2915.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2915


commit 82342491b2996f5086fd17f3b24ca588e783b8d4
Author: wrighe3 
Date:   2016-11-17T19:33:40Z

[FLINK-4921] Upgrade to Mesos 1.0.1

Updated the Mesos dependency, to unlock some new features (notably the
ability to fetch into sandbox sub-directories).

Shaded the protobuf dependency because the new Mesos library depends on
a newer version than does akka-remoting.

commit abaf2b4b3eb7c9978bdbcdc4d8c59617b86be664
Author: wrighe3 
Date:   2016-11-30T08:34:58Z

[FLINK-5091] Formalize the Mesos AppMaster environment for docker 
compability

Fixes FLINK-5091 and FLINK-4826.

- introduced ContainerSpecification.
- reworked how the TM container environment is constructed; eliminated
special-case environment variables, file layout.
- added dynamic configuration support to GlobalConfiguration.
- integrated the SecurityContext into AM/TM runners.
- added config setting for Mesos framework user.
- support DCOS side-channel authentication.
- set the FS default scheme.
- made the artifact server more generic (no assumption about existence
of dispatcher, Path-based).
- moved some test code related to overriding the JVM’s env.

commit e0db6f4c3d8630b11ed022453de2296e0d5c03ef
Author: wrighe3 
Date:   2016-11-30T23:22:00Z

Merge branch 'master' of https://github.com/apache/flink into 
feature-FLINK-5091

commit 08f9921eb3d5883d5ab1c1f786a4cac9f777b310
Author: wrighe3 
Date:   2016-12-01T01:25:41Z

[FLINK-5091] Formalize the Mesos AppMaster environment for docker 
compatibility

- remove dead code




> Formalize the AppMaster environment for docker compability
> --
>
> Key: FLINK-5091
> URL: https://issues.apache.org/jira/browse/FLINK-5091
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management, Mesos
>Reporter: Eron Wright 
>Assignee: Eron Wright 
> Fix For: 1.2.0
>
>
> For scenarios where the AppMaster is launched from a docker image, it would 
> be ideal to use the installed Flink rather than rely on a special file layout 
> in the sandbox directory.
> This is related to DCOS integration, which (in 1.2) will launch the AppMaster 
> via Marathon (as a top-level DCOS service).  The existing code assumed that 
> only the dispatcher (coming in 1.3) would launch the AppMaster.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5091) Formalize the AppMaster environment for docker compability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15709299#comment-15709299
 ] 

ASF GitHub Bot commented on FLINK-5091:
---

Github user EronWright closed the pull request at:

https://github.com/apache/flink/pull/2904


> Formalize the AppMaster environment for docker compability
> --
>
> Key: FLINK-5091
> URL: https://issues.apache.org/jira/browse/FLINK-5091
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management, Mesos
>Reporter: Eron Wright 
>Assignee: Eron Wright 
> Fix For: 1.2.0
>
>
> For scenarios where the AppMaster is launched from a docker image, it would 
> be ideal to use the installed Flink rather than rely on a special file layout 
> in the sandbox directory.
> This is related to DCOS integration, which (in 1.2) will launch the AppMaster 
> via Marathon (as a top-level DCOS service).  The existing code assumed that 
> only the dispatcher (coming in 1.3) would launch the AppMaster.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-5091) Formalize the AppMaster environment for docker compability

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15707963#comment-15707963
 ] 

ASF GitHub Bot commented on FLINK-5091:
---

GitHub user EronWright opened a pull request:

https://github.com/apache/flink/pull/2904

[FLINK-5091] Formalize the Mesos AppMaster environment for docker 
compatibility

Fixes FLINK-5091 and FLINK-4826.

- introduced ContainerSpecification.
- reworked how the TM container environment is constructed; eliminated
special-case environment variables, file layout.
- added dynamic configuration support to GlobalConfiguration.
- integrated the SecurityContext into AM/TM runners.
- added config setting for Mesos framework user.
- support DCOS side-channel authentication.
- set the FS default scheme.
- made the artifact server more generic (no assumption about existence
of dispatcher, Path-based).
- moved some test code related to overriding the JVM’s env.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/EronWright/flink feature-FLINK-5091

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2904.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2904


commit 82342491b2996f5086fd17f3b24ca588e783b8d4
Author: wrighe3 
Date:   2016-11-17T19:33:40Z

[FLINK-4921] Upgrade to Mesos 1.0.1

Updated the Mesos dependency, to unlock some new features (notably the
ability to fetch into sandbox sub-directories).

Shaded the protobuf dependency because the new Mesos library depends on
a newer version than does akka-remoting.

commit abaf2b4b3eb7c9978bdbcdc4d8c59617b86be664
Author: wrighe3 
Date:   2016-11-30T08:34:58Z

[FLINK-5091] Formalize the Mesos AppMaster environment for docker 
compability

Fixes FLINK-5091 and FLINK-4826.

- introduced ContainerSpecification.
- reworked how the TM container environment is constructed; eliminated
special-case environment variables, file layout.
- added dynamic configuration support to GlobalConfiguration.
- integrated the SecurityContext into AM/TM runners.
- added config setting for Mesos framework user.
- support DCOS side-channel authentication.
- set the FS default scheme.
- made the artifact server more generic (no assumption about existence
of dispatcher, Path-based).
- moved some test code related to overriding the JVM’s env.




> Formalize the AppMaster environment for docker compability
> --
>
> Key: FLINK-5091
> URL: https://issues.apache.org/jira/browse/FLINK-5091
> Project: Flink
>  Issue Type: Sub-task
>  Components: Cluster Management, Mesos
>Reporter: Eron Wright 
>Assignee: Eron Wright 
> Fix For: 1.2.0
>
>
> For scenarios where the AppMaster is launched from a docker image, it would 
> be ideal to use the installed Flink rather than rely on a special file layout 
> in the sandbox directory.
> This is related to DCOS integration, which (in 1.2) will launch the AppMaster 
> via Marathon (as a top-level DCOS service).  The existing code assumed that 
> only the dispatcher (coming in 1.3) would launch the AppMaster.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)