[GitHub] flink pull request #2915: [FLINK-5091] Formalize the Mesos AppMaster environ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2915


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2915: [FLINK-5091] Formalize the Mesos AppMaster environ...

[mxm]

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2915#discussion_r90664938
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/overlays/SSLStoreOverlay.java
 ---
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.clusterframework.overlays;
+
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.clusterframework.ContainerSpecification;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+import java.io.File;
+import java.io.IOException;
+
+
+/**
+ * Overlays an SSL keystore/truststore into a container.
+ *
+ * The following files are placed into the container:
+ *  - keystore.jks
+ *  - truststore.jks
+ *
+ * The following Flink configuration entries are set:
+ *  - security.ssl.keystore
+ *  - security.ssl.truststore
+ */
+public class SSLStoreOverlay extends AbstractContainerOverlay {
+
+   private static final Logger LOG = 
LoggerFactory.getLogger(SSLStoreOverlay.class);
+
+   static final Path TARGET_KEYSTORE_PATH = new Path("keystore.jks");
+   static final Path TARGET_TRUSTSTORE_PATH = new Path("truststore.jks");
+
+   final Path keystore;
+   final Path truststore;
+
+   public SSLStoreOverlay(@Nullable File keystoreFile, @Nullable File 
truststoreFile) {
+   this.keystore = keystoreFile != null ? new 
Path(keystoreFile.toURI()) : null;
+   this.truststore = truststoreFile != null ? new 
Path(truststoreFile.toURI()) : null;
+   }
+
+   @Override
+   public void configure(ContainerSpecification container) throws 
IOException {
+   if(keystore != null) {
+   
container.getArtifacts().add(ContainerSpecification.Artifact.newBuilder()
+   .setSource(keystore)
+   .setDest(TARGET_KEYSTORE_PATH)
+   .setCachable(false)
+   .build());
+   
container.getDynamicConfiguration().setString(ConfigConstants.SECURITY_SSL_KEYSTORE,
 TARGET_KEYSTORE_PATH.getPath());
+   }
+   if(truststore != null) {
+   
container.getArtifacts().add(ContainerSpecification.Artifact.newBuilder()
+   .setSource(truststore)
+   .setDest(TARGET_TRUSTSTORE_PATH)
+   .setCachable(false)
+   .build());
+   
container.getDynamicConfiguration().setString(ConfigConstants.SECURITY_SSL_TRUSTSTORE,
 TARGET_TRUSTSTORE_PATH.getPath());
+   }
+   }
+
+   public static Builder newBuilder() {
+   return new Builder();
+   }
+
+   /**
+* A builder for the {@link Krb5ConfOverlay}.
+*/
+   public static class Builder {
+
+   File keystorePath;
+
+   File truststorePath;
+
+   /**
+* Configures the overlay using the current environment (and 
global configuration).
+*
+* The following Flink configuration settings are used to 
source the keystore and truststore:
+*  - security.ssl.keystore
+*  - security.ssl.truststore
+ */
--- End diff --

indention is off here


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2915: [FLINK-5091] Formalize the Mesos AppMaster environ...

[mxm]

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2915#discussion_r90664759
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/overlays/HadoopUserOverlay.java
 ---
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.clusterframework.overlays;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.clusterframework.ContainerSpecification;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+import java.io.IOException;
+
+/**
+ * Overlays a Hadoop user context into a container.
+ *
+ * The overlay essentially configures Hadoop's {@link 
UserGroupInformation} class,
+ * establishing the effective username for filesystem calls to HDFS in 
non-secure clusters.
+ *
+ * In secure clusters, the configured keytab establishes the effective 
user.
+ *
+ * The following environment variables are set in the container:
+ *  - HADOOP_USER_NAME
+ */
+public class HadoopUserOverlay implements ContainerOverlay {
+
+   private static final Logger LOG = 
LoggerFactory.getLogger(HadoopUserOverlay.class);
+
+   private final UserGroupInformation ugi;
+
+   public HadoopUserOverlay(@Nullable UserGroupInformation ugi) {
+   this.ugi = ugi;
+   }
+
+   @Override
+   public void configure(ContainerSpecification container) throws 
IOException {
+   if(ugi != null) {
+   // overlay the Hadoop user identity (w/ tokens)
+   
container.getEnvironmentVariables().put("HADOOP_USER_NAME", ugi.getUserName());
+   }
+   }
+
+   public static Builder newBuilder() {
+   return new Builder();
+   }
+
+   /**
+* A builder for the {@link HadoopUserOverlay}.
+*/
+   public static class Builder {
+
+   UserGroupInformation ugi;
+
+   /**
+* Configures the overlay using the current Hadoop user 
information (from {@link UserGroupInformation}).
+ */
--- End diff --

indention is off here


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2915: [FLINK-5091] Formalize the Mesos AppMaster environ...

[mxm]

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2915#discussion_r90666953
  
--- Diff: 
flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/test/util/TestBaseUtils.java
 ---
@@ -542,42 +542,10 @@ protected static File asFile(String path) {
return configs;
}
 
-   // This code is taken from: http://stackoverflow.com/a/7201825/568695
-   // it changes the environment variables of this JVM. Use only for 
testing purposes!
-   @SuppressWarnings("unchecked")
public static void setEnv(Map newenv) {
--- End diff --

How about removing this method and redirecting the calls? Was there a 
dependency management reason you didn't want to do that?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2915: [FLINK-5091] Formalize the Mesos AppMaster environ...

[mxm]

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2915#discussion_r90454287
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/overlays/ContainerOverlay.java
 ---
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.clusterframework.overlays;
+
+import org.apache.flink.runtime.clusterframework.ContainerSpecification;
+
+import java.io.IOException;
+
+/**
+ * A container overlay to produce a container specification.
+ *
+ * An overlay applies configuration elements, environment variables,
+ * system properties, and artifacts to a container specification.
+ */
+public interface ContainerOverlay {
+
+   /**
+* Configure the given container specification.
+ */
--- End diff --

Indention is off here


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2915: [FLINK-5091] Formalize the Mesos AppMaster environ...

[mxm]

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2915#discussion_r90454802
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/overlays/AbstractContainerOverlay.java
 ---
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.clusterframework.overlays;
+
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.clusterframework.ContainerSpecification;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.FileVisitResult;
+import java.nio.file.Files;
+import java.nio.file.SimpleFileVisitor;
+import java.nio.file.attribute.BasicFileAttributes;
+
+/**
+ * An abstract container overlay.
--- End diff --

Could you elaborate a bit here?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2915: [FLINK-5091] Formalize the Mesos AppMaster environ...

[mxm]

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2915#discussion_r90664795
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/overlays/Krb5ConfOverlay.java
 ---
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.clusterframework.overlays;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.clusterframework.ContainerSpecification;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+import java.io.File;
+import java.io.IOException;
+
+
+/**
+ * Overlays a Kerberos configuration file into a container.
+ *
+ * The following files are copied to the container:
+ *  - krb5.conf
+ *
+ * The following Java system properties are set in the container:
+ *  - java.security.krb5.conf
+ */
+public class Krb5ConfOverlay extends AbstractContainerOverlay {
+
+   private static final Logger LOG = 
LoggerFactory.getLogger(Krb5ConfOverlay.class);
+
+   static final String JAVA_SECURITY_KRB5_CONF = "java.security.krb5.conf";
+
+   static final Path TARGET_PATH = new Path("krb5.conf");
+   final Path krb5Conf;
+
+   public Krb5ConfOverlay(@Nullable File krb5Conf) {
+   this.krb5Conf = krb5Conf != null ? new Path(krb5Conf.toURI()) : 
null;
+   }
+
+   public Krb5ConfOverlay(@Nullable Path krb5Conf) {
+   this.krb5Conf = krb5Conf;
+   }
+
+   @Override
+   public void configure(ContainerSpecification container) throws 
IOException {
+   if(krb5Conf != null) {
+   
container.getArtifacts().add(ContainerSpecification.Artifact.newBuilder()
+   .setSource(krb5Conf)
+   .setDest(TARGET_PATH)
+   .setCachable(true)
+   .build());
+   
container.getSystemProperties().setString(JAVA_SECURITY_KRB5_CONF, 
TARGET_PATH.getPath());
+   }
+   }
+
+   public static Builder newBuilder() {
+   return new Builder();
+   }
+
+   /**
+* A builder for the {@link Krb5ConfOverlay}.
+*/
+   public static class Builder {
+
+   File krb5ConfPath;
+
+   /**
+* Configures the overlay using the current environment.
+*
+* Locates the krb5.conf configuration file as per
+* https://docs.oracle.com/javase/8/docs/technotes/guides/security/jgss/tutorials/KerberosReq.html;>Java
 documentation.
+* Note that the JRE doesn't support the KRB5_CONFIG 
environment variable (JDK-7045913).
+ */
--- End diff --

indention is off here


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2915: [FLINK-5091] Formalize the Mesos AppMaster environ...

[mxm]

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2915#discussion_r90665396
  
--- Diff: 
flink-runtime/src/test/java/org/apache/flink/runtime/clusterframework/overlays/HadoopConfOverlayTest.java
 ---
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.clusterframework.overlays;
+
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.core.testutils.CommonTestUtils;
+import org.apache.flink.runtime.clusterframework.ContainerSpecification;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+
+import static 
org.apache.flink.runtime.clusterframework.overlays.HadoopConfOverlay.TARGET_CONF_DIR;
+
+public class HadoopConfOverlayTest extends ContainerOverlayTestBase {
+
+   @Rule
+   public TemporaryFolder tempFolder = new TemporaryFolder();
+
+   @Test
+   public void testConfigure() throws Exception {
+
+   File confDir = tempFolder.newFolder();
+   initConfDir(confDir);
+
+   HadoopConfOverlay overlay = new HadoopConfOverlay(confDir);
+
+   ContainerSpecification spec = new ContainerSpecification();
+   overlay.configure(spec);
+
+   assertEquals(TARGET_CONF_DIR.getPath(), 
spec.getEnvironmentVariables().get("HADOOP_CONF_DIR"));
+   assertEquals(TARGET_CONF_DIR.getPath(), 
spec.getDynamicConfiguration().getString(ConfigConstants.PATH_HADOOP_CONFIG, 
null));
+
+   checkArtifact(spec, new Path(TARGET_CONF_DIR, "core-site.xml"));
+   checkArtifact(spec, new Path(TARGET_CONF_DIR, "hdfs-site.xml"));
+   }
+
+   @Test
+   public void testNoConf() throws Exception {
+   HadoopConfOverlay overlay = new HadoopConfOverlay(null);
+
+   ContainerSpecification containerSpecification = new 
ContainerSpecification();
+   overlay.configure(containerSpecification);
+   }
+
+   @Test
+   public void testBuilderFromEnvironment() throws Exception {
--- End diff --

indention is off here


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2915: [FLINK-5091] Formalize the Mesos AppMaster environ...

[mxm]

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2915#discussion_r90664782
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/overlays/KeytabOverlay.java
 ---
@@ -0,0 +1,102 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.clusterframework.overlays;
+
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.clusterframework.ContainerSpecification;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+import java.io.File;
+import java.io.IOException;
+
+
+/**
+ * Overlays cluster-level Kerberos credentials (i.e. keytab) into a 
container.
+ *
+ * The folloowing Flink configuration entries are updated:
+ *  - security.keytab
+ */
+public class KeytabOverlay extends AbstractContainerOverlay {
+
+   private static final Logger LOG = 
LoggerFactory.getLogger(KeytabOverlay.class);
+
+   static final Path TARGET_PATH = new Path("krb5.keytab");
+
+   final Path keytab;
+
+   public KeytabOverlay(@Nullable File keytab) {
+   this.keytab = keytab != null ? new Path(keytab.toURI()) : null;
+   }
+
+   public KeytabOverlay(@Nullable Path keytab) {
+   this.keytab = keytab;
+   }
+
+   @Override
+   public void configure(ContainerSpecification container) throws 
IOException {
+   if(keytab != null) {
+   
container.getArtifacts().add(ContainerSpecification.Artifact.newBuilder()
+   .setSource(keytab)
+   .setDest(TARGET_PATH)
+   .setCachable(false)
+   .build());
+   
container.getDynamicConfiguration().setString(ConfigConstants.SECURITY_KEYTAB_KEY,
 TARGET_PATH.getPath());
+   }
+   }
+
+   public static Builder newBuilder() {
+   return new Builder();
+   }
+
+   /**
+* A builder for the {@link HadoopUserOverlay}.
+*/
+   public static class Builder {
+
+   File keytabPath;
+
+   /**
+* Configures the overlay using the current environment (and 
global configuration).
+*
+* The following Flink configuration settings are checked for a 
keytab:
+*  - security.keytab
+ */
--- End diff --

indention is off here


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2915: [FLINK-5091] Formalize the Mesos AppMaster environ...

[mxm]

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2915#discussion_r90666707
  
--- Diff: 
flink-runtime/src/test/java/org/apache/flink/runtime/clusterframework/overlays/HadoopConfOverlayTest.java
 ---
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.clusterframework.overlays;
+
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.core.testutils.CommonTestUtils;
+import org.apache.flink.runtime.clusterframework.ContainerSpecification;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.junit.Assert.assertEquals;
+
+import static 
org.apache.flink.runtime.clusterframework.overlays.HadoopConfOverlay.TARGET_CONF_DIR;
+
+public class HadoopConfOverlayTest extends ContainerOverlayTestBase {
+
+   @Rule
+   public TemporaryFolder tempFolder = new TemporaryFolder();
+
+   @Test
+   public void testConfigure() throws Exception {
+
+   File confDir = tempFolder.newFolder();
+   initConfDir(confDir);
+
+   HadoopConfOverlay overlay = new HadoopConfOverlay(confDir);
+
+   ContainerSpecification spec = new ContainerSpecification();
+   overlay.configure(spec);
+
+   assertEquals(TARGET_CONF_DIR.getPath(), 
spec.getEnvironmentVariables().get("HADOOP_CONF_DIR"));
+   assertEquals(TARGET_CONF_DIR.getPath(), 
spec.getDynamicConfiguration().getString(ConfigConstants.PATH_HADOOP_CONFIG, 
null));
+
+   checkArtifact(spec, new Path(TARGET_CONF_DIR, "core-site.xml"));
+   checkArtifact(spec, new Path(TARGET_CONF_DIR, "hdfs-site.xml"));
+   }
+
+   @Test
+   public void testNoConf() throws Exception {
+   HadoopConfOverlay overlay = new HadoopConfOverlay(null);
+
+   ContainerSpecification containerSpecification = new 
ContainerSpecification();
+   overlay.configure(containerSpecification);
+   }
+
+   @Test
+   public void testBuilderFromEnvironment() throws Exception {
+
+   // verify that the builder picks up various environment 
locations
+   HadoopConfOverlay.Builder builder;
+   Map env;
+
+   // fs.hdfs.hadoopconf
+   File confDir = tempFolder.newFolder();
+   initConfDir(confDir);
+   Configuration conf = new Configuration();
+   conf.setString(ConfigConstants.PATH_HADOOP_CONFIG, 
confDir.getAbsolutePath());
+   builder = HadoopConfOverlay.newBuilder().fromEnvironment(conf);
+   assertEquals(confDir, builder.hadoopConfDir);
+
+   // HADOOP_CONF_DIR
+   env = new HashMap(System.getenv());
+   env.remove("HADOOP_HOME");
+   env.put("HADOOP_CONF_DIR", confDir.getAbsolutePath());
+   CommonTestUtils.setEnv(env);
+   builder = HadoopConfOverlay.newBuilder().fromEnvironment(new 
Configuration());
+   assertEquals(confDir, builder.hadoopConfDir);
+
+   // HADOOP_HOME/conf
+   File homeDir = tempFolder.newFolder();
+   confDir = initConfDir(new File(homeDir, "conf"));
+   env = new HashMap(System.getenv());
+   env.remove("HADOOP_CONF_DIR");
+   env.put("HADOOP_HOME", homeDir.getAbsolutePath());
--- End diff --

Should we restore the original environment for all test cases in an 
@AfterTest method?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the 

[GitHub] flink pull request #2915: [FLINK-5091] Formalize the Mesos AppMaster environ...

[EronWright]

Github user EronWright commented on a diff in the pull request:

https://github.com/apache/flink/pull/2915#discussion_r90498149
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java ---
@@ -1372,6 +1381,12 @@
/** The environment variable name which contains the location of the 
lib folder */
public static final String ENV_FLINK_LIB_DIR = "FLINK_LIB_DIR";
 
+   /** The environment variable name which contains the location of the 
bin directory */
+   public static final String ENV_FLINK_BIN_DIR = "FLINK_BIN_DIR";
+
+   /** The environment variable name which contains the Flink installation 
root directory */
+   public static final String ENV_FLINK_HOME_DIR = "FLINK_HOME";
+
--- End diff --

@Makman2  the standalone TM relies on the `FLINK_CONF_DIR` environment 
variable to find its configuration.  Now the Mesos TM does too, for 
consistency, rather than looking to the working directory.   

When the AM launches the TM as a Mesos task, it installs Flink to a 
subdirectory within the sandbox directory, and sets the `FLINK_HOME` 
environment variable accordingly.   The variable value is a relative path 
(based on the sandbox directory); it is assumed that the TM working directory 
is the sandbox directory.

Does this address your concern?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2915: [FLINK-5091] Formalize the Mesos AppMaster environ...

[Makman2]

Github user Makman2 commented on a diff in the pull request:

https://github.com/apache/flink/pull/2915#discussion_r90423889
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java ---
@@ -1372,6 +1381,12 @@
/** The environment variable name which contains the location of the 
lib folder */
public static final String ENV_FLINK_LIB_DIR = "FLINK_LIB_DIR";
 
+   /** The environment variable name which contains the location of the 
bin directory */
+   public static final String ENV_FLINK_BIN_DIR = "FLINK_BIN_DIR";
+
+   /** The environment variable name which contains the Flink installation 
root directory */
+   public static final String ENV_FLINK_HOME_DIR = "FLINK_HOME";
+
--- End diff --

you sure these should be config values? What if someone wants to set these 
to `$MESOS_SANDBOX`, then we need to rely on shell expansion on an already 
started container^^ (Not sure this is a usecase)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2915: [FLINK-5091] Formalize the Mesos AppMaster environ...

[Makman2]

Github user Makman2 commented on a diff in the pull request:

https://github.com/apache/flink/pull/2915#discussion_r90423588
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/configuration/GlobalConfiguration.java
 ---
@@ -39,12 +39,31 @@
 
public static final String FLINK_CONF_FILENAME = "flink-conf.yaml";
 
+
--- End diff --

unrelated change :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2915: [FLINK-5091] Formalize the Mesos AppMaster environ...

[EronWright]

GitHub user EronWright opened a pull request:

https://github.com/apache/flink/pull/2915

[FLINK-5091] Formalize the Mesos AppMaster environment for docker 
compatibility

Fixes FLINK-5091.

- introduced ContainerSpecification.
- reworked how the TM container environment is constructed; eliminated
special-case environment variables, file layout.
- added dynamic configuration support to GlobalConfiguration.
- integrated the SecurityContext into AM/TM runners.
- added config setting for Mesos framework user.
- support DCOS side-channel authentication.
- set the FS default scheme.
- made the artifact server more generic (no assumption about existence
of dispatcher, Path-based).
- moved some test code related to overriding the JVM’s env.
- moved the Mesos containerizer config code to the 
MesosTaskManagerParameters.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/EronWright/flink feature-FLINK-5091

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/2915.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2915


commit 82342491b2996f5086fd17f3b24ca588e783b8d4
Author: wrighe3 
Date:   2016-11-17T19:33:40Z

[FLINK-4921] Upgrade to Mesos 1.0.1

Updated the Mesos dependency, to unlock some new features (notably the
ability to fetch into sandbox sub-directories).

Shaded the protobuf dependency because the new Mesos library depends on
a newer version than does akka-remoting.

commit abaf2b4b3eb7c9978bdbcdc4d8c59617b86be664
Author: wrighe3 
Date:   2016-11-30T08:34:58Z

[FLINK-5091] Formalize the Mesos AppMaster environment for docker 
compability

Fixes FLINK-5091 and FLINK-4826.

- introduced ContainerSpecification.
- reworked how the TM container environment is constructed; eliminated
special-case environment variables, file layout.
- added dynamic configuration support to GlobalConfiguration.
- integrated the SecurityContext into AM/TM runners.
- added config setting for Mesos framework user.
- support DCOS side-channel authentication.
- set the FS default scheme.
- made the artifact server more generic (no assumption about existence
of dispatcher, Path-based).
- moved some test code related to overriding the JVM’s env.

commit e0db6f4c3d8630b11ed022453de2296e0d5c03ef
Author: wrighe3 
Date:   2016-11-30T23:22:00Z

Merge branch 'master' of https://github.com/apache/flink into 
feature-FLINK-5091

commit 08f9921eb3d5883d5ab1c1f786a4cac9f777b310
Author: wrighe3 
Date:   2016-12-01T01:25:41Z

[FLINK-5091] Formalize the Mesos AppMaster environment for docker 
compatibility

- remove dead code




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---