[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[John Lonergan (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17013767#comment-17013767
 ] 

John Lonergan commented on FLINK-3929:
--

This approach alone doesn't consider the impact of password changes on the 
kerberos keytab and session. 
I imagine many apps exist in env's where a password must be rolled every so 
often. 

When this happens then the distributed keytab will be invalidated and the job 
will fail. 
What options are there to avoid this failure?
In general we have to assume that a process beyond our control will peridically 
roll the  password and there will be no notification to our job.

Therefore presumably our job needs to either be able to attempt recovery from 
this (a just in time attempt at recreating the keytab) or we need a process 
that preemptively refreshes the keytab so that the next call to 
UserGroupInformation.loginFromKeytab in the HDFS client (or wherever) causes 
the new keytab to be loaded.

And of course this will depend on the coding of the client lib to cooperate 
with the refresh of the keytab.

What is the scheme for long running considering password rolls?



> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>  Components: Runtime / Coordination
>Reporter: Eron Wright
>Assignee: Vijay Srinivasaraghavan
>Priority: Major
>  Labels: kerberos, security
> Fix For: 1.2.0
>
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15507665#comment-15507665
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/2275


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503996#comment-15503996
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2275
  
Tests seem to pass on my Travis. Running some more tests to ensure 
everything is running smoothly. Thank you for bearing with me!


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503262#comment-15503262
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79356049
  
--- Diff: 
flink-yarn-tests/src/test/java/org/apache/flink/yarn/YarnTestBase.java ---
@@ -374,7 +409,39 @@ public static void startYARNWithConfig(Configuration 
conf) {
File flinkConfDirPath = findFile(flinkDistRootDir, new 
ContainsName(new String[]{"flink-conf.yaml"}));
Assert.assertNotNull(flinkConfDirPath);
 
-   map.put(ConfigConstants.ENV_FLINK_CONF_DIR, 
flinkConfDirPath.getParent());
+   if(!StringUtils.isBlank(principal) && 
!StringUtils.isBlank(keytab)) {
+   //copy conf dir to test temporary workspace 
location
+   tempConfPathForSecureRun = 
tmp.newFolder("conf");
+
+   String confDirPath = 
flinkConfDirPath.getParentFile().getAbsolutePath();
+   FileUtils.copyDirectory(new File(confDirPath), 
tempConfPathForSecureRun);
+
+   try(FileWriter fw = new FileWriter(new 
File(tempConfPathForSecureRun,"flink-conf.yaml"), true);
+   BufferedWriter bw = new 
BufferedWriter(fw);
+   PrintWriter out = new PrintWriter(bw))
+   {
+   LOG.info("writing keytab: " + keytab + 
" and principal: " + principal + " to config file");
+   out.println("");
+   out.println("#Security Configurations 
Auto Populated ");
+   
out.println(ConfigConstants.SECURITY_KEYTAB_KEY + ": " + keytab);
+   
out.println(ConfigConstants.SECURITY_PRINCIPAL_KEY + ": " + principal);
+   out.println("");
+   } catch (IOException e) {
+   LOG.error("Exception occured while 
trying to append the security configurations. Reason: {}", e.getMessage());
--- End diff --

Should be `LOG.error("Exception occured while trying to append the security 
configurations.", e);`


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503258#comment-15503258
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79355282
  
--- Diff: 
flink-yarn/src/main/java/org/apache/flink/yarn/cli/FlinkYarnSessionCli.java ---
@@ -523,6 +544,7 @@ public YarnClusterClient createCluster(String 
applicationName, CommandLine cmdLi
try {
return yarnClusterDescriptor.deploy();
} catch (Exception e) {
+   LOG.error("Error while deploying YARN cluster: 
"+e.getMessage(), e);
--- End diff --

The message will be printed anyways when you pass the exception to the 
`error(String msg, Throwable t)` method.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503261#comment-15503261
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79355908
  
--- Diff: 
flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/test/util/SecureTestEnvironment.java
 ---
@@ -0,0 +1,249 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.test.util;
+
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.security.SecurityContext;
+import org.apache.hadoop.minikdc.MiniKdc;
+import org.junit.rules.TemporaryFolder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+import java.io.File;
+import java.io.FileWriter;
+import java.io.BufferedWriter;
+import java.io.PrintWriter;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Properties;
+
+/**
+ * Helper {@link SecureTestEnvironment} to handle MiniKDC lifecycle.
+ * This class can be used to start/stop MiniKDC and create secure 
configurations for MiniDFSCluster
+ * and MiniYarn
+ */
+
+public class SecureTestEnvironment {
+
+   protected static final Logger LOG = 
LoggerFactory.getLogger(SecureTestEnvironment.class);
+
+   private static MiniKdc kdc;
+
+   private static String testKeytab = null;
+
+   private static String testPrincipal = null;
+
+   private static String testZkServerPrincipal = null;
+
+   private static String testZkClientPrincipal = null;
+
+   private static String testKafkaServerPrincipal = null;
+
+   private static String hadoopServicePrincipal = null;
+
+   private static File baseDirForSecureRun = null;
+
+   public static void prepare(TemporaryFolder tempFolder) {
+
+   try {
+   baseDirForSecureRun = tempFolder.newFolder();
+   LOG.info("Base Directory for Secure Environment: {}", 
baseDirForSecureRun);
+
+   String hostName = "localhost";
+   Properties kdcConf = MiniKdc.createConf();
+   if(LOG.isDebugEnabled()) {
+   kdcConf.setProperty(MiniKdc.DEBUG, "true");
+   }
+   kdcConf.setProperty(MiniKdc.KDC_BIND_ADDRESS, hostName);
+   kdc = new MiniKdc(kdcConf, baseDirForSecureRun);
+   kdc.start();
+   LOG.info("Started Mini KDC");
+
+   File keytabFile = new File(baseDirForSecureRun, 
"test-users.keytab");
+   testKeytab = keytabFile.getAbsolutePath();
+   testZkServerPrincipal = "zookeeper/127.0.0.1";
+   testZkClientPrincipal = "zk-client/127.0.0.1";
+   testKafkaServerPrincipal = "kafka/" + hostName;
+   hadoopServicePrincipal = "hadoop/" + hostName;
+   testPrincipal = "client/" + hostName;
+
+   kdc.createPrincipal(keytabFile, testPrincipal, 
testZkServerPrincipal,
+   hadoopServicePrincipal,
+   testZkClientPrincipal,
+   testKafkaServerPrincipal);
+
+   testPrincipal = testPrincipal + "@" + kdc.getRealm();
+   testZkServerPrincipal = testZkServerPrincipal + "@" + 
kdc.getRealm();
+   testZkClientPrincipal = testZkClientPrincipal + "@" + 
kdc.getRealm();
+   testKafkaServerPrincipal = testKafkaServerPrincipal + 
"@" + kdc.getRealm();
+   hadoopServicePrincipal = 

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503255#comment-15503255
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79355653
  
--- Diff: 
flink-yarn/src/main/java/org/apache/flink/yarn/YarnTaskManagerRunner.java ---
@@ -75,34 +85,66 @@ public static void runYarnTaskManager(String[] args, 
final Class toks : 
UserGroupInformation.getCurrentUser().getTokens()) {
-   ugi.addToken(toks);
+   String keytabPath = null;
+   if(remoteKeytabPath != null) {
+   File f = new File(currDir, Utils.KEYTAB_FILE_NAME);
+   keytabPath = f.getAbsolutePath();
+   LOG.info("keytabPath: {}", keytabPath);
}
 
+   UserGroupInformation currentUser = 
UserGroupInformation.getCurrentUser();
+
+   LOG.info("YARN daemon is running as: {} Yarn client user 
obtainer: {}",
+   currentUser.getShortUserName(), 
yarnClientUsername );
+
// Infer the resource identifier from the environment variable
String containerID = 
Preconditions.checkNotNull(envs.get(YarnFlinkResourceManager.ENV_FLINK_CONTAINER_ID));
final ResourceID resourceId = new ResourceID(containerID);
LOG.info("ResourceID assigned for this container: {}", 
resourceId);
 
-   ugi.doAs(new PrivilegedAction() {
-   @Override
-   public Object run() {
-   try {
-   
TaskManager.selectNetworkInterfaceAndRunTaskManager(configuration, resourceId, 
taskManager);
-   }
-   catch (Throwable t) {
-   LOG.error("Error while starting the 
TaskManager", t);
-   
System.exit(TaskManager.STARTUP_FAILURE_RETURN_CODE());
-   }
-   return null;
+   try {
+
+   SecurityContext.SecurityConfiguration sc = new 
SecurityContext.SecurityConfiguration();
+
+   //To support Yarn Secure Integration Test Scenario
+   File krb5Conf = new File(currDir, Utils.KRB5_FILE_NAME);
+   if(krb5Conf.exists() && krb5Conf.canRead()) {
+   String krb5Path = krb5Conf.getAbsolutePath();
+   LOG.info("KRB5 Conf: {}", krb5Path);
+   org.apache.hadoop.conf.Configuration conf = new 
org.apache.hadoop.conf.Configuration();
+   
conf.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION, 
"kerberos");
+   
conf.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHORIZATION, "true");
+   sc.setHadoopConfiguration(conf);
+   }
+
+   if(keytabPath != null && remoteKeytabPrincipal != null) 
{
+   
configuration.setString(ConfigConstants.SECURITY_KEYTAB_KEY, keytabPath);
+   
configuration.setString(ConfigConstants.SECURITY_PRINCIPAL_KEY, 
remoteKeytabPrincipal);
}
-   });
+   
configuration.setString(ConfigConstants.FLINK_BASE_DIR_PATH_KEY, currDir);
+
+   
SecurityContext.install(sc.setFlinkConfiguration(configuration));
+
+   SecurityContext.getInstalled().runSecured(new 
SecurityContext.FlinkSecuredRunner() {
+   @Override
+   public Integer run() {
+   try {
+   
TaskManager.selectNetworkInterfaceAndRunTaskManager(configuration, resourceId, 
taskManager);
+   }
+   catch (Throwable t) {
+   LOG.error("Error while starting 
the TaskManager", t);
+   
System.exit(TaskManager.STARTUP_FAILURE_RETURN_CODE());
+   }
+   return null;
+   }
+   });
+   } catch(Exception e) {
+   LOG.error("Exception occurred while launching Task 
Manager. Reason: {}", e);
--- End diff --

The signature is error(String msg, Throwable t). You can remove the 
"Reason: {}".


> Support for Kerberos Authentication with Keytab Credential
> 

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503257#comment-15503257
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79355451
  
--- Diff: pom.xml ---
@@ -107,6 +107,13 @@ under the License.
2.7.4
3.1.0
4.11
+   
+   2.7.2
--- End diff --

Does that work across different Hadoop versions?


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503263#comment-15503263
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79377160
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/security/SecurityContext.java
 ---
@@ -0,0 +1,313 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.security;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.IllegalConfigurationException;
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.security.Credentials;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.apache.hadoop.security.token.Token;
+import org.apache.hadoop.security.token.TokenIdentifier;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.security.auth.Subject;
+import java.io.File;
+import java.lang.reflect.Method;
+import java.security.PrivilegedExceptionAction;
+import java.util.Collection;
+
+/*
+ * Process-wide security context object which initializes UGI with 
appropriate security credentials and also it
+ * creates in-memory JAAS configuration object which will serve 
appropriate ApplicationConfigurationEntry for the
+ * connector login module implementation that authenticates Kerberos 
identity using SASL/JAAS based mechanism.
+ */
+@Internal
+public class SecurityContext {
+
+   private static final Logger LOG = 
LoggerFactory.getLogger(SecurityContext.class);
+
+   public static final String JAAS_CONF_FILENAME = "flink-jaas.conf";
+
+   private static final String JAVA_SECURITY_AUTH_LOGIN_CONFIG = 
"java.security.auth.login.config";
+
+   private static final String ZOOKEEPER_SASL_CLIENT = 
"zookeeper.sasl.client";
+
+   private static final String ZOOKEEPER_SASL_CLIENT_USERNAME = 
"zookeeper.sasl.client.username";
+
+   private static SecurityContext installedContext;
+
+   public static SecurityContext getInstalled() { return installedContext; 
}
+
+   private UserGroupInformation ugi;
+
+   SecurityContext(UserGroupInformation ugi) {
+   if(ugi == null) {
+   throw new RuntimeException("UGI passed cannot be null");
+   }
+   this.ugi = ugi;
+   }
+
+   public  T runSecured(final FlinkSecuredRunner runner) throws 
Exception {
+   return ugi.doAs(new PrivilegedExceptionAction() {
+   @Override
+   public T run() throws Exception {
+   return runner.run();
+   }
+   });
+   }
+
+   public static void install(SecurityConfiguration config) throws 
Exception {
+
+   // perform static initialization of UGI, JAAS
+   if(installedContext != null) {
+   LOG.warn("overriding previous security context");
+   }
+
+   // establish the JAAS config
+   JaasConfiguration jaasConfig = new 
JaasConfiguration(config.keytab, config.principal);
+   
javax.security.auth.login.Configuration.setConfiguration(jaasConfig);
+
+   populateSystemSecurityProperties(config.flinkConf);
+
+   // establish the UGI login user
+   UserGroupInformation.setConfiguration(config.hadoopConf);
+
+   UserGroupInformation loginUser;
+
+   if(UserGroupInformation.isSecurityEnabled() &&
+   config.keytab != null && 
!StringUtils.isBlank(config.principal)) {
+   String keytabPath = (new 

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503256#comment-15503256
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79355618
  
--- Diff: 
flink-yarn/src/main/java/org/apache/flink/yarn/cli/FlinkYarnSessionCli.java ---
@@ -460,9 +463,27 @@ public static void runInteractiveCli(YarnClusterClient 
yarnCluster, boolean read
}
}
 
-   public static void main(String[] args) {
-   FlinkYarnSessionCli cli = new FlinkYarnSessionCli("", ""); // 
no prefix for the YARN session
-   System.exit(cli.run(args));
+   public static void main(final String[] args) {
+   final FlinkYarnSessionCli cli = new FlinkYarnSessionCli("", "", 
true); // no prefix for the YARN session
+
+   String confDirPath = 
CliFrontend.getConfigurationDirectoryFromEnv();
+   GlobalConfiguration.loadConfiguration(confDirPath);
+   Configuration flinkConfiguration = 
GlobalConfiguration.loadConfiguration();
+   
flinkConfiguration.setString(ConfigConstants.FLINK_BASE_DIR_PATH_KEY, 
confDirPath);
+   try {
+   SecurityContext.install(new 
SecurityContext.SecurityConfiguration().setFlinkConfiguration(flinkConfiguration));
+   int retCode = 
SecurityContext.getInstalled().runSecured(new 
SecurityContext.FlinkSecuredRunner() {
+   @Override
+   public Integer run() {
+   return cli.run(args);
+   }
+   });
+   System.exit(retCode);
+   } catch(Exception e) {
+   e.printStackTrace();
+   LOG.error("Exception Occured. Reason: {}", e);
--- End diff --

The signature is `error(String msg, Throwable t)`. You can remove the 
"Reason: {}".


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503259#comment-15503259
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79355820
  
--- Diff: 
flink-streaming-connectors/flink-connector-filesystem/src/test/java/org/apache/flink/streaming/connectors/fs/RollingSinkSecuredITCase.java
 ---
@@ -0,0 +1,232 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.fs;
+
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.security.SecurityContext;
+import org.apache.flink.streaming.util.TestStreamEnvironment;
+import org.apache.flink.test.util.SecureTestEnvironment;
+import org.apache.flink.test.util.TestingSecurityContext;
+import org.apache.flink.test.util.TestBaseUtils;
+import org.apache.flink.util.NetUtils;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hadoop.http.HttpConfig;
+import org.apache.hadoop.security.SecurityUtil;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Ignore;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+
+import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_HTTP_ADDRESS_KEY;
+import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_NAMENODE_KEYTAB_FILE_KEY;
+import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_NAMENODE_USER_NAME_KEY;
+import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_USER_NAME_KEY;
+import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_KEYTAB_FILE_KEY;
+import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_WEB_AUTHENTICATION_KERBEROS_PRINCIPAL_KEY;
+import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_BLOCK_ACCESS_TOKEN_ENABLE_KEY;
+import static org.apache.hadoop.hdfs.DFSConfigKeys.DFS_HTTP_POLICY_KEY;
+import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_ENCRYPT_DATA_TRANSFER_KEY;
+import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_ADDRESS_KEY;
+import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_HOST_NAME_KEY;
+
+/**
+ * Tests for running {@link RollingSinkSecuredITCase} which is an 
extension of {@link RollingSink} in secure environment
+ */
+
+//The test is disabled since MiniDFS secure run requires lower order ports 
to be used.
+//We can enable the test when the fix is available (HDFS-9213)
+@Ignore
+public class RollingSinkSecuredITCase extends RollingSinkITCase {
+
+   protected static final Logger LOG = 
LoggerFactory.getLogger(RollingSinkSecuredITCase.class);
+
+   /*
+* override super class static methods to avoid creating MiniDFS and 
MiniFlink with wrong configurations
+* and out-of-order sequence for secure cluster
+*/
+   @BeforeClass
+   public static void setup() throws Exception {}
+
+   @AfterClass
+   public static void teardown() throws Exception {}
+
+   @BeforeClass
+   public static void createHDFS() throws IOException {}
+
+   @AfterClass
+   public static void destroyHDFS() {}
+
+   @BeforeClass
+   public static void startSecureCluster() throws Exception {
+
+   LOG.info("starting secure cluster environment for testing");
+
+   dataDir = tempFolder.newFolder();
+
+   conf.set(MiniDFSCluster.HDFS_MINIDFS_BASEDIR, 
dataDir.getAbsolutePath());
+
+   SecureTestEnvironment.prepare(tempFolder);
+
+   populateSecureConfigurations();
+
   

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15503260#comment-15503260
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79355845
  
--- Diff: 
flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/test/util/SecureTestEnvironment.java
 ---
@@ -0,0 +1,249 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.test.util;
+
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.runtime.security.SecurityContext;
+import org.apache.hadoop.minikdc.MiniKdc;
+import org.junit.rules.TemporaryFolder;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+import java.io.File;
+import java.io.FileWriter;
+import java.io.BufferedWriter;
+import java.io.PrintWriter;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Properties;
+
+/**
+ * Helper {@link SecureTestEnvironment} to handle MiniKDC lifecycle.
+ * This class can be used to start/stop MiniKDC and create secure 
configurations for MiniDFSCluster
+ * and MiniYarn
+ */
+
+public class SecureTestEnvironment {
+
+   protected static final Logger LOG = 
LoggerFactory.getLogger(SecureTestEnvironment.class);
+
+   private static MiniKdc kdc;
+
+   private static String testKeytab = null;
+
+   private static String testPrincipal = null;
+
+   private static String testZkServerPrincipal = null;
+
+   private static String testZkClientPrincipal = null;
+
+   private static String testKafkaServerPrincipal = null;
+
+   private static String hadoopServicePrincipal = null;
+
+   private static File baseDirForSecureRun = null;
+
+   public static void prepare(TemporaryFolder tempFolder) {
+
+   try {
+   baseDirForSecureRun = tempFolder.newFolder();
+   LOG.info("Base Directory for Secure Environment: {}", 
baseDirForSecureRun);
+
+   String hostName = "localhost";
+   Properties kdcConf = MiniKdc.createConf();
+   if(LOG.isDebugEnabled()) {
+   kdcConf.setProperty(MiniKdc.DEBUG, "true");
+   }
+   kdcConf.setProperty(MiniKdc.KDC_BIND_ADDRESS, hostName);
+   kdc = new MiniKdc(kdcConf, baseDirForSecureRun);
+   kdc.start();
+   LOG.info("Started Mini KDC");
+
+   File keytabFile = new File(baseDirForSecureRun, 
"test-users.keytab");
+   testKeytab = keytabFile.getAbsolutePath();
+   testZkServerPrincipal = "zookeeper/127.0.0.1";
+   testZkClientPrincipal = "zk-client/127.0.0.1";
+   testKafkaServerPrincipal = "kafka/" + hostName;
+   hadoopServicePrincipal = "hadoop/" + hostName;
+   testPrincipal = "client/" + hostName;
+
+   kdc.createPrincipal(keytabFile, testPrincipal, 
testZkServerPrincipal,
+   hadoopServicePrincipal,
+   testZkClientPrincipal,
+   testKafkaServerPrincipal);
+
+   testPrincipal = testPrincipal + "@" + kdc.getRealm();
+   testZkServerPrincipal = testZkServerPrincipal + "@" + 
kdc.getRealm();
+   testZkClientPrincipal = testZkClientPrincipal + "@" + 
kdc.getRealm();
+   testKafkaServerPrincipal = testKafkaServerPrincipal + 
"@" + kdc.getRealm();
+   hadoopServicePrincipal = 

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496973#comment-15496973
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on the issue:

https://github.com/apache/flink/pull/2275
  
@mxm I have addressed some of the review feedback and rebased to upstream 
master.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496957#comment-15496957
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79221254
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/security/SecurityContext.java
 ---
@@ -155,6 +157,58 @@ public static void install(SecurityConfiguration 
config) throws Exception {
installedContext = new SecurityContext(loginUser);
}
 
+   /*
+* This is a temporary fix to support both Kafka and ZK client libraries
+* that are expecting the system variable to determine secure cluster
+*/
+   private static void populateJaasConfigSystemProperty(Configuration 
configuration) {
+
+   //hack since Kafka Login Handler explicitly looks for the 
property or else it throws an exception
+   
//https://github.com/apache/kafka/blob/0.9.0/clients/src/main/java/org/apache/kafka/common/security/kerberos/Login.java#L289
+   if(null == configuration) {
+   System.setProperty("java.security.auth.login.config", 
"");
--- End diff --

Moved all the hard coded configuration property to static variable


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496955#comment-15496955
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79221107
  
--- Diff: 
flink-clients/src/main/java/org/apache/flink/client/CliFrontend.java ---
@@ -161,6 +161,8 @@ public CliFrontend(String configDir) throws Exception {
"filesystem scheme from configuration.", e);
}
 
+   this.config.setString(ConfigConstants.FLINK_BASE_DIR_PATH_KEY, 
configDirectory.getAbsolutePath());
--- End diff --

Good catch, though the implementation takes care of handling base directory 
and the immediate "conf" directory. Will change it to point to parent directory.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496555#comment-15496555
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79190433
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java ---
@@ -1233,6 +1239,9 @@
/** ZooKeeper default leader port. */
public static final int DEFAULT_ZOOKEEPER_LEADER_PORT = 3888;
 
+   /** Defaults for ZK client security **/
+   public static final boolean DEFAULT_ZOOKEEPER_SASL_DISABLE = true;
--- End diff --

I agree but it can be argued both ways. We could keep the default to false 
(enable SASL client auth if not disabled explicitly through configuration file) 
or expect an explicit ask to enable SASL through the configuration settings. I 
chose later since secure ZK is not a common deployment (mostly) and moreover we 
also have introduced new security configurations to enable security and one 
could configure/adjust ZK configuration at that time. 


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496542#comment-15496542
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on the issue:

https://github.com/apache/flink/pull/2275
  
>
@vijikarthi Thanks for the update. Great to see the tests are passing now. 
I'm curious, why did this issue only appear on Travis and not locally?

Kafka/ZK connection is unusually longer in Travis and I don't know the 
reason? I have noticed a similar comment in the Kafka Test code too where we 
have changed the timeout from 6 sec (default) to 30 sec. I have increased the 
timeout interval further for the secure run since occasionally it fails with 
the lower timeout settings.

The combination of longer ZK connection time and client code waiting for 
incorrect event (SysConnected) was the reason why in Travis it was failing.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496158#comment-15496158
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79156099
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java ---
@@ -1233,6 +1239,9 @@
/** ZooKeeper default leader port. */
public static final int DEFAULT_ZOOKEEPER_LEADER_PORT = 3888;
 
+   /** Defaults for ZK client security **/
+   public static final boolean DEFAULT_ZOOKEEPER_SASL_DISABLE = true;
--- End diff --

I'm wondering, shouldn't the default be true? If I configure Jaas, then I 
probably want to connect to a secured Zookeeper by default.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496153#comment-15496153
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79153742
  
--- Diff: 
flink-streaming-connectors/flink-connector-kafka-0.9/src/test/java/org/apache/flink/streaming/connectors/kafka/KafkaTestEnvironmentImpl.java
 ---
@@ -369,6 +385,11 @@ public Properties getSecureProperties() {
prop.put("security.inter.broker.protocol", 
"SASL_PLAINTEXT");
prop.put("security.protocol", "SASL_PLAINTEXT");
prop.put("sasl.kerberos.service.name", "kafka");
+
+   //add special timeout for Travis
+   prop.setProperty("zookeeper.session.timeout.ms", 
zkTimeout);
+   prop.setProperty("zookeeper.connection.timeout.ms", 
zkTimeout);
+   prop.setProperty("metadata.fetch.timeout.ms","12");
--- End diff --

That timeout seems very high.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496157#comment-15496157
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79153795
  
--- Diff: 
flink-streaming-connectors/flink-connector-kafka-0.9/src/test/java/org/apache/flink/streaming/connectors/kafka/KafkaTestEnvironmentImpl.java
 ---
@@ -141,9 +141,11 @@ public boolean isSecureRunSupported() {
@Override
public void prepare(int numKafkaServers, Properties 
additionalServerProperties, boolean secureMode) {
 
-   //increase the timeout since in Travis it is failing.
+   //increase the timeout since in Travis ZK connection takes long 
time for secure connection.
if(secureMode) {
-   zkTimeout = "27";
+   //run only one kafka server to avoid multiple ZK 
connections from many instances - Travis timeout
+   numKafkaServers = 1;
+   zkTimeout = String.valueOf(Integer.parseInt(zkTimeout) 
* 15);
--- End diff --

The timeout here seems arbitrary.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496154#comment-15496154
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79155885
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/security/SecurityContext.java
 ---
@@ -158,22 +157,37 @@ public static void install(SecurityConfiguration 
config) throws Exception {
}
 
/*
-* This is a temporary fix to support both Kafka and ZK client libraries
-* that are expecting the system variable to determine secure cluster
+* This method configures some of the system property that are require 
for ZK and Kafka SASL authentication
+* See: 
https://github.com/apache/kafka/blob/0.9.0/clients/src/main/java/org/apache/kafka/common/security/kerberos/Login.java#L289
+* See: 
https://github.com/sgroschupf/zkclient/blob/master/src/main/java/org/I0Itec/zkclient/ZkClient.java#L900
+* In this method, setting java.security.auth.login.config 
configuration is temporary hack only to support ZK and
+* Kafka current code behavior
 */
-   private static void populateJaasConfigSystemProperty(Configuration 
configuration) {
+   private static void populateSystemSecurityProperties(Configuration 
configuration) {
 
-   //hack since Kafka Login Handler explicitly looks for the 
property or else it throws an exception
-   
//https://github.com/apache/kafka/blob/0.9.0/clients/src/main/java/org/apache/kafka/common/security/kerberos/Login.java#L289
-   if(null == configuration) {
-   System.setProperty("java.security.auth.login.config", 
"");
+   //required to be empty for Kafka but we will override the 
property
+   //with pseudo JAAS configuration file if SASL auth is enabled 
for ZK
+   System.setProperty("java.security.auth.login.config", "");
+
+   if(configuration == null) {
+   return;
+   }
+
+   boolean disableSaslClient = 
configuration.getBoolean(ConfigConstants.ZOOKEEPER_SASL_DISABLE,
+   ConfigConstants.DEFAULT_ZOOKEEPER_SASL_DISABLE);
+   if(disableSaslClient) {
+   LOG.info("SASL client auth for ZK will be disabled");
+   //SASL auth is disabled by default but will be enabled 
if specified in configuration
+   System.setProperty("zookeeper.sasl.client","false");
--- End diff --

This is to disable SASL with Zookeeper if a Jaas configuration is used but 
Zookeeper is not configured to use it?


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496150#comment-15496150
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79154968
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/security/SecurityContext.java
 ---
@@ -155,6 +157,58 @@ public static void install(SecurityConfiguration 
config) throws Exception {
installedContext = new SecurityContext(loginUser);
}
 
+   /*
+* This is a temporary fix to support both Kafka and ZK client libraries
+* that are expecting the system variable to determine secure cluster
+*/
+   private static void populateJaasConfigSystemProperty(Configuration 
configuration) {
+
+   //hack since Kafka Login Handler explicitly looks for the 
property or else it throws an exception
+   
//https://github.com/apache/kafka/blob/0.9.0/clients/src/main/java/org/apache/kafka/common/security/kerberos/Login.java#L289
+   if(null == configuration) {
+   System.setProperty("java.security.auth.login.config", 
"");
+   return;
+   }
+
+   String baseDir = 
configuration.getString(ConfigConstants.FLINK_BASE_DIR_PATH_KEY, null);
+   if(baseDir == null) {
+   System.setProperty("java.security.auth.login.config", 
"");
+   return;
+   }
+
+   File f = new File(baseDir);
+   if(!f.exists() || !f.isDirectory()) {
+   LOG.error("Invalid flink base directory {} 
configuration provided", baseDir);
+   throw new IllegalConfigurationException("Invalid flink 
base directory configuration provided");
+   }
--- End diff --

This check should really be somewhere else. I think you can use 
`ENV_FLINK_CONF_DIR` which is always set. The `FLINK_BASE_DIR_PATH_KEY` should 
be deprecated. It's not used anymore.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496155#comment-15496155
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79152195
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/security/SecurityContext.java
 ---
@@ -155,6 +157,58 @@ public static void install(SecurityConfiguration 
config) throws Exception {
installedContext = new SecurityContext(loginUser);
}
 
+   /*
+* This is a temporary fix to support both Kafka and ZK client libraries
+* that are expecting the system variable to determine secure cluster
+*/
+   private static void populateJaasConfigSystemProperty(Configuration 
configuration) {
+
+   //hack since Kafka Login Handler explicitly looks for the 
property or else it throws an exception
+   
//https://github.com/apache/kafka/blob/0.9.0/clients/src/main/java/org/apache/kafka/common/security/kerberos/Login.java#L289
+   if(null == configuration) {
+   System.setProperty("java.security.auth.login.config", 
"");
--- End diff --

Could we factor out `"java.security.auth.login.config"` to a static 
variable?


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496156#comment-15496156
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79151542
  
--- Diff: 
flink-clients/src/main/java/org/apache/flink/client/CliFrontend.java ---
@@ -161,6 +161,8 @@ public CliFrontend(String configDir) throws Exception {
"filesystem scheme from configuration.", e);
}
 
+   this.config.setString(ConfigConstants.FLINK_BASE_DIR_PATH_KEY, 
configDirectory.getAbsolutePath());
--- End diff --

You're setting the base dir to the config directory here. Why?


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496151#comment-15496151
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79154987
  
--- Diff: 
flink-runtime/src/main/scala/org/apache/flink/runtime/taskmanager/TaskManager.scala
 ---
@@ -1585,6 +1585,8 @@ object TaskManager {
   }
 }
 
+conf.setString(ConfigConstants.FLINK_BASE_DIR_PATH_KEY, 
cliConfig.getConfigDir() + "/..")
--- End diff --

Here you're setting the base dir path to the root directory instead of the 
config directory.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496149#comment-15496149
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79152436
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/security/SecurityContext.java
 ---
@@ -79,9 +82,8 @@ public static void install(SecurityConfiguration config) 
throws Exception {
JaasConfiguration jaasConfig = new 
JaasConfiguration(config.keytab, config.principal);

javax.security.auth.login.Configuration.setConfiguration(jaasConfig);
 
-   //hack since Kafka Login Handler explicitly looks for the 
property or else it throws an exception
-   
//https://github.com/apache/kafka/blob/0.9.0/clients/src/main/java/org/apache/kafka/common/security/kerberos/Login.java#L289
-   System.setProperty("java.security.auth.login.config", "");
+   //temporary fix
+   populateJaasConfigSystemProperty(config.flinkConf);
--- End diff --

From my experience comments like "temporary fix" tend to stay forever. I 
would replace it with something that explains the reasoning. That helps to fix 
the code when it is reviewed later on.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496152#comment-15496152
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79153661
  
--- Diff: 
flink-streaming-connectors/flink-connector-kafka-0.9/src/test/java/org/apache/flink/streaming/connectors/kafka/KafkaTestEnvironmentImpl.java
 ---
@@ -266,23 +268,37 @@ public void createTestTopic(String topic, int 
numberOfPartitions, int replicatio
zkUtils.close();
}
 
+   LOG.info("Topic {} create request is successfully posted", 
topic);
+
// validate that the topic has been created
-   final long deadline = System.currentTimeMillis() + 3;
+   final long deadline = System.currentTimeMillis() + 
Integer.parseInt(zkTimeout);
do {
try {
-   Thread.sleep(100);
+   if(secureMode) {
+   //increase wait time since in Travis ZK 
timeout occurs frequently
+   int wait = Integer.parseInt(zkTimeout) 
/ 100;
+   LOG.info("waiting for {} msecs before 
the topic {} can be checked", wait, topic);
+   Thread.sleep(wait);
+   } else {
+   Thread.sleep(100);
--- End diff --

I would rather retry here instead of waiting.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496148#comment-15496148
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r79151877
  
--- Diff: flink-dist/src/main/flink-bin/conf/flink-jaas.conf ---
@@ -0,0 +1,26 @@

+
+#  Licensed to the Apache Software Foundation (ASF) under one
+#  or more contributor license agreements.  See the NOTICE file
+#  distributed with this work for additional information
+#  regarding copyright ownership.  The ASF licenses this file
+#  to you under the Apache License, Version 2.0 (the
+#  "License"); you may not use this file except in compliance
+#  with the License.  You may obtain a copy of the License at
+#
+#  http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+# limitations under the License.

+
+# We are using this file as an workaround for the Kafka and ZK SASL 
implementation
+# since they explicitly look for java.security.auth.login.config 
environment variable
--- End diff --

In the code link it looks like a Java property and not an environment 
variable: 
https://github.com/sgroschupf/zkclient/blob/master/src/main/java/org/I0Itec/zkclient/ZkClient.java#L900


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15496069#comment-15496069
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2275
  
@vijikarthi Thanks for the update. Great to see the tests are passing now. 
I'm curious, why did this issue only appear on Travis and not locally?


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15492419#comment-15492419
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on the issue:

https://github.com/apache/flink/pull/2275
  
@mxm The issue is apparently due to ZK client API implementation which 
lookup the system configuration property to determined the type of event 
(SASLAuthenticated/SysConnected) that it should wait for? I have fixed the 
issue by adding a dummy configuration file that will be used only to set the 
system property when SASL authentication is enabled (Apparently both Kafka and 
ZK rely on the configuration property and I believe they are moving away from 
using these variables, but for now we need the hack for secure deployment). 
Please take a look at the two new commits to find the fix details and I will 
squash the commits after your review. Travis build is not failing anymore.  


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15492393#comment-15492393
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on the issue:

https://github.com/apache/flink/pull/2275
  
@nielsbasjes Thanks for the link. The issue however is related to ZK SASL 
client API implementation and it took a while to figure out the actual cause.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15483603#comment-15483603
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user nielsbasjes commented on the issue:

https://github.com/apache/flink/pull/2275
  
Perhaps you are running into similar problems as described here: 
http://stackoverflow.com/questions/2890259/running-each-junit-test-in-a-separate-jvm-in-eclipse

#JustTryingToHelp


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15477588#comment-15477588
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on the issue:

https://github.com/apache/flink/pull/2275
  
Thanks, will try that.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15477573#comment-15477573
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2275
  
I would suggest to just you your own Travis account. It is free, you just 
need to connect it with your account. Then push to any branch in your GitHub 
account and Travis will run.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15477315#comment-15477315
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on the issue:

https://github.com/apache/flink/pull/2275
  
hmmm.. it's getting complicated. will try to debug the issue. Do you know 
why on Travis it has to fail but not on Jenkins?

How do I simulate this on Travis for my own testing?


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15476431#comment-15476431
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2275
  
Interestingly, this just fails on Travis but on Jenkins the test completes: 
https://builds.apache.org/job/flink-github-ci/434/org.apache.flink$flink-connector-kafka-0.9_2.10/testReport/org.apache.flink.streaming.connectors.kafka/


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15476299#comment-15476299
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2275
  
I would suspect the tests suffer from is a network configuration or port 
issue when they run in parallel.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15476290#comment-15476290
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2275
  
Unfortunately, that didn't fix it. The builds still consistently fails on 
Travis in `Kafka09SecureRunITCase`: 
https://travis-ci.org/apache/flink/builds/158627315

Under these circumstances, I can't merge the pull request because it would 
break our CI testing.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15475694#comment-15475694
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on the issue:

https://github.com/apache/flink/pull/2275
  
I have noticed the issue occasionally in master branch too but very 
inconsistent. I just rebased against the latest master and ran "mvn clean 
verify". I don't see any errors. I have tried couple of times to see if it 
reoccurs but it looks good. Not sure if the issue is addressed by some other 
patches in the master branch?  Could you please give it a try?


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15474208#comment-15474208
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2275
  
We will have to fix the issue before we can merge this PR. The timeout 
doesn't occur in non-secure Kafka tests. I think we need to increase the 
timeout or change the test layout. Could you look into that?


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15466699#comment-15466699
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on the issue:

https://github.com/apache/flink/pull/2275
  
rebased again with the latest master


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15465855#comment-15465855
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on the issue:

https://github.com/apache/flink/pull/2275
  
@mxm I believe the ZK timeout issue occurs from 
LocalFlinkMiniClusterITCase->testLocalFlinkMiniClusterWithMultipleTaskManagers 
test case but it is not consistent. I ran the Kafka test case alone and it 
worked. I also ran "mvn clean verify" and I don't see any errors (after couple 
of retry - same ZK timeout error from LocalFlinkMiniClusterITCase) . It looks 
like there is some inconsistency in some of the integration test scenarios.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15465144#comment-15465144
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2275
  
The Kafka tests fail:

```
Running org.apache.flink.streaming.connectors.kafka.Kafka09SecureRunITCase
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to 
zookeeper server within timeout: 3
at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1223)
at org.I0Itec.zkclient.ZkClient.(ZkClient.java:155)
at org.I0Itec.zkclient.ZkClient.(ZkClient.java:129)
at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:89)
at kafka.utils.ZkUtils$.apply(ZkUtils.scala:71)
at kafka.server.KafkaServer.initZk(KafkaServer.scala:278)
at kafka.server.KafkaServer.startup(KafkaServer.scala:168)
at 
org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl.getKafkaServer(KafkaTestEnvironmentImpl.java:336)
at 
org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl.prepare(KafkaTestEnvironmentImpl.java:170)
at 
org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironment.prepare(KafkaTestEnvironment.java:41)
at 
org.apache.flink.streaming.connectors.kafka.KafkaTestBase.startClusters(KafkaTestBase.java:131)
at 
org.apache.flink.streaming.connectors.kafka.Kafka09SecureRunITCase.prepare(Kafka09SecureRunITCase.java:45)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 47.945 sec 
<<< FAILURE! - in 
org.apache.flink.streaming.connectors.kafka.Kafka09SecureRunITCase
```

The goal should be that all builds pass on Travis before we merge this.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15464874#comment-15464874
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2275
  
Thank you. Testing again and merging if nothing fails.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15463483#comment-15463483
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on the issue:

https://github.com/apache/flink/pull/2275
  
@mxm, I was able to reproduce this issue on some other machine. The issue 
is that KRB5 config file which is mounted as a local resource is not visible 
though we set the system property (java.security.krb5.conf) within the 
container. I have modified the implementation to pass the system property as 
part of container JVM parameter. I have rebased the code against upstream 
master. Please give it a try.   


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15459081#comment-15459081
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on the issue:

https://github.com/apache/flink/pull/2275
  
@mxm, I am not seeing the error messages when I run the Yarn test. Could 
you please run "secure" test case alone and share the logs?

>
mvn test integration-test -Dtest="YARNSessionFIFOSecuredITCase" 
-Pinclude-yarn-tests -pl flink-yarn-tests -DfailIfNoTests=false


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15458351#comment-15458351
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2275
  
Yarn tests still not passing:

```
2016-09-02 11:35:04,244 ERROR 
org.apache.flink.yarn.YarnApplicationMasterRunner - YARN 
Application Master initialization failed
java.lang.IllegalArgumentException: Can't get Kerberos realm
at 
org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:65)
at 
org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:275)
at 
org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:311)
at 
org.apache.flink.runtime.security.SecurityContext.install(SecurityContext.java:87)
at 
org.apache.flink.yarn.YarnApplicationMasterRunner.run(YarnApplicationMasterRunner.java:167)
at 
org.apache.flink.yarn.YarnApplicationMasterRunner.main(YarnApplicationMasterRunner.java:112)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at 
org.apache.hadoop.security.authentication.util.KerberosUtil.getDefaultRealm(KerberosUtil.java:84)
at 
org.apache.hadoop.security.HadoopKerberosName.setConfiguration(HadoopKerberosName.java:63)
... 5 more
Caused by: KrbException: Cannot locate default realm
at sun.security.krb5.Config.getDefaultRealm(Config.java:1006)
... 11 more
Caused by: KrbException: Generic error (description in e-text) (60) - 
Unable to locate Kerberos realm
at sun.security.krb5.Config.getRealmFromDNS(Config.java:1102)
at sun.security.krb5.Config.getDefaultRealm(Config.java:987)
... 11 more
```


```
2016-09-02 11:34:00,491 ERROR 
org.apache.flink.shaded.org.apache.curator.ConnectionState- Authentication 
failed
```


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15458147#comment-15458147
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2275
  
Running some last tests before merging.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15450140#comment-15450140
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on the issue:

https://github.com/apache/flink/pull/2275
  
@mxm I have rebased the code to the latest master. Please take a look.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15449163#comment-15449163
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2275
  
Could you please rebase to the latest master and remove any merge commits?


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15449160#comment-15449160
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2275
  
Thanks for testing this on an actual secure cluster. 

Hmpf, only allowing to bind to privileged ports doesn't really improve 
security but we have to deal with this limitation somehow. I was initially 
thinking we could just include a custom version of the `MiniKDC` dependency in 
the Flink code. As you pointed out, this is not possible because the problem 
lies in the `SecureDataNodeStarter` which returns the secure resources. This 
class is part of `hadoop-hdfs`which we can't include a custom version because 
it depends on the Hadoop version provided during build time.

Once the patch is out, we can run this test in a special profile where we 
set the Hadoop version which supports non-privileged ports. As of now, I don't 
see a feasible solution other than skipping this test.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15444044#comment-15444044
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on the issue:

https://github.com/apache/flink/pull/2275
  
> 
YARNSessionFIFOSecuredITCase gives me the following:
17:49:58,097 INFO SecurityLogger.org.apache.hadoop.ipc.Server - Auth 
successful for appattempt_1471880990715_0001_01 (auth:SIMPLE)
It is not using Kerberos it seems. We should check that security is really 
enabled and fail the test if not.

@mxm I am not sure why the log statements from IPC layers are using 
auth:SIMPLE but I have verified the same messages (NM/RM logs) on a running HDP 
(secure) cluster too. I would imagine this is the default implementation and we 
can ignore those messages. However, while investigating this issue, I have 
found an interesting problem with YarnMiniCluster. The containers created does 
not have the Yarn Configuration that we pass through the test code. The KRB5 
file is also not visible and hence the UGI/security context that we create was 
missing proper Hadoop configurations. I have fixed the issue and patched it.

I have also disabled the RollingSinkSecure IT test case since secure MiniFS 
cluster requires privileged ports. We can enable the test case when the patch 
(HDFS-9213) is made in to main stream.

Please take a look and let me know if you can deploy and run the code.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15435356#comment-15435356
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on the issue:

https://github.com/apache/flink/pull/2275
  
>
It seems like the privileged port issues can be circumvented by setting 
conf.getBoolean("dfs.datanode.require.secure.ports", false)?

It is not supported yet  https://github.com/apache/hadoop/blob/branch-2.3.0/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/SecureDataNodeStarter.java#L106;>ref.
 The trunk https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/SecureDataNodeStarter.java#L115;>
 code  also have similar logic.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434557#comment-15434557
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2275
  
It seems like the privileged port issues can be circumvented by setting ` 
conf.getBoolean("dfs.datanode.require.secure.ports", false)`?


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433133#comment-15433133
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on the issue:

https://github.com/apache/flink/pull/2275
  
> If we have to use privileged ports then we won't be able to use our CI 
system. Are you sure it can only run in privileged mode? Is it not possible to 
change the port binding to port >= 1024?

Unfortunately this is how the underlying code is implemented and the patch 
from HDFS-9213 will get us to use higher order ports. I can mark the test class 
@Ignore for now if we have to wait until the patch is ported?


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432757#comment-15432757
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2275
  
>The RollingFileSink error might be due to 
https://issues.apache.org/jira/browse/HDFS-9213. The secure MiniDFS cluster 
requires privileged ports to be used and we need to enable the java process to 
grant access if it is not running as root.

If we have to use privileged ports then we won't be able to use our CI 
system. Are you sure it can only run in privileged mode? Is it not possible to 
change the port binding to port >= 1024?

If you have previously ran `mvn install -DskipTests`, then you can use the 
following to run the test:

```
mvn test integration-test -Dtest="YARNSessionFIFOSecuredITCase" 
-Pinclude-yarn-tests -pl flink-yarn-tests
```


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431703#comment-15431703
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on the issue:

https://github.com/apache/flink/pull/2275
  
> YARNSessionFIFOSecuredITCase gives me the following:
17:49:58,097 INFO SecurityLogger.org.apache.hadoop.ipc.Server - Auth 
successful for appattempt_1471880990715_0001_01 (auth:SIMPLE)
 It is not using Kerberos it seems. We should check that security is really 
enabled and fail the test if not.

How did you run the Yarn IT test? I am seeing some compile issues when I 
try to run the below command?

```
> mvn verify -pl flink-yarn-tests -Pinclude-yarn-tests
[INFO] Scanning for projects...
[INFO]  
   
[INFO] 

[INFO] Building flink-yarn-tests 1.2-SNAPSHOT
[INFO] 

[INFO] 
[INFO] --- maven-checkstyle-plugin:2.16:check (validate) @ 
flink-yarn-tests_2.10 ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-maven) @ 
flink-yarn-tests_2.10 ---
[INFO] 
[INFO] --- build-helper-maven-plugin:1.7:add-source (add-source) @ 
flink-yarn-tests_2.10 ---
[INFO] Source directory: 
/workspace/git-projects/junk/apache-flink/flink-yarn-tests/src/main/scala added.
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ 
flink-yarn-tests_2.10 ---
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ 
flink-yarn-tests_2.10 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory 
/workspace/git-projects/junk/apache-flink/flink-yarn-tests/src/main/resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- scala-maven-plugin:3.1.4:compile (scala-compile-first) @ 
flink-yarn-tests_2.10 ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ 
flink-yarn-tests_2.10 ---
[INFO] No sources to compile
[INFO] 
[INFO] --- build-helper-maven-plugin:1.7:add-test-source (add-test-source) 
@ flink-yarn-tests_2.10 ---
[INFO] Test Source directory: 
/workspace/git-projects/junk/apache-flink/flink-yarn-tests/src/test/scala added.
[INFO] 
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) 
@ flink-yarn-tests_2.10 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 3 resources
[INFO] 
[INFO] --- scala-maven-plugin:3.1.4:testCompile (scala-test-compile) @ 
flink-yarn-tests_2.10 ---
[INFO] Nothing to compile - all classes are up to date
[INFO] 
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ 
flink-yarn-tests_2.10 ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 12 source files to 
/workspace/git-projects/junk/apache-flink/flink-yarn-tests/target/test-classes
[INFO] -
[WARNING] COMPILATION WARNING : 
[INFO] -
[WARNING] 
/workspace/git-projects/junk/apache-flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/UtilsTest.java:[81,47]
 YARN_HEAP_CUTOFF_RATIO in org.apache.flink.configuration.ConfigConstants has 
been deprecated
[WARNING] 
/workspace/git-projects/junk/apache-flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/UtilsTest.java:[82,48]
 YARN_HEAP_CUTOFF_MIN in org.apache.flink.configuration.ConfigConstants has 
been deprecated
[INFO] 2 warnings 
[INFO] -
[INFO] -
[ERROR] COMPILATION ERROR : 
[INFO] -
[ERROR] 
/workspace/git-projects/junk/apache-flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionFIFOSecuredITCase.java:[21,41]
 cannot find symbol
  symbol:   class SecurityContext
  location: package org.apache.flink.runtime.security
[ERROR] 
/workspace/git-projects/junk/apache-flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionFIFOSecuredITCase.java:[22,34]
 cannot find symbol
  symbol:   class SecureTestEnvironment
  location: package org.apache.flink.test.util
[ERROR] 
/workspace/git-projects/junk/apache-flink/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionFIFOSecuredITCase.java:[23,34]
 cannot find symbol
  symbol:   class TestingSecurityContext
  location: package org.apache.flink.test.util

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431658#comment-15431658
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on the issue:

https://github.com/apache/flink/pull/2275
  
The `RollingFileSink` error might be due to 
https://issues.apache.org/jira/browse/HDFS-9213. The secure MiniDFS 
cluster requires privileged ports to be used and we need to enable the java 
process to grant access if it is not running as root.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431106#comment-15431106
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2275
  
Seems like this is not yet in a mergeable state and needs another pass.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431064#comment-15431064
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2275
  
`RollingFileSink`

```
---
 T E S T S
---
Running org.apache.flink.streaming.connectors.fs.RollingSinkSecuredITCase
Formatting using clusterid: testClusterID
java.net.BindException: Permission denied
at java.net.PlainSocketImpl.socketBind(Native Method)
at 
java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:382)
at java.net.ServerSocket.bind(ServerSocket.java:375)
at 
org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.getSecureResources(SecureDataNodeStarter.java:103)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1213)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:684)
at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:351)
at 
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:332)
at 
org.apache.flink.streaming.connectors.fs.RollingSinkSecuredITCase.startSecureCluster(RollingSinkSecuredITCase.java:117)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 6.144 sec 
<<< FAILURE! - in 
org.apache.flink.streaming.connectors.fs.RollingSinkSecuredITCase
org.apache.flink.streaming.connectors.fs.RollingSinkSecuredITCase  Time 
elapsed: 6.143 sec  <<< ERROR!
java.lang.RuntimeException: Cannot start secure cluster without privileged 
resources.
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:734)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.(DataNode.java:316)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1848)
at 
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1748)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:1218)
at 
org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:684)
at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:351)
at 
org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:332)
at 
org.apache.flink.streaming.connectors.fs.RollingSinkSecuredITCase.startSecureCluster(RollingSinkSecuredITCase.java:117)

org.apache.flink.streaming.connectors.fs.RollingSinkSecuredITCase  Time 
elapsed: 6.144 sec  <<< ERROR!
java.lang.NullPointerException: null
at 
org.apache.flink.streaming.connectors.fs.RollingSinkSecuredITCase.teardownSecureCluster(RollingSinkSecuredITCase.java:135)
```


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron 

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431059#comment-15431059
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2275
  
YARNSessionFIFOSecuredITCase gives me the following:

>17:49:58,097 INFO  SecurityLogger.org.apache.hadoop.ipc.Server 
  - Auth successful for appattempt_1471880990715_0001_01 (auth:SIMPLE)

It is not using Kerberos it seems. We should check that security is really 
enabled and fail the test if not.



> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431043#comment-15431043
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2275
  
Hi @vijikarthi! Sorry for the late reply. I intend to merge this as soon as 
possible. When I run the secure test cases from IntelliJ I get 
>java.lang.NoClassDefFoundError: jdbm/helper/CachePolicy

Investigating why this could be the case. 


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424865#comment-15424865
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on the issue:

https://github.com/apache/flink/pull/2275
  
@mxm - Could you please take a look and let me know if you need any further 
changes.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15418405#comment-15418405
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on the issue:

https://github.com/apache/flink/pull/2275
  
Thanks Max. I have reorganized the secure test case and removed custom 
JRunner implementation for Kafka. Kept single secure test case for HDFS, Kafka 
and Yarn modules.  Please take a look.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15417431#comment-15417431
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2275
  
>HDFS and Yarn are handled through the @BeforeClass and @AfterClass style 
and they do not use custom JRunner implementation. As you have suggested, I 
could keep just one or two tests for each of the modules to cut down the 
running time, if that's okay with you?

Thanks! Yes, please just one test per entity (HDFS, Yarn, Kafka/Zookeeper). 
Could you also convert the Kafka test to using `@BeforeClass` and 
`@AfterClass`? You don't necessarily have to duplicate code. How about changing 
the test base to include a method to instantiate the secure settings? I think 
you'll need to add an abstract method in `KafkaTestEnvironment`, e.g. 
loadSecureSettings(), then add another one in `KafkaTestBase`, e.g. 
getTestEnvironmentClass(), to load the appropriate test environment 
(secure/non-secure). You will have additional classes that implement these two 
methods. These classes will be very short as they just overload the method. 
This is cleaner although a bit more verbose.

>I am open to keep just 3 classes for each scenarios (HDFS, Yarn & Kafka) 
as you have suggested but in my opinion that will defeat the idea of reusing 
existing test program.

I understand but we're actually just increasing the testing time and not 
gaining much from running multiple security tests for the same component.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15413789#comment-15413789
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on the issue:

https://github.com/apache/flink/pull/2275
  
@mxm - Could you please let me know if you are okay with the modifications 
to the integration test case scenarios that I have mentioned. I am open to keep 
just 3 classes for each scenarios (HDFS, Yarn & Kafka) as you have suggested 
but in my opinion that will defeat the idea of reusing existing test program. 
Please let me know either way and I will fix the code accordingly.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405073#comment-15405073
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on the issue:

https://github.com/apache/flink/pull/2275
  
@mxm - I have addressed most of the feedback and pushed the changes. Please 
take a look when you get a chance. 

Regarding the secure test cases, 
-  HDFS and Yarn are handled through the @BeforeClass and @AfterClass style 
and they do not use custom JRunner implementation. As you have suggested, I 
could keep just one or two tests for each of the modules to cut down the 
running time, if that's okay with you?
- Kafka tests are handled with custom JRunner and if we need to move it to 
@BeforeClass and @AfterClass, then we may have to duplicate the code used in 
the base classes which may not look good. For e.g., Have a look at 
Kafka09ProducerSecuredITCase which is extended from Kafka09ProducerITCase which 
has 2 level of parent classes (KafkaProducerTestBase & KafkaTestBase). If we 
write a separate test case for secure cluster, then I may have to duplicate 
some of these base class code which can be avoided if we use custom JRunner. We 
could still limit the number of test cases to just one or two to minimize the 
running time. Please let me know your thoughts?


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404880#comment-15404880
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r73245570
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java ---
@@ -1016,6 +1016,23 @@
/** The environment variable name which contains the location of the 
lib folder */
public static final String ENV_FLINK_LIB_DIR = "FLINK_LIB_DIR";
 
+   //  Security 
---
+
+   /**
+* The config parameter defining security credentials required
+* for securing Flink cluster.
+*/
+
+   /** Keytab file key name to be used in flink configuration file */
+   public static final String SECURITY_KEYTAB_KEY = "security.keytab";
+
+   /** Kerberos security principal key name to be used in flink 
configuration file */
+   public static final String SECURITY_PRINCIPAL_KEY = 
"security.principal";
+
+   /** Keytab file name populated in YARN container */
+   public static final String KEYTAB_FILE_NAME = "krb5.keytab";
--- End diff --

Okay, will move it to flink-yarn module


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404021#comment-15404021
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r73160331
  
--- Diff: 
flink-streaming-connectors/flink-connector-kafka-base/src/test/java/org/apache/flink/streaming/connectors/kafka/KafkaShortRetentionTestBase.java
 ---
@@ -60,18 +64,33 @@
private static Properties standardProps;
private static ForkableFlinkMiniCluster flink;
 
+   @ClassRule
+   public static TemporaryFolder tempFolder = new TemporaryFolder();
+
+   protected static Properties secureProps = new Properties();
+
@BeforeClass
public static void prepare() throws IOException, ClassNotFoundException 
{

LOG.info("-");
LOG.info("Starting KafkaShortRetentionTestBase ");

LOG.info("-");
 
+   Configuration flinkConfig = new Configuration();
+
// dynamically load the implementation for the test
Class clazz = 
Class.forName("org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl");
kafkaServer = (KafkaTestEnvironment) 
InstantiationUtil.instantiate(clazz);
 
LOG.info("Starting KafkaTestBase.prepare() for Kafka " + 
kafkaServer.getVersion());
 
+   LOG.info("Runtype: {}", RunTypeHolder.get());
+   if(RunTypeHolder.get().equals(RunTypeHolder.RunType.SECURE)
+   && kafkaServer.isSecureRunSupported()) {
+   SecureTestEnvironment.prepare(tempFolder);
+   
SecureTestEnvironment.getSecurityEnabledFlinkConfiguration(flinkConfig);
--- End diff --

The name of the method doesn't suggest that though.

`SecureTestEnvironment.writeSecurityConfiguration(flinkConfig);` would make 
it more explicit.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404003#comment-15404003
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r73159225
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java ---
@@ -1016,6 +1016,23 @@
/** The environment variable name which contains the location of the 
lib folder */
public static final String ENV_FLINK_LIB_DIR = "FLINK_LIB_DIR";
 
+   //  Security 
---
+
+   /**
+* The config parameter defining security credentials required
+* for securing Flink cluster.
+*/
+
+   /** Keytab file key name to be used in flink configuration file */
+   public static final String SECURITY_KEYTAB_KEY = "security.keytab";
+
+   /** Kerberos security principal key name to be used in flink 
configuration file */
+   public static final String SECURITY_PRINCIPAL_KEY = 
"security.principal";
+
+   /** Keytab file name populated in YARN container */
+   public static final String KEYTAB_FILE_NAME = "krb5.keytab";
--- End diff --

My point is that this class is declared `@Public` and we will have to stick 
with this config key name. It could also reside in the `flink-yarn` module 
because it doesn't have to be exposed to the user.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404010#comment-15404010
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r73159602
  
--- Diff: 
flink-streaming-connectors/flink-connector-filesystem/src/test/java/org/apache/flink/streaming/connectors/fs/RollingSinkSecuredITCase.java
 ---
@@ -0,0 +1,195 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.fs;
+
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.runtime.security.SecurityContext;
+import org.apache.flink.streaming.util.TestStreamEnvironment;
+import org.apache.flink.test.util.SecureTestEnvironment;
+import org.apache.flink.test.util.TestingSecurityContext;
+import org.apache.flink.test.util.TestBaseUtils;
+import org.apache.flink.util.NetUtils;
+import org.apache.hadoop.fs.FileUtil;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hadoop.http.HttpConfig;
+import org.apache.hadoop.security.SecurityUtil;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hadoop.hdfs.DFSConfigKeys.*;
+import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_HTTP_ADDRESS_KEY;
--- End diff --

Actually, Checkstyle doesn't run on the testing code but still nice to keep 
a consistent style. Thanks! 


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404008#comment-15404008
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r73159441
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/security/SecurityContext.java
 ---
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.security;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.Preconditions;
+import org.apache.hadoop.security.Credentials;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.security.auth.Subject;
+import java.io.File;
+import java.lang.reflect.Method;
+import java.security.PrivilegedExceptionAction;
+/*
+ * Process-wide security context object which initializes UGI with 
appropriate security credentials and also it
+ * creates in-memory JAAS configuration object which will serve 
appropriate ApplicationConfigurationEntry for the
+ * connector login module implementation that authenticates Kerberos 
identity using SASL/JAAS based mechanism.
+ */
+@Internal
+public class SecurityContext {
+
+   private static final Logger LOG = 
LoggerFactory.getLogger(SecurityContext.class);
+
+   private static SecurityContext installedContext;
+
+   public static SecurityContext getInstalled() { return installedContext; 
}
+
+   private UserGroupInformation ugi;
+
+   SecurityContext(UserGroupInformation ugi) {
+   this.ugi = ugi;
+   }
+
+   public  T runSecured(final FlinkSecuredRunner runner) throws 
Exception {
+   return ugi.doAs(new PrivilegedExceptionAction() {
+   @Override
+   public T run() throws Exception {
+   return runner.run();
+   }
+   });
+   }
+
+   public static void install(SecurityConfiguration config) throws 
Exception {
+
+   // perform static initialization of UGI, JAAS
+   if(installedContext != null) {
+   LOG.warn("overriding previous security context");
+   }
+
+   // establish the JAAS config
+   JaasConfiguration jaasConfig = new 
JaasConfiguration(config.keytab, config.principal);
+   
javax.security.auth.login.Configuration.setConfiguration(jaasConfig);
+
+   //hack since Kafka Login Handler explicitly look for the 
property or else it throws an exception
+   
//https://github.com/apache/kafka/blob/0.9.0/clients/src/main/java/org/apache/kafka/common/security/kerberos/Login.java#L289
+   System.setProperty("java.security.auth.login.config", "");
--- End diff --

I see. Thanks!


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the 

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404005#comment-15404005
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r73159333
  
--- Diff: docs/internals/flink_security.md ---
@@ -0,0 +1,87 @@
+---
+title:  "Flink Security"
+# Top navigation
+top-nav-group: internals
+top-nav-pos: 10
+top-nav-title: Flink Security
+---
+
+
+This document briefly describes how Flink security works in the context of 
various deployment mechanism (Standalone/Cluster vs YARN) 
+and the connectors that participates in Flink Job execution stage. This 
documentation can be helpful for both administrators and developers 
+who plans to run Flink on a secure environment.
+
+## Objective
+
+The primary goal of Flink security model is to enable secure data access 
for jobs within a cluster via connectors. In production deployment scenario, 
+streaming jobs are understood to run for longer period of time 
(days/weeks/months) and the system must be  able to authenticate against secure 
+data sources throughout the life of the job. The current implementation 
supports running Flink cluster (Job Manager/Task Manager/Jobs) under the 
+context of a Kerberos identity based on Keytab credential supplied during 
deployment time. Any jobs submitted will continue to run in the identity of the 
cluster.
+
+## How Flink Security works
+Flink deployment includes running Job Manager/ZooKeeper, Task Manager(s), 
Web UI and Job(s). Jobs (user code) can be submitted through web UI and/or CLI. 
+A Job program may use one or more connectors (Kafka, HDFS, Cassandra, 
Flume, Kinesis etc.,) and each connector may have a specific security 
+requirements (Kerberos, database based, SSL/TLS, custom etc.,). While 
satisfying the security requirements for all the connectors evolve over a 
period 
+of time but at this time of writing, the following connectors/services are 
tested for Kerberos/Keytab based security.
+
+- Kafka (0.9)
+- HDFS
+- ZooKeeper
+
+Hadoop uses UserGroupInformation (UGI) class to manage security. UGI is a 
static implementation that takes care of handling Kerberos authentication. 
Flink bootstrap implementation
+(JM/TM/CLI) takes care of instantiating UGI with appropriate security 
credentials to establish necessary security context.
+
+Services like Kafka and ZooKeeper uses SASL/JAAS based authentication 
mechanism to authenticate against a Kerberos server. It expects JAAS 
configuration with platform-specific login 
+module *name* to be provided. Managing per-connector configuration files 
will be an overhead and to overcome this requirement, a process-wide JAAS 
configuration object is 
+instantiated which serves standard ApplicationConfigurationEntry for the 
connectors that authenticates using SASL/JAAS mechanism.
+
+It is important to understand that the Flink processes (JM/TM/UI/Jobs) 
itself uses UGI's doAS() implementation to run under specific user context 
i.e., if Hadoop security is enabled 
+then the Flink processes will be running under secure user account or else 
it will run as the OS login user account who starts Flink cluster.
+
+## Security Configurations
+
+Secure credentials can be supplied by adding below configuration elements 
to Flink configuration file:
+
+- `security.keytab`: Absolute path to Kerberos keytab file that contains 
the user credentials/secret.
+
+- `security.principal`: User principal name that the Flink cluster should 
run as.
--- End diff --

Is the keytab file enough then? Do we need an additional principal?


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15404000#comment-15404000
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r73159038
  
--- Diff: flink-core/src/main/java/org/apache/flink/util/Preconditions.java 
---
@@ -293,6 +293,16 @@ private static String format(@Nullable String 
template, @Nullable Object... args
 
return builder.toString();
}
+
+   /**
+* Returns {@code true} if the given string is null or is the empty 
string.
+* @param string a string reference to check
+* @return {@code true} if the string is null or is the empty string
+*/
+   public static boolean isNullOrEmpty(@Nullable String string) {
--- End diff --

Yes, even better :)


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403999#comment-15403999
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r73158919
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/security/JaasConfiguration.java
 ---
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.security;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.util.Preconditions;
+import org.apache.hadoop.security.authentication.util.KerberosUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.security.auth.login.AppConfigurationEntry;
+import javax.security.auth.login.Configuration;
+import java.io.File;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ *
+ * JAAS configuration provider object that provides default LoginModule 
for various connectors that supports
+ * JAAS/SASL based Kerberos authentication. The implementation is inspired 
from Hadoop UGI class.
+ *
+ * Different connectors uses different login module name to implement JAAS 
based authentication support.
+ * For example, Kafka expects the login module name to be "kafkaClient" 
whereas ZooKeeper expect the
+ * name to be "client". This sets onus on the Flink cluster administrator 
to configure/provide right
+ * JAAS config entries. To simplify this requirement, we have introduced 
this abstraction that provides
+ * a standard lookup to get the login module entry for the JAAS based 
authentication to work.
+ *
+ * HDFS connector will not be impacted with this configuration since it 
uses UGI based mechanism to authenticate.
+ *
+ * https://docs.oracle.com/javase/7/docs/api/javax/security/auth/login/Configuration.html;>Configuration
+ *
+ */
+
+@Internal
+public class JaasConfiguration extends Configuration {
+
+   private static final Logger LOG = 
LoggerFactory.getLogger(JaasConfiguration.class);
+
+   public static final String JAVA_VENDOR_NAME = 
System.getProperty("java.vendor");
+
+   public static final boolean IBM_JAVA;
+
+   static {
+   IBM_JAVA = JAVA_VENDOR_NAME.contains("IBM");
+   }
+
+   public JaasConfiguration(String keytab, String principal) {
--- End diff --

Then making it protected should work.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15403938#comment-15403938
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2275
  
> Yes, it is not essential to run the secure test case all the time as it 
consumes more cycles. Do you have any suggestion on controlling this through 
some mvn/surefire plugin configuration?

I would suggest that we keep three integration test classes for testing 
security:
1. Kafka (with Zookeeper)
2. HDFS
3. Yarn

In addition, we need a unit test to verify that the security context is 
correctly set up. I think we're covered then and don't need any more tests. 
After all, tests that are disabled by default and we almost never run are not 
very useful.

>It is little overhead but works out well with minimal changes to the code. 
We could revisit and make any changes if it creates any issues.

I would really like to move all security testing code to a utility class 
and call this code through `@AfterClass` and `@BeforeClass` methods. If that is 
fine with you. This is simpler and easier to understand than the custom runner 
implementation.

>I believe you are referring to the 
https://ci.apache.org/projects/flink/flink-docs-master/setup/config.html. If 
so, yes it certainly helps and I will be happy to add the details but I don't 
have access to edit the page.

You can actually change this page in the repository at 
[/docs/setup/config.md](/apache/flink/blob/master/docs/setup/config.md).

Concerning `SASL_PLAINTEXT`: I suppose we're making an effort to encrypt 
the authentication channel in the changes according to the security document? 
For now this seems fine.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402673#comment-15402673
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r73037620
  
--- Diff: docs/internals/flink_security.md ---
@@ -0,0 +1,87 @@
+---
+title:  "Flink Security"
+# Top navigation
+top-nav-group: internals
+top-nav-pos: 10
+top-nav-title: Flink Security
+---
+
+
+This document briefly describes how Flink security works in the context of 
various deployment mechanism (Standalone/Cluster vs YARN) 
+and the connectors that participates in Flink Job execution stage. This 
documentation can be helpful for both administrators and developers 
+who plans to run Flink on a secure environment.
+
+## Objective
+
+The primary goal of Flink security model is to enable secure data access 
for jobs within a cluster via connectors. In production deployment scenario, 
+streaming jobs are understood to run for longer period of time 
(days/weeks/months) and the system must be  able to authenticate against secure 
+data sources throughout the life of the job. The current implementation 
supports running Flink cluster (Job Manager/Task Manager/Jobs) under the 
+context of a Kerberos identity based on Keytab credential supplied during 
deployment time. Any jobs submitted will continue to run in the identity of the 
cluster.
+
+## How Flink Security works
+Flink deployment includes running Job Manager/ZooKeeper, Task Manager(s), 
Web UI and Job(s). Jobs (user code) can be submitted through web UI and/or CLI. 
+A Job program may use one or more connectors (Kafka, HDFS, Cassandra, 
Flume, Kinesis etc.,) and each connector may have a specific security 
+requirements (Kerberos, database based, SSL/TLS, custom etc.,). While 
satisfying the security requirements for all the connectors evolve over a 
period 
+of time but at this time of writing, the following connectors/services are 
tested for Kerberos/Keytab based security.
+
+- Kafka (0.9)
+- HDFS
+- ZooKeeper
+
+Hadoop uses UserGroupInformation (UGI) class to manage security. UGI is a 
static implementation that takes care of handling Kerberos authentication. 
Flink bootstrap implementation
+(JM/TM/CLI) takes care of instantiating UGI with appropriate security 
credentials to establish necessary security context.
+
+Services like Kafka and ZooKeeper uses SASL/JAAS based authentication 
mechanism to authenticate against a Kerberos server. It expects JAAS 
configuration with platform-specific login 
+module *name* to be provided. Managing per-connector configuration files 
will be an overhead and to overcome this requirement, a process-wide JAAS 
configuration object is 
+instantiated which serves standard ApplicationConfigurationEntry for the 
connectors that authenticates using SASL/JAAS mechanism.
+
+It is important to understand that the Flink processes (JM/TM/UI/Jobs) 
itself uses UGI's doAS() implementation to run under specific user context 
i.e., if Hadoop security is enabled 
+then the Flink processes will be running under secure user account or else 
it will run as the OS login user account who starts Flink cluster.
+
+## Security Configurations
+
+Secure credentials can be supplied by adding below configuration elements 
to Flink configuration file:
+
+- `security.keytab`: Absolute path to Kerberos keytab file that contains 
the user credentials/secret.
+
+- `security.principal`: User principal name that the Flink cluster should 
run as.
--- End diff --

The keytab file contains both principal and encrypted keys (password)


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402626#comment-15402626
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on the issue:

https://github.com/apache/flink/pull/2275
  
@mxm - Thanks for your feedback and here is my response to some of your 
comments.

- Do we need to run all the Yarn tests normally and secured? We already 
have problems with our test execution time. Perhaps we could have one dedicated 
test for secure setups and disable the other ones by default to run them 
manually if needed.
[Vijay] - Yes, it is not essential to run the secure test case all the time 
as it consumes more cycles. Do you have any suggestion on controlling this 
through some mvn/surefire plugin configuration?

- The testing code seems overly complicated using the custom JUnit Runner. 
I think we could achieve the same with @BeforeClass and @AfterClass methods in 
the secure IT cases.
[Vijay] - It is little overhead but works out well with minimal changes to 
the code. We could revisit and make any changes if it creates any issues.

- There is no dedicated test for the SecurityContext and the 
JaasConfiguration classes
[Vijay] - Yes, will add UT for those classes.

- It would be nice to add some documentation to the configuration web page.
[Vijay] - I believe you are referring to the 
https://ci.apache.org/projects/flink/flink-docs-master/setup/config.html. If 
so, yes it certainly helps and I will be happy to add the details but I don't 
have access to edit the page.

- We should throw exceptions if the secure configuration is not complete 
instead of falling back to non-authenticated execution for either Hadoop or the 
Jaas configuration. Otherwise, users might end up with a partly secure 
environment.
[Vijay] - Yes, will add the validation logic


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402606#comment-15402606
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on the issue:

https://github.com/apache/flink/pull/2275
  
@nielsbasjes - In most deployments, the KRB5 configuration file will be 
located in a well known (for e.g., /etc/krb5.conf) but in scenarios where 
custom location needs to be provided, we could pass the value through  
"-Djava.security.krb5.conf"


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402585#comment-15402585
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r73028461
  
--- Diff: 
flink-yarn/src/main/java/org/apache/flink/yarn/YarnTaskManagerRunner.java ---
@@ -75,34 +84,47 @@ public static void runYarnTaskManager(String[] args, 
final Class toks : 
UserGroupInformation.getCurrentUser().getTokens()) {
-   ugi.addToken(toks);
+   String keytabPath = null;
+   if(remoteKeytabPath != null) {
+   File f = new File(currDir, 
ConfigConstants.KEYTAB_FILE_NAME);
--- End diff --

The name is not configurable (user provided) but we use a constant value. 
Is there any reason to keep the name unique? 


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402570#comment-15402570
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r73027093
  
--- Diff: 
flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/test/util/RunTypeSelectionRunner.java
 ---
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.test.util;
+
+import org.junit.runners.BlockJUnit4ClassRunner;
+import org.junit.runners.model.FrameworkMethod;
+import org.junit.runners.model.InitializationError;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/*
+ * Custom JRunner used to run integration tests for Kafka in secure mode. 
Since the test code implementation uses
+ * abstract base code to run the MiniFlinkCluster and also start/stop 
Kafka brokers depending on the version (0.8/0.9)
+ * we need to know the run mode (CLEAR/SECURE) ahead of time (before the 
beforeClass). This Runner instance
+ * take care of setting SECURE flag on the holder class for secure testing 
to work seamless
+ *
+ */
+public class RunTypeSelectionRunner extends BlockJUnit4ClassRunner {
--- End diff --

I agree with you that it will be nice to control this behavior in 
@BeforeClass but the challenge is controlling test run (secure vs insecure 
mode). The base class should be aware of the run mode (secure/insecure) ahead 
of time and I found it very challenging especially when the lifecycle of the 
services (MiniFlink, Kafka/ZK etc.,) are handled at different levels. I ended 
up in using custom JRunner which gives the flexibility in handling the run 
level from the top most layer (@Test class) without having to readjust the 
codebase much.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402540#comment-15402540
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r73023816
  
--- Diff: 
flink-streaming-connectors/flink-connector-kafka-base/src/test/java/org/apache/flink/streaming/connectors/kafka/KafkaTestBase.java
 ---
@@ -81,13 +90,21 @@ public static void prepare() throws IOException, 
ClassNotFoundException {

LOG.info("-");
LOG.info("Starting KafkaTestBase ");

LOG.info("-");
-   
 
+   Configuration flinkConfig = new Configuration();
 
// dynamically load the implementation for the test
Class clazz = 
Class.forName("org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl");
kafkaServer = (KafkaTestEnvironment) 
InstantiationUtil.instantiate(clazz);
 
+   LOG.info("Runtype: {}", RunTypeHolder.get());
+   if(RunTypeHolder.get().equals(RunTypeHolder.RunType.SECURE)
--- End diff --

Yes == is more appropriate


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402541#comment-15402541
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r73023852
  
--- Diff: 
flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/test/util/RunTypeHolder.java
 ---
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.test.util;
+
+public class RunTypeHolder {
--- End diff --

Will add the desc


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402535#comment-15402535
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r73023705
  
--- Diff: 
flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/test/util/RunTypeHolder.java
 ---
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.test.util;
+
+public class RunTypeHolder {
+
+   private static RunType runType = RunType.CLEAR;
--- End diff --

Will change it to PLAIN


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402485#comment-15402485
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r73020806
  
--- Diff: 
flink-streaming-connectors/flink-connector-kafka-base/src/test/java/org/apache/flink/streaming/connectors/kafka/KafkaShortRetentionTestBase.java
 ---
@@ -60,18 +64,33 @@
private static Properties standardProps;
private static ForkableFlinkMiniCluster flink;
 
+   @ClassRule
+   public static TemporaryFolder tempFolder = new TemporaryFolder();
+
+   protected static Properties secureProps = new Properties();
+
@BeforeClass
public static void prepare() throws IOException, ClassNotFoundException 
{

LOG.info("-");
LOG.info("Starting KafkaShortRetentionTestBase ");

LOG.info("-");
 
+   Configuration flinkConfig = new Configuration();
+
// dynamically load the implementation for the test
Class clazz = 
Class.forName("org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl");
kafkaServer = (KafkaTestEnvironment) 
InstantiationUtil.instantiate(clazz);
 
LOG.info("Starting KafkaTestBase.prepare() for Kafka " + 
kafkaServer.getVersion());
 
+   LOG.info("Runtype: {}", RunTypeHolder.get());
+   if(RunTypeHolder.get().equals(RunTypeHolder.RunType.SECURE)
+   && kafkaServer.isSecureRunSupported()) {
+   SecureTestEnvironment.prepare(tempFolder);
+   
SecureTestEnvironment.getSecurityEnabledFlinkConfiguration(flinkConfig);
--- End diff --

No. The passed config object will be populated with keytab and principal 
security configurations.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402457#comment-15402457
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r73017860
  
--- Diff: 
flink-streaming-connectors/flink-connector-filesystem/src/test/java/org/apache/flink/streaming/connectors/fs/RollingSinkSecuredITCase.java
 ---
@@ -0,0 +1,195 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.fs;
+
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.runtime.security.SecurityContext;
+import org.apache.flink.streaming.util.TestStreamEnvironment;
+import org.apache.flink.test.util.SecureTestEnvironment;
+import org.apache.flink.test.util.TestingSecurityContext;
+import org.apache.flink.test.util.TestBaseUtils;
+import org.apache.flink.util.NetUtils;
+import org.apache.hadoop.fs.FileUtil;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.apache.hadoop.http.HttpConfig;
+import org.apache.hadoop.security.SecurityUtil;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.io.FileWriter;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+
+import static org.apache.hadoop.hdfs.DFSConfigKeys.*;
+import static 
org.apache.hadoop.hdfs.DFSConfigKeys.DFS_DATANODE_HTTP_ADDRESS_KEY;
--- End diff --

Sorry, this might be from IntelliJ auto-import stuff. Will revert to use 
appropriate imports.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402438#comment-15402438
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r73015565
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/security/SecurityContext.java
 ---
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.security;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.Preconditions;
+import org.apache.hadoop.security.Credentials;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.security.auth.Subject;
+import java.io.File;
+import java.lang.reflect.Method;
+import java.security.PrivilegedExceptionAction;
+/*
+ * Process-wide security context object which initializes UGI with 
appropriate security credentials and also it
+ * creates in-memory JAAS configuration object which will serve 
appropriate ApplicationConfigurationEntry for the
+ * connector login module implementation that authenticates Kerberos 
identity using SASL/JAAS based mechanism.
+ */
+@Internal
+public class SecurityContext {
+
+   private static final Logger LOG = 
LoggerFactory.getLogger(SecurityContext.class);
+
+   private static SecurityContext installedContext;
+
+   public static SecurityContext getInstalled() { return installedContext; 
}
+
+   private UserGroupInformation ugi;
+
+   SecurityContext(UserGroupInformation ugi) {
+   this.ugi = ugi;
+   }
+
+   public  T runSecured(final FlinkSecuredRunner runner) throws 
Exception {
+   return ugi.doAs(new PrivilegedExceptionAction() {
+   @Override
+   public T run() throws Exception {
+   return runner.run();
+   }
+   });
+   }
+
+   public static void install(SecurityConfiguration config) throws 
Exception {
+
+   // perform static initialization of UGI, JAAS
+   if(installedContext != null) {
+   LOG.warn("overriding previous security context");
+   }
+
+   // establish the JAAS config
+   JaasConfiguration jaasConfig = new 
JaasConfiguration(config.keytab, config.principal);
+   
javax.security.auth.login.Configuration.setConfiguration(jaasConfig);
+
+   //hack since Kafka Login Handler explicitly look for the 
property or else it throws an exception
+   
//https://github.com/apache/kafka/blob/0.9.0/clients/src/main/java/org/apache/kafka/common/security/kerberos/Login.java#L289
+   System.setProperty("java.security.auth.login.config", "");
+
+   // establish the UGI login user
+   UserGroupInformation.setConfiguration(config.hadoopConf);
+   UserGroupInformation loginUser;
+   if(UserGroupInformation.isSecurityEnabled() && config.keytab != 
null && !Preconditions.isNullOrEmpty(config.principal)) {
--- End diff --

Sure. Will handle it in JAASConfiguration.java as that looks more 
appropriate place.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a 

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402428#comment-15402428
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r73014755
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/security/SecurityContext.java
 ---
@@ -0,0 +1,218 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.security;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.util.Preconditions;
+import org.apache.hadoop.security.Credentials;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.security.auth.Subject;
+import java.io.File;
+import java.lang.reflect.Method;
+import java.security.PrivilegedExceptionAction;
+/*
+ * Process-wide security context object which initializes UGI with 
appropriate security credentials and also it
+ * creates in-memory JAAS configuration object which will serve 
appropriate ApplicationConfigurationEntry for the
+ * connector login module implementation that authenticates Kerberos 
identity using SASL/JAAS based mechanism.
+ */
+@Internal
+public class SecurityContext {
+
+   private static final Logger LOG = 
LoggerFactory.getLogger(SecurityContext.class);
+
+   private static SecurityContext installedContext;
+
+   public static SecurityContext getInstalled() { return installedContext; 
}
+
+   private UserGroupInformation ugi;
+
+   SecurityContext(UserGroupInformation ugi) {
+   this.ugi = ugi;
+   }
+
+   public  T runSecured(final FlinkSecuredRunner runner) throws 
Exception {
+   return ugi.doAs(new PrivilegedExceptionAction() {
+   @Override
+   public T run() throws Exception {
+   return runner.run();
+   }
+   });
+   }
+
+   public static void install(SecurityConfiguration config) throws 
Exception {
+
+   // perform static initialization of UGI, JAAS
+   if(installedContext != null) {
+   LOG.warn("overriding previous security context");
+   }
+
+   // establish the JAAS config
+   JaasConfiguration jaasConfig = new 
JaasConfiguration(config.keytab, config.principal);
+   
javax.security.auth.login.Configuration.setConfiguration(jaasConfig);
+
+   //hack since Kafka Login Handler explicitly look for the 
property or else it throws an exception
+   
//https://github.com/apache/kafka/blob/0.9.0/clients/src/main/java/org/apache/kafka/common/security/kerberos/Login.java#L289
+   System.setProperty("java.security.auth.login.config", "");
--- End diff --

It was never set earlier and not really essential. We are adding it only 
because Kafka code implementation (09 branch) explicitly looks for this.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink 

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402414#comment-15402414
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r73013795
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/security/JaasConfiguration.java
 ---
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.security;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.util.Preconditions;
+import org.apache.hadoop.security.authentication.util.KerberosUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.security.auth.login.AppConfigurationEntry;
+import javax.security.auth.login.Configuration;
+import java.io.File;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ *
+ * JAAS configuration provider object that provides default LoginModule 
for various connectors that supports
+ * JAAS/SASL based Kerberos authentication. The implementation is inspired 
from Hadoop UGI class.
+ *
+ * Different connectors uses different login module name to implement JAAS 
based authentication support.
+ * For example, Kafka expects the login module name to be "kafkaClient" 
whereas ZooKeeper expect the
+ * name to be "client". This sets onus on the Flink cluster administrator 
to configure/provide right
+ * JAAS config entries. To simplify this requirement, we have introduced 
this abstraction that provides
+ * a standard lookup to get the login module entry for the JAAS based 
authentication to work.
+ *
+ * HDFS connector will not be impacted with this configuration since it 
uses UGI based mechanism to authenticate.
+ *
+ * https://docs.oracle.com/javase/7/docs/api/javax/security/auth/login/Configuration.html;>Configuration
+ *
+ */
+
+@Internal
+public class JaasConfiguration extends Configuration {
+
+   private static final Logger LOG = 
LoggerFactory.getLogger(JaasConfiguration.class);
+
+   public static final String JAVA_VENDOR_NAME = 
System.getProperty("java.vendor");
+
+   public static final boolean IBM_JAVA;
+
+   static {
+   IBM_JAVA = JAVA_VENDOR_NAME.contains("IBM");
+   }
+
+   public JaasConfiguration(String keytab, String principal) {
--- End diff --

I had this as package local earlier but the test framework code is making 
use of this class (extending) and hence I changed it to public constructor.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402412#comment-15402412
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r73013657
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/security/JaasConfiguration.java
 ---
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.security;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.util.Preconditions;
+import org.apache.hadoop.security.authentication.util.KerberosUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.security.auth.login.AppConfigurationEntry;
+import javax.security.auth.login.Configuration;
+import java.io.File;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ *
+ * JAAS configuration provider object that provides default LoginModule 
for various connectors that supports
+ * JAAS/SASL based Kerberos authentication. The implementation is inspired 
from Hadoop UGI class.
+ *
+ * Different connectors uses different login module name to implement JAAS 
based authentication support.
+ * For example, Kafka expects the login module name to be "kafkaClient" 
whereas ZooKeeper expect the
+ * name to be "client". This sets onus on the Flink cluster administrator 
to configure/provide right
+ * JAAS config entries. To simplify this requirement, we have introduced 
this abstraction that provides
+ * a standard lookup to get the login module entry for the JAAS based 
authentication to work.
+ *
+ * HDFS connector will not be impacted with this configuration since it 
uses UGI based mechanism to authenticate.
+ *
+ * https://docs.oracle.com/javase/7/docs/api/javax/security/auth/login/Configuration.html;>Configuration
+ *
+ */
+
+@Internal
+public class JaasConfiguration extends Configuration {
+
+   private static final Logger LOG = 
LoggerFactory.getLogger(JaasConfiguration.class);
+
+   public static final String JAVA_VENDOR_NAME = 
System.getProperty("java.vendor");
+
+   public static final boolean IBM_JAVA;
+
+   static {
+   IBM_JAVA = JAVA_VENDOR_NAME.contains("IBM");
+   }
+
+   public JaasConfiguration(String keytab, String principal) {
+
+   LOG.info("Initializing JAAS configuration instance. Parameters: 
{}, {}", keytab, principal);
+
+   if(!Preconditions.isNullOrEmpty(keytab) && 
!Preconditions.isNullOrEmpty(principal)) {
+
+   if(IBM_JAVA) {
+   keytabKerberosOptions.put("useKeytab", 
prependFileUri(keytab));
+   keytabKerberosOptions.put("credsType", "both");
+   } else {
+   keytabKerberosOptions.put("keyTab", keytab);
+   keytabKerberosOptions.put("doNotPrompt", 
"true");
+   keytabKerberosOptions.put("useKeyTab", "true");
+   keytabKerberosOptions.put("storeKey", "true");
+   }
+
+   keytabKerberosOptions.put("principal", principal);
+   keytabKerberosOptions.put("refreshKrb5Config", "true");
+   keytabKerberosOptions.putAll(debugOptions);
+
+   keytabKerberosAce = new AppConfigurationEntry(
+   KerberosUtil.getKrb5LoginModuleName(),
+   
AppConfigurationEntry.LoginModuleControlFlag.REQUIRED,
+   keytabKerberosOptions);
+   }
+   }
+
+   private static final Map debugOptions = new HashMap<>();
--- End diff --

Yes, it makes sense to use log4j 

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402398#comment-15402398
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r73012870
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/security/JaasConfiguration.java
 ---
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.security;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.util.Preconditions;
+import org.apache.hadoop.security.authentication.util.KerberosUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.security.auth.login.AppConfigurationEntry;
+import javax.security.auth.login.Configuration;
+import java.io.File;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ *
+ * JAAS configuration provider object that provides default LoginModule 
for various connectors that supports
+ * JAAS/SASL based Kerberos authentication. The implementation is inspired 
from Hadoop UGI class.
+ *
+ * Different connectors uses different login module name to implement JAAS 
based authentication support.
+ * For example, Kafka expects the login module name to be "kafkaClient" 
whereas ZooKeeper expect the
+ * name to be "client". This sets onus on the Flink cluster administrator 
to configure/provide right
+ * JAAS config entries. To simplify this requirement, we have introduced 
this abstraction that provides
+ * a standard lookup to get the login module entry for the JAAS based 
authentication to work.
+ *
+ * HDFS connector will not be impacted with this configuration since it 
uses UGI based mechanism to authenticate.
+ *
+ * https://docs.oracle.com/javase/7/docs/api/javax/security/auth/login/Configuration.html;>Configuration
+ *
+ */
+
+@Internal
+public class JaasConfiguration extends Configuration {
+
+   private static final Logger LOG = 
LoggerFactory.getLogger(JaasConfiguration.class);
+
+   public static final String JAVA_VENDOR_NAME = 
System.getProperty("java.vendor");
+
+   public static final boolean IBM_JAVA;
+
+   static {
+   IBM_JAVA = JAVA_VENDOR_NAME.contains("IBM");
+   }
+
+   public JaasConfiguration(String keytab, String principal) {
+
+   LOG.info("Initializing JAAS configuration instance. Parameters: 
{}, {}", keytab, principal);
+
+   if(!Preconditions.isNullOrEmpty(keytab) && 
!Preconditions.isNullOrEmpty(principal)) {
+
+   if(IBM_JAVA) {
+   keytabKerberosOptions.put("useKeytab", 
prependFileUri(keytab));
+   keytabKerberosOptions.put("credsType", "both");
+   } else {
+   keytabKerberosOptions.put("keyTab", keytab);
+   keytabKerberosOptions.put("doNotPrompt", 
"true");
+   keytabKerberosOptions.put("useKeyTab", "true");
+   keytabKerberosOptions.put("storeKey", "true");
+   }
+
+   keytabKerberosOptions.put("principal", principal);
+   keytabKerberosOptions.put("refreshKrb5Config", "true");
+   keytabKerberosOptions.putAll(debugOptions);
+
+   keytabKerberosAce = new AppConfigurationEntry(
+   KerberosUtil.getKrb5LoginModuleName(),
+   
AppConfigurationEntry.LoginModuleControlFlag.REQUIRED,
+   keytabKerberosOptions);
+   }
+   }
+
+   private static final Map debugOptions = new HashMap<>();
+
+   private static final Map 

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402395#comment-15402395
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r73012593
  
--- Diff: 
flink-runtime/src/main/java/org/apache/flink/runtime/security/JaasConfiguration.java
 ---
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.security;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.util.Preconditions;
+import org.apache.hadoop.security.authentication.util.KerberosUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.security.auth.login.AppConfigurationEntry;
+import javax.security.auth.login.Configuration;
+import java.io.File;
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ *
+ * JAAS configuration provider object that provides default LoginModule 
for various connectors that supports
+ * JAAS/SASL based Kerberos authentication. The implementation is inspired 
from Hadoop UGI class.
+ *
+ * Different connectors uses different login module name to implement JAAS 
based authentication support.
+ * For example, Kafka expects the login module name to be "kafkaClient" 
whereas ZooKeeper expect the
+ * name to be "client". This sets onus on the Flink cluster administrator 
to configure/provide right
+ * JAAS config entries. To simplify this requirement, we have introduced 
this abstraction that provides
+ * a standard lookup to get the login module entry for the JAAS based 
authentication to work.
+ *
+ * HDFS connector will not be impacted with this configuration since it 
uses UGI based mechanism to authenticate.
+ *
+ * https://docs.oracle.com/javase/7/docs/api/javax/security/auth/login/Configuration.html;>Configuration
+ *
+ */
+
+@Internal
+public class JaasConfiguration extends Configuration {
+
+   private static final Logger LOG = 
LoggerFactory.getLogger(JaasConfiguration.class);
+
+   public static final String JAVA_VENDOR_NAME = 
System.getProperty("java.vendor");
+
+   public static final boolean IBM_JAVA;
+
+   static {
+   IBM_JAVA = JAVA_VENDOR_NAME.contains("IBM");
+   }
+
+   public JaasConfiguration(String keytab, String principal) {
+
+   LOG.info("Initializing JAAS configuration instance. Parameters: 
{}, {}", keytab, principal);
+
+   if(!Preconditions.isNullOrEmpty(keytab) && 
!Preconditions.isNullOrEmpty(principal)) {
+
+   if(IBM_JAVA) {
--- End diff --

I have tested with OpenJDK (1.8.0_91) only. Since the code fragment is 
derived from Hadoop UGI implementation, I assume it will work for IBM based 
JVM:)


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402387#comment-15402387
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r73012055
  
--- Diff: 
flink-core/src/main/java/org/apache/flink/configuration/ConfigConstants.java ---
@@ -1016,6 +1016,23 @@
/** The environment variable name which contains the location of the 
lib folder */
public static final String ENV_FLINK_LIB_DIR = "FLINK_LIB_DIR";
 
+   //  Security 
---
+
+   /**
+* The config parameter defining security credentials required
+* for securing Flink cluster.
+*/
+
+   /** Keytab file key name to be used in flink configuration file */
+   public static final String SECURITY_KEYTAB_KEY = "security.keytab";
+
+   /** Kerberos security principal key name to be used in flink 
configuration file */
+   public static final String SECURITY_PRINCIPAL_KEY = 
"security.principal";
+
+   /** Keytab file name populated in YARN container */
+   public static final String KEYTAB_FILE_NAME = "krb5.keytab";
--- End diff --

It is just a constant literal and not exposed in the Flink configuration 
file.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15401863#comment-15401863
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user nielsbasjes commented on the issue:

https://github.com/apache/flink/pull/2275
  
Question: Does this also work when starting yarn-session.sh?


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15401787#comment-15401787
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user nielsbasjes commented on the issue:

https://github.com/apache/flink/pull/2275
  
I noticed that the system property java.security.krb5.conf isn't set 
anywhere in the code.
Shouldn't there be a config setting/property that allows the user to set 
this?


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15396614#comment-15396614
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user vijikarthi commented on the issue:

https://github.com/apache/flink/pull/2275
  
Thanks @mxm for the review and feedback. I will respond to the comments and 
incorporate any changes required.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395923#comment-15395923
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user smarthi commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r72472262
  
--- Diff: flink-core/src/main/java/org/apache/flink/util/Preconditions.java 
---
@@ -293,6 +293,16 @@ private static String format(@Nullable String 
template, @Nullable Object... args
 
return builder.toString();
}
+
+   /**
+* Returns {@code true} if the given string is null or is the empty 
string.
+* @param string a string reference to check
+* @return {@code true} if the string is null or is the empty string
+*/
+   public static boolean isNullOrEmpty(@Nullable String string) {
--- End diff --

Consider using Apache Commons StringUtils.isBlank() for this.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395843#comment-15395843
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on the issue:

https://github.com/apache/flink/pull/2275
  
I've passed through your changes. Great work! Thanks a lot for the 
documentation, it was very helpful. Please don't be scared by all the comments, 
most of them are minor. The most important points:

- Do we need to run all the Yarn tests normally and secured? We already 
have problems with our test execution time. Perhaps we could have one dedicated 
test for secure setups and disable the other ones by default to run them 
manually if needed.

- The testing code seems overly complicated using the custom JUnit Runner. 
I think we could achieve the same with `@BeforeClass` and `@AfterClass` methods 
in the secure IT cases. 

- There is no dedicated test for the `SecurityContext` and the 
`JaasConfiguration` classes

- It would be nice to add some documentation to the configuration web page.

- We should throw exceptions if the secure configuration is not complete 
instead of falling back to non-authenticated execution for either Hadoop or the 
Jaas configuration. Otherwise, users might end up with a partly secure 
environment.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395809#comment-15395809
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r72456655
  
--- Diff: 
flink-yarn/src/main/java/org/apache/flink/yarn/YarnTaskManagerRunner.java ---
@@ -75,34 +84,47 @@ public static void runYarnTaskManager(String[] args, 
final Class toks : 
UserGroupInformation.getCurrentUser().getTokens()) {
-   ugi.addToken(toks);
+   String keytabPath = null;
+   if(remoteKeytabPath != null) {
+   File f = new File(currDir, 
ConfigConstants.KEYTAB_FILE_NAME);
--- End diff --

We should choose an internal unique name for the keytab file. We can 
transfer the name via the Yarn environment variables.


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3929) Support for Kerberos Authentication with Keytab Credential

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15395805#comment-15395805
 ] 

ASF GitHub Bot commented on FLINK-3929:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/2275#discussion_r72456165
  
--- Diff: 
flink-yarn/src/main/java/org/apache/flink/yarn/YarnApplicationMasterRunner.java 
---
@@ -125,29 +124,35 @@ protected int run(String[] args) {
try {
LOG.debug("All environment variables: {}", ENV);
 
-   final String yarnClientUsername = 
ENV.get(YarnConfigKeys.ENV_CLIENT_USERNAME);
+   final String yarnClientUsername = 
ENV.get(YarnConfigKeys.ENV_HADOOP_USER_NAME);
require(yarnClientUsername != null, "YARN client user 
name environment variable {} not set",
-   YarnConfigKeys.ENV_CLIENT_USERNAME);
+   YarnConfigKeys.ENV_HADOOP_USER_NAME);
 
-   final UserGroupInformation currentUser;
-   try {
-   currentUser = 
UserGroupInformation.getCurrentUser();
-   } catch (Throwable t) {
-   throw new Exception("Cannot access 
UserGroupInformation information for current user", t);
-   }
+   final String currDir = ENV.get(Environment.PWD.key());
+   require(currDir != null, "Current working directory 
variable (%s) not set", Environment.PWD.key());
+   LOG.debug("Current working Directory: {}", currDir);
 
-   LOG.info("YARN daemon runs as user {}. Running Flink 
Application Master/JobManager as user {}",
--- End diff --

Could we keep this log output which prints out the user?


> Support for Kerberos Authentication with Keytab Credential
> --
>
> Key: FLINK-3929
> URL: https://issues.apache.org/jira/browse/FLINK-3929
> Project: Flink
>  Issue Type: New Feature
>Reporter: Eron Wright 
>Assignee: Vijay Srinivasaraghavan
>  Labels: kerberos, security
>   Original Estimate: 672h
>  Remaining Estimate: 672h
>
> _This issue is part of a series of improvements detailed in the [Secure Data 
> Access|https://docs.google.com/document/d/1-GQB6uVOyoaXGwtqwqLV8BHDxWiMO2WnVzBoJ8oPaAs/edit?usp=sharing]
>  design doc._
> Add support for a keytab credential to be associated with the Flink cluster, 
> to facilitate:
> - Kerberos-authenticated data access for connectors
> - Kerberos-authenticated ZooKeeper access
> Support both the standalone and YARN deployment modes.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)