[jira] [Commented] (FLINK-9235) Add Integration test for Flink-Yarn-Kerberos integration for flip-6

[Till Rohrmann (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16623209#comment-16623209
 ] 

Till Rohrmann commented on FLINK-9235:
--

What's the status of this issue [~suez1224] and [~aljoscha]?

> Add Integration test for Flink-Yarn-Kerberos integration for flip-6
> ---
>
> Key: FLINK-9235
> URL: https://issues.apache.org/jira/browse/FLINK-9235
> Project: Flink
>  Issue Type: Test
>Affects Versions: 1.5.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Major
>  Labels: pull-request-available
>
> We need to provide an integration test for flip-6 similar to 
> YARNSessionFIFOSecuredITCase for the legacy deployment mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9235) Add Integration test for Flink-Yarn-Kerberos integration for flip-6

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16531150#comment-16531150
 ] 

ASF GitHub Bot commented on FLINK-9235:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5901


> Add Integration test for Flink-Yarn-Kerberos integration for flip-6
> ---
>
> Key: FLINK-9235
> URL: https://issues.apache.org/jira/browse/FLINK-9235
> Project: Flink
>  Issue Type: Test
>Affects Versions: 1.5.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Major
>  Labels: pull-request-available
>
> We need to provide an integration test for flip-6 similar to 
> YARNSessionFIFOSecuredITCase for the legacy deployment mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9235) Add Integration test for Flink-Yarn-Kerberos integration for flip-6

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467282#comment-16467282
 ] 

ASF GitHub Bot commented on FLINK-9235:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/5901
  
By the way, the test still doesn't catch the bug that you're fixing 
(https://issues.apache.org/jira/browse/FLINK-8286). I think we need proper 
end-to-end tests that really test Flink-Kerberos integration on an actual YARN 
cluster. I have started looking onto using Docker Compose for that, i.e. 
bringing up a hadoop cluster in docker with Kerberos and then running Flink on 
that as an end-to-end test.


> Add Integration test for Flink-Yarn-Kerberos integration for flip-6
> ---
>
> Key: FLINK-9235
> URL: https://issues.apache.org/jira/browse/FLINK-9235
> Project: Flink
>  Issue Type: Test
>Affects Versions: 1.5.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Major
>
> We need to provide an integration test for flip-6 similar to 
> YARNSessionFIFOSecuredITCase for the legacy deployment mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9235) Add Integration test for Flink-Yarn-Kerberos integration for flip-6

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467275#comment-16467275
 ] 

ASF GitHub Bot commented on FLINK-9235:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/5901
  
@suez1224 I resolve https://issues.apache.org/jira/browse/FLINK-9306 via 
#5953. This supersedes this PR, could you please close it?

Thanks a lot for looking into this, by the way, and discovering the 
problem!  


> Add Integration test for Flink-Yarn-Kerberos integration for flip-6
> ---
>
> Key: FLINK-9235
> URL: https://issues.apache.org/jira/browse/FLINK-9235
> Project: Flink
>  Issue Type: Test
>Affects Versions: 1.5.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Major
>
> We need to provide an integration test for flip-6 similar to 
> YARNSessionFIFOSecuredITCase for the legacy deployment mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9235) Add Integration test for Flink-Yarn-Kerberos integration for flip-6

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467177#comment-16467177
 ] 

ASF GitHub Bot commented on FLINK-9235:
---

Github user aljoscha closed the pull request at:

https://github.com/apache/flink/pull/5953


> Add Integration test for Flink-Yarn-Kerberos integration for flip-6
> ---
>
> Key: FLINK-9235
> URL: https://issues.apache.org/jira/browse/FLINK-9235
> Project: Flink
>  Issue Type: Test
>Affects Versions: 1.5.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Major
>
> We need to provide an integration test for flip-6 similar to 
> YARNSessionFIFOSecuredITCase for the legacy deployment mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9235) Add Integration test for Flink-Yarn-Kerberos integration for flip-6

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465888#comment-16465888
 ] 

ASF GitHub Bot commented on FLINK-9235:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/5953
  
hey, it's working now  

cc @zentol 


> Add Integration test for Flink-Yarn-Kerberos integration for flip-6
> ---
>
> Key: FLINK-9235
> URL: https://issues.apache.org/jira/browse/FLINK-9235
> Project: Flink
>  Issue Type: Test
>Affects Versions: 1.5.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Major
>
> We need to provide an integration test for flip-6 similar to 
> YARNSessionFIFOSecuredITCase for the legacy deployment mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9235) Add Integration test for Flink-Yarn-Kerberos integration for flip-6

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465577#comment-16465577
 ] 

ASF GitHub Bot commented on FLINK-9235:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/5953#discussion_r186350834
  
--- Diff: 
flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionFIFOITCase.java 
---
@@ -85,21 +91,56 @@ public void checkForProhibitedLogContents() {
public void testDetachedMode() throws InterruptedException, IOException 
{
LOG.info("Starting testDetachedMode()");
addTestAppender(FlinkYarnSessionCli.class, Level.INFO);
-   Runner runner =
-   startWithArgs(new String[]{"-j", 
flinkUberjar.getAbsolutePath(),
-   "-t", 
flinkLibFolder.getAbsolutePath(),
-   "-n", "1",
-   "-jm", "768",
-   "-tm", "1024",
-   "--name", "MyCustomName", // 
test setting a custom name
-   "--detached"},
+
+   File exampleJarLocation = 
getTestJarPath("StreamingWordCount.jar");
+   // get temporary file for reading input data for wordcount 
example
+   File tmpInFile = tmp.newFile();
+   FileUtils.writeStringToFile(tmpInFile, WordCountData.TEXT);
+
+   ArrayList args = new ArrayList<>();
+   args.add("-j"); args.add(flinkUberjar.getAbsolutePath());
+   args.add("-t"); args.add(flinkLibFolder.getAbsolutePath());
+   args.add("-n"); args.add("1");
+   args.add("-jm"); args.add("768");
+   args.add("-tm"); args.add("1024");
+   if (SecureTestEnvironment.getTestKeytab() != null) {
+   args.add("-D" + 
SecurityOptions.KERBEROS_LOGIN_KEYTAB.key() + "=" + 
SecureTestEnvironment.getTestKeytab());
+   }
+   if (SecureTestEnvironment.getHadoopServicePrincipal() != null) {
+   args.add("-D" + 
SecurityOptions.KERBEROS_LOGIN_PRINCIPAL.key() + "=" + 
SecureTestEnvironment.getHadoopServicePrincipal());
+   }
+   args.add("--name"); args.add("MyCustomName");
+   args.add("--detached");
+   Runner clusterRunner =
+   startWithArgs(
+   args.toArray(new String[args.size()]),
"Flink JobManager is now running on", 
RunTypes.YARN_SESSION);
 
// before checking any strings outputted by the CLI, first give 
it time to return
-   runner.join();
-   checkForLogString("The Flink YARN client has been started in 
detached mode");
+   clusterRunner.join();
 
if (!isNewMode) {
+   checkForLogString("The Flink YARN client has been 
started in detached mode");
+
+   // in legacy mode we have to wait until the TMs are up 
until we can submit the job
+   LOG.info("Waiting until two containers are running");
+   // wait until two containers are running
+   while (getRunningContainers() < 2) {
--- End diff --

there a comment at L 151: `// additional sleep for the JM/TM to start and 
establish connection`, the job execution should probably happen after that...


> Add Integration test for Flink-Yarn-Kerberos integration for flip-6
> ---
>
> Key: FLINK-9235
> URL: https://issues.apache.org/jira/browse/FLINK-9235
> Project: Flink
>  Issue Type: Test
>Affects Versions: 1.5.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Major
>
> We need to provide an integration test for flip-6 similar to 
> YARNSessionFIFOSecuredITCase for the legacy deployment mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9235) Add Integration test for Flink-Yarn-Kerberos integration for flip-6

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465574#comment-16465574
 ] 

ASF GitHub Bot commented on FLINK-9235:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/5953#discussion_r186350483
  
--- Diff: 
flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionFIFOITCase.java 
---
@@ -85,21 +91,56 @@ public void checkForProhibitedLogContents() {
public void testDetachedMode() throws InterruptedException, IOException 
{
LOG.info("Starting testDetachedMode()");
addTestAppender(FlinkYarnSessionCli.class, Level.INFO);
-   Runner runner =
-   startWithArgs(new String[]{"-j", 
flinkUberjar.getAbsolutePath(),
-   "-t", 
flinkLibFolder.getAbsolutePath(),
-   "-n", "1",
-   "-jm", "768",
-   "-tm", "1024",
-   "--name", "MyCustomName", // 
test setting a custom name
-   "--detached"},
+
+   File exampleJarLocation = 
getTestJarPath("StreamingWordCount.jar");
+   // get temporary file for reading input data for wordcount 
example
+   File tmpInFile = tmp.newFile();
+   FileUtils.writeStringToFile(tmpInFile, WordCountData.TEXT);
+
+   ArrayList args = new ArrayList<>();
+   args.add("-j"); args.add(flinkUberjar.getAbsolutePath());
+   args.add("-t"); args.add(flinkLibFolder.getAbsolutePath());
+   args.add("-n"); args.add("1");
+   args.add("-jm"); args.add("768");
+   args.add("-tm"); args.add("1024");
+   if (SecureTestEnvironment.getTestKeytab() != null) {
+   args.add("-D" + 
SecurityOptions.KERBEROS_LOGIN_KEYTAB.key() + "=" + 
SecureTestEnvironment.getTestKeytab());
+   }
+   if (SecureTestEnvironment.getHadoopServicePrincipal() != null) {
+   args.add("-D" + 
SecurityOptions.KERBEROS_LOGIN_PRINCIPAL.key() + "=" + 
SecureTestEnvironment.getHadoopServicePrincipal());
+   }
+   args.add("--name"); args.add("MyCustomName");
+   args.add("--detached");
+   Runner clusterRunner =
+   startWithArgs(
+   args.toArray(new String[args.size()]),
"Flink JobManager is now running on", 
RunTypes.YARN_SESSION);
 
// before checking any strings outputted by the CLI, first give 
it time to return
-   runner.join();
-   checkForLogString("The Flink YARN client has been started in 
detached mode");
+   clusterRunner.join();
 
if (!isNewMode) {
+   checkForLogString("The Flink YARN client has been 
started in detached mode");
+
+   // in legacy mode we have to wait until the TMs are up 
until we can submit the job
+   LOG.info("Waiting until two containers are running");
+   // wait until two containers are running
+   while (getRunningContainers() < 2) {
--- End diff --

is it guaranteed that if the container runs it is already registered with 
the JobManager?


> Add Integration test for Flink-Yarn-Kerberos integration for flip-6
> ---
>
> Key: FLINK-9235
> URL: https://issues.apache.org/jira/browse/FLINK-9235
> Project: Flink
>  Issue Type: Test
>Affects Versions: 1.5.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Major
>
> We need to provide an integration test for flip-6 similar to 
> YARNSessionFIFOSecuredITCase for the legacy deployment mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9235) Add Integration test for Flink-Yarn-Kerberos integration for flip-6

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465561#comment-16465561
 ] 

ASF GitHub Bot commented on FLINK-9235:
---

Github user zentol commented on a diff in the pull request:

https://github.com/apache/flink/pull/5953#discussion_r186343907
  
--- Diff: 
flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionFIFOITCase.java 
---
@@ -85,21 +91,56 @@ public void checkForProhibitedLogContents() {
public void testDetachedMode() throws InterruptedException, IOException 
{
LOG.info("Starting testDetachedMode()");
addTestAppender(FlinkYarnSessionCli.class, Level.INFO);
-   Runner runner =
-   startWithArgs(new String[]{"-j", 
flinkUberjar.getAbsolutePath(),
-   "-t", 
flinkLibFolder.getAbsolutePath(),
-   "-n", "1",
-   "-jm", "768",
-   "-tm", "1024",
-   "--name", "MyCustomName", // 
test setting a custom name
-   "--detached"},
+
+   File exampleJarLocation = 
getTestJarPath("StreamingWordCount.jar");
+   // get temporary file for reading input data for wordcount 
example
+   File tmpInFile = tmp.newFile();
+   FileUtils.writeStringToFile(tmpInFile, WordCountData.TEXT);
+
+   ArrayList args = new ArrayList<>();
+   args.add("-j"); args.add(flinkUberjar.getAbsolutePath());
--- End diff --

  I see what you're trying to do here, but I'm fairly certain this will 
get auto-formatted at some point.

I would do it like this:
```
ArrayList args = new ArrayList<>();
args.add("-j");
args.add(flinkUberjar.getAbsolutePath());

args.add("-t");
args.add(flinkLibFolder.getAbsolutePath());

...
```


> Add Integration test for Flink-Yarn-Kerberos integration for flip-6
> ---
>
> Key: FLINK-9235
> URL: https://issues.apache.org/jira/browse/FLINK-9235
> Project: Flink
>  Issue Type: Test
>Affects Versions: 1.5.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Major
>
> We need to provide an integration test for flip-6 similar to 
> YARNSessionFIFOSecuredITCase for the legacy deployment mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9235) Add Integration test for Flink-Yarn-Kerberos integration for flip-6

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465243#comment-16465243
 ] 

ASF GitHub Bot commented on FLINK-9235:
---

Github user suez1224 commented on a diff in the pull request:

https://github.com/apache/flink/pull/5901#discussion_r186298337
  
--- Diff: 
flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionFIFOITCase.java 
---
@@ -85,54 +88,75 @@ public void checkForProhibitedLogContents() {
public void testDetachedMode() throws InterruptedException, IOException 
{
LOG.info("Starting testDetachedMode()");
addTestAppender(FlinkYarnSessionCli.class, Level.INFO);
-   Runner runner =
-   startWithArgs(new String[]{"-j", 
flinkUberjar.getAbsolutePath(),
-   "-t", 
flinkLibFolder.getAbsolutePath(),
-   "-n", "1",
-   "-jm", "768",
-   "-tm", "1024",
-   "--name", "MyCustomName", // 
test setting a custom name
-   "--detached"},
-   "Flink JobManager is now running on", 
RunTypes.YARN_SESSION);
-
+   File exampleJarLocation = 
getTestJarPath("StreamingWordCount.jar");
--- End diff --

We need to add the jar for the new mode, otherwise no TaskExecutor will be 
brought up.


> Add Integration test for Flink-Yarn-Kerberos integration for flip-6
> ---
>
> Key: FLINK-9235
> URL: https://issues.apache.org/jira/browse/FLINK-9235
> Project: Flink
>  Issue Type: Test
>Affects Versions: 1.5.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Major
>
> We need to provide an integration test for flip-6 similar to 
> YARNSessionFIFOSecuredITCase for the legacy deployment mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9235) Add Integration test for Flink-Yarn-Kerberos integration for flip-6

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462643#comment-16462643
 ] 

ASF GitHub Bot commented on FLINK-9235:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/5953
  
R: @zentol @suez1224 

At least the secured ITCase is currently failing for legacy mode. 
Investigating...


> Add Integration test for Flink-Yarn-Kerberos integration for flip-6
> ---
>
> Key: FLINK-9235
> URL: https://issues.apache.org/jira/browse/FLINK-9235
> Project: Flink
>  Issue Type: Test
>Affects Versions: 1.5.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Major
>
> We need to provide an integration test for flip-6 similar to 
> YARNSessionFIFOSecuredITCase for the legacy deployment mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9235) Add Integration test for Flink-Yarn-Kerberos integration for flip-6

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462641#comment-16462641
 ] 

ASF GitHub Bot commented on FLINK-9235:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/5901
  
@suez1224 I did an alternative version in #5953. All YARN tests currently 
have the problem that they are only executed with the "new" (FLIP-6) mode or 
legacy mode (only the secured IT Case). In my PR I change that to use the 
legacy flag that we specify on Travis. Meaning that all YARN tests now run for 
both configurations.


> Add Integration test for Flink-Yarn-Kerberos integration for flip-6
> ---
>
> Key: FLINK-9235
> URL: https://issues.apache.org/jira/browse/FLINK-9235
> Project: Flink
>  Issue Type: Test
>Affects Versions: 1.5.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Major
>
> We need to provide an integration test for flip-6 similar to 
> YARNSessionFIFOSecuredITCase for the legacy deployment mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9235) Add Integration test for Flink-Yarn-Kerberos integration for flip-6

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462615#comment-16462615
 ] 

ASF GitHub Bot commented on FLINK-9235:
---

GitHub user aljoscha opened a pull request:

https://github.com/apache/flink/pull/5953

[FLINK-9235] Add Integration test for Flink-Yarn-Kerberos integration for 
flip-6

Alternative version of #5901

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aljoscha/flink 
jira-9235-flip-6-yarn-secured-test

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5953.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5953


commit 9ca7852fe1a6f1a86d05ba2f7851199e43411579
Author: Aljoscha Krettek 
Date:   2018-05-03T14:23:29Z

Remove special-case krb5.conf code from YARN runners

commit b4fb889c911ccef0521cd9ed8839190a12f2d4a5
Author: Aljoscha Krettek 
Date:   2018-05-03T14:27:40Z

[FLINK-9235] Test new FLIP-6 code in YARNSessionFIFOSecuredITCase

Before, always setting mode to LEGACY_MODE when security settings are
present caused the test never to run with the new code.

For this, we also need to actually execute an example. Otherwise, no
TaskExecutors would be brought up.




> Add Integration test for Flink-Yarn-Kerberos integration for flip-6
> ---
>
> Key: FLINK-9235
> URL: https://issues.apache.org/jira/browse/FLINK-9235
> Project: Flink
>  Issue Type: Test
>Affects Versions: 1.5.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Major
>
> We need to provide an integration test for flip-6 similar to 
> YARNSessionFIFOSecuredITCase for the legacy deployment mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9235) Add Integration test for Flink-Yarn-Kerberos integration for flip-6

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462393#comment-16462393
 ] 

ASF GitHub Bot commented on FLINK-9235:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/5901
  
@suez1224 Ah, I think the reason this test passes even without your fix in 
#5896 is that the test doesn't really test submission to a YARN cluster on 
different machines. The TaskManager runner will pick up the path of the key tab 
that exists on the local filesystem.


> Add Integration test for Flink-Yarn-Kerberos integration for flip-6
> ---
>
> Key: FLINK-9235
> URL: https://issues.apache.org/jira/browse/FLINK-9235
> Project: Flink
>  Issue Type: Test
>Affects Versions: 1.5.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Major
>
> We need to provide an integration test for flip-6 similar to 
> YARNSessionFIFOSecuredITCase for the legacy deployment mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9235) Add Integration test for Flink-Yarn-Kerberos integration for flip-6

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460815#comment-16460815
 ] 

ASF GitHub Bot commented on FLINK-9235:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/5901
  
If I only remove this line I mentioned earlier from `master` the test 
actually fails for me: 
https://github.com/apache/flink/blob/af5279e9bd3dec18512a54c59982e3dc1f253cd2/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YarnTestBase.java#L531

Because #5896 is not in. I'm still wondering why this PR here succeeds 
without the fixes.


> Add Integration test for Flink-Yarn-Kerberos integration for flip-6
> ---
>
> Key: FLINK-9235
> URL: https://issues.apache.org/jira/browse/FLINK-9235
> Project: Flink
>  Issue Type: Test
>Affects Versions: 1.5.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Major
>
> We need to provide an integration test for flip-6 similar to 
> YARNSessionFIFOSecuredITCase for the legacy deployment mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9235) Add Integration test for Flink-Yarn-Kerberos integration for flip-6

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460810#comment-16460810
 ] 

ASF GitHub Bot commented on FLINK-9235:
---

Github user aljoscha commented on a diff in the pull request:

https://github.com/apache/flink/pull/5901#discussion_r185449426
  
--- Diff: 
flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionFIFOITCase.java 
---
@@ -85,54 +88,75 @@ public void checkForProhibitedLogContents() {
public void testDetachedMode() throws InterruptedException, IOException 
{
LOG.info("Starting testDetachedMode()");
addTestAppender(FlinkYarnSessionCli.class, Level.INFO);
-   Runner runner =
-   startWithArgs(new String[]{"-j", 
flinkUberjar.getAbsolutePath(),
-   "-t", 
flinkLibFolder.getAbsolutePath(),
-   "-n", "1",
-   "-jm", "768",
-   "-tm", "1024",
-   "--name", "MyCustomName", // 
test setting a custom name
-   "--detached"},
-   "Flink JobManager is now running on", 
RunTypes.YARN_SESSION);
-
+   File exampleJarLocation = 
getTestJarPath("StreamingWordCount.jar");
+   // get temporary file for reading input data for wordcount 
example
+   File tmpInFile;
+   try {
+   tmpInFile = tmp.newFile();
+   FileUtils.writeStringToFile(tmpInFile, 
WordCountData.TEXT);
+   }
+   catch (IOException e) {
+   throw new RuntimeException(e);
+   }
+   Runner runner = isNewMode ?
+   startWithArgs(new String[]{"run", "-m", 
"yarn-cluster",
+   "-yj", 
flinkUberjar.getAbsolutePath(),
+   "-yt", 
flinkLibFolder.getAbsolutePath(),
+   "-yn", "1",
+   "-yjm", "768",
+   "-ytm", "1024",
+   "-ynm", 
"MyCustomName", // test setting a custom name
+   "--detached", 
exampleJarLocation.getAbsolutePath(),
+   "--input", 
tmpInFile.getAbsoluteFile().toString()},
+   "Job has been submitted with 
JobID", RunTypes.CLI_FRONTEND) :
+   startWithArgs(new String[]{"-j", 
flinkUberjar.getAbsolutePath(),
+   "-t", 
flinkLibFolder.getAbsolutePath(),
+   "-n", "1",
+   "-jm", "768",
+   "-tm", "1024",
+   "--name", 
"MyCustomName", // test setting a custom name
+   "--detached"},
+   "Flink JobManager is now 
running on", RunTypes.YARN_SESSION);
// before checking any strings outputted by the CLI, first give 
it time to return
runner.join();
-   checkForLogString("The Flink YARN client has been started in 
detached mode");
 
-   if (!isNewMode) {
-   LOG.info("Waiting until two containers are running");
-   // wait until two containers are running
-   while (getRunningContainers() < 2) {
-   sleep(500);
-   }
+   LOG.info("Waiting until two containers are running");
+   // wait until two containers are running
+   while (getRunningContainers() < 2) {
+   sleep(500);
}
 
// additional sleep for the JM/TM to start and establish 
connection
long startTime = System.nanoTime();
while (System.nanoTime() - startTime < 
TimeUnit.NANOSECONDS.convert(10, TimeUnit.SECONDS) &&
!(verifyStringsInNamedLogFiles(
-   new String[]{"YARN Application 
Master started"}, "jobmanager.log") &&
+   new String[]{isNewMode ? 
"JobManager 

[jira] [Commented] (FLINK-9235) Add Integration test for Flink-Yarn-Kerberos integration for flip-6

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460812#comment-16460812
 ] 

ASF GitHub Bot commented on FLINK-9235:
---

Github user aljoscha commented on a diff in the pull request:

https://github.com/apache/flink/pull/5901#discussion_r185448995
  
--- Diff: 
flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionFIFOITCase.java 
---
@@ -85,54 +88,75 @@ public void checkForProhibitedLogContents() {
public void testDetachedMode() throws InterruptedException, IOException 
{
LOG.info("Starting testDetachedMode()");
addTestAppender(FlinkYarnSessionCli.class, Level.INFO);
-   Runner runner =
-   startWithArgs(new String[]{"-j", 
flinkUberjar.getAbsolutePath(),
-   "-t", 
flinkLibFolder.getAbsolutePath(),
-   "-n", "1",
-   "-jm", "768",
-   "-tm", "1024",
-   "--name", "MyCustomName", // 
test setting a custom name
-   "--detached"},
-   "Flink JobManager is now running on", 
RunTypes.YARN_SESSION);
-
+   File exampleJarLocation = 
getTestJarPath("StreamingWordCount.jar");
--- End diff --

I think we shouldn't change the example we run as part of this PR. It's 
somewhat unrelated.


> Add Integration test for Flink-Yarn-Kerberos integration for flip-6
> ---
>
> Key: FLINK-9235
> URL: https://issues.apache.org/jira/browse/FLINK-9235
> Project: Flink
>  Issue Type: Test
>Affects Versions: 1.5.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Major
>
> We need to provide an integration test for flip-6 similar to 
> YARNSessionFIFOSecuredITCase for the legacy deployment mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9235) Add Integration test for Flink-Yarn-Kerberos integration for flip-6

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460811#comment-16460811
 ] 

ASF GitHub Bot commented on FLINK-9235:
---

Github user aljoscha commented on a diff in the pull request:

https://github.com/apache/flink/pull/5901#discussion_r185449249
  
--- Diff: 
flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionFIFOITCase.java 
---
@@ -85,54 +88,75 @@ public void checkForProhibitedLogContents() {
public void testDetachedMode() throws InterruptedException, IOException 
{
LOG.info("Starting testDetachedMode()");
addTestAppender(FlinkYarnSessionCli.class, Level.INFO);
-   Runner runner =
-   startWithArgs(new String[]{"-j", 
flinkUberjar.getAbsolutePath(),
-   "-t", 
flinkLibFolder.getAbsolutePath(),
-   "-n", "1",
-   "-jm", "768",
-   "-tm", "1024",
-   "--name", "MyCustomName", // 
test setting a custom name
-   "--detached"},
-   "Flink JobManager is now running on", 
RunTypes.YARN_SESSION);
-
+   File exampleJarLocation = 
getTestJarPath("StreamingWordCount.jar");
+   // get temporary file for reading input data for wordcount 
example
+   File tmpInFile;
+   try {
+   tmpInFile = tmp.newFile();
+   FileUtils.writeStringToFile(tmpInFile, 
WordCountData.TEXT);
+   }
+   catch (IOException e) {
+   throw new RuntimeException(e);
+   }
+   Runner runner = isNewMode ?
+   startWithArgs(new String[]{"run", "-m", 
"yarn-cluster",
+   "-yj", 
flinkUberjar.getAbsolutePath(),
+   "-yt", 
flinkLibFolder.getAbsolutePath(),
+   "-yn", "1",
+   "-yjm", "768",
+   "-ytm", "1024",
+   "-ynm", 
"MyCustomName", // test setting a custom name
+   "--detached", 
exampleJarLocation.getAbsolutePath(),
+   "--input", 
tmpInFile.getAbsoluteFile().toString()},
+   "Job has been submitted with 
JobID", RunTypes.CLI_FRONTEND) :
--- End diff --

This is now testing per-job mode, which is tested in `YARNITCase`. This 
ITCase is for YARN session mode.

`YARNITCase` should probably be called `YARNPerJobITCase` and we should 
also have a `YARNPerJobSecuredITCase`.


> Add Integration test for Flink-Yarn-Kerberos integration for flip-6
> ---
>
> Key: FLINK-9235
> URL: https://issues.apache.org/jira/browse/FLINK-9235
> Project: Flink
>  Issue Type: Test
>Affects Versions: 1.5.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Major
>
> We need to provide an integration test for flip-6 similar to 
> YARNSessionFIFOSecuredITCase for the legacy deployment mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9235) Add Integration test for Flink-Yarn-Kerberos integration for flip-6

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460790#comment-16460790
 ] 

ASF GitHub Bot commented on FLINK-9235:
---

Github user aljoscha commented on a diff in the pull request:

https://github.com/apache/flink/pull/5901#discussion_r185445414
  
--- Diff: 
flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionFIFOITCase.java 
---
@@ -85,54 +88,75 @@ public void checkForProhibitedLogContents() {
public void testDetachedMode() throws InterruptedException, IOException 
{
LOG.info("Starting testDetachedMode()");
addTestAppender(FlinkYarnSessionCli.class, Level.INFO);
-   Runner runner =
-   startWithArgs(new String[]{"-j", 
flinkUberjar.getAbsolutePath(),
-   "-t", 
flinkLibFolder.getAbsolutePath(),
-   "-n", "1",
-   "-jm", "768",
-   "-tm", "1024",
-   "--name", "MyCustomName", // 
test setting a custom name
-   "--detached"},
-   "Flink JobManager is now running on", 
RunTypes.YARN_SESSION);
-
+   File exampleJarLocation = 
getTestJarPath("StreamingWordCount.jar");
+   // get temporary file for reading input data for wordcount 
example
+   File tmpInFile;
+   try {
+   tmpInFile = tmp.newFile();
+   FileUtils.writeStringToFile(tmpInFile, 
WordCountData.TEXT);
+   }
+   catch (IOException e) {
+   throw new RuntimeException(e);
+   }
+   Runner runner = isNewMode ?
--- End diff --

The reason why the other method always starts a legacy cluster is this 
line: 
https://github.com/apache/flink/blob/af5279e9bd3dec18512a54c59982e3dc1f253cd2/flink-yarn-tests/src/test/java/org/apache/flink/yarn/YarnTestBase.java#L531

We always set mode to legacy if there's Kerberos stuff in the config. I 
don't know yet why that line is there.


> Add Integration test for Flink-Yarn-Kerberos integration for flip-6
> ---
>
> Key: FLINK-9235
> URL: https://issues.apache.org/jira/browse/FLINK-9235
> Project: Flink
>  Issue Type: Test
>Affects Versions: 1.5.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Major
>
> We need to provide an integration test for flip-6 similar to 
> YARNSessionFIFOSecuredITCase for the legacy deployment mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9235) Add Integration test for Flink-Yarn-Kerberos integration for flip-6

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460703#comment-16460703
 ] 

ASF GitHub Bot commented on FLINK-9235:
---

Github user aljoscha commented on a diff in the pull request:

https://github.com/apache/flink/pull/5901#discussion_r185423908
  
--- Diff: 
flink-yarn-tests/src/test/java/org/apache/flink/yarn/YARNSessionFIFOITCase.java 
---
@@ -85,54 +88,75 @@ public void checkForProhibitedLogContents() {
public void testDetachedMode() throws InterruptedException, IOException 
{
LOG.info("Starting testDetachedMode()");
addTestAppender(FlinkYarnSessionCli.class, Level.INFO);
-   Runner runner =
-   startWithArgs(new String[]{"-j", 
flinkUberjar.getAbsolutePath(),
-   "-t", 
flinkLibFolder.getAbsolutePath(),
-   "-n", "1",
-   "-jm", "768",
-   "-tm", "1024",
-   "--name", "MyCustomName", // 
test setting a custom name
-   "--detached"},
-   "Flink JobManager is now running on", 
RunTypes.YARN_SESSION);
-
+   File exampleJarLocation = 
getTestJarPath("StreamingWordCount.jar");
+   // get temporary file for reading input data for wordcount 
example
+   File tmpInFile;
+   try {
+   tmpInFile = tmp.newFile();
+   FileUtils.writeStringToFile(tmpInFile, 
WordCountData.TEXT);
+   }
+   catch (IOException e) {
+   throw new RuntimeException(e);
+   }
+   Runner runner = isNewMode ?
--- End diff --

Why do we need this change? Was it always starting in legacy mode before?


> Add Integration test for Flink-Yarn-Kerberos integration for flip-6
> ---
>
> Key: FLINK-9235
> URL: https://issues.apache.org/jira/browse/FLINK-9235
> Project: Flink
>  Issue Type: Test
>Affects Versions: 1.5.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Major
>
> We need to provide an integration test for flip-6 similar to 
> YARNSessionFIFOSecuredITCase for the legacy deployment mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9235) Add Integration test for Flink-Yarn-Kerberos integration for flip-6

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460680#comment-16460680
 ] 

ASF GitHub Bot commented on FLINK-9235:
---

Github user aljoscha commented on the issue:

https://github.com/apache/flink/pull/5901
  
Shouldn't this fail without the changes in #5896?


> Add Integration test for Flink-Yarn-Kerberos integration for flip-6
> ---
>
> Key: FLINK-9235
> URL: https://issues.apache.org/jira/browse/FLINK-9235
> Project: Flink
>  Issue Type: Test
>Affects Versions: 1.5.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Major
>
> We need to provide an integration test for flip-6 similar to 
> YARNSessionFIFOSecuredITCase for the legacy deployment mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-9235) Add Integration test for Flink-Yarn-Kerberos integration for flip-6

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-9235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16448635#comment-16448635
 ] 

ASF GitHub Bot commented on FLINK-9235:
---

GitHub user suez1224 opened a pull request:

https://github.com/apache/flink/pull/5901

[FLINK-9235][Security] Add integration tests for YARN kerberos integration.

## What is the purpose of the change

Add integration tests for YARN kerberos integration.


## Brief change log

  - Adapte YARN kerberos integration test to run either in legacy mode and 
new mode.


## Verifying this change

This change added tests and can be verified as follows:

  - *Added integration tests for end-to-end deployment on YARN cluster with 
Kerberos and a streaming job.
 

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): ( no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: ( no)
  - The serializers: (no )
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
  - The S3 file system connector: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/suez1224/flink flip6-kerberos

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5901.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5901


commit 021f741ee1dc1a3d3f48e9b44af0e6298c999812
Author: Shuyi Chen 
Date:   2018-04-22T18:13:32Z

Add integration tests for YARN kerberos integration.




> Add Integration test for Flink-Yarn-Kerberos integration for flip-6
> ---
>
> Key: FLINK-9235
> URL: https://issues.apache.org/jira/browse/FLINK-9235
> Project: Flink
>  Issue Type: Test
>Affects Versions: 1.5.0
>Reporter: Shuyi Chen
>Assignee: Shuyi Chen
>Priority: Major
>
> We need to provide an integration test for flip-6 similar to 
> YARNSessionFIFOSecuredITCase for the legacy deployment mode.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)