[jira] [Commented] (FLINK-8347) Make Cluster id typesafe

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324650#comment-16324650
 ] 

ASF GitHub Bot commented on FLINK-8347:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5232


> Make Cluster id typesafe
> 
>
> Key: FLINK-8347
> URL: https://issues.apache.org/jira/browse/FLINK-8347
> Project: Flink
>  Issue Type: Sub-task
>  Components: Client
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>  Labels: flip-6
> Fix For: 1.5.0
>
>
> Currently, the cluster id is of type {{String}}. We should make the id 
> typesafe to avoid mixups between different {{CustomCommandLines}} and 
> {{ClusterDescriptors}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-8347) Make Cluster id typesafe

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323798#comment-16323798
 ] 

ASF GitHub Bot commented on FLINK-8347:
---

Github user tillrohrmann commented on the issue:

https://github.com/apache/flink/pull/5232
  
Thanks a lot for the review @GJL. Rebased onto the latest master. Merging 
once Travis gives green light.


> Make Cluster id typesafe
> 
>
> Key: FLINK-8347
> URL: https://issues.apache.org/jira/browse/FLINK-8347
> Project: Flink
>  Issue Type: Sub-task
>  Components: Client
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>  Labels: flip-6
> Fix For: 1.5.0
>
>
> Currently, the cluster id is of type {{String}}. We should make the id 
> typesafe to avoid mixups between different {{CustomCommandLines}} and 
> {{ClusterDescriptors}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-8347) Make Cluster id typesafe

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323107#comment-16323107
 ] 

ASF GitHub Bot commented on FLINK-8347:
---

Github user tillrohrmann commented on a diff in the pull request:

https://github.com/apache/flink/pull/5232#discussion_r161094149
  
--- Diff: 
flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java
 ---
@@ -367,40 +366,31 @@ public ClusterClient retrieve(String applicationID) {
flinkConfiguration,
false);
} catch (Exception e) {
-   throw new RuntimeException("Couldn't retrieve Yarn 
cluster", e);
+   throw new ClusterRetrieveException("Couldn't retrieve 
Yarn cluster", e);
}
}
 
@Override
-   public YarnClusterClient deploySessionCluster(ClusterSpecification 
clusterSpecification) {
+   public YarnClusterClient deploySessionCluster(ClusterSpecification 
clusterSpecification) throws ClusterDeploymentException {
try {
return deployInternal(
clusterSpecification,
getYarnSessionClusterEntrypoint(),
null);
} catch (Exception e) {
-   throw new RuntimeException("Couldn't deploy Yarn 
session cluster", e);
+   throw new ClusterDeploymentException("Couldn't deploy 
Yarn session cluster", e);
}
}
 
@Override
-   public YarnClusterClient deployJobCluster(ClusterSpecification 
clusterSpecification, JobGraph jobGraph) {
+   public YarnClusterClient deployJobCluster(ClusterSpecification 
clusterSpecification, JobGraph jobGraph) throws ClusterDeploymentException {
try {
return deployInternal(
clusterSpecification,
getYarnJobClusterEntrypoint(),
jobGraph);
} catch (Exception e) {
-   throw new RuntimeException("Could not deploy Yarn job 
cluster.", e);
-   }
-   }
-
-   @Override
--- End diff --

Good catch. Will add it.


> Make Cluster id typesafe
> 
>
> Key: FLINK-8347
> URL: https://issues.apache.org/jira/browse/FLINK-8347
> Project: Flink
>  Issue Type: Sub-task
>  Components: Client
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>  Labels: flip-6
> Fix For: 1.5.0
>
>
> Currently, the cluster id is of type {{String}}. We should make the id 
> typesafe to avoid mixups between different {{CustomCommandLines}} and 
> {{ClusterDescriptors}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-8347) Make Cluster id typesafe

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16316731#comment-16316731
 ] 

ASF GitHub Bot commented on FLINK-8347:
---

Github user GJL commented on a diff in the pull request:

https://github.com/apache/flink/pull/5232#discussion_r160216156
  
--- Diff: 
flink-clients/src/test/java/org/apache/flink/client/cli/CliFrontendStopTest.java
 ---
@@ -98,24 +103,26 @@ public void testUnknownJobId() throws Exception {
JobID jid = new JobID();
 
String[] parameters = { jid.toString() };
-   final ClusterClient clusterClient = createClusterClient(true);
+   String expectedMessage = "Test exception";
+   FlinkException testException = new 
FlinkException(expectedMessage);
+   final ClusterClient clusterClient = 
createClusterClient(testException);
MockedCliFrontend testFrontend = new 
MockedCliFrontend(clusterClient);
 
try {
testFrontend.stop(parameters);
fail("Should have failed.");
-   } catch (IllegalArgumentException ignored) {
-   // expected
+   } catch (FlinkException e) {
+   assertTrue(ExceptionUtils.findThrowableWithMessage(e, 
expectedMessage).isPresent());
}
 
Mockito.verify(clusterClient, times(1)).stop(any(JobID.class));
--- End diff --

I think I have written it on a previous PR: `verify` shouldn't be needed 
because you already assert if the exception was thrown.


> Make Cluster id typesafe
> 
>
> Key: FLINK-8347
> URL: https://issues.apache.org/jira/browse/FLINK-8347
> Project: Flink
>  Issue Type: Sub-task
>  Components: Client
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>  Labels: flip-6
> Fix For: 1.5.0
>
>
> Currently, the cluster id is of type {{String}}. We should make the id 
> typesafe to avoid mixups between different {{CustomCommandLines}} and 
> {{ClusterDescriptors}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-8347) Make Cluster id typesafe

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16316730#comment-16316730
 ] 

ASF GitHub Bot commented on FLINK-8347:
---

Github user GJL commented on a diff in the pull request:

https://github.com/apache/flink/pull/5232#discussion_r160209875
  
--- Diff: 
flink-yarn/src/test/java/org/apache/flink/yarn/AbstractYarnClusterTest.java ---
@@ -83,7 +83,7 @@ public void 
testClusterClientRetrievalOfFinishedYarnApplication() throws IOExcep
temporaryFolder.newFolder().getAbsolutePath(),
yarnClient);
 
-   clusterDescriptor.retrieve(clusterId);
+   clusterDescriptor.retrieve(applicationId);
 
fail("We should not be able to retrieve ClusterClient for a 
finished Yarn application.");
--- End diff --

As written on a previous PR, `fail` should not be used in this case.


> Make Cluster id typesafe
> 
>
> Key: FLINK-8347
> URL: https://issues.apache.org/jira/browse/FLINK-8347
> Project: Flink
>  Issue Type: Sub-task
>  Components: Client
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>  Labels: flip-6
> Fix For: 1.5.0
>
>
> Currently, the cluster id is of type {{String}}. We should make the id 
> typesafe to avoid mixups between different {{CustomCommandLines}} and 
> {{ClusterDescriptors}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-8347) Make Cluster id typesafe

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16316656#comment-16316656
 ] 

ASF GitHub Bot commented on FLINK-8347:
---

Github user GJL commented on a diff in the pull request:

https://github.com/apache/flink/pull/5232#discussion_r160208748
  
--- Diff: 
flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java
 ---
@@ -367,40 +366,31 @@ public ClusterClient retrieve(String applicationID) {
flinkConfiguration,
false);
} catch (Exception e) {
-   throw new RuntimeException("Couldn't retrieve Yarn 
cluster", e);
+   throw new ClusterRetrieveException("Couldn't retrieve 
Yarn cluster", e);
}
}
 
@Override
-   public YarnClusterClient deploySessionCluster(ClusterSpecification 
clusterSpecification) {
+   public YarnClusterClient deploySessionCluster(ClusterSpecification 
clusterSpecification) throws ClusterDeploymentException {
try {
return deployInternal(
clusterSpecification,
getYarnSessionClusterEntrypoint(),
null);
} catch (Exception e) {
-   throw new RuntimeException("Couldn't deploy Yarn 
session cluster", e);
+   throw new ClusterDeploymentException("Couldn't deploy 
Yarn session cluster", e);
}
}
 
@Override
-   public YarnClusterClient deployJobCluster(ClusterSpecification 
clusterSpecification, JobGraph jobGraph) {
+   public YarnClusterClient deployJobCluster(ClusterSpecification 
clusterSpecification, JobGraph jobGraph) throws ClusterDeploymentException {
try {
return deployInternal(
clusterSpecification,
getYarnJobClusterEntrypoint(),
jobGraph);
} catch (Exception e) {
-   throw new RuntimeException("Could not deploy Yarn job 
cluster.", e);
-   }
-   }
-
-   @Override
--- End diff --

You should add `@Override` to the other overload of `terminateCluster` now.


> Make Cluster id typesafe
> 
>
> Key: FLINK-8347
> URL: https://issues.apache.org/jira/browse/FLINK-8347
> Project: Flink
>  Issue Type: Sub-task
>  Components: Client
>Affects Versions: 1.5.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>  Labels: flip-6
> Fix For: 1.5.0
>
>
> Currently, the cluster id is of type {{String}}. We should make the id 
> typesafe to avoid mixups between different {{CustomCommandLines}} and 
> {{ClusterDescriptors}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (FLINK-8347) Make Cluster id typesafe

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-8347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308073#comment-16308073
 ] 

ASF GitHub Bot commented on FLINK-8347:
---

GitHub user tillrohrmann opened a pull request:

https://github.com/apache/flink/pull/5232

[FLINK-8347] [flip6] Make cluster id used by ClusterDescriptor typesafe

## What is the purpose of the change

The ClusterDescriptor uses a typed cluster id for the ClusterClient 
retrieval.
Moreover, the ClusterClient and the CustomCommandLine are typed accordingly.

This PR is based on #5229.

## Brief change log

- Introduce generic type parameter to `ClusterClient`, `ClusterDescriptor` 
and `CustomCommandLine`
- Make cluster id adhere to generic type parameter

## Verifying this change

- Covered by existing tests

## Does this pull request potentially affect one of the following parts:

  - Dependencies (does it add or upgrade a dependency): (no)
  - The public API, i.e., is any changed class annotated with 
`@Public(Evolving)`: (no)
  - The serializers: (no)
  - The runtime per-record code paths (performance sensitive): (no)
  - Anything that affects deployment or recovery: JobManager (and its 
components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes)
  - The S3 file system connector: (no)

## Documentation

  - Does this pull request introduce a new feature? (no)
  - If yes, how is the feature documented? (not applicable)


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tillrohrmann/flink introduceTypedClusterId

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/5232.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #5232


commit c48324702b0ebff3fd7a16e480ead90fc4c9a30b
Author: Till Rohrmann 
Date:   2017-12-07T12:57:24Z

[FLINK-8328] [flip6] Move Yarn ApplicationStatus polling out of 
YarnClusterClient

Introduce YarnApplicationStatusMonitor which does the Yarn 
ApplicationStatus polling in
the FlinkYarnSessionCli. This decouples the YarnClusterClient from the 
actual communication
with Yarn and, thus, gives a better separation of concerns.

commit 783bd1daf36b34a74cbe718a52a1043ba38d5a44
Author: Till Rohrmann 
Date:   2017-12-20T15:43:21Z

[FLINK-8329] [flip6] Move YarnClient to AbstractYarnClusterDescriptor

Moves the YarnClient from the YarnClusterClient to the 
AbstractYarnClusterDescriptor.
This makes the latter responsible for the lifecycle management of the 
client and gives
a better separation of concerns.

commit 192adb786a48d19b71e797355500652d51de6296
Author: Till Rohrmann 
Date:   2017-12-18T17:59:30Z

[FLINK-8332] [flip6] Move savepoint dispose into ClusterClient

Move the savepoint disposal logic from the CliFrontend into the 
ClusterClient. This gives
a better separation of concerns and allows the CliFrontend to be used with 
different
ClusterClient implementations.

commit a73b6e2850d4b2445a835914ba570d1057e59dfb
Author: Till Rohrmann 
Date:   2018-01-02T06:42:18Z

[FLINK-8333] [flip6] Separate deployment options from command options

This commit separates the parsing of command options and deployment options 
into two
steps. This makes it easier to make the CustomCommandLines non-static.

Moreover, this commit moves the CliFrontend into the cli sub package.

commit 57df8f9e1c5f4015cd6543ded43849a71003ef36
Author: Till Rohrmann 
Date:   2018-01-02T06:59:34Z

[FLINK-8338] [flip6] Make CustomCommandLines non static in CliFrontend

This commit changes how CustomCommandLines are registered at the 
CliFrontend.
Henceforth, the CliFrontend is initialized with the set of 
CustomCommandLines
instead of registering them statically. This improves maintainability and
testability.

commit 2ee8059b949b4a9c2e165f3e78716ae531e5c0e7
Author: Till Rohrmann 
Date:   2018-01-02T08:22:12Z

[FLINK-8339] [flip6] Let CustomCommandLine return ClusterDescriptor

Instead of directly retrieving or deploying a Flink cluster, the
CustomCommandLine now only returns a ClusterDescriptor which can be used
for these operations. This disentangles the ClusterDescriptor and the
CustomCommandLine a bit better supporting a proper lifecycle management
of the former.

commit 9ff62c7cfadc977d02f414ee26fae6b76a568eda
Author: Till Rohrmann 
Date:   2018-01-02T08:55:13Z

[FLINK-8340] [flip6] Remove passing of Configuration to CustomCommandLine

Since the Configuration does not change over the lifetime of a