[GitHub] flink pull request: [FLINK-1616] [client] Overhaul of the client.
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/451#issuecomment-77194681 Allright. I was able to start a YARN session from the frontend. Also the `-h` is working now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1616] [client] Overhaul of the client.
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/451#issuecomment-77206172 Manually merged in 5385e48d94a2df81c8fd6102a889cf42dd93fe2f --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1616] [client] Overhaul of the client.
Github user tillrohrmann commented on the pull request: https://github.com/apache/flink/pull/451#issuecomment-77132462 Except for the YARN test cases which fail, LGTM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1616] [client] Overhaul of the client.
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/451#issuecomment-77131171 The two last build profiles are probably failing because the tests are checking on the output of the CLI frontend. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1616] [client] Overhaul of the client.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/451#discussion_r25763642
--- Diff:
flink-clients/src/main/java/org/apache/flink/client/CliFrontend.java ---
@@ -601,452 +482,295 @@ public int compare(ExecutionGraph o1,
ExecutionGraph o2) {
}
/**
-* Executes the cancel action.
+* Executes the CANCEL action.
*
* @param args Command line arguments for the cancel action.
*/
protected int cancel(String[] args) {
- // Parse command line options
- CommandLine line;
+ LOG.info(Running 'cancel' command.);
+
+ CancelOptions options;
try {
- line = parser.parse(CANCEL_OPTIONS, args, false);
- evaluateGeneralOptions(line);
- }
- catch (MissingOptionException e) {
- return handleArgException(e);
- }
- catch (MissingArgumentException e) {
- return handleArgException(e);
+ options = CliFrontendParser.parseCancelCommand(args);
}
- catch (UnrecognizedOptionException e) {
+ catch (CliArgsException e) {
return handleArgException(e);
}
- catch (Exception e) {
- return handleError(e);
+ catch (Throwable t) {
+ return handleError(t);
}
-
- if (printHelp) {
- printHelpForCancel();
+
+ // evaluate help flag
+ if (options.isPrintHelp()) {
+ CliFrontendParser.printHelpForCancel();
return 0;
}
- String[] cleanedArgs = line.getArgs();
+ String[] cleanedArgs = options.getArgs();
JobID jobId;
if (cleanedArgs.length 0) {
String jobIdString = cleanedArgs[0];
try {
jobId = new
JobID(StringUtils.hexStringToByte(jobIdString));
- } catch (Exception e) {
+ }
+ catch (Exception e) {
+ LOG.error(Error: The value for the Job ID is
not a valid ID.);
System.out.println(Error: The value for the
Job ID is not a valid ID.);
return 1;
}
- } else {
+ }
+ else {
+ LOG.error(Missing JobID in the command line
arguments.);
System.out.println(Error: Specify a Job ID to cancel a
job.);
return 1;
}
try {
- ActorRef jobManager = getJobManager(line,
getGlobalConfiguration());
-
- if (jobManager == null) {
- return 1;
- }
-
- final FutureObject response =
Patterns.ask(jobManager, new CancelJob(jobId),
- new Timeout(getAkkaTimeout()));
+ ActorRef jobManager = getJobManager(options);
+ FutureObject response = Patterns.ask(jobManager, new
CancelJob(jobId), new Timeout(askTimeout));
try {
- Await.ready(response, getAkkaTimeout());
- } catch (Exception exception) {
- throw new IOException(Canceling the job with
job ID + jobId + failed.,
- exception);
+ Await.result(response, askTimeout);
--- End diff --
```Await.ready``` should be sufficient, because we don't do anything with
the return value.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1616] [client] Overhaul of the client.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/451#discussion_r25763670
--- Diff:
flink-clients/src/main/java/org/apache/flink/client/CliFrontend.java ---
@@ -601,452 +482,295 @@ public int compare(ExecutionGraph o1,
ExecutionGraph o2) {
}
/**
-* Executes the cancel action.
+* Executes the CANCEL action.
*
* @param args Command line arguments for the cancel action.
*/
protected int cancel(String[] args) {
- // Parse command line options
- CommandLine line;
+ LOG.info(Running 'cancel' command.);
+
+ CancelOptions options;
try {
- line = parser.parse(CANCEL_OPTIONS, args, false);
- evaluateGeneralOptions(line);
- }
- catch (MissingOptionException e) {
- return handleArgException(e);
- }
- catch (MissingArgumentException e) {
- return handleArgException(e);
+ options = CliFrontendParser.parseCancelCommand(args);
}
- catch (UnrecognizedOptionException e) {
+ catch (CliArgsException e) {
return handleArgException(e);
}
- catch (Exception e) {
- return handleError(e);
+ catch (Throwable t) {
+ return handleError(t);
}
-
- if (printHelp) {
- printHelpForCancel();
+
+ // evaluate help flag
+ if (options.isPrintHelp()) {
+ CliFrontendParser.printHelpForCancel();
return 0;
}
- String[] cleanedArgs = line.getArgs();
+ String[] cleanedArgs = options.getArgs();
JobID jobId;
if (cleanedArgs.length 0) {
String jobIdString = cleanedArgs[0];
try {
jobId = new
JobID(StringUtils.hexStringToByte(jobIdString));
- } catch (Exception e) {
+ }
+ catch (Exception e) {
+ LOG.error(Error: The value for the Job ID is
not a valid ID.);
System.out.println(Error: The value for the
Job ID is not a valid ID.);
return 1;
}
- } else {
+ }
+ else {
+ LOG.error(Missing JobID in the command line
arguments.);
System.out.println(Error: Specify a Job ID to cancel a
job.);
return 1;
}
try {
- ActorRef jobManager = getJobManager(line,
getGlobalConfiguration());
-
- if (jobManager == null) {
- return 1;
- }
-
- final FutureObject response =
Patterns.ask(jobManager, new CancelJob(jobId),
- new Timeout(getAkkaTimeout()));
+ ActorRef jobManager = getJobManager(options);
+ FutureObject response = Patterns.ask(jobManager, new
CancelJob(jobId), new Timeout(askTimeout));
try {
- Await.ready(response, getAkkaTimeout());
- } catch (Exception exception) {
- throw new IOException(Canceling the job with
job ID + jobId + failed.,
- exception);
+ Await.result(response, askTimeout);
--- End diff --
My bad, we want the exception.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1616] [client] Overhaul of the client.
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/451#discussion_r25765068
--- Diff:
flink-runtime/src/main/scala/org/apache/flink/runtime/messages/JobManagerMessages.scala
---
@@ -19,11 +19,14 @@
package org.apache.flink.runtime.messages
import org.apache.flink.runtime.accumulators.AccumulatorEvent
+import org.apache.flink.runtime.client.JobStatusMessage
import org.apache.flink.runtime.executiongraph.{ExecutionAttemptID,
ExecutionGraph}
import org.apache.flink.runtime.instance.{InstanceID, Instance}
import org.apache.flink.runtime.jobgraph.{JobGraph, JobID, JobStatus,
JobVertexID}
import org.apache.flink.runtime.taskmanager.TaskExecutionState
+import scala.collection.JavaConverters._
--- End diff --
There are also explicit imports of ```JavaConverters._``` in
```RegisteredTaskManagers``` which then should be removed if putting the import
here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-1616] [client] Overhaul of the client.
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/451#issuecomment-77134702 The `-h` command is not working for the actions (tested with info and run) ``` [root@sandbox flink-yarn-0.9-SNAPSHOT]# ./bin/flink info -h 10:28:01,440 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable The program JAR file was not specified. Use the help option (-h or --help) to get help on the command. ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1616] [client] Overhaul of the client.
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/451#issuecomment-77135314 The YARN application master doesn't start anymore with the following exception: ``` 10:31:01,526 ERROR org.apache.flink.yarn.ApplicationMaster$ - Error while running the application master. java.lang.Exception: Missing parameter '--executionMode' at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$parseArgs$1.apply(JobManager.scala:796) at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$parseArgs$1.apply(JobManager.scala:790) at scala.Option.map(Option.scala:145) at org.apache.flink.runtime.jobmanager.JobManager$.parseArgs(JobManager.scala:789) at org.apache.flink.yarn.ApplicationMaster$.startJobManager(ApplicationMaster.scala:200) at org.apache.flink.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:88) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:356) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1471) at org.apache.flink.yarn.ApplicationMaster$.main(ApplicationMaster.scala:58) at org.apache.flink.yarn.ApplicationMaster.main(ApplicationMaster.scala) ``` That's probably also the reason why the YARN tests are failing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1616] [client] Overhaul of the client.
GitHub user StephanEwen opened a pull request: https://github.com/apache/flink/pull/451 [FLINK-1616] [client] Overhaul of the client. - Fix bugs with non-serializable messages - Separate parser and action logic - Clean up tests - Vastly improve logging in CLI client - Additional tests for parsing / config setup in the command line client You can merge this pull request into a Git repository by running: $ git pull https://github.com/StephanEwen/incubator-flink master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/451.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #451 commit 23190a553f3af015494e72d93a6d9615972c9b2a Author: Stephan Ewen se...@apache.org Date: 2015-03-03T20:49:37Z [FLINK-1631] [client] Overhaul of the client. - Fix bugs with non-serializable messages - Separate parser and action logic - Clean up tests - Vastly improve logging in CLI client - Additional tests for parsing / config setup in the command line client --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
