[GitHub] flink pull request #6140: [FLINK-9554] flink scala shell doesn't work in yar...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/6140 ---
[GitHub] flink pull request #6140: [FLINK-9554] flink scala shell doesn't work in yar...
Github user zjffdu commented on a diff in the pull request: https://github.com/apache/flink/pull/6140#discussion_r199817183 --- Diff: flink-scala-shell/start-script/start-scala-shell.sh --- @@ -19,6 +19,9 @@ # from scala-lang 2.10.4 +# Uncomment the following line to enable remote debug +# export FLINK_SCALA_SHELL_JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005" --- End diff -- Created PR #6245 for it ---
[GitHub] flink pull request #6140: [FLINK-9554] flink scala shell doesn't work in yar...
Github user zjffdu commented on a diff in the pull request: https://github.com/apache/flink/pull/6140#discussion_r199813552 --- Diff: flink-scala-shell/start-script/start-scala-shell.sh --- @@ -19,6 +19,9 @@ # from scala-lang 2.10.4 +# Uncomment the following line to enable remote debug +# export FLINK_SCALA_SHELL_JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005" --- End diff -- Created FLINK-9718 for it ---
[GitHub] flink pull request #6140: [FLINK-9554] flink scala shell doesn't work in yar...
Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/6140#discussion_r199760246 --- Diff: flink-scala-shell/start-script/start-scala-shell.sh --- @@ -19,6 +19,9 @@ # from scala-lang 2.10.4 +# Uncomment the following line to enable remote debug +# export FLINK_SCALA_SHELL_JAVA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005" --- End diff -- please move this into a separate commit ---
[GitHub] flink pull request #6140: [FLINK-9554] flink scala shell doesn't work in yar...
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/6140#discussion_r199760506
--- Diff:
flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontend.java ---
@@ -166,6 +166,10 @@ public Configuration getConfiguration() {
return copiedConfiguration;
}
+ public static Options getCustomCommandLineOptions() {
--- End diff --
it should be possible for this getter to be non-static. Then the changes to
the `customCommandLineOptions` field are unnecessary, and the in `FlinkShell`
replace `CliFrontend.getCustomCommandLineOptions()` with
`frontend.getCustomCommandLineOptions`
---
[GitHub] flink pull request #6140: [FLINK-9554] flink scala shell doesn't work in yar...
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/6140#discussion_r199752165
--- Diff:
flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontend.java ---
@@ -166,6 +166,10 @@ public Configuration getConfiguration() {
return copiedConfiguration;
}
+ public static Options getCustomCommandLineOptions() {
--- End diff --
so.why does this getter have to be static?
---
[GitHub] flink pull request #6140: [FLINK-9554] flink scala shell doesn't work in yar...
Github user zjffdu commented on a diff in the pull request:
https://github.com/apache/flink/pull/6140#discussion_r199750956
--- Diff:
flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkShell.scala ---
@@ -255,14 +257,25 @@ object FlinkShell {
yarnConfig.queue.foreach((queue) => args ++= Seq("-yqu",
queue.toString))
yarnConfig.slots.foreach((slots) => args ++= Seq("-ys",
slots.toString))
+val customCommandLines = CliFrontend.loadCustomCommandLines(
+ configuration,configurationDirectory)
+val commandOptions = CliFrontendParser.getRunCommandOptions
+val customCommandLineOptions = new Options()
+customCommandLines.asScala.foreach(cmd => {
--- End diff --
@zentol I have fixed that, but I have to make `customCommandLineOptions`
of `CliFrontend` to be static (a little ugly)
---
[GitHub] flink pull request #6140: [FLINK-9554] flink scala shell doesn't work in yar...
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/6140#discussion_r199724031
--- Diff:
flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkShell.scala ---
@@ -255,14 +257,25 @@ object FlinkShell {
yarnConfig.queue.foreach((queue) => args ++= Seq("-yqu",
queue.toString))
yarnConfig.slots.foreach((slots) => args ++= Seq("-ys",
slots.toString))
+val customCommandLines = CliFrontend.loadCustomCommandLines(
+ configuration,configurationDirectory)
+val commandOptions = CliFrontendParser.getRunCommandOptions
+val customCommandLineOptions = new Options()
+customCommandLines.asScala.foreach(cmd => {
--- End diff --
this is already done int he `CliFrontend` constructor. It may be sufficient
to switch the initialization of `frontend` and `commandLine`.
---
[GitHub] flink pull request #6140: [FLINK-9554] flink scala shell doesn't work in yar...
GitHub user zjffdu opened a pull request: https://github.com/apache/flink/pull/6140 [FLINK-9554] flink scala shell doesn't work in yarn mode ## What is the purpose of the change This PR is trying to fix the issue of scala-shell unable to run in yarn mode. ## Brief change log The root cause is the options of CustomCommandLine is missed which cause the "-m yarn-cluster" can not be parsed correctly. ## Verifying this change I verify it manually on a single node hadoop cluster. I think it is better to add integration for that so that we can avoid the regression issue in future. If necessary, I can create a ticket for it and do it in another PR. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): ( no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no ) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? ( not documented) You can merge this pull request into a Git repository by running: $ git pull https://github.com/zjffdu/flink FLINK-9554 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/6140.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6140 commit f3d37d4799e51323fbbcd52cf972ac1d874d2514 Author: Jeff Zhang Date: 2018-06-07T09:47:32Z save commit d65058133ce220e2d4212b906d521b38b9ef53dd Author: Jeff Zhang Date: 2018-06-08T02:39:27Z [FLINK-9554] flink scala shell doesn't work in yarn mode ---
