[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15159386#comment-15159386 ] ASF GitHub Bot commented on FLINK-2021: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/1581 > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Assignee: Stefano Baghino >Priority: Minor > Labels: starter > Fix For: 1.0.0 > > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15159091#comment-15159091 ] ASF GitHub Bot commented on FLINK-2021: --- Github user stefanobaghino commented on the pull request: https://github.com/apache/flink/pull/1581#issuecomment-187764378 @rmetzger Thanks, it's been a nice starter, thanks to both you and @fhueske for the help! > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Assignee: Stefano Baghino >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15159056#comment-15159056 ] ASF GitHub Bot commented on FLINK-2021: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1581#issuecomment-187752847 +1 to merge the changes. I rebased the code already locally. Right now, I'm running a final travis test. If it passes, I'll merge the change. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Assignee: Stefano Baghino >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15157611#comment-15157611 ] ASF GitHub Bot commented on FLINK-2021: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1581#issuecomment-187364918 Hi, the error is not related to your changes. Its an issue of our build system (an inconsistent file has been cached). From a CI point of view, the changes look good to be merged. I try to review your changes tomorrow. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Assignee: Stefano Baghino >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15157102#comment-15157102 ] ASF GitHub Bot commented on FLINK-2021: --- Github user stefanobaghino commented on the pull request: https://github.com/apache/flink/pull/1581#issuecomment-187221864 Was this a hiccup in the build or an issue related to my changes? I really can't tell from the error message. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Assignee: Stefano Baghino >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15152442#comment-15152442 ] ASF GitHub Bot commented on FLINK-2021: --- Github user stefanobaghino commented on the pull request: https://github.com/apache/flink/pull/1581#issuecomment-185772362 Three build configurations passes, one is currently running, the other fails and I've spotted this stacktrace: ``` [ERROR] error: error while loading , Error accessing /home/travis/.m2/repository/org/scalanlp/breeze_2.11/0.11.2/breeze_2.11-0.11.2.jar [ERROR] error: scala.reflect.internal.MissingRequirementError: object java.lang.Object in compiler mirror not found. [ERROR] at scala.reflect.internal.MissingRequirementError$.signal(MissingRequirementError.scala:17) [ERROR] at scala.reflect.internal.MissingRequirementError$.notFound(MissingRequirementError.scala:18) ``` > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Assignee: Stefano Baghino >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15150496#comment-15150496 ] ASF GitHub Bot commented on FLINK-2021: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1581#issuecomment-185204685 You can rebase your PR's branch to our current master and then force push it again. This will update the PR. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Assignee: Stefano Baghino >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15150484#comment-15150484 ] ASF GitHub Bot commented on FLINK-2021: --- Github user stefanobaghino commented on the pull request: https://github.com/apache/flink/pull/1581#issuecomment-185200365 I see there's a conflict, perhaps due to my minor fix in `YARNSessionFIFOITCase.java`, can I help fix this one? At this point I'm pretty sure it's the last thing missing. Thanks for the help with the YARN tests. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Assignee: Stefano Baghino >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15149186#comment-15149186 ] ASF GitHub Bot commented on FLINK-2021: --- Github user stefanobaghino commented on the pull request: https://github.com/apache/flink/pull/1581#issuecomment-184850520 Thanks for the tip, I'll try to replicate the S3 setup on our fork as well, it'll probably come in handy for similar cases. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Assignee: Stefano Baghino >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15148863#comment-15148863 ] ASF GitHub Bot commented on FLINK-2021: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1581#issuecomment-184754971 To get to the logs, I executed your branch in my travis account. I have a S3 bucket configured where we upload the full logs to (travis has a log limit of ~5 mb): https://travis-ci.org/rmetzger/flink/builds/109557167 Then, in the log of one build: https://s3.amazonaws.com/archive.travis-ci.org/jobs/109557168/log.txt you'll find the link to the s3 bucket: https://s3.amazonaws.com/flink-logs-us/travis-artifacts/rmetzger/flink/1419/1419.1.tar.gz unpacking that archive, you'll find the output in the `1.log` file. The YARN test is executing the WordCount example, and I guess the arguments passed are wrong. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Assignee: Stefano Baghino >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15148860#comment-15148860 ] ASF GitHub Bot commented on FLINK-2021: --- Github user stefanobaghino commented on the pull request: https://github.com/apache/flink/pull/1581#issuecomment-184753539 @rmetzger I'm looking at the logs as well but I couldn't find that stacktrace in particular. I've looked for `ParameterTool.fromArgs` (that is present in the stacktrace you extracted) in the raw logs of the three failing builds but found nothing... > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Assignee: Stefano Baghino >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15148843#comment-15148843 ] ASF GitHub Bot commented on FLINK-2021: --- Github user stefanobaghino commented on the pull request: https://github.com/apache/flink/pull/1581#issuecomment-184747590 Thanks for digging up the tests, I missed that exception in particular, I'll fix it right away. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Assignee: Stefano Baghino >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15148448#comment-15148448 ] ASF GitHub Bot commented on FLINK-2021: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1581#issuecomment-184624521 The `testDetachedPerJobYarnCluster()` YARN test is failing with ``` 10:33:26,969 INFO org.apache.flink.client.CliFrontend - Starting execution of program 10:33:26,969 INFO org.apache.flink.client.program.Client - Starting program in interactive mode 10:33:26,971 ERROR org.apache.flink.client.CliFrontend - Error while running the command. org.apache.flink.client.program.ProgramInvocationException: The main method caused an error. at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:520) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:403) at org.apache.flink.client.program.Client.runDetached(Client.java:277) at org.apache.flink.client.CliFrontend.executeProgramDetached(CliFrontend.java:774) at org.apache.flink.client.CliFrontend.run(CliFrontend.java:320) at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1112) at org.apache.flink.yarn.YarnTestBase$Runner.run(YarnTestBase.java:565) Caused by: java.lang.RuntimeException: Error parsing arguments '[/tmp/junit2827618452146338513/junit4708065236621010716.tmp, /tmp/junit2827618452146338513/junit2411517217475739253]' on '/tmp/junit2827618452146338513/junit4708065236621010716.tmp'. Unexpected value. Please prefix values with -- or -. at org.apache.flink.api.java.utils.ParameterTool.fromArgs(ParameterTool.java:107) at org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:58) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:505) ... 6 more 10:33:26,974 INFO akka.remote.RemoteActorRefProvider$RemotingTerminator - Shutting down remote daemon. ``` > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Assignee: Stefano Baghino >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15148316#comment-15148316 ] ASF GitHub Bot commented on FLINK-2021: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1581#issuecomment-184594465 Looks like the YARN tests are failing. I'll look into it. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Assignee: Stefano Baghino >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15148303#comment-15148303 ] ASF GitHub Bot commented on FLINK-2021: --- Github user stefanobaghino commented on the pull request: https://github.com/apache/flink/pull/1581#issuecomment-184589232 I think this should do it. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Assignee: Stefano Baghino >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15144281#comment-15144281 ] ASF GitHub Bot commented on FLINK-2021: --- Github user fhueske commented on the pull request: https://github.com/apache/flink/pull/1581#issuecomment-183240236 Hi @stefanobaghino, that looks good! Maybe print the usage string (`"Usage: KMeans --points --centroids --output --iterations "`) always at the beginning of the program such that all parameters are shown? Otherwise, the iterations parameter will never be shown. Thanks! > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Assignee: Stefano Baghino >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15144053#comment-15144053 ] ASF GitHub Bot commented on FLINK-2021: --- Github user stefanobaghino commented on the pull request: https://github.com/apache/flink/pull/1581#issuecomment-183181049 @fhueske If this is what you had in mind I can go on and apply it to the rest of the examples. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Assignee: Stefano Baghino >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15141564#comment-15141564 ] ASF GitHub Bot commented on FLINK-2021: --- Github user fhueske commented on the pull request: https://github.com/apache/flink/pull/1581#issuecomment-182555170 Thanks a lot for updating the PR @stefanobaghino! It's up to you if you want to squash the commits. We can also do it before we merge the PR. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Assignee: Stefano Baghino >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15141409#comment-15141409 ] ASF GitHub Bot commented on FLINK-2021: --- Github user stefanobaghino commented on the pull request: https://github.com/apache/flink/pull/1581#issuecomment-182520553 That's absolutely not a problem @fhueske, your remarks are reasonable and I do appreciate the care you all take in making sure Flink turns out great. It's my first non-trivial PR on Flink (or any major open source project, for that matter) and the guidance you are both providing is extremely precious. I agree with your remarks and will make sure to provide the required fixes ASAP. Feel free to feed any more suggestions asynchronously, it's in the best interest for the project to have a solid set of examples for newcomers and I'd like to give a significant contribution. One final question: when we feel that the refactoring is complete, should I squash the commits coming from the PR review as well? Thanks in advance. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Assignee: Stefano Baghino >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15141319#comment-15141319 ] ASF GitHub Bot commented on FLINK-2021: --- Github user fhueske commented on the pull request: https://github.com/apache/flink/pull/1581#issuecomment-182505103 Hi @stefanobaghino, sorry to bug you again. :-( I understand if you don't want to refactor this PR again. Please drop a note if you update it one last time (*promise*) or not. I can also take over and do the final changes before merging it. Thanks, Fabian > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Assignee: Stefano Baghino >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15141306#comment-15141306 ] ASF GitHub Bot commented on FLINK-2021: --- Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/1581#discussion_r52496326 --- Diff: flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java/clustering/KMeans.java --- @@ -105,119 +110,127 @@ public static void main(String[] args) throws Exception { .groupBy(0).reduce(new CentroidAccumulator()) // compute new centroids from point counts and coordinate sums .map(new CentroidAverager()); - + // feed new centroids back into next iteration DataSet finalCentroids = loop.closeWith(newCentroids); - + DataSet> clusteredPoints = points - // assign points to final clusters - .map(new SelectNearestCenter()).withBroadcastSet(finalCentroids, "centroids"); - + // assign points to final clusters + .map(new SelectNearestCenter()).withBroadcastSet(finalCentroids, "centroids"); + // emit result - if (fileOutput) { - clusteredPoints.writeAsCsv(outputPath, "\n", " "); + if (params.has("output")) { + clusteredPoints.writeAsCsv(params.get("output"), "\n", " "); // since file sinks are lazy, we trigger the execution explicitly env.execute("KMeans Example"); - } - else { + } else { clusteredPoints.print(); --- End diff -- Add a message saying something like "Printing result to std-out. Use --output to specify output path." > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Assignee: Stefano Baghino >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15141305#comment-15141305 ] ASF GitHub Bot commented on FLINK-2021: --- Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/1581#discussion_r52496167 --- Diff: flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java/clustering/KMeans.java --- @@ -61,42 +62,46 @@ * Cluster centers are represented by an integer id and a point value. * For example "1 6.2 3.2\n2 2.9 5.7\n" gives two centers (id=1, x=6.2, y=3.2) and (id=2, x=2.9, y=5.7). * - * + * * - * Usage: KMeans- * If no parameters are provided, the program is run with default data from {@link org.apache.flink.examples.java.clustering.util.KMeansData} and 10 iterations. - * + * Usage: KMeans --points --centroids --output --iterations + * If no parameters are provided, the program is run with default data from {@link org.apache.flink.examples.java.clustering.util.KMeansData} and 10 iterations. + * * * This example shows how to use: * * Bulk iterations * Broadcast variables in bulk iterations - * Custom Java objects (PoJos) + * Custom Java objects (POJOs) * */ @SuppressWarnings("serial") public class KMeans { - - // * - // PROGRAM - // * - + public static void main(String[] args) throws Exception { - - if(!parseParameters(args)) { - return; + + // Checking input parameters + final ParameterTool params = ParameterTool.fromArgs(args); + if (params.getNumberOfParameters() < 4) { --- End diff -- Otherwise we will use the file input if a user provides only the --points parameter although the printed message says differently. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Assignee: Stefano Baghino >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15141302#comment-15141302 ] ASF GitHub Bot commented on FLINK-2021: --- Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/1581#discussion_r52495987 --- Diff: flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java/clustering/KMeans.java --- @@ -61,42 +62,46 @@ * Cluster centers are represented by an integer id and a point value. * For example "1 6.2 3.2\n2 2.9 5.7\n" gives two centers (id=1, x=6.2, y=3.2) and (id=2, x=2.9, y=5.7). * - * + * * - * Usage: KMeans- * If no parameters are provided, the program is run with default data from {@link org.apache.flink.examples.java.clustering.util.KMeansData} and 10 iterations. - * + * Usage: KMeans --points --centroids --output --iterations + * If no parameters are provided, the program is run with default data from {@link org.apache.flink.examples.java.clustering.util.KMeansData} and 10 iterations. + * * * This example shows how to use: * * Bulk iterations * Broadcast variables in bulk iterations - * Custom Java objects (PoJos) + * Custom Java objects (POJOs) * */ @SuppressWarnings("serial") public class KMeans { - - // * - // PROGRAM - // * - + public static void main(String[] args) throws Exception { - - if(!parseParameters(args)) { - return; + + // Checking input parameters + final ParameterTool params = ParameterTool.fromArgs(args); + if (params.getNumberOfParameters() < 4) { --- End diff -- We used this pattern to check unnamed parameters. I think we can do better now and a shorter message into the `else` clause of the named parameter check. For this example print something like "Executing K-Means example with default point data set. Use --points to specify file input." in the `else` case of the `getPointDataSet()` method. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Assignee: Stefano Baghino >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136221#comment-15136221 ] ASF GitHub Bot commented on FLINK-2021: --- Github user stefanobaghino commented on the pull request: https://github.com/apache/flink/pull/1581#issuecomment-180995095 Ok, thanks! Should I squash the changes coming from the PR feedback now? > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Assignee: Stefano Baghino >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136216#comment-15136216 ] ASF GitHub Bot commented on FLINK-2021: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1581#issuecomment-180994099 The first test failed because some files on travis were corrupted, the two other tests failed probably because the YARN tests became unstable recently. I think you can consider your changes stable > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Assignee: Stefano Baghino >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15136205#comment-15136205 ] ASF GitHub Bot commented on FLINK-2021: --- Github user stefanobaghino commented on the pull request: https://github.com/apache/flink/pull/1581#issuecomment-180985129 I'm not sure of why some tests keep failing. I've also noticed that the build of our fork failed differently. They don't seem to affect `flink-batch-examples`, `flink-streaming-examples` and `flink-tests`. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Assignee: Stefano Baghino >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15134383#comment-15134383 ] ASF GitHub Bot commented on FLINK-2021: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1581#issuecomment-180422247 Awesome, thank you! > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15134081#comment-15134081 ] ASF GitHub Bot commented on FLINK-2021: --- Github user stefanobaghino commented on the pull request: https://github.com/apache/flink/pull/1581#issuecomment-180329011 Oh, ok, I obviously got it wrong, thanks for the explanation. Fine with me, I'll fix it along the other issues of the PR. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133923#comment-15133923 ] ASF GitHub Bot commented on FLINK-2021: --- Github user fhueske commented on the pull request: https://github.com/apache/flink/pull/1581#issuecomment-180279675 Hi Stefano, thanks for the update! - you are right, it is good practice to not squash commits the address PR feedback. That make following the changes much easier. These commits will be squashed before merging to the master branch. - I am not sure if we are talking about the same thing. Robert suggested to move the code for reading inputs into the main method for examples that have only a single input such as TwitterStream or WordCount. This is the point we discussed before. Usually, there is only one parameter to check, i.e., whether an input path was provided or not. We do not need to move that into a `checkParameters` method, IMO. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15132602#comment-15132602 ] ASF GitHub Bot commented on FLINK-2021: --- Github user stefanobaghino commented on the pull request: https://github.com/apache/flink/pull/1581#issuecomment-179949096 Following your suggestions, thanks. * squash: should I squash the commits coming from the review as well? * tests: I thought it was about some Travis configuration on our fork, I'll have to look into it * docs: thanks for pointing out the documentation, I missed it! * get: I'm not sure about it, I think it would be cleaner logically but it would make the `params` checking a little bit heavier on the reader — I can work on that but I'd wrap the parameter checking in a `checkParameters` method that is used for the initial checking and the one in the `get*` methods, would that work for you? > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15132379#comment-15132379 ] ASF GitHub Bot commented on FLINK-2021: --- Github user fhueske commented on the pull request: https://github.com/apache/flink/pull/1581#issuecomment-179867266 OK, let's do it. +1 for @rmetzger's suggestion. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15132369#comment-15132369 ] ASF GitHub Bot commented on FLINK-2021: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1581#issuecomment-179859441 I just checked out your pull request locally and looked through it. I think the following things need to be fixed before we can merge it: - Squash the 60+ commits in the PR into one - Fix the failing tests - Adopt also the "http://localhost:4000/apis/batch/examples.html"; page. - If @fhueske agrees, do not use a `get*` (for example `TwitterStream.getTextDataStream`) method if there is only one input. I'm okay with using those methods for examples with input >= 2. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15132358#comment-15132358 ] ASF GitHub Bot commented on FLINK-2021: --- Github user stefanobaghino commented on a diff in the pull request: https://github.com/apache/flink/pull/1581#discussion_r51875652 --- Diff: flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java/distcp/DistCp.java --- @@ -62,18 +63,32 @@ public static final String FILES_COPIED_CNT_NAME = "FILES_COPIED"; public static void main(String[] args) throws Exception { - if (args.length != 3) { - printHelp(); + + // set up the execution environment + final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment(); + + ParameterTool params = ParameterTool.fromArgs(args); + if (!params.has("input") || !params.has("output")) { + System.err.println("Usage: --input --output [--parallelism ]"); return; } - final Path sourcePath = new Path(args[0]); - final Path targetPath = new Path(args[1]); - int parallelism = Integer.valueOf(args[2], 10); + final Path sourcePath = new Path(params.get("input")); + final Path targetPath = new Path(params.get("output")); + if (!isLocal(env) && !(isOnDistributedFS(sourcePath) && isOnDistributedFS(targetPath))) { + System.out.println("In a distributed mode only HDFS input/output paths are supported"); + return; + } + + final int parallelism = Integer.valueOf(params.getInt("parallelism", 10)); --- End diff -- Right, fixing it right away, thanks! > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15132348#comment-15132348 ] ASF GitHub Bot commented on FLINK-2021: --- Github user rmetzger commented on a diff in the pull request: https://github.com/apache/flink/pull/1581#discussion_r51874764 --- Diff: flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java/distcp/DistCp.java --- @@ -62,18 +63,32 @@ public static final String FILES_COPIED_CNT_NAME = "FILES_COPIED"; public static void main(String[] args) throws Exception { - if (args.length != 3) { - printHelp(); + + // set up the execution environment + final ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment(); + + ParameterTool params = ParameterTool.fromArgs(args); + if (!params.has("input") || !params.has("output")) { + System.err.println("Usage: --input --output [--parallelism ]"); return; } - final Path sourcePath = new Path(args[0]); - final Path targetPath = new Path(args[1]); - int parallelism = Integer.valueOf(args[2], 10); + final Path sourcePath = new Path(params.get("input")); + final Path targetPath = new Path(params.get("output")); + if (!isLocal(env) && !(isOnDistributedFS(sourcePath) && isOnDistributedFS(targetPath))) { + System.out.println("In a distributed mode only HDFS input/output paths are supported"); + return; + } + + final int parallelism = Integer.valueOf(params.getInt("parallelism", 10)); --- End diff -- The Integer.valueOf() is not needed. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15130372#comment-15130372 ] ASF GitHub Bot commented on FLINK-2021: --- Github user stefanobaghino commented on the pull request: https://github.com/apache/flink/pull/1536#issuecomment-179225452 Branch recreated to remove merge commits, now opened as #1581. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15130367#comment-15130367 ] ASF GitHub Bot commented on FLINK-2021: --- GitHub user stefanobaghino opened a pull request: https://github.com/apache/flink/pull/1581 [FLINK-2021] Rework examples to use ParameterTool This should fix [FLINK-2021]. I was unable to run a full build because of permission issues but it should be ready to merged, after any fixes that may be highlighted by a review. You can merge this pull request into a Git repository by running: $ git pull https://github.com/radicalbit/flink 2021 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1581.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1581 commit f68cb95af1c27dd8bf99e1a2374969915dd567f9 Author: Stefano Baghino Date: 2016-01-21T11:20:49Z [FLINK-2021] Rework the K-Means examples to use built-in argument parsing facilities instead of ad-hoc functions commit a432cc204741dd0a2d505886b987c4113e781c84 Author: Stefano Baghino Date: 2016-01-21T11:33:15Z [FLINK-2021] Renamed user-facing names for options commit a6d7f38a0a41f5e7e1290afd267eb6b260f3ae06 Author: Stefano Baghino Date: 2016-01-21T12:07:39Z [FLINK-2021] Updated documentation to be in sync with reworked examples commit 31a6a1e4d1bae900866fc74aff9402b2cd3cee94 Author: Stefano Baghino Date: 2016-01-21T13:40:39Z [FLINK-2021] Removed return statement in Scala, updated usage message to reflect the changes commit 2c4978dce0d4a8a8cd11c2e32319a71b2b57427a Author: Stefano Baghino Date: 2016-01-21T14:01:45Z [FLINK-2021] Delegating the usage line to the the object commit 65b64746a006c01b92b02f4549d744776ed00980 Author: Stefano Baghino Date: 2016-01-21T14:05:04Z [FLINK-2021] Removed unused imports (checkstyle compliance) commit 47c0d26f3fa7b3d9913114e45e8895b70c80e538 Author: Stefano Baghino Date: 2016-01-21T14:47:00Z [FLINK-2021] Removed the parseParameters and printRunWithDefaultParams methods commit 63bb710babcd1b0fb8d7a5253e442612ef3a86e6 Author: Stefano Baghino Date: 2016-01-21T17:08:47Z [FLINK-2021] Removing static members, ditching RequiredParameters to improve readability commit 857e625d29490112bfd63d0595aa6bcfe4ab8890 Author: Stefano Baghino Date: 2016-01-22T09:11:46Z [FLINK-2021] Reworked the Scala example to keep it up to date with the Java one commit fbd5f8ee0a06238d7f1d7b10be2b12f2c1428599 Author: Stefano Baghino Date: 2016-01-26T11:44:13Z [FLINK-2021] Restored input source reading in separate methods commit dc2480c8fe97c698a31918c825c63e193fe84302 Author: Stefano Baghino Date: 2016-01-26T13:46:09Z [FLINK-2021] Using POJO types instead of tuple converters commit 32b51c4e4b9ec43e3ea54bc7ae2573e96a961e58 Author: Stefano Baghino Date: 2016-01-29T11:35:44Z [FLINK-2021] Restored source reading methods in Scala examples, introduced case classes commit 567a3e01d18919ab4f570398de0ef7a0002d1a2d Author: Stefano Baghino Date: 2016-01-29T11:39:54Z [FLINK-2021] Documentation updated to reflect changes in the example commit 179ca81b4f9d41a83c130d5c9edd436a1fdf3a03 Author: Stefano Baghino Date: 2016-01-29T13:32:19Z [FLINK-2021] Indentation problem in docs fixed commit ef04e9aa13bf85f8d0d8018f7f6db4239ba5a01d Author: Stefano Baghino Date: 2016-01-29T13:33:41Z [FLINK-2021] ScalaDoc comment re-indented to comply with coding style commit d163105d0b04fa085853d4f577f55a3ffcdc4e3e Author: Stefano Baghino Date: 2016-01-29T14:50:42Z [FLINK-2021] Tests updated to reflect usage of ParameterTools in KMeans commit a715cb150eae18881c106610f02f8ea80f553f92 Author: Stefano Baghino Date: 2016-01-29T16:44:20Z [FLINK-2021] Boy Scout rule: fixed typo in method name commit 2083bc66efb98e33f07707b5f69d6fa69d8db273 Author: Stefano Baghino Date: 2016-02-01T13:06:04Z [FLINK-2021] Reworked Java distcp example to use ParameterTool commit 844f2dc050c2ac28122b2c306fed637d9dd0e678 Author: Stefano Baghino Date: 2016-02-01T13:49:55Z [FLINK-2021] Parameter validation simplified and brought into main method commit 92dcf3da9f7e79c5f65b23256f1861cb6299d5d1 Author: Stefano Baghino Date: 2016-02-01T14:00:12Z [FLINK-2021] Reworked Java connected components example to use ParameterTool commit 6b1a63667c4c532ff57a0033b431762501a77d20 Author: Stefano Baghino Date: 2016-02-01T14:31:28Z [FLINK-2021] Reworked Java linear regression example to use ParameterTool commit acdafd58beed35e03f1474e7f3d49477d4697adc Author: Stefano Baghino Date: 2016-02-01T14:43:00Z [FLINK-2021] Reworked Java word count examples to use ParameterTool commit 02bbf7e2ce7708254228dffacfa2506c80c686
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15126191#comment-15126191 ] ASF GitHub Bot commented on FLINK-2021: --- Github user stefanobaghino commented on the pull request: https://github.com/apache/flink/pull/1536#issuecomment-177954808 Since it looks like I've gathered enough feedback, I'm closing the PR so that I can keep working on it and not burden on the CI system. Sorry for the WIP PR, I'll reopen it as it's presumably ready for merge. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15126192#comment-15126192 ] ASF GitHub Bot commented on FLINK-2021: --- Github user stefanobaghino closed the pull request at: https://github.com/apache/flink/pull/1536 > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15123359#comment-15123359 ] ASF GitHub Bot commented on FLINK-2021: --- Github user stefanobaghino commented on the pull request: https://github.com/apache/flink/pull/1536#issuecomment-176700143 It works, I'll be pushing the code in a few minutes. If everything looks good I can move on to the other examples. Thank you all the precious feedback provided so far. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15123289#comment-15123289 ] ASF GitHub Bot commented on FLINK-2021: --- Github user fhueske commented on the pull request: https://github.com/apache/flink/pull/1536#issuecomment-176675459 Thanks for the update! +1 using case classes in the Scala example. Can you also check if we can directly read the CSV file into the case classes (not sure if that works). Looks good otherwise. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15117138#comment-15117138 ] ASF GitHub Bot commented on FLINK-2021: --- Github user fhueske commented on the pull request: https://github.com/apache/flink/pull/1536#issuecomment-174991021 Good point! When the example was added, only Tuples were supported by CsvInputFormats. In the meantime, we Pojo support was added for CsvInputFormats. IMO, it makes sense to use this feature for this example. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15117086#comment-15117086 ] ASF GitHub Bot commented on FLINK-2021: --- Github user stefanobaghino commented on the pull request: https://github.com/apache/flink/pull/1536#issuecomment-174966751 This can be a starting point for further discussion. The only weak point I see (but maybe I'm wrong) is that right now we're reading `Tuple`s and then converting them to `Point`s and `Centroid`s. Since one of the stated aims of the example is to showcase the usage of POJOs, should I also use the `pojoType` method on input sources and drop the `TuplePointConverter` and `TupleCentroidConverter`? > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15117072#comment-15117072 ] ASF GitHub Bot commented on FLINK-2021: --- Github user fhueske commented on the pull request: https://github.com/apache/flink/pull/1536#issuecomment-174962013 Thanks @stefanobaghino! > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15117070#comment-15117070 ] ASF GitHub Bot commented on FLINK-2021: --- Github user stefanobaghino commented on the pull request: https://github.com/apache/flink/pull/1536#issuecomment-174960226 I'll edit the code so that we can all read the result and maybe discuss about the pros and cons of this approach. Again, thank you both for the feedback. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15117069#comment-15117069 ] ASF GitHub Bot commented on FLINK-2021: --- Github user fhueske commented on the pull request: https://github.com/apache/flink/pull/1536#issuecomment-174959098 The examples handle inputs differently than most other DataSet programs because they use default input data if no input paths are specified. This adds complexity which is not present in common programs. I agree that reading input is an important aspect of a DataSet program, but adding 15 lines for each input to each example seems unnecessary to me. Extracting code into methods can be good practice, even if a method is only called once. Reading code can become a lot easier if it is clear what a method does (good method name + comments) because it can hide all little nasty details and helps to focus on the relevant parts. The only question is how do we define relevance ;-) @stefanobaghino, I think this is a good idea. I would keep the parameter parsing in the main function of the examples, but only move the code to either read a CSV file or get a collection DataSource to a method. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15117057#comment-15117057 ] ASF GitHub Bot commented on FLINK-2021: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1536#issuecomment-174953115 I don't really see the points of encapsulating parts of the examples into methods: Usually, methods are used when functionality is used in different places. In this case, every method is only called once (there is no generic abstraction). Its adding an unnecessary level of indirection, making it harder to read the code. I don't see why reading the initial dataset is less important than the rest of the application. Reading the file is just 5 lines of code (relevant code, I woudn't call that boilerplate). > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15117052#comment-15117052 ] ASF GitHub Bot commented on FLINK-2021: --- Github user stefanobaghino commented on the pull request: https://github.com/apache/flink/pull/1536#issuecomment-174951427 Maybe I can rework the example so that secondary needs (like parameter parsing and source reading) are encapsulated in functions kept at the bottom, while eventually adding one further example with a spotlights on these secondary (but important) tasks. What do you think? > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15117050#comment-15117050 ] ASF GitHub Bot commented on FLINK-2021: --- Github user fhueske commented on the pull request: https://github.com/apache/flink/pull/1536#issuecomment-174950907 Right, but this is boilerplate code would still be accessible at the bottom of the file. It is not gone. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15117048#comment-15117048 ] ASF GitHub Bot commented on FLINK-2021: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1536#issuecomment-174950736 I think its important that users also see how to read data from sources. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15117046#comment-15117046 ] ASF GitHub Bot commented on FLINK-2021: --- Github user stefanobaghino commented on the pull request: https://github.com/apache/flink/pull/1536#issuecomment-174950384 Good, I'll keep on working on the PR with the other examples, thank you for the feedback and for the guidance provided so far. :smiley: > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15117047#comment-15117047 ] ASF GitHub Bot commented on FLINK-2021: --- Github user fhueske commented on the pull request: https://github.com/apache/flink/pull/1536#issuecomment-174950409 Sorry for the late notice, but why did we move the code to fetch the data to the top of the main function? IMO, it distracts from the main part of the example code, no? > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15117042#comment-15117042 ] ASF GitHub Bot commented on FLINK-2021: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1536#issuecomment-174946246 Okay, since nobody objected on the mailing list, we can start changing all examples. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15112288#comment-15112288 ] ASF GitHub Bot commented on FLINK-2021: --- Github user stefanobaghino commented on the pull request: https://github.com/apache/flink/pull/1536#issuecomment-173876414 I'll take some time. I've spotted another possible issue in the docs, I'll look into it as we gain more feedback from the mailing list. Thank you! > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15112281#comment-15112281 ] ASF GitHub Bot commented on FLINK-2021: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1536#issuecomment-173874750 mh, I understand that you are motivated to move on with this task. I think most likely other Flink committers agree to the change, but it might also happen that the discussion is not over yet. Worst case: You'll have to redo some of the changes because the discussion goes into a different direction. If you are confident enough and you are aware of the risks, you can also do it ... its up to you :) > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15112257#comment-15112257 ] ASF GitHub Bot commented on FLINK-2021: --- Github user stefanobaghino commented on the pull request: https://github.com/apache/flink/pull/1536#issuecomment-173871987 It seems like your improvements have had a positive feedback from the ML. Shall I proceed replicating those on other examples as well? > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15111848#comment-15111848 ] ASF GitHub Bot commented on FLINK-2021: --- Github user chiwanpark commented on a diff in the pull request: https://github.com/apache/flink/pull/1536#discussion_r50498429 --- Diff: flink-examples/flink-examples-batch/src/main/scala/org/apache/flink/examples/scala/clustering/KMeans.scala --- @@ -26,53 +27,84 @@ import org.apache.flink.examples.java.clustering.util.KMeansData import scala.collection.JavaConverters._ /** - * This example implements a basic K-Means clustering algorithm. - * - * K-Means is an iterative clustering algorithm and works as follows: - * K-Means is given a set of data points to be clustered and an initial set of ''K'' cluster - * centers. - * In each iteration, the algorithm computes the distance of each data point to each cluster center. - * Each point is assigned to the cluster center which is closest to it. - * Subsequently, each cluster center is moved to the center (''mean'') of all points that have - * been assigned to it. - * The moved cluster centers are fed into the next iteration. - * The algorithm terminates after a fixed number of iterations (as in this implementation) - * or if cluster centers do not (significantly) move in an iteration. - * This is the Wikipedia entry for the [[http://en.wikipedia - * .org/wiki/K-means_clustering K-Means Clustering algorithm]]. - * - * This implementation works on two-dimensional data points. - * It computes an assignment of data points to cluster centers, i.e., - * each data point is annotated with the id of the final cluster (center) it belongs to. - * - * Input files are plain text files and must be formatted as follows: - * - * - Data points are represented as two double values separated by a blank character. - *Data points are separated by newline characters. - *For example `"1.2 2.3\n5.3 7.2\n"` gives two data points (x=1.2, y=2.3) and (x=5.3, - *y=7.2). - * - Cluster centers are represented by an integer id and a point value. - *For example `"1 6.2 3.2\n2 2.9 5.7\n"` gives two centers (id=1, x=6.2, - *y=3.2) and (id=2, x=2.9, y=5.7). - * - * Usage: - * {{{ - * KMeans - * }}} - * If no parameters are provided, the program is run with default data from - * [[org.apache.flink.examples.java.clustering.util.KMeansData]] - * and 10 iterations. - * - * This example shows how to use: - * - * - Bulk iterations - * - Broadcast variables in bulk iterations - * - Custom Java objects (PoJos) - */ + * This example implements a basic K-Means clustering algorithm. + * + * K-Means is an iterative clustering algorithm and works as follows: + * K-Means is given a set of data points to be clustered and an initial set of ''K'' cluster + * centers. + * In each iteration, the algorithm computes the distance of each data point to each cluster center. + * Each point is assigned to the cluster center which is closest to it. + * Subsequently, each cluster center is moved to the center (''mean'') of all points that have + * been assigned to it. + * The moved cluster centers are fed into the next iteration. + * The algorithm terminates after a fixed number of iterations (as in this implementation) + * or if cluster centers do not (significantly) move in an iteration. + * This is the Wikipedia entry for the [[http://en.wikipedia + * .org/wiki/K-means_clustering K-Means Clustering algorithm]]. + * + * This implementation works on two-dimensional data points. + * It computes an assignment of data points to cluster centers, i.e., + * each data point is annotated with the id of the final cluster (center) it belongs to. + * + * Input files are plain text files and must be formatted as follows: + * + * - Data points are represented as two double values separated by a blank character. + * Data points are separated by newline characters. + * For example `"1.2 2.3\n5.3 7.2\n"` gives two data points (x=1.2, y=2.3) and (x=5.3, + * y=7.2). + * - Cluster centers are represented by an integer id and a point value. + * For example `"1 6.2 3.2\n2 2.9 5.7\n"` gives two centers (id=1, x=6.2, + * y=3.2) and (id=2, x=2.9, y=5.7). + * + * Usage: + * {{{ + * KMeans + * }}} + * If no parameters are provided, the program is run with default data from + * [[org.apache.flink.examples.java.clustering.util.KMeansData]] + * and 10 iterations. + * + * This example shows how to use: + * + * - Bulk iterations + * - Broadcas
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110804#comment-15110804 ] ASF GitHub Bot commented on FLINK-2021: --- Github user uce commented on the pull request: https://github.com/apache/flink/pull/1536#issuecomment-173612372 Nice to see it reworked. Thanks for the contribution. I think Robert raises a good point with the readability improvements though. I would be in favour of Robert's take on the ParameterTool. ;) > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110776#comment-15110776 ] ASF GitHub Bot commented on FLINK-2021: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1536#issuecomment-173607167 I have quickly checked out your branch and came up with another variant for reworking the KMeans example with the `ParameterTool`: https://github.com/rmetzger/flink/blob/4c3b569b522172043fe1f49da2858bae37519eef/flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java/clustering/KMeans.java I didn't use the required parameters, because they make the code too clumsy. I also think that getting rid of all these static variables and methods is a great improvement in readability. Users new to Flink should be able to focus on understanding the Flink's APIs and abstractions and not be distracted by our examples. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110666#comment-15110666 ] ASF GitHub Bot commented on FLINK-2021: --- Github user stefanobaghino commented on a diff in the pull request: https://github.com/apache/flink/pull/1536#discussion_r50406845 --- Diff: flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java/clustering/KMeans.java --- @@ -291,31 +297,54 @@ public Centroid map(Tuple3 value) { private static String centersPath = null; private static String outputPath = null; private static int numIterations = 10; - - private static boolean parseParameters(String[] programArguments) { - - if(programArguments.length > 0) { - // parse input arguments + + private static final Option POINTS_PATH_OPTION = + new Option("points").alt("P").help("The path to the input points"); + private static final Option CENTERS_PATH_OPTION = + new Option("centroids").alt("C").help("The path to the input centroids"); + private static final Option OUTPUT_PATH_OPTION = + new Option("output").alt("O").help("The path where the output will be written"); + private static final Option NUM_ITERATIONS_OPTION = + new Option("iterations").alt("I").help("The number of iteration performed by the K-Means algorithm"); + + private static boolean parseParameters(final ParameterTool params) throws RequiredParametersException { --- End diff -- I thought about it but was afraid of polluting the ´main´ method with "secondary" concerns. I'm going to fix this right now. I'll keep this in mind for other examples as well, thanks! > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110609#comment-15110609 ] ASF GitHub Bot commented on FLINK-2021: --- Github user stefanobaghino commented on a diff in the pull request: https://github.com/apache/flink/pull/1536#discussion_r50402286 --- Diff: flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java/clustering/KMeans.java --- @@ -291,31 +297,54 @@ public Centroid map(Tuple3 value) { private static String centersPath = null; private static String outputPath = null; private static int numIterations = 10; - - private static boolean parseParameters(String[] programArguments) { - - if(programArguments.length > 0) { - // parse input arguments + + private static final Option POINTS_PATH_OPTION = + new Option("points").alt("P").help("The path to the input points"); + private static final Option CENTERS_PATH_OPTION = + new Option("centroids").alt("C").help("The path to the input centroids"); + private static final Option OUTPUT_PATH_OPTION = + new Option("output").alt("O").help("The path where the output will be written"); + private static final Option NUM_ITERATIONS_OPTION = + new Option("iterations").alt("I").help("The number of iteration performed by the K-Means algorithm"); + + private static boolean parseParameters(final ParameterTool params) throws RequiredParametersException { + + final RequiredParameters requiredParameters = new RequiredParameters(); + boolean parseStatus = false; + + requiredParameters.add(POINTS_PATH_OPTION); + requiredParameters.add(CENTERS_PATH_OPTION); + requiredParameters.add(OUTPUT_PATH_OPTION); + requiredParameters.add(NUM_ITERATIONS_OPTION); + + try { + requiredParameters.applyTo(params); + pointsPath = params.get(POINTS_PATH_OPTION.getName()); + centersPath = params.get(CENTERS_PATH_OPTION.getName()); + outputPath = params.get(OUTPUT_PATH_OPTION.getName()); + numIterations = params.getInt(NUM_ITERATIONS_OPTION.getName()); fileOutput = true; - if(programArguments.length == 4) { - pointsPath = programArguments[0]; - centersPath = programArguments[1]; - outputPath = programArguments[2]; - numIterations = Integer.parseInt(programArguments[3]); + parseStatus = true; + } catch (RequiredParametersException e) { + if (params.getNumberOfParameters() == 0) { + printRunWithDefaultParams(); + parseStatus = true; } else { - System.err.println("Usage: KMeans "); - return false; + System.out.println(requiredParameters.getHelp(e.getMissingArguments())); } - } else { - System.out.println("Executing K-Means example with default parameters and built-in default data."); - System.out.println(" Provide parameters to read input data from files."); - System.out.println(" See the documentation for the correct format of input files."); - System.out.println(" We provide a data generator to create synthetic input files for this program."); - System.out.println(" Usage: KMeans "); } - return true; + + return parseStatus; } - + + private static void printRunWithDefaultParams() { + System.out.println("Executing K-Means example with default parameters and built-in default data."); + System.out.println(" Provide parameters to read input data from files."); + System.out.println(" See the documentation for the correct format of input files."); + System.out.println(" We provide a data generator to create synthetic input files for this program."); + System.out.println(" Usage: KMeans --points --centroids --output --iterations "); --- End diff -- Clever! I'll do it, thanks! > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Typ
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110599#comment-15110599 ] ASF GitHub Bot commented on FLINK-2021: --- Github user rmetzger commented on a diff in the pull request: https://github.com/apache/flink/pull/1536#discussion_r50401470 --- Diff: flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java/clustering/KMeans.java --- @@ -291,31 +297,54 @@ public Centroid map(Tuple3 value) { private static String centersPath = null; private static String outputPath = null; private static int numIterations = 10; - - private static boolean parseParameters(String[] programArguments) { - - if(programArguments.length > 0) { - // parse input arguments + + private static final Option POINTS_PATH_OPTION = + new Option("points").alt("P").help("The path to the input points"); + private static final Option CENTERS_PATH_OPTION = + new Option("centroids").alt("C").help("The path to the input centroids"); + private static final Option OUTPUT_PATH_OPTION = + new Option("output").alt("O").help("The path where the output will be written"); + private static final Option NUM_ITERATIONS_OPTION = + new Option("iterations").alt("I").help("The number of iteration performed by the K-Means algorithm"); + + private static boolean parseParameters(final ParameterTool params) throws RequiredParametersException { + + final RequiredParameters requiredParameters = new RequiredParameters(); + boolean parseStatus = false; + + requiredParameters.add(POINTS_PATH_OPTION); + requiredParameters.add(CENTERS_PATH_OPTION); + requiredParameters.add(OUTPUT_PATH_OPTION); + requiredParameters.add(NUM_ITERATIONS_OPTION); + + try { + requiredParameters.applyTo(params); + pointsPath = params.get(POINTS_PATH_OPTION.getName()); + centersPath = params.get(CENTERS_PATH_OPTION.getName()); + outputPath = params.get(OUTPUT_PATH_OPTION.getName()); + numIterations = params.getInt(NUM_ITERATIONS_OPTION.getName()); fileOutput = true; - if(programArguments.length == 4) { - pointsPath = programArguments[0]; - centersPath = programArguments[1]; - outputPath = programArguments[2]; - numIterations = Integer.parseInt(programArguments[3]); + parseStatus = true; + } catch (RequiredParametersException e) { + if (params.getNumberOfParameters() == 0) { + printRunWithDefaultParams(); + parseStatus = true; } else { - System.err.println("Usage: KMeans "); - return false; + System.out.println(requiredParameters.getHelp(e.getMissingArguments())); } - } else { - System.out.println("Executing K-Means example with default parameters and built-in default data."); - System.out.println(" Provide parameters to read input data from files."); - System.out.println(" See the documentation for the correct format of input files."); - System.out.println(" We provide a data generator to create synthetic input files for this program."); - System.out.println(" Usage: KMeans "); } - return true; + + return parseStatus; } - + + private static void printRunWithDefaultParams() { + System.out.println("Executing K-Means example with default parameters and built-in default data."); + System.out.println(" Provide parameters to read input data from files."); + System.out.println(" See the documentation for the correct format of input files."); + System.out.println(" We provide a data generator to create synthetic input files for this program."); + System.out.println(" Usage: KMeans --points --centroids --output --iterations "); --- End diff -- I think you can generate the usage information from the required parameters > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 >
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110591#comment-15110591 ] ASF GitHub Bot commented on FLINK-2021: --- Github user rmetzger commented on a diff in the pull request: https://github.com/apache/flink/pull/1536#discussion_r50401269 --- Diff: flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java/clustering/KMeans.java --- @@ -291,31 +297,54 @@ public Centroid map(Tuple3 value) { private static String centersPath = null; private static String outputPath = null; private static int numIterations = 10; - - private static boolean parseParameters(String[] programArguments) { - - if(programArguments.length > 0) { - // parse input arguments + + private static final Option POINTS_PATH_OPTION = + new Option("points").alt("P").help("The path to the input points"); + private static final Option CENTERS_PATH_OPTION = + new Option("centroids").alt("C").help("The path to the input centroids"); + private static final Option OUTPUT_PATH_OPTION = + new Option("output").alt("O").help("The path where the output will be written"); + private static final Option NUM_ITERATIONS_OPTION = + new Option("iterations").alt("I").help("The number of iteration performed by the K-Means algorithm"); + + private static boolean parseParameters(final ParameterTool params) throws RequiredParametersException { --- End diff -- I would actually remove the `parseParameters` method > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110577#comment-15110577 ] ASF GitHub Bot commented on FLINK-2021: --- Github user stefanobaghino commented on a diff in the pull request: https://github.com/apache/flink/pull/1536#discussion_r50400122 --- Diff: flink-examples/flink-examples-batch/src/main/scala/org/apache/flink/examples/scala/clustering/KMeans.scala --- @@ -104,36 +105,52 @@ object KMeans { } - private def parseParameters(programArguments: Array[String]): Boolean = { -if (programArguments.length > 0) { + private val POINTS_PATH_OPTION: Option = +new Option("points").alt("P").help("The path to the input points") + private val CENTERS_PATH_OPTION: Option = +new Option("centroids").alt("C").help("The path to the input centroids") + private val OUTPUT_PATH_OPTION: Option = +new Option("output").alt("O").help("The path where the output will be written") + private val NUM_ITERATIONS_OPTION: Option = +new Option("iterations").alt("I").help("The number of iteration performed by the K-Means algorithm") + + @throws(classOf[RequiredParametersException]) --- End diff -- Should I ignore this remark, too? You can find a more detailed explanation of why I kept the checked exceptions for the `add` calls in the PR description. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110574#comment-15110574 ] ASF GitHub Bot commented on FLINK-2021: --- Github user stefanobaghino commented on a diff in the pull request: https://github.com/apache/flink/pull/1536#discussion_r50399628 --- Diff: flink-examples/flink-examples-batch/src/main/scala/org/apache/flink/examples/scala/clustering/KMeans.scala --- @@ -104,36 +105,52 @@ object KMeans { } - private def parseParameters(programArguments: Array[String]): Boolean = { -if (programArguments.length > 0) { + private val POINTS_PATH_OPTION: Option = +new Option("points").alt("P").help("The path to the input points") + private val CENTERS_PATH_OPTION: Option = +new Option("centroids").alt("C").help("The path to the input centroids") + private val OUTPUT_PATH_OPTION: Option = +new Option("output").alt("O").help("The path where the output will be written") + private val NUM_ITERATIONS_OPTION: Option = +new Option("iterations").alt("I").help("The number of iteration performed by the K-Means algorithm") + + @throws(classOf[RequiredParametersException]) + private def parseParameters(params: ParameterTool): Boolean = { +val requiredParameters: RequiredParameters = new RequiredParameters +var parseStatus: Boolean = false +requiredParameters.add(POINTS_PATH_OPTION) +requiredParameters.add(CENTERS_PATH_OPTION) +requiredParameters.add(OUTPUT_PATH_OPTION) +requiredParameters.add(NUM_ITERATIONS_OPTION) +try { + requiredParameters.applyTo(params) + pointsPath = params.get(POINTS_PATH_OPTION.getName) + centersPath = params.get(CENTERS_PATH_OPTION.getName) + outputPath = params.get(OUTPUT_PATH_OPTION.getName) + numIterations = params.getInt(NUM_ITERATIONS_OPTION.getName) fileOutput = true - if (programArguments.length == 4) { -pointsPath = programArguments(0) -centersPath = programArguments(1) -outputPath = programArguments(2) -numIterations = Integer.parseInt(programArguments(3)) - -true - } - else { -System.err.println("Usage: KMeans ") - -false + parseStatus = true +} +catch { + case e: RequiredParametersException => { +if (params.getNumberOfParameters == 0) { + printRunWithDefaultParams() + parseStatus = true +} +else { + println(requiredParameters.getHelp(e.getMissingArguments)) +} } } -else { - System.out.println("Executing K-Means example with default parameters and built-in default " + -"data.") - System.out.println(" Provide parameters to read input data from files.") - System.out.println(" See the documentation for the correct format of input files.") - System.out.println(" We provide a data generator to create synthetic input files for this " + -"program.") - System.out.println(" Usage: KMeans ") +return parseStatus --- End diff -- Ok, I kept because I tend to do it on "Java-like" code, but I'll fix this as well. Thanks! > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110570#comment-15110570 ] ASF GitHub Bot commented on FLINK-2021: --- Github user stefanobaghino commented on a diff in the pull request: https://github.com/apache/flink/pull/1536#discussion_r50399418 --- Diff: flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java/clustering/KMeans.java --- @@ -291,31 +297,54 @@ public Centroid map(Tuple3 value) { private static String centersPath = null; private static String outputPath = null; private static int numIterations = 10; - - private static boolean parseParameters(String[] programArguments) { - - if(programArguments.length > 0) { - // parse input arguments + + private static final Option POINTS_PATH_OPTION = + new Option("points").alt("P").help("The path to the input points"); + private static final Option CENTERS_PATH_OPTION = + new Option("centroids").alt("C").help("The path to the input centroids"); + private static final Option OUTPUT_PATH_OPTION = + new Option("output").alt("O").help("The path where the output will be written"); + private static final Option NUM_ITERATIONS_OPTION = + new Option("iterations").alt("I").help("The number of iteration performed by the K-Means algorithm"); + + private static boolean parseParameters(final ParameterTool params) throws RequiredParametersException { + + final RequiredParameters requiredParameters = new RequiredParameters(); + boolean parseStatus = false; + + requiredParameters.add(POINTS_PATH_OPTION); + requiredParameters.add(CENTERS_PATH_OPTION); + requiredParameters.add(OUTPUT_PATH_OPTION); + requiredParameters.add(NUM_ITERATIONS_OPTION); + + try { + requiredParameters.applyTo(params); + pointsPath = params.get(POINTS_PATH_OPTION.getName()); + centersPath = params.get(CENTERS_PATH_OPTION.getName()); + outputPath = params.get(OUTPUT_PATH_OPTION.getName()); + numIterations = params.getInt(NUM_ITERATIONS_OPTION.getName()); fileOutput = true; - if(programArguments.length == 4) { - pointsPath = programArguments[0]; - centersPath = programArguments[1]; - outputPath = programArguments[2]; - numIterations = Integer.parseInt(programArguments[3]); + parseStatus = true; + } catch (RequiredParametersException e) { + if (params.getNumberOfParameters() == 0) { + printRunWithDefaultParams(); + parseStatus = true; } else { - System.err.println("Usage: KMeans "); - return false; + System.out.println(requiredParameters.getHelp(e.getMissingArguments())); } - } else { - System.out.println("Executing K-Means example with default parameters and built-in default data."); - System.out.println(" Provide parameters to read input data from files."); - System.out.println(" See the documentation for the correct format of input files."); - System.out.println(" We provide a data generator to create synthetic input files for this program."); - System.out.println(" Usage: KMeans "); } - return true; + + return parseStatus; } - + + private static void printRunWithDefaultParams() { + System.out.println("Executing K-Means example with default parameters and built-in default data."); + System.out.println(" Provide parameters to read input data from files."); + System.out.println(" See the documentation for the correct format of input files."); + System.out.println(" We provide a data generator to create synthetic input files for this program."); + System.out.println(" Usage: KMeans"); --- End diff -- Thanks for making me notice it, I'll fix it right away! > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement >
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110534#comment-15110534 ] ASF GitHub Bot commented on FLINK-2021: --- Github user chiwanpark commented on a diff in the pull request: https://github.com/apache/flink/pull/1536#discussion_r50395674 --- Diff: flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java/clustering/KMeans.java --- @@ -291,31 +297,54 @@ public Centroid map(Tuple3 value) { private static String centersPath = null; private static String outputPath = null; private static int numIterations = 10; - - private static boolean parseParameters(String[] programArguments) { - - if(programArguments.length > 0) { - // parse input arguments + + private static final Option POINTS_PATH_OPTION = + new Option("points").alt("P").help("The path to the input points"); + private static final Option CENTERS_PATH_OPTION = + new Option("centroids").alt("C").help("The path to the input centroids"); + private static final Option OUTPUT_PATH_OPTION = + new Option("output").alt("O").help("The path where the output will be written"); + private static final Option NUM_ITERATIONS_OPTION = + new Option("iterations").alt("I").help("The number of iteration performed by the K-Means algorithm"); + + private static boolean parseParameters(final ParameterTool params) throws RequiredParametersException { --- End diff -- Oh, `add` method in `RequiredParameters` can throw `RequireParametersException`. Ignore my previous comment. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110521#comment-15110521 ] ASF GitHub Bot commented on FLINK-2021: --- Github user chiwanpark commented on a diff in the pull request: https://github.com/apache/flink/pull/1536#discussion_r50395062 --- Diff: flink-examples/flink-examples-batch/src/main/scala/org/apache/flink/examples/scala/clustering/KMeans.scala --- @@ -104,36 +105,52 @@ object KMeans { } - private def parseParameters(programArguments: Array[String]): Boolean = { -if (programArguments.length > 0) { + private val POINTS_PATH_OPTION: Option = +new Option("points").alt("P").help("The path to the input points") + private val CENTERS_PATH_OPTION: Option = +new Option("centroids").alt("C").help("The path to the input centroids") + private val OUTPUT_PATH_OPTION: Option = +new Option("output").alt("O").help("The path where the output will be written") + private val NUM_ITERATIONS_OPTION: Option = +new Option("iterations").alt("I").help("The number of iteration performed by the K-Means algorithm") + + @throws(classOf[RequiredParametersException]) --- End diff -- This statement doesn't need also. We catch `RequiredParametersException` in `catch` statement. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110520#comment-15110520 ] ASF GitHub Bot commented on FLINK-2021: --- Github user chiwanpark commented on a diff in the pull request: https://github.com/apache/flink/pull/1536#discussion_r50395017 --- Diff: flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java/clustering/KMeans.java --- @@ -291,31 +297,54 @@ public Centroid map(Tuple3 value) { private static String centersPath = null; private static String outputPath = null; private static int numIterations = 10; - - private static boolean parseParameters(String[] programArguments) { - - if(programArguments.length > 0) { - // parse input arguments + + private static final Option POINTS_PATH_OPTION = + new Option("points").alt("P").help("The path to the input points"); + private static final Option CENTERS_PATH_OPTION = + new Option("centroids").alt("C").help("The path to the input centroids"); + private static final Option OUTPUT_PATH_OPTION = + new Option("output").alt("O").help("The path where the output will be written"); + private static final Option NUM_ITERATIONS_OPTION = + new Option("iterations").alt("I").help("The number of iteration performed by the K-Means algorithm"); + + private static boolean parseParameters(final ParameterTool params) throws RequiredParametersException { --- End diff -- Does this method throw `RequiredParametersException`? The exception seems caught in `catch` statement. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110519#comment-15110519 ] ASF GitHub Bot commented on FLINK-2021: --- Github user chiwanpark commented on a diff in the pull request: https://github.com/apache/flink/pull/1536#discussion_r50394888 --- Diff: flink-examples/flink-examples-batch/src/main/scala/org/apache/flink/examples/scala/clustering/KMeans.scala --- @@ -104,36 +105,52 @@ object KMeans { } - private def parseParameters(programArguments: Array[String]): Boolean = { -if (programArguments.length > 0) { + private val POINTS_PATH_OPTION: Option = +new Option("points").alt("P").help("The path to the input points") + private val CENTERS_PATH_OPTION: Option = +new Option("centroids").alt("C").help("The path to the input centroids") + private val OUTPUT_PATH_OPTION: Option = +new Option("output").alt("O").help("The path where the output will be written") + private val NUM_ITERATIONS_OPTION: Option = +new Option("iterations").alt("I").help("The number of iteration performed by the K-Means algorithm") + + @throws(classOf[RequiredParametersException]) + private def parseParameters(params: ParameterTool): Boolean = { +val requiredParameters: RequiredParameters = new RequiredParameters +var parseStatus: Boolean = false +requiredParameters.add(POINTS_PATH_OPTION) +requiredParameters.add(CENTERS_PATH_OPTION) +requiredParameters.add(OUTPUT_PATH_OPTION) +requiredParameters.add(NUM_ITERATIONS_OPTION) +try { + requiredParameters.applyTo(params) + pointsPath = params.get(POINTS_PATH_OPTION.getName) + centersPath = params.get(CENTERS_PATH_OPTION.getName) + outputPath = params.get(OUTPUT_PATH_OPTION.getName) + numIterations = params.getInt(NUM_ITERATIONS_OPTION.getName) fileOutput = true - if (programArguments.length == 4) { -pointsPath = programArguments(0) -centersPath = programArguments(1) -outputPath = programArguments(2) -numIterations = Integer.parseInt(programArguments(3)) - -true - } - else { -System.err.println("Usage: KMeans ") - -false + parseStatus = true +} +catch { + case e: RequiredParametersException => { +if (params.getNumberOfParameters == 0) { + printRunWithDefaultParams() + parseStatus = true +} +else { + println(requiredParameters.getHelp(e.getMissingArguments)) +} } } -else { - System.out.println("Executing K-Means example with default parameters and built-in default " + -"data.") - System.out.println(" Provide parameters to read input data from files.") - System.out.println(" See the documentation for the correct format of input files.") - System.out.println(" We provide a data generator to create synthetic input files for this " + -"program.") - System.out.println(" Usage: KMeans ") +return parseStatus --- End diff -- In Scala, we don't need `return` keyword. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110514#comment-15110514 ] ASF GitHub Bot commented on FLINK-2021: --- Github user chiwanpark commented on a diff in the pull request: https://github.com/apache/flink/pull/1536#discussion_r50394801 --- Diff: flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java/clustering/KMeans.java --- @@ -291,31 +297,54 @@ public Centroid map(Tuple3 value) { private static String centersPath = null; private static String outputPath = null; private static int numIterations = 10; - - private static boolean parseParameters(String[] programArguments) { - - if(programArguments.length > 0) { - // parse input arguments + + private static final Option POINTS_PATH_OPTION = + new Option("points").alt("P").help("The path to the input points"); + private static final Option CENTERS_PATH_OPTION = + new Option("centroids").alt("C").help("The path to the input centroids"); + private static final Option OUTPUT_PATH_OPTION = + new Option("output").alt("O").help("The path where the output will be written"); + private static final Option NUM_ITERATIONS_OPTION = + new Option("iterations").alt("I").help("The number of iteration performed by the K-Means algorithm"); + + private static boolean parseParameters(final ParameterTool params) throws RequiredParametersException { + + final RequiredParameters requiredParameters = new RequiredParameters(); + boolean parseStatus = false; + + requiredParameters.add(POINTS_PATH_OPTION); + requiredParameters.add(CENTERS_PATH_OPTION); + requiredParameters.add(OUTPUT_PATH_OPTION); + requiredParameters.add(NUM_ITERATIONS_OPTION); + + try { + requiredParameters.applyTo(params); + pointsPath = params.get(POINTS_PATH_OPTION.getName()); + centersPath = params.get(CENTERS_PATH_OPTION.getName()); + outputPath = params.get(OUTPUT_PATH_OPTION.getName()); + numIterations = params.getInt(NUM_ITERATIONS_OPTION.getName()); fileOutput = true; - if(programArguments.length == 4) { - pointsPath = programArguments[0]; - centersPath = programArguments[1]; - outputPath = programArguments[2]; - numIterations = Integer.parseInt(programArguments[3]); + parseStatus = true; + } catch (RequiredParametersException e) { + if (params.getNumberOfParameters() == 0) { + printRunWithDefaultParams(); + parseStatus = true; } else { - System.err.println("Usage: KMeans "); - return false; + System.out.println(requiredParameters.getHelp(e.getMissingArguments())); } - } else { - System.out.println("Executing K-Means example with default parameters and built-in default data."); - System.out.println(" Provide parameters to read input data from files."); - System.out.println(" See the documentation for the correct format of input files."); - System.out.println(" We provide a data generator to create synthetic input files for this program."); - System.out.println(" Usage: KMeans "); } - return true; + + return parseStatus; } - + + private static void printRunWithDefaultParams() { + System.out.println("Executing K-Means example with default parameters and built-in default data."); + System.out.println(" Provide parameters to read input data from files."); + System.out.println(" See the documentation for the correct format of input files."); + System.out.println(" We provide a data generator to create synthetic input files for this program."); + System.out.println(" Usage: KMeans"); --- End diff -- This parameter description should be updated. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Comp
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110516#comment-15110516 ] ASF GitHub Bot commented on FLINK-2021: --- Github user chiwanpark commented on a diff in the pull request: https://github.com/apache/flink/pull/1536#discussion_r50394812 --- Diff: flink-examples/flink-examples-batch/src/main/scala/org/apache/flink/examples/scala/clustering/KMeans.scala --- @@ -104,36 +105,52 @@ object KMeans { } - private def parseParameters(programArguments: Array[String]): Boolean = { -if (programArguments.length > 0) { + private val POINTS_PATH_OPTION: Option = +new Option("points").alt("P").help("The path to the input points") + private val CENTERS_PATH_OPTION: Option = +new Option("centroids").alt("C").help("The path to the input centroids") + private val OUTPUT_PATH_OPTION: Option = +new Option("output").alt("O").help("The path where the output will be written") + private val NUM_ITERATIONS_OPTION: Option = +new Option("iterations").alt("I").help("The number of iteration performed by the K-Means algorithm") + + @throws(classOf[RequiredParametersException]) + private def parseParameters(params: ParameterTool): Boolean = { +val requiredParameters: RequiredParameters = new RequiredParameters +var parseStatus: Boolean = false +requiredParameters.add(POINTS_PATH_OPTION) +requiredParameters.add(CENTERS_PATH_OPTION) +requiredParameters.add(OUTPUT_PATH_OPTION) +requiredParameters.add(NUM_ITERATIONS_OPTION) +try { + requiredParameters.applyTo(params) + pointsPath = params.get(POINTS_PATH_OPTION.getName) + centersPath = params.get(CENTERS_PATH_OPTION.getName) + outputPath = params.get(OUTPUT_PATH_OPTION.getName) + numIterations = params.getInt(NUM_ITERATIONS_OPTION.getName) fileOutput = true - if (programArguments.length == 4) { -pointsPath = programArguments(0) -centersPath = programArguments(1) -outputPath = programArguments(2) -numIterations = Integer.parseInt(programArguments(3)) - -true - } - else { -System.err.println("Usage: KMeans ") - -false + parseStatus = true +} +catch { + case e: RequiredParametersException => { +if (params.getNumberOfParameters == 0) { + printRunWithDefaultParams() + parseStatus = true +} +else { + println(requiredParameters.getHelp(e.getMissingArguments)) +} } } -else { - System.out.println("Executing K-Means example with default parameters and built-in default " + -"data.") - System.out.println(" Provide parameters to read input data from files.") - System.out.println(" See the documentation for the correct format of input files.") - System.out.println(" We provide a data generator to create synthetic input files for this " + -"program.") - System.out.println(" Usage: KMeans ") +return parseStatus + } - true -} + private def printRunWithDefaultParams() { +println("Executing K-Means example with default parameters and built-in default data.") +println(" Provide parameters to read input data from files.") +println(" See the documentation for the correct format of input files.") +println(" We provide a data generator to create synthetic input files for this program.") +println(" Usage: KMeans ") --- End diff -- This parameter description should be updated also. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final Exec
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110469#comment-15110469 ] ASF GitHub Bot commented on FLINK-2021: --- GitHub user stefanobaghino opened a pull request: https://github.com/apache/flink/pull/1536 [FLINK-2021] Rework examples to use ParameterTool (K-Means) I've reworked the K-Means examples in Java and Scala to get a feeling of what it would mean on the whole examples code base. Just a remark: I've caught the `RequiredParametersException` on the requirements application but not on their definition because I've felt the errors are different in nature (the application depends on user input while the definition should be somehow caught sooner — it would be great to have a way to catch it at compile-time). You can merge this pull request into a Git repository by running: $ git pull https://github.com/radicalbit/flink 2021 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/1536.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1536 commit f68cb95af1c27dd8bf99e1a2374969915dd567f9 Author: Stefano Baghino Date: 2016-01-21T11:20:49Z [FLINK-2021] Rework the K-Means examples to use built-in argument parsing facilities instead of ad-hoc functions commit a432cc204741dd0a2d505886b987c4113e781c84 Author: Stefano Baghino Date: 2016-01-21T11:33:15Z [FLINK-2021] Renamed user-facing names for options > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15108537#comment-15108537 ] Stefano Baghino commented on FLINK-2021: Sounds fine, thanks for the tips, I'll start working on it. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-2021) Rework examples to use ParameterTool
[ https://issues.apache.org/jira/browse/FLINK-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15108398#comment-15108398 ] Robert Metzger commented on FLINK-2021: --- Hi Stefano, cool. I think nobody is currently working on this issue. Some thoughts on this: I think that using the ParameterTool will improve the readability of the examples. In the beginning I was a bit hesitant with pushing the ParameterTool into all examples because I wasn't sure if users would like it. But I've seen many happy users and a wide adoption of it. We recently added RequiredParameters into Flink as well. They'll also allow you to print a help text. Maybe it makes sense to use the RequiredParameters in the examples. How about you introduce the tool into one or two examples, we review the change together and then you apply it to all examples? Also note, that we need to carefully check and rework the examples in the documentation to be in sync after the change. > Rework examples to use ParameterTool > > > Key: FLINK-2021 > URL: https://issues.apache.org/jira/browse/FLINK-2021 > Project: Flink > Issue Type: Improvement > Components: Examples >Affects Versions: 0.9 >Reporter: Robert Metzger >Priority: Minor > Labels: starter > > In FLINK-1525, we introduced the {{ParameterTool}}. > We should port the examples to use the tool. > The examples could look like this (we should maybe discuss it first on the > mailing lists): > {code} > public static void main(String[] args) throws Exception { > ParameterTool pt = ParameterTool.fromArgs(args); > boolean fileOutput = pt.getNumberOfParameters() == 2; > String textPath = null; > String outputPath = null; > if(fileOutput) { > textPath = pt.getRequired("input"); > outputPath = pt.getRequired("output"); > } > // set up the execution environment > final ExecutionEnvironment env = > ExecutionEnvironment.getExecutionEnvironment(); > env.getConfig().setUserConfig(pt); > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)