I get a similar error when I use a negative value for the default parallelism in the flink-config.yaml
I'll push a patch that gives better error messages and sanity checks the config value, prints a warning, and falls back to DOP 1 if the config is invalid and no manual value is provided. Stephan On Wed, Sep 24, 2014 at 5:48 PM, rmetzger0 <[email protected]> wrote: > Hi, > > this looks like a minor issue in your flink configuration. > Can you have a look into your config (in the conf/flink-conf.yaml file) > and tell me the value of the "parallelization.degree.default" property? > If it is set to 0 or a negative value, thats the source of the issue. Set > it to something higher (I recommend numOfNodes*numOfCoresPerNode). > > But I have to admit that our error reporting here is not very helpful, > we'll improve that... > > > Cheers, > Robert > > > On Wed, Sep 24, 2014 at 4:23 PM, nirvanesque [via Apache Flink (Incubator) > User Mailing List archive.] <[hidden email] > <http://user/SendEmail.jtp?type=node&node=103&i=0>> wrote: > >> Hello robert, >> >> Apologies for getting back to you on this one :-( >> Es tut mir wirklich Leid! >> >> I've been trying today several times in vain this programme, and >> following is the error output : >> >> [QUOTE] >> abasu@edel-4:~$ ./flink/bin/flink run -v >> flink/examples/flink-java-examples-0.6-incubating-WebLogAnalysis.jar >> file:///home/abasu/examples/Weblogs/documents >> file:///home/abasu/examples/Weblogs/ranks >> file:///home/abasu/examples/Weblogs/visits >> file:///home/abasu/examples/Weblogs/results >> Error: The main method caused an error. >> org.apache.flink.client.program.ProgramInvocationException: The main >> method caused an error. >> at >> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:404) >> at >> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:307) >> at org.apache.flink.client.program.Client.run(Client.java:244) >> at >> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:332) >> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:319) >> at >> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:930) >> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:954) >> Caused by: java.lang.IllegalArgumentException >> at >> org.apache.flink.compiler.dag.OptimizerNode.setDegreeOfParallelism(OptimizerNode.java:414) >> at >> org.apache.flink.compiler.PactCompiler$GraphCreatingVisitor.preVisit(PactCompiler.java:772) >> at >> org.apache.flink.compiler.PactCompiler$GraphCreatingVisitor.preVisit(PactCompiler.java:616) >> at >> org.apache.flink.api.common.operators.base.GenericDataSinkBase.accept(GenericDataSinkBase.java:287) >> at org.apache.flink.api.common.Plan.accept(Plan.java:281) >> at >> org.apache.flink.compiler.PactCompiler.compile(PactCompiler.java:511) >> at >> org.apache.flink.compiler.PactCompiler.compile(PactCompiler.java:460) >> at >> org.apache.flink.client.program.Client.getOptimizedPlan(Client.java:196) >> at >> org.apache.flink.client.program.Client.getOptimizedPlan(Client.java:209) >> at org.apache.flink.client.program.Client.run(Client.java:285) >> at >> org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:54) >> at >> org.apache.flink.example.java.relational.WebLogAnalysis.main(WebLogAnalysis.java:148) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at >> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:389) >> ... 6 more >> abasu@edel-4:~$ >> >> [/QUOTE] >> >> >> I'm using Flink version 0.6 on a cluster of 10 nodes. >> Job launched from Job manager node. >> >> As per your instructions, I've used the set for <<documents>> <<ranks>> >> <<visits>> <<results>> >> that I downloaded from your scratch tree branch. >> >> It seems that the main function fails because of other reasons (even >> before entering the programme logic). >> I'm trying to break my head on the exact reason .... >> >> If you could help me in this direction ... ? >> >> Thanks a million in advance, >> Anirvan >> >> >> Le 23/09/2014 17:22, rmetzger0 [via Apache Flink (Incubator) User Mailing >> List archive.] a écrit : >> >> Hi, >> >> you have to use the "WebLogDataGenerator" found here: >> https://github.com/apache/incubator-flink/blob/master/flink-examples/flink-java-examples/src/main/java/org/apache/flink/examples/java/relational/util/WebLogDataGenerator.java >> >> It accepts two arguments, the number of documents and visits. >> The generated files are located in /tmp/documents /tmp/ranks and >> /tmp/visits. >> I've generated some sample data for you, located here: >> https://github.com/rmetzger/scratch/tree/weblogdataexample/weblog >> >> >> Best, >> Robert >> >> >> On Tue, Sep 23, 2014 at 4:05 PM, nirvanesque [via Apache Flink >> (Incubator) User Mailing List archive.] <[hidden email] >> <http://user/SendEmail.jtp?type=node&node=100&i=0>> wrote: >> >>> Hello Robert, >>> >>> Thanks as usual for all your help with the information. >>> >>> I'm trying in vain to create the different input files from the program >>> source code but running into difficulties. >>> >>> Could you (or anyone else) please post here samples of the 4 inputs that >>> are required to run this program ? >>> >>> Thanks in advance, >>> Anirvan >>> >>> Le 09/09/2014 23:54, rmetzger0 [via Apache Flink (Incubator) User >>> Mailing List archive.] a écrit : >>> >>> Hi Anirvan, >>> >>> sorry for the late response. You've posted the question to Nabble, which >>> is only a mirror of our actual mailing list at [hidden email] >>> <http://user/SendEmail.jtp?type=node&node=99&i=0>. Sadly, the message >>> is not automatically posted to the apache list because the apache server is >>> rejecting the mails from nabble. >>> I've already asked and there is no way to change this behavior. >>> So I actually saw the two messages you posted here by accident. >>> >>> Regarding your actual question: >>> - The command line arguments for the WebLogAnalysis example are: >>> "WebLogAnalysis <documents path> <ranks path> <visits path> <result >>> path>" >>> >>> - Regarding the "info -d" command. I think its an artifact from our old >>> java API. I've filed an issue in JIRA: >>> https://issues.apache.org/jira/browse/FLINK-1095 Lets see how we >>> resolve it. Thanks for reporting this! >>> >>> You can find the source code of all of our examples in the source >>> release of Flink (in the flink-examples/flink-java-examples project. You >>> can also access the source (and hence the examples) through GitHub: >>> https://github.com/apache/incubator-flink/blob/master/flink-examples/flink-java-examples/src/main/java/org/apache/flink/example/java/relational/WebLogAnalysis.java. >>> >>> >>> To build the examples, you can run: "mvn clean package -DskipTests" in >>> the "flink-examples/flink-java-examples" directory. This will re-build >>> them. >>> >>> If you don't want to import the whole Flink project just for playing >>> around with the examples, you can also create an empty maven project. This >>> script: >>> curl >>> https://raw.githubusercontent.com/apache/incubator-flink/master/flink-quickstart/quickstart.sh >>> | >>> bash >>> >>> will automatically set everything up for you. Just import the >>> "quickstart" project into Eclipse or IntelliJ. It will download all >>> dependencies and package everything correctly. If you want to use an >>> example there, just copy the Java file into the "quickstart" project. >>> >>> The examples are indeed a very good way to learn how to write Flink >>> jobs. >>> >>> Please continue asking if you have further questions! >>> >>> Best, >>> Robert >>> >>> >>> ------------------------------ >>> If you reply to this email, your message will be added to the >>> discussion below: >>> >>> http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/Looking-for-instructions-source-for-flink-java-examples-0-6-incubating-WebLogAnalysis-jar-tp66p67.html >>> To unsubscribe from Looking for instructions & source for >>> flink-java-examples-0.6-incubating-WebLogAnalysis.jar, click here. >>> NAML >>> <http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >>> >>> >>> >>> >>> ------------------------------ >>> If you reply to this email, your message will be added to the >>> discussion below: >>> >>> http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/Looking-for-instructions-source-for-flink-java-examples-0-6-incubating-WebLogAnalysis-jar-tp66p99.html >>> To unsubscribe from Looking for instructions & source for >>> flink-java-examples-0.6-incubating-WebLogAnalysis.jar, click here. >>> NAML >>> <http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >>> >> >> >> >> ------------------------------ >> If you reply to this email, your message will be added to the >> discussion below: >> >> http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/Looking-for-instructions-source-for-flink-java-examples-0-6-incubating-WebLogAnalysis-jar-tp66p100.html >> To unsubscribe from Looking for instructions & source for >> flink-java-examples-0.6-incubating-WebLogAnalysis.jar, click here. >> NAML >> <http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >> >> >> >> >> ------------------------------ >> If you reply to this email, your message will be added to the >> discussion below: >> >> http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/Looking-for-instructions-source-for-flink-java-examples-0-6-incubating-WebLogAnalysis-jar-tp66p102.html >> To unsubscribe from Looking for instructions & source for >> flink-java-examples-0.6-incubating-WebLogAnalysis.jar, click here. >> NAML >> <http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >> > > > ------------------------------ > View this message in context: Re: Looking for instructions & source for > flink-java-examples-0.6-incubating-WebLogAnalysis.jar > <http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/Looking-for-instructions-source-for-flink-java-examples-0-6-incubating-WebLogAnalysis-jar-tp66p103.html> > > Sent from the Apache Flink (Incubator) User Mailing List archive. mailing > list archive > <http://apache-flink-incubator-user-mailing-list-archive.2336050.n4.nabble.com/> > at Nabble.com. >
