Okay got it. I think i'll have to run using spark-submit. On Wed, May 25, 2016 at 3:06 PM, Punit Naik <[email protected]> wrote:
> Hi Everyone > > I was able to run the example with Spark runner on files present in /tmp > directory > . What if I want to use the files from HDFS? > > On Wed, May 25, 2016 at 2:04 PM, Punit Naik <[email protected]> > wrote: > >> Hi Ismael >> >> I am using maven 3.3.9 >> >> On Wed, May 25, 2016 at 1:31 PM, Ismaël Mejía <[email protected]> wrote: >> >>> Hi Punit, >>> >>> Can you confirm the version of Maven are you using ? >>> >>> I ask this because I had some issues building the project with the >>> native version of maven that came with debian/ubuntu, but since I moved to >>> the 'real' one I have no problmems. And I just did git pull from master and >>> everything seems to work ok. >>> >>> Ismaël >>> >>> >>> On Wed, May 25, 2016 at 10:00 AM, Amit Sela <[email protected]> >>> wrote: >>> >>>> I'll take a look first chance I get and get back to you. >>>> >>>> On Wed, May 25, 2016, 10:57 Punit Naik <[email protected]> wrote: >>>> >>>>> Hi Amit >>>>> >>>>> Thanks, it ran but got stuck at: >>>>> >>>>> [INFO] --- exec-maven-plugin:1.4.0:exec (default-cli) @ spark --- >>>>> log4j:WARN No appenders could be found for logger >>>>> (org.apache.beam.sdk.options.PipelineOptionsFactory). >>>>> log4j:WARN Please initialize the log4j system properly. >>>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig >>>>> for more info. >>>>> >>>>> >>>>> On Wed, May 25, 2016 at 1:17 PM, Amit Sela <[email protected]> >>>>> wrote: >>>>> >>>>>> I'm not near a computer but are you running from the parent project >>>>>> dir or the runner ? You need to run from the runner dir. >>>>>> >>>>>> On Wed, May 25, 2016, 10:44 Punit Naik <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Hi Amit >>>>>>> >>>>>>> Just tried that and still got an error: >>>>>>> >>>>>>> [ERROR] Failed to execute goal >>>>>>> org.codehaus.mojo:exec-maven-plugin:1.4.0:exec (default-cli) on project >>>>>>> parent: The parameters 'executable' for goal >>>>>>> org.codehaus.mojo:exec-maven-plugin:1.4.0:exec are missing or invalid -> >>>>>>> [Help 1] >>>>>>> >>>>>>> On Wed, May 25, 2016 at 1:01 PM, Amit Sela <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Would you try running with mvn exec:exec ? >>>>>>>> See here: >>>>>>>> https://github.com/apache/incubator-beam/blob/master/runners/spark/README.md >>>>>>>> >>>>>>>> On Wed, May 25, 2016, 10:14 Punit Naik <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Everyone >>>>>>>>> >>>>>>>>> I compiled beam project with the command "mvn clean install >>>>>>>>> -DskipTests" and then ran the WordCount example with the following >>>>>>>>> command: >>>>>>>>> >>>>>>>>> mvn compile exec:java -pl examples/java >>>>>>>>> -Dexec.mainClass=org.apache.beam.examples.WordCount >>>>>>>>> -Dexec.args="--runner=SparkPipelineRunner --sparkMaster=local >>>>>>>>> --inputFile=/in --output=/beam-out" >>>>>>>>> >>>>>>>>> But it gave me an error saying: >>>>>>>>> >>>>>>>>> [ERROR] Failed to execute goal >>>>>>>>> org.codehaus.mojo:exec-maven-plugin:1.4.0:java (default-cli) on >>>>>>>>> project >>>>>>>>> java-examples-all: An exception occured while executing the Java >>>>>>>>> class. >>>>>>>>> null: InvocationTargetException: Unknown 'runner' specified >>>>>>>>> 'SparkPipelineRunner', supported pipeline runners >>>>>>>>> [BlockingDataflowPipelineRunner, DataflowPipelineRunner, >>>>>>>>> DirectPipelineRunner] -> [Help 1] >>>>>>>>> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to >>>>>>>>> execute goal org.codehaus.mojo:exec-maven-plugin:1.4.0:java >>>>>>>>> (default-cli) >>>>>>>>> on project java-examples-all: An exception occured while executing >>>>>>>>> the Java >>>>>>>>> class. null >>>>>>>>> at >>>>>>>>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212) >>>>>>>>> at >>>>>>>>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153) >>>>>>>>> at >>>>>>>>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145) >>>>>>>>> at >>>>>>>>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116) >>>>>>>>> at >>>>>>>>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80) >>>>>>>>> at >>>>>>>>> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51) >>>>>>>>> at >>>>>>>>> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128) >>>>>>>>> at >>>>>>>>> org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307) >>>>>>>>> at >>>>>>>>> org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193) >>>>>>>>> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106) >>>>>>>>> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863) >>>>>>>>> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288) >>>>>>>>> at org.apache.maven.cli.MavenCli.main(MavenCli.java:199) >>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>>>>> at >>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>>>>>>> at >>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:498) >>>>>>>>> at >>>>>>>>> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) >>>>>>>>> at >>>>>>>>> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) >>>>>>>>> at >>>>>>>>> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) >>>>>>>>> at >>>>>>>>> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356) >>>>>>>>> Caused by: org.apache.maven.plugin.MojoExecutionException: An >>>>>>>>> exception occured while executing the Java class. null >>>>>>>>> at >>>>>>>>> org.codehaus.mojo.exec.ExecJavaMojo.execute(ExecJavaMojo.java:345) >>>>>>>>> at >>>>>>>>> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134) >>>>>>>>> at >>>>>>>>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207) >>>>>>>>> ... 20 more >>>>>>>>> Caused by: java.lang.reflect.InvocationTargetException >>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>>>>>> at >>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>>>>>>> at >>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:498) >>>>>>>>> at >>>>>>>>> org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293) >>>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>>> Caused by: java.lang.IllegalArgumentException: Unknown 'runner' >>>>>>>>> specified 'SparkPipelineRunner', supported pipeline runners >>>>>>>>> [BlockingDataflowPipelineRunner, DataflowPipelineRunner, >>>>>>>>> DirectPipelineRunner] >>>>>>>>> at >>>>>>>>> org.apache.beam.sdk.options.PipelineOptionsFactory.parseObjects(PipelineOptionsFactory.java:1434) >>>>>>>>> at >>>>>>>>> org.apache.beam.sdk.options.PipelineOptionsFactory.access$400(PipelineOptionsFactory.java:107) >>>>>>>>> at >>>>>>>>> org.apache.beam.sdk.options.PipelineOptionsFactory$Builder.as(PipelineOptionsFactory.java:292) >>>>>>>>> at org.apache.beam.examples.WordCount.main(WordCount.java:192) >>>>>>>>> ... 6 more >>>>>>>>> Caused by: java.lang.ClassNotFoundException: SparkPipelineRunner >>>>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) >>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>>>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>>>>>>>> at java.lang.Class.forName0(Native Method) >>>>>>>>> at java.lang.Class.forName(Class.java:264) >>>>>>>>> at >>>>>>>>> org.apache.beam.sdk.options.PipelineOptionsFactory.parseObjects(PipelineOptionsFactory.java:1421) >>>>>>>>> ... 9 more >>>>>>>>> >>>>>>>>> What is the problem? >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Thank You >>>>>>>>> >>>>>>>>> Regards >>>>>>>>> >>>>>>>>> Punit Naik >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Thank You >>>>>>> >>>>>>> Regards >>>>>>> >>>>>>> Punit Naik >>>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Thank You >>>>> >>>>> Regards >>>>> >>>>> Punit Naik >>>>> >>>> >>> >> >> >> -- >> Thank You >> >> Regards >> >> Punit Naik >> > > > > -- > Thank You > > Regards > > Punit Naik > -- Thank You Regards Punit Naik
