I am creating a jar with only my dependencies and run spark-submit through
my project mvn build. I have configured the mvn exec goal to the location
of the script. Here is how I have set it up for my app. The mainClass is my
driver program, and I am able to send my custom args too. Hope this helps.
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>exec</goal>
</goals>
</execution>
</executions>
<configuration>
<executable>/home/sgoyal/spark/bin/spark-submit</executable>
<arguments>
<argument>${jars}</argument>
<argument>--class</argument>
<argument>${mainClass}</argument>
<argument>--arg</argument>
<argument>${spark.master}</argument>
<argument>--arg</argument>
<argument>${my app arg 1}</argument>
<argument>--arg</argument>
<argument>${my arg 2}</argument>
</arguments>
</configuration>
</plugin>
Best Regards,
Sonal
Nube Technologies <http://www.nubetech.co>
<http://in.linkedin.com/in/sonalgoyal>
On Wed, May 7, 2014 at 6:57 AM, Tathagata Das
<tathagata.das1...@gmail.com>wrote:
> Doesnt the run-example script work for you? Also, are you on the latest
> commit of branch-1.0 ?
>
> TD
>
>
> On Mon, May 5, 2014 at 7:51 PM, Soumya Simanta
> <soumya.sima...@gmail.com>wrote:
>
>>
>>
>> Yes, I'm struggling with a similar problem where my class are not found
>> on the worker nodes. I'm using 1.0.0_SNAPSHOT. I would really appreciate
>> if someone can provide some documentation on the usage of spark-submit.
>>
>> Thanks
>>
>> > On May 5, 2014, at 10:24 PM, Stephen Boesch <java...@gmail.com> wrote:
>> >
>> >
>> > I have a spark streaming application that uses the external streaming
>> modules (e.g. kafka, mqtt, ..) as well. It is not clear how to properly
>> invoke the spark-submit script: what are the ---driver-class-path and/or
>> -Dspark.executor.extraClassPath parameters required?
>> >
>> > For reference, the following error is proving difficult to resolve:
>> >
>> > java.lang.ClassNotFoundException:
>> org.apache.spark.streaming.examples.StreamingExamples
>> >
>>
>
>
