Just discovered sbt-pack: that addresses (quite well) the last item for
identifying and packaging the external jars.
2014-05-11 12:34 GMT-07:00 Stephen Boesch <java...@gmail.com>:
> HI Sonal,
> Yes I am working towards that same idea. How did you go about
> creating the non-spark-jar dependencies ? The way I am doing it is a
> separate straw-man project that does not include spark but has the external
> third party jars included. Then running sbt compile:managedClasspath and
> reverse engineering the lib jars from it. That is obviously not ideal.
>
> The maven "run" will be useful for other projects built by maven: i will
> keep in my notes.
>
> AFA sbt run-example, it requires additional libraries to be added for my
> external dependencies. I tried several items including ADD_JARS,
> --driver-class-path and combinations of extraClassPath. I have deferred
> that ad-hoc approach to finding a systematic one.
>
>
>
>
> 2014-05-08 5:26 GMT-07:00 Sonal Goyal <sonalgoy...@gmail.com>:
>
> I am creating a jar with only my dependencies and run spark-submit through
>> my project mvn build. I have configured the mvn exec goal to the location
>> of the script. Here is how I have set it up for my app. The mainClass is my
>> driver program, and I am able to send my custom args too. Hope this helps.
>>
>> <plugin>
>> <groupId>org.codehaus.mojo</groupId>
>> <artifactId>exec-maven-plugin</artifactId>
>> <executions>
>> <execution>
>> <goals>
>> <goal>exec</goal>
>> </goals>
>> </execution>
>> </executions>
>> <configuration>
>> <executable>/home/sgoyal/spark/bin/spark-submit</executable>
>> <arguments>
>> <argument>${jars}</argument>
>> <argument>--class</argument>
>> <argument>${mainClass}</argument>
>> <argument>--arg</argument>
>> <argument>${spark.master}</argument>
>> <argument>--arg</argument>
>> <argument>${my app arg 1}</argument>
>> <argument>--arg</argument>
>> <argument>${my arg 2}</argument>
>> </arguments>
>> </configuration>
>> </plugin>
>>
>>
>> Best Regards,
>> Sonal
>> Nube Technologies <http://www.nubetech.co>
>>
>> <http://in.linkedin.com/in/sonalgoyal>
>>
>>
>>
>>
>> On Wed, May 7, 2014 at 6:57 AM, Tathagata Das <
>> tathagata.das1...@gmail.com> wrote:
>>
>>> Doesnt the run-example script work for you? Also, are you on the latest
>>> commit of branch-1.0 ?
>>>
>>> TD
>>>
>>>
>>> On Mon, May 5, 2014 at 7:51 PM, Soumya Simanta <soumya.sima...@gmail.com
>>> > wrote:
>>>
>>>>
>>>>
>>>> Yes, I'm struggling with a similar problem where my class are not found
>>>> on the worker nodes. I'm using 1.0.0_SNAPSHOT. I would really appreciate
>>>> if someone can provide some documentation on the usage of spark-submit.
>>>>
>>>> Thanks
>>>>
>>>> > On May 5, 2014, at 10:24 PM, Stephen Boesch <java...@gmail.com>
>>>> wrote:
>>>> >
>>>> >
>>>> > I have a spark streaming application that uses the external streaming
>>>> modules (e.g. kafka, mqtt, ..) as well. It is not clear how to properly
>>>> invoke the spark-submit script: what are the ---driver-class-path and/or
>>>> -Dspark.executor.extraClassPath parameters required?
>>>> >
>>>> > For reference, the following error is proving difficult to resolve:
>>>> >
>>>> > java.lang.ClassNotFoundException:
>>>> org.apache.spark.streaming.examples.StreamingExamples
>>>> >
>>>>
>>>
>>>
>>
>
