Re: java.lang.ClassNotFoundException: org.apache.spark.streaming.twitter.TwitterReceiver

2015-11-09 Thread Tathagata Das 
You cannot submit from eclipse to a cluster that easily. You can run
locally (master set to local...), and it should work with just the pom.

On Mon, Nov 9, 2015 at 2:49 AM, أنس الليثي  wrote:

> If I packaged the application and submit it, it works fine but I need to
> run it from eclipse.
>
> Is there any problem running the application from eclipse ?
>
>
>
> On 9 November 2015 at 12:27, Tathagata Das  wrote:
>
>> How are you submitting the spark application?
>> You are supposed to submit the fat-jar of the application that include
>> the spark-streaming-twitter dependency (and its subdeps) but not
>> spark-streaming and spark-core.
>>
>> On Mon, Nov 9, 2015 at 1:02 AM, أنس الليثي  wrote:
>>
>>> I tried to remove maven and adding the dependencies manually using build
>>> path > configure build path > add external jars, then adding the jars
>>> manually but it did not work.
>>>
>>> I tried to create another project and copied the code from the first app
>>> but the problem still the same.
>>>
>>> I event tried to change eclipse with another version, but the same
>>> problem exist.
>>>
>>> :( :( :( :(
>>>
>>> On 9 November 2015 at 10:47, أنس الليثي  wrote:
>>>
 I tried both, the same exception still thrown

 On 9 November 2015 at 10:45, Sean Owen  wrote:

> You included a very old version of the Twitter jar - 1.0.0. Did you
> mean 1.5.1?
>
> On Mon, Nov 9, 2015 at 7:36 AM, fanooos  wrote:
> > This is my first Spark Stream application. The setup is as following
> >
> > 3 nodes running a spark cluster. One master node and two slaves.
> >
> > The application is a simple java application streaming from twitter
> and
> > dependencies managed by maven.
> >
> > Here is the code of the application
> >
> > public class SimpleApp {
> >
> > public static void main(String[] args) {
> >
> > SparkConf conf = new SparkConf().setAppName("Simple
> > Application").setMaster("spark://rethink-node01:7077");
> >
> > JavaStreamingContext sc = new JavaStreamingContext(conf, new
> > Duration(1000));
> >
> > ConfigurationBuilder cb = new ConfigurationBuilder();
> >
> > cb.setDebugEnabled(true).setOAuthConsumerKey("ConsumerKey")
> > .setOAuthConsumerSecret("ConsumerSecret")
> > .setOAuthAccessToken("AccessToken")
> > .setOAuthAccessTokenSecret("TokenSecret");
> >
> > OAuthAuthorization auth = new OAuthAuthorization(cb.build());
> >
> > JavaDStream tweets = TwitterUtils.createStream(sc,
> auth);
> >
> >  JavaDStream statuses = tweets.map(new
> Function > String>() {
> >  public String call(Status status) throws Exception {
> > return status.getText();
> > }
> > });
> >
> >  statuses.print();;
> >
> >  sc.start();
> >
> >  sc.awaitTermination();
> >
> > }
> >
> > }
> >
> >
> > here is the pom file
> >
> > http://maven.apache.org/POM/4.0.0;
> > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> > xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> > http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> > 4.0.0
> > SparkFirstTry
> > SparkFirstTry
> > 0.0.1-SNAPSHOT
> >
> > 
> > 
> > org.apache.spark
> > spark-core_2.10
> > 1.5.1
> > provided
> > 
> >
> > 
> > org.apache.spark
> > spark-streaming_2.10
> > 1.5.1
> > provided
> > 
> >
> > 
> > org.twitter4j
> > twitter4j-stream
> > 3.0.3
> > 
> > 
> > org.apache.spark
> > spark-streaming-twitter_2.10
> > 1.0.0
> > 
> >
> > 
> >
> > 
> > src
> > 
> > 
> > maven-compiler-plugin
> > 3.3
> > 
> > 1.8
> > 1.8
> > 
> > 
> > 
> > maven-assembly-plugin
> > 
> > 
> > 
> >
> > com.test.sparkTest.SimpleApp
> > 
> > 
> > 
> >
>  jar-with-dependencies
> > 
> > 
> > 
> >
> >

Re: java.lang.ClassNotFoundException: org.apache.spark.streaming.twitter.TwitterReceiver

2015-11-09 Thread DW @ Gmail 
Try to reproduce what the spark-submit shell script does, setting up the class 
path etc. 

Sent from my rotary phone. 


> On Nov 9, 2015, at 7:07 AM, Tathagata Das  wrote:
> 
> You cannot submit from eclipse to a cluster that easily. You can run locally 
> (master set to local...), and it should work with just the pom.
> 
>> On Mon, Nov 9, 2015 at 2:49 AM, أنس الليثي  wrote:
>> If I packaged the application and submit it, it works fine but I need to run 
>> it from eclipse.  
>> 
>> Is there any problem running the application from eclipse ? 
>> 
>> 
>> 
>>> On 9 November 2015 at 12:27, Tathagata Das  wrote:
>>> How are you submitting the spark application? 
>>> You are supposed to submit the fat-jar of the application that include the 
>>> spark-streaming-twitter dependency (and its subdeps) but not 
>>> spark-streaming and spark-core. 
>>> 
 On Mon, Nov 9, 2015 at 1:02 AM, أنس الليثي  wrote:
 I tried to remove maven and adding the dependencies manually using build 
 path > configure build path > add external jars, then adding the jars 
 manually but it did not work.
 
 I tried to create another project and copied the code from the first app 
 but the problem still the same. 
 
 I event tried to change eclipse with another version, but the same problem 
 exist.
 
 :( :( :( :( 
 
> On 9 November 2015 at 10:47, أنس الليثي  wrote:
> I tried both, the same exception still thrown 
> 
>> On 9 November 2015 at 10:45, Sean Owen  wrote:
>> You included a very old version of the Twitter jar - 1.0.0. Did you mean 
>> 1.5.1?
>> 
>> On Mon, Nov 9, 2015 at 7:36 AM, fanooos  wrote:
>> > This is my first Spark Stream application. The setup is as following
>> >
>> > 3 nodes running a spark cluster. One master node and two slaves.
>> >
>> > The application is a simple java application streaming from twitter and
>> > dependencies managed by maven.
>> >
>> > Here is the code of the application
>> >
>> > public class SimpleApp {
>> >
>> > public static void main(String[] args) {
>> >
>> > SparkConf conf = new SparkConf().setAppName("Simple
>> > Application").setMaster("spark://rethink-node01:7077");
>> >
>> > JavaStreamingContext sc = new JavaStreamingContext(conf, new
>> > Duration(1000));
>> >
>> > ConfigurationBuilder cb = new ConfigurationBuilder();
>> >
>> > cb.setDebugEnabled(true).setOAuthConsumerKey("ConsumerKey")
>> > .setOAuthConsumerSecret("ConsumerSecret")
>> > .setOAuthAccessToken("AccessToken")
>> > .setOAuthAccessTokenSecret("TokenSecret");
>> >
>> > OAuthAuthorization auth = new OAuthAuthorization(cb.build());
>> >
>> > JavaDStream tweets = TwitterUtils.createStream(sc, 
>> > auth);
>> >
>> >  JavaDStream statuses = tweets.map(new Function> > String>() {
>> >  public String call(Status status) throws Exception {
>> > return status.getText();
>> > }
>> > });
>> >
>> >  statuses.print();;
>> >
>> >  sc.start();
>> >
>> >  sc.awaitTermination();
>> >
>> > }
>> >
>> > }
>> >
>> >
>> > here is the pom file
>> >
>> > http://maven.apache.org/POM/4.0.0;
>> > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
>> > xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
>> > http://maven.apache.org/xsd/maven-4.0.0.xsd;>
>> > 4.0.0
>> > SparkFirstTry
>> > SparkFirstTry
>> > 0.0.1-SNAPSHOT
>> >
>> > 
>> > 
>> > org.apache.spark
>> > spark-core_2.10
>> > 1.5.1
>> > provided
>> > 
>> >
>> > 
>> > org.apache.spark
>> > spark-streaming_2.10
>> > 1.5.1
>> > provided
>> > 
>> >
>> > 
>> > org.twitter4j
>> > twitter4j-stream
>> > 3.0.3
>> > 
>> > 
>> > org.apache.spark
>> > spark-streaming-twitter_2.10
>> > 1.0.0
>> > 
>> >
>> > 
>> >
>> > 
>> > src
>> > 
>> > 
>> > maven-compiler-plugin
>> > 3.3
>> > 
>> > 1.8
>> > 1.8
>> > 
>> > 
>> > 
>> > maven-assembly-plugin
>> >

Re: java.lang.ClassNotFoundException: org.apache.spark.streaming.twitter.TwitterReceiver

2015-11-09 Thread أنس الليثي 
If I packaged the application and submit it, it works fine but I need to
run it from eclipse.

Is there any problem running the application from eclipse ?



On 9 November 2015 at 12:27, Tathagata Das  wrote:

> How are you submitting the spark application?
> You are supposed to submit the fat-jar of the application that include the
> spark-streaming-twitter dependency (and its subdeps) but not
> spark-streaming and spark-core.
>
> On Mon, Nov 9, 2015 at 1:02 AM, أنس الليثي  wrote:
>
>> I tried to remove maven and adding the dependencies manually using build
>> path > configure build path > add external jars, then adding the jars
>> manually but it did not work.
>>
>> I tried to create another project and copied the code from the first app
>> but the problem still the same.
>>
>> I event tried to change eclipse with another version, but the same
>> problem exist.
>>
>> :( :( :( :(
>>
>> On 9 November 2015 at 10:47, أنس الليثي  wrote:
>>
>>> I tried both, the same exception still thrown
>>>
>>> On 9 November 2015 at 10:45, Sean Owen  wrote:
>>>
 You included a very old version of the Twitter jar - 1.0.0. Did you
 mean 1.5.1?

 On Mon, Nov 9, 2015 at 7:36 AM, fanooos  wrote:
 > This is my first Spark Stream application. The setup is as following
 >
 > 3 nodes running a spark cluster. One master node and two slaves.
 >
 > The application is a simple java application streaming from twitter
 and
 > dependencies managed by maven.
 >
 > Here is the code of the application
 >
 > public class SimpleApp {
 >
 > public static void main(String[] args) {
 >
 > SparkConf conf = new SparkConf().setAppName("Simple
 > Application").setMaster("spark://rethink-node01:7077");
 >
 > JavaStreamingContext sc = new JavaStreamingContext(conf, new
 > Duration(1000));
 >
 > ConfigurationBuilder cb = new ConfigurationBuilder();
 >
 > cb.setDebugEnabled(true).setOAuthConsumerKey("ConsumerKey")
 > .setOAuthConsumerSecret("ConsumerSecret")
 > .setOAuthAccessToken("AccessToken")
 > .setOAuthAccessTokenSecret("TokenSecret");
 >
 > OAuthAuthorization auth = new OAuthAuthorization(cb.build());
 >
 > JavaDStream tweets = TwitterUtils.createStream(sc,
 auth);
 >
 >  JavaDStream statuses = tweets.map(new
 Function String>() {
 >  public String call(Status status) throws Exception {
 > return status.getText();
 > }
 > });
 >
 >  statuses.print();;
 >
 >  sc.start();
 >
 >  sc.awaitTermination();
 >
 > }
 >
 > }
 >
 >
 > here is the pom file
 >
 > http://maven.apache.org/POM/4.0.0;
 > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
 > xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
 > http://maven.apache.org/xsd/maven-4.0.0.xsd;>
 > 4.0.0
 > SparkFirstTry
 > SparkFirstTry
 > 0.0.1-SNAPSHOT
 >
 > 
 > 
 > org.apache.spark
 > spark-core_2.10
 > 1.5.1
 > provided
 > 
 >
 > 
 > org.apache.spark
 > spark-streaming_2.10
 > 1.5.1
 > provided
 > 
 >
 > 
 > org.twitter4j
 > twitter4j-stream
 > 3.0.3
 > 
 > 
 > org.apache.spark
 > spark-streaming-twitter_2.10
 > 1.0.0
 > 
 >
 > 
 >
 > 
 > src
 > 
 > 
 > maven-compiler-plugin
 > 3.3
 > 
 > 1.8
 > 1.8
 > 
 > 
 > 
 > maven-assembly-plugin
 > 
 > 
 > 
 >
 > com.test.sparkTest.SimpleApp
 > 
 > 
 > 
 >
  jar-with-dependencies
 > 
 > 
 > 
 >
 > 
 > 
 > 
 >
 >
 > The application starts successfully but no tweets comes and this
 exception
 > is thrown
 >
 > 15/11/08 15:55:46 WARN TaskSetManager: Lost task 0.0 in stage 4.0
 (TID 78,
 > 192.168.122.39): java.io.IOException:
 java.lang.ClassNotFoundException:
 > org.apache.spark.streaming.twitter.TwitterReceiver
 >

Re: java.lang.ClassNotFoundException: org.apache.spark.streaming.twitter.TwitterReceiver

2015-11-09 Thread Tathagata Das 
How are you submitting the spark application?
You are supposed to submit the fat-jar of the application that include the
spark-streaming-twitter dependency (and its subdeps) but not
spark-streaming and spark-core.

On Mon, Nov 9, 2015 at 1:02 AM, أنس الليثي  wrote:

> I tried to remove maven and adding the dependencies manually using build
> path > configure build path > add external jars, then adding the jars
> manually but it did not work.
>
> I tried to create another project and copied the code from the first app
> but the problem still the same.
>
> I event tried to change eclipse with another version, but the same problem
> exist.
>
> :( :( :( :(
>
> On 9 November 2015 at 10:47, أنس الليثي  wrote:
>
>> I tried both, the same exception still thrown
>>
>> On 9 November 2015 at 10:45, Sean Owen  wrote:
>>
>>> You included a very old version of the Twitter jar - 1.0.0. Did you mean
>>> 1.5.1?
>>>
>>> On Mon, Nov 9, 2015 at 7:36 AM, fanooos  wrote:
>>> > This is my first Spark Stream application. The setup is as following
>>> >
>>> > 3 nodes running a spark cluster. One master node and two slaves.
>>> >
>>> > The application is a simple java application streaming from twitter and
>>> > dependencies managed by maven.
>>> >
>>> > Here is the code of the application
>>> >
>>> > public class SimpleApp {
>>> >
>>> > public static void main(String[] args) {
>>> >
>>> > SparkConf conf = new SparkConf().setAppName("Simple
>>> > Application").setMaster("spark://rethink-node01:7077");
>>> >
>>> > JavaStreamingContext sc = new JavaStreamingContext(conf, new
>>> > Duration(1000));
>>> >
>>> > ConfigurationBuilder cb = new ConfigurationBuilder();
>>> >
>>> > cb.setDebugEnabled(true).setOAuthConsumerKey("ConsumerKey")
>>> > .setOAuthConsumerSecret("ConsumerSecret")
>>> > .setOAuthAccessToken("AccessToken")
>>> > .setOAuthAccessTokenSecret("TokenSecret");
>>> >
>>> > OAuthAuthorization auth = new OAuthAuthorization(cb.build());
>>> >
>>> > JavaDStream tweets = TwitterUtils.createStream(sc,
>>> auth);
>>> >
>>> >  JavaDStream statuses = tweets.map(new Function>> > String>() {
>>> >  public String call(Status status) throws Exception {
>>> > return status.getText();
>>> > }
>>> > });
>>> >
>>> >  statuses.print();;
>>> >
>>> >  sc.start();
>>> >
>>> >  sc.awaitTermination();
>>> >
>>> > }
>>> >
>>> > }
>>> >
>>> >
>>> > here is the pom file
>>> >
>>> > http://maven.apache.org/POM/4.0.0;
>>> > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
>>> > xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
>>> > http://maven.apache.org/xsd/maven-4.0.0.xsd;>
>>> > 4.0.0
>>> > SparkFirstTry
>>> > SparkFirstTry
>>> > 0.0.1-SNAPSHOT
>>> >
>>> > 
>>> > 
>>> > org.apache.spark
>>> > spark-core_2.10
>>> > 1.5.1
>>> > provided
>>> > 
>>> >
>>> > 
>>> > org.apache.spark
>>> > spark-streaming_2.10
>>> > 1.5.1
>>> > provided
>>> > 
>>> >
>>> > 
>>> > org.twitter4j
>>> > twitter4j-stream
>>> > 3.0.3
>>> > 
>>> > 
>>> > org.apache.spark
>>> > spark-streaming-twitter_2.10
>>> > 1.0.0
>>> > 
>>> >
>>> > 
>>> >
>>> > 
>>> > src
>>> > 
>>> > 
>>> > maven-compiler-plugin
>>> > 3.3
>>> > 
>>> > 1.8
>>> > 1.8
>>> > 
>>> > 
>>> > 
>>> > maven-assembly-plugin
>>> > 
>>> > 
>>> > 
>>> >
>>> > com.test.sparkTest.SimpleApp
>>> > 
>>> > 
>>> > 
>>> >
>>>  jar-with-dependencies
>>> > 
>>> > 
>>> > 
>>> >
>>> > 
>>> > 
>>> > 
>>> >
>>> >
>>> > The application starts successfully but no tweets comes and this
>>> exception
>>> > is thrown
>>> >
>>> > 15/11/08 15:55:46 WARN TaskSetManager: Lost task 0.0 in stage 4.0 (TID
>>> 78,
>>> > 192.168.122.39): java.io.IOException: java.lang.ClassNotFoundException:
>>> > org.apache.spark.streaming.twitter.TwitterReceiver
>>> > at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1163)
>>> > at
>>> >
>>> org.apache.spark.rdd.ParallelCollectionPartition.readObject(ParallelCollectionRDD.scala:70)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>> > at
>>> >
>>>

Re: java.lang.ClassNotFoundException: org.apache.spark.streaming.twitter.TwitterReceiver

2015-11-09 Thread أنس الليثي 
I tried to remove maven and adding the dependencies manually using build
path > configure build path > add external jars, then adding the jars
manually but it did not work.

I tried to create another project and copied the code from the first app
but the problem still the same.

I event tried to change eclipse with another version, but the same problem
exist.

:( :( :( :(

On 9 November 2015 at 10:47, أنس الليثي  wrote:

> I tried both, the same exception still thrown
>
> On 9 November 2015 at 10:45, Sean Owen  wrote:
>
>> You included a very old version of the Twitter jar - 1.0.0. Did you mean
>> 1.5.1?
>>
>> On Mon, Nov 9, 2015 at 7:36 AM, fanooos  wrote:
>> > This is my first Spark Stream application. The setup is as following
>> >
>> > 3 nodes running a spark cluster. One master node and two slaves.
>> >
>> > The application is a simple java application streaming from twitter and
>> > dependencies managed by maven.
>> >
>> > Here is the code of the application
>> >
>> > public class SimpleApp {
>> >
>> > public static void main(String[] args) {
>> >
>> > SparkConf conf = new SparkConf().setAppName("Simple
>> > Application").setMaster("spark://rethink-node01:7077");
>> >
>> > JavaStreamingContext sc = new JavaStreamingContext(conf, new
>> > Duration(1000));
>> >
>> > ConfigurationBuilder cb = new ConfigurationBuilder();
>> >
>> > cb.setDebugEnabled(true).setOAuthConsumerKey("ConsumerKey")
>> > .setOAuthConsumerSecret("ConsumerSecret")
>> > .setOAuthAccessToken("AccessToken")
>> > .setOAuthAccessTokenSecret("TokenSecret");
>> >
>> > OAuthAuthorization auth = new OAuthAuthorization(cb.build());
>> >
>> > JavaDStream tweets = TwitterUtils.createStream(sc,
>> auth);
>> >
>> >  JavaDStream statuses = tweets.map(new Function> > String>() {
>> >  public String call(Status status) throws Exception {
>> > return status.getText();
>> > }
>> > });
>> >
>> >  statuses.print();;
>> >
>> >  sc.start();
>> >
>> >  sc.awaitTermination();
>> >
>> > }
>> >
>> > }
>> >
>> >
>> > here is the pom file
>> >
>> > http://maven.apache.org/POM/4.0.0;
>> > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
>> > xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
>> > http://maven.apache.org/xsd/maven-4.0.0.xsd;>
>> > 4.0.0
>> > SparkFirstTry
>> > SparkFirstTry
>> > 0.0.1-SNAPSHOT
>> >
>> > 
>> > 
>> > org.apache.spark
>> > spark-core_2.10
>> > 1.5.1
>> > provided
>> > 
>> >
>> > 
>> > org.apache.spark
>> > spark-streaming_2.10
>> > 1.5.1
>> > provided
>> > 
>> >
>> > 
>> > org.twitter4j
>> > twitter4j-stream
>> > 3.0.3
>> > 
>> > 
>> > org.apache.spark
>> > spark-streaming-twitter_2.10
>> > 1.0.0
>> > 
>> >
>> > 
>> >
>> > 
>> > src
>> > 
>> > 
>> > maven-compiler-plugin
>> > 3.3
>> > 
>> > 1.8
>> > 1.8
>> > 
>> > 
>> > 
>> > maven-assembly-plugin
>> > 
>> > 
>> > 
>> >
>> > com.test.sparkTest.SimpleApp
>> > 
>> > 
>> > 
>> >
>>  jar-with-dependencies
>> > 
>> > 
>> > 
>> >
>> > 
>> > 
>> > 
>> >
>> >
>> > The application starts successfully but no tweets comes and this
>> exception
>> > is thrown
>> >
>> > 15/11/08 15:55:46 WARN TaskSetManager: Lost task 0.0 in stage 4.0 (TID
>> 78,
>> > 192.168.122.39): java.io.IOException: java.lang.ClassNotFoundException:
>> > org.apache.spark.streaming.twitter.TwitterReceiver
>> > at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1163)
>> > at
>> >
>> org.apache.spark.rdd.ParallelCollectionPartition.readObject(ParallelCollectionRDD.scala:70)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> > at java.lang.reflect.Method.invoke(Method.java:497)
>> > at
>> > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
>> > at
>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900)
>> > at
>> >
>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
>> > at
>>

Re: java.lang.ClassNotFoundException: org.apache.spark.streaming.twitter.TwitterReceiver

2015-11-08 Thread Sean Owen 
You included a very old version of the Twitter jar - 1.0.0. Did you mean 1.5.1?

On Mon, Nov 9, 2015 at 7:36 AM, fanooos  wrote:
> This is my first Spark Stream application. The setup is as following
>
> 3 nodes running a spark cluster. One master node and two slaves.
>
> The application is a simple java application streaming from twitter and
> dependencies managed by maven.
>
> Here is the code of the application
>
> public class SimpleApp {
>
> public static void main(String[] args) {
>
> SparkConf conf = new SparkConf().setAppName("Simple
> Application").setMaster("spark://rethink-node01:7077");
>
> JavaStreamingContext sc = new JavaStreamingContext(conf, new
> Duration(1000));
>
> ConfigurationBuilder cb = new ConfigurationBuilder();
>
> cb.setDebugEnabled(true).setOAuthConsumerKey("ConsumerKey")
> .setOAuthConsumerSecret("ConsumerSecret")
> .setOAuthAccessToken("AccessToken")
> .setOAuthAccessTokenSecret("TokenSecret");
>
> OAuthAuthorization auth = new OAuthAuthorization(cb.build());
>
> JavaDStream tweets = TwitterUtils.createStream(sc, auth);
>
>  JavaDStream statuses = tweets.map(new Function String>() {
>  public String call(Status status) throws Exception {
> return status.getText();
> }
> });
>
>  statuses.print();;
>
>  sc.start();
>
>  sc.awaitTermination();
>
> }
>
> }
>
>
> here is the pom file
>
> http://maven.apache.org/POM/4.0.0;
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd;>
> 4.0.0
> SparkFirstTry
> SparkFirstTry
> 0.0.1-SNAPSHOT
>
> 
> 
> org.apache.spark
> spark-core_2.10
> 1.5.1
> provided
> 
>
> 
> org.apache.spark
> spark-streaming_2.10
> 1.5.1
> provided
> 
>
> 
> org.twitter4j
> twitter4j-stream
> 3.0.3
> 
> 
> org.apache.spark
> spark-streaming-twitter_2.10
> 1.0.0
> 
>
> 
>
> 
> src
> 
> 
> maven-compiler-plugin
> 3.3
> 
> 1.8
> 1.8
> 
> 
> 
> maven-assembly-plugin
> 
> 
> 
>
> com.test.sparkTest.SimpleApp
> 
> 
> 
> jar-with-dependencies
> 
> 
> 
>
> 
> 
> 
>
>
> The application starts successfully but no tweets comes and this exception
> is thrown
>
> 15/11/08 15:55:46 WARN TaskSetManager: Lost task 0.0 in stage 4.0 (TID 78,
> 192.168.122.39): java.io.IOException: java.lang.ClassNotFoundException:
> org.apache.spark.streaming.twitter.TwitterReceiver
> at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1163)
> at
> org.apache.spark.rdd.ParallelCollectionPartition.readObject(ParallelCollectionRDD.scala:70)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2000)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1801)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
> at
> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:72)
> at
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:98)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:194)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ClassNotFoundException:
>