Dear All,
I am facing problem with Spark Twitter Streaming code, When ever twitter4j
throws exception, i am unable to catch that exception. Could anyone help me
catching that exception.
Here is Pseudo Code:
SparkConf sparkConf = new
SparkConf().setMaster("local[2]").setAppName("Test");
// SparkConf sparkConf = new
SparkConf().setMaster("yarn-client").setAppName("Test");
// SparkTwitterStreaming sss = new SparkTwitterStreaming();
final int batchIntervalInSec = 60; // choose long interval to
reduce
// num
// files created
Duration batchInterval = Durations.seconds(batchIntervalInSec);
JavaStreamingContext jssc = new JavaStreamingContext(sparkConf,
batchInterval);
try{
JavaReceiverInputDStream<Status> receiverStream = null;
receiverStream = TwitterUtils.createStream(jssc, a2, params);
JavaDStream<String> tweets = receiverStream.map(new
Function<Status, String>() {
public String call(Status tweet) throws Exception {
String json = "";
Gson gson = new Gson();
jssc.start();
jssc.awaitTermination();
}
I have no issue with streaming the twitter data, When ever twitter account
had expired, i have to catch this exception and do work around for this
exception.
Here is the exception trace:
INFO spark.streaming.receiver.BlockGenerator - Started BlockGenerator
INFO spark.streaming.scheduler.ReceiverTracker - Registered receiver for
stream 0 from 172.16.28.183:34829
INFO spark.streaming.receiver.ReceiverSupervisorImpl - Starting receiver
INFO spark.streaming.twitter.TwitterReceiver - Twitter receiver started
INFO spark.streaming.receiver.ReceiverSupervisorImpl - Called receiver
onStart
INFO spark.streaming.receiver.ReceiverSupervisorImpl - Waiting for
receiver to be stopped
INFO twitter4j.TwitterStreamImpl - Establishing connection.
INFO twitter4j.TwitterStreamImpl - 401:Authentication credentials (
https://dev.twitter.com/pages/auth) were missing or incorrect. Ensure that
you have set valid consumer key/secret, access token/secret, and the system
clock is in sync.
<html>\n<head>\n<meta http-equiv="Content-Type" content="text/html;
charset=utf-8"/>\n<title>Error 401 Unauthorized</title>
</head>
<body>
<h2>HTTP ERROR: 401</h2>
<p>Problem accessing '/1.1/statuses/filter.json'. Reason:
<pre> Unauthorized</pre>
</body>
</html>
INFO twitter4j.TwitterStreamImpl - Waiting for 10000 milliseconds
WARN spark.streaming.receiver.ReceiverSupervisorImpl - Restarting receiver
with delay 2000 ms: Error receiving tweets
401:Authentication credentials (https://dev.twitter.com/pages/auth) were
missing or incorrect. Ensure that you have set valid consumer key/secret,
access token/secret, and the system clock is in sync.
<html>\n<head>\n<meta http-equiv="Content-Type" content="text/html;
charset=utf-8"/>\n<title>Error 401 Unauthorized</title>
</head>
<body>
<h2>HTTP ERROR: 401</h2>
<p>Problem accessing '/1.1/statuses/filter.json'. Reason:
<pre> Unauthorized</pre>
</body>
</html>
Relevant discussions can be found on the Internet at:
http://www.google.co.jp/search?q=944a924a or
http://www.google.co.jp/search?q=24fd66dc
TwitterException{exceptionCode=[944a924a-24fd66dc], statusCode=401,
message=null, code=-1, retryAfter=-1, rateLimitStatus=null, version=3.0.3}
at
twitter4j.internal.http.HttpClientImpl.request(HttpClientImpl.java:177)
at
twitter4j.internal.http.HttpClientWrapper.request(HttpClientWrapper.java:61)
at
twitter4j.internal.http.HttpClientWrapper.post(HttpClientWrapper.java:98)
at
twitter4j.TwitterStreamImpl.getFilterStream(TwitterStreamImpl.java:304)
at twitter4j.TwitterStreamImpl$7.getStream(TwitterStreamImpl.java:292)
at
twitter4j.TwitterStreamImpl$TwitterStreamConsumer.run(TwitterStreamImpl.java:462)
Please let me know if you need some more clarity my question.
Thanks,
Sony.
