GitHub user mubarak opened a pull request:
https://github.com/apache/spark/pull/1723
[SPARK-1853] Show Streaming application code context (file, line number) in
Spark Stages UI
Please review: [SPARK-1853] Show Streaming application code context (file,
line number) in Spark Stages UI
_Screenshot after fix:_
_Sample code used: Tutorial.scala_
```scala
1 package org.apache.spark.examples.streaming
2
3 import org.apache.spark._
4 import org.apache.spark.SparkContext._
5 import org.apache.spark.streaming._
6 import org.apache.spark.streaming.twitter._
7 import org.apache.spark.streaming.StreamingContext._
8 import org.apache.spark.examples.streaming.TutorialHelper._
9 import org.apache.spark.util.{CallSite, Utils}
10
11 object Tutorial {
12 def main(args: Array[String]) {
13
14 // Checkpoint directory
15 val checkpointDir = TutorialHelper.getCheckpointDirectory()
16
17 // Configure Twitter credentials
18 val apiKey = ""
19 val apiSecret = ""
20 val accessToken = ""
21 val accessTokenSecret = ""
22 TutorialHelper.configureTwitterCredentials(apiKey, apiSecret,
accessToken, accessTokenSecret)
23
24 // Your code goes here
25 val sc: SparkContext = new SparkContext(new
SparkConf().setAppName("Twitter
Example").setMaster("spark://ec2-54-241-226-42.us-west-1.compute.amazonaws.com:7077"))
26 val ssc = new StreamingContext(sc, Seconds(5))
27 val tweets = TwitterUtils.createStream(ssc, None)
28
29 val statuses = tweets.map(status => status.getText())
30 val words = statuses.flatMap(status => status.split(" "))
31
32 val hashtags = words.filter(word => word.startsWith("#"))
33 val counts = hashtags.map(tag => (tag, 1)).reduceByKeyAndWindow(_ +
_, _ - _, Seconds(60 * 5), Seconds(10))
34
35 val sortedCounts = counts.map { case(tag, count) => (count, tag) }
36 .transform(rdd => rdd.sortByKey(false))
37 sortedCounts.foreach(rdd =>
38 println("\nTop 10 hashtags:\n" + rdd.take(10).mkString("\n")))
39
40 ssc.checkpoint(checkpointDir)
41 ssc.start()
42 ssc.awaitTermination()
43
44 }
45 }
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mubarak/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1723.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1723
----
commit d466d75341bf246bf57341ca6b9ea3ed11679040
Author: Mubarak Seyed <[email protected]>
Date: 2014-07-18T01:15:05Z
Changes for spark streaming UI
commit 9d38d3cecedc37775cb5bd70c661c3acd2dd7f43
Author: Mubarak Seyed <[email protected]>
Date: 2014-07-18T01:30:46Z
[SPARK-1853] Show Streaming application code context (file, line number) in
Spark Stages UI
commit 1500deb44c7c484fae725144c288240b5b477fdb
Author: Mubarak Seyed <[email protected]>
Date: 2014-07-18T15:11:58Z
Changes in Spark Streaming UI
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
