GitHub user zhichao-li opened a pull request:
https://github.com/apache/spark/pull/4828
[SPARK-6077] update listener for the existing streamingTab instead of
adding an one new one
Currently we would create a new streaming tab for each streamingContext
even if there's already one on the same sparkContext which would cause
duplicate StreamingTab created and none of them is taking effect.
snapshot:
https://www.dropbox.com/s/t4gd6hqyqo0nivz/bad%20multiple%20streamings.png?dl=0
How to reproduce:
1)
import org.apache.spark.SparkConf
import org.apache.spark.streaming.
{Seconds, StreamingContext}
import org.apache.spark.storage.StorageLevel
val ssc = new StreamingContext(sc, Seconds(1))
val lines = ssc.socketTextStream("localhost", 9999,
StorageLevel.MEMORY_AND_DISK_SER)
val words = lines.flatMap(_.split(" "))
val wordCounts = words.map(x => (x, 1)).reduceByKey(_ + _)
wordCounts.print()
ssc.start()
.....
2)
ssc.stop(false)
val ssc = new StreamingContext(sc, Seconds(1))
val lines = ssc.socketTextStream("localhost", 9999,
StorageLevel.MEMORY_AND_DISK_SER)
val words = lines.flatMap(_.split(" "))
val wordCounts = words.map(x => (x, 1)).reduceByKey(_ + _)
wordCounts.print()
ssc.start()
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zhichao-li/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/4828.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4828
----
commit a711920518fe6c0c569d490a2c6ee33cdb1f14c0
Author: lisurprise <[email protected]>
Date: 2015-02-28T06:34:31Z
update listener for the existing streamingTab instead of adding one new tab
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
