[
https://issues.apache.org/jira/browse/SPARK-13650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15179347#comment-15179347
]
Mario Briggs commented on SPARK-13650:
--------------------------------------
Running locally on my MAC, here is the output
{quote}
[Stage 0:> (0 + 0) /
2]16/03/04 10:38:18 INFO VerifiableProperties: Verifying properties
16/03/04 10:38:18 INFO VerifiableProperties: Property auto.offset.reset is
overridden to smallest
16/03/04 10:38:18 INFO VerifiableProperties: Property group.id is overridden to
16/03/04 10:38:18 INFO VerifiableProperties: Property zookeeper.connect is
overridden to
-------------------------------------------
Time: 1457068098000 ms
-------------------------------------------
10
[Stage 6:=======================================> (2 + 0) / 3]
[Stage 6:=======================================> (2 + 0) / 3]
{quote}
The '10' is the output from the first batch interval at time '1457068098000 ms'
. Thereafter, the only output for the next ~2 minutes is the 'stage 6'. A ^C
fails to stop the app and need to do a 'kill -9 pid'
> Usage of the window() function on DStream
> -----------------------------------------
>
> Key: SPARK-13650
> URL: https://issues.apache.org/jira/browse/SPARK-13650
> Project: Spark
> Issue Type: Bug
> Components: Streaming
> Affects Versions: 1.5.2, 1.6.0, 2.0.0
> Reporter: Mario Briggs
> Priority: Minor
>
> Is there some guidance of the usage of the Window() function on DStream. Here
> is my academic use-case for which it fails.
> Standard word count
> val ssc = new StreamingContext(sparkConf, Seconds(6))
> val messages = KafkaUtils.createDirectStream(...)
> val words = messages.map(_._2).flatMap(_.split(" "))
> val window = words.window(Seconds(12), Seconds(6))
> window.count().print()
> For the first batch interval it gives the count and then it hangs (inside the
> unionRDD)
> I say the above use-case is academic since one can achieve similar
> fuctionality by using instead the more compact API
> words.countByWindow(Seconds(12), Seconds(6))
> which works fine.
> Is the first approach above not the intended way of using the .window() API
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
