[
https://issues.apache.org/jira/browse/SPARK-29426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16949892#comment-16949892
]
Jungtaek Lim commented on SPARK-29426:
--------------------------------------
Could you attach query listener and see the actual event time in each batch as
well? And I suggest you to provide simple and complete reproducer, as to
reproduce mes has to exist.
> Watermark does not take effect
> ------------------------------
>
> Key: SPARK-29426
> URL: https://issues.apache.org/jira/browse/SPARK-29426
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 2.4.3
> Reporter: jingshanglu
> Priority: Major
>
> I use withWatermark and window to express windowed aggregations, but the
> Watermark does not take effect.
> my code:
> {code:java}
> // code placeholder
> Dataset<Row> clientSqlIpCount = mes.withWatermark("timestamp","1 minute")
> .groupBy(
> functions.window(mes.col("timestamp"),"10 minutes","5
> minutes"),
> mes.col("sql"),mes.col("client"),mes.col("ip"))
> .count();
> StreamingQuery query = clientSqlIpCount
> .writeStream()
> .outputMode("Update")
> .format("console")
> .start();
> spark.streams().awaitAnyTermination();
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
