[
https://issues.apache.org/jira/browse/SPARK-22480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-22480:
---------------------------------
Labels: bulk-closed (was: )
> Dynamic Watermarking
> --------------------
>
> Key: SPARK-22480
> URL: https://issues.apache.org/jira/browse/SPARK-22480
> Project: Spark
> Issue Type: Wish
> Components: Structured Streaming
> Affects Versions: 2.2.0
> Reporter: Jochen Niebuhr
> Priority: Major
> Labels: bulk-closed
>
> When you're using the watermark feature, you're forced to provide an absolute
> duration to identify late events. For the case we're using structured
> streaming for, this is not completely working. In our case, late events will
> be possible on the next business day. So I'd like to use 24 hours watermark
> for Sunday-Thursday, 72 hours for Friday, 48 for Saturday. What I would
> suggest is being able to use a function or expression to {{withWatermark}} so
> people can implement this or similar behaviors. If this sounds like a good
> idea, I can probably supply a pull request.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
