[jira] [Commented] (FLINK-22300) Why TimeEvictor of Keyed Windows evictor do not support ProcessingTime of TimeCharacteristic
[ https://issues.apache.org/jira/browse/FLINK-22300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450970#comment-17450970 ] Paul Lin commented on FLINK-22300: -- [~arvid] For a usage case, how about a global window (no window size is defined) that relies on a processing time TimeEvictor to clean up the elements? RWT the solution, I think maybe in case of processing time we can attach the processing timestamp to the TimestampedValue in window assigners and erase them after evictors. WDYT? > Why TimeEvictor of Keyed Windows evictor do not support ProcessingTime of > TimeCharacteristic > > > Key: FLINK-22300 > URL: https://issues.apache.org/jira/browse/FLINK-22300 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.11.3 >Reporter: Bo Huang >Priority: Minor > Labels: auto-deprioritized-major, stale-minor > > StreamExecutionEnvironment.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime). > The data after Windows process can not be evicted by TimeEvictor of Keyed > Windows Beause TimestampedValue have no timestamp value. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-22300) Why TimeEvictor of Keyed Windows evictor do not support ProcessingTime of TimeCharacteristic
[
https://issues.apache.org/jira/browse/FLINK-22300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450921#comment-17450921
]
Arvid Heise commented on FLINK-22300:
-
Okay I see, so Flink would never evict the data. That is indeed a bug.
What do you think is the proper fix? I have a hard time to come up with proper
semantics beyond saying that users can't use {{TimeEvictor}} with processing
time. For a window in processing time, a user probably wants to retain all
elements before trigger and evict all elements after the trigger in all cases.
For other cases, they would use ingestion or event time.
> Why TimeEvictor of Keyed Windows evictor do not support ProcessingTime of
> TimeCharacteristic
>
>
> Key: FLINK-22300
> URL: https://issues.apache.org/jira/browse/FLINK-22300
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.11.3
>Reporter: Bo Huang
>Priority: Minor
> Labels: auto-deprioritized-major, stale-minor
>
> StreamExecutionEnvironment.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime).
> The data after Windows process can not be evicted by TimeEvictor of Keyed
> Windows Beause TimestampedValue have no timestamp value.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (FLINK-22300) Why TimeEvictor of Keyed Windows evictor do not support ProcessingTime of TimeCharacteristic
[ https://issues.apache.org/jira/browse/FLINK-22300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450847#comment-17450847 ] Paul Lin commented on FLINK-22300: -- [~arvid] Please let me clarify a bit more. The problem is that TimeEvictor would not be effective with processing time, because instead of getting the current system timestamp (as `now()` you mentioned), it would extract timestamps from TimestampedValue(backed by StreamRecord), see [code|https://github.com/apache/flink/blob/6f05e6fdadd5cd8f27983bc8140d05f1c3557bb6/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/windowing/evictors/TimeEvictor.java#L70]. However, with processing time, TimestampedValue has no timestamp, so `evict()` would always return directly. > Why TimeEvictor of Keyed Windows evictor do not support ProcessingTime of > TimeCharacteristic > > > Key: FLINK-22300 > URL: https://issues.apache.org/jira/browse/FLINK-22300 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.11.3 >Reporter: Bo Huang >Priority: Minor > Labels: auto-deprioritized-major, stale-minor > > StreamExecutionEnvironment.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime). > The data after Windows process can not be evicted by TimeEvictor of Keyed > Windows Beause TimestampedValue have no timestamp value. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-22300) Why TimeEvictor of Keyed Windows evictor do not support ProcessingTime of TimeCharacteristic
[ https://issues.apache.org/jira/browse/FLINK-22300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450532#comment-17450532 ] Arvid Heise commented on FLINK-22300: - Sorry for not following it up. I have a hard time to understand what you exactly expect. The timestamp of a record in processing time is always `now()` by definition and from your description it sounds like indeed returns it (I may be wrong here). I think what you are looking at is to use [ingestion time|https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/learn-flink/streaming_analytics/]. Then a record gets a proper timestamp assigned on first arrival in the system and windowing and eviction are deterministic tasks. However, I might have misunderstood your needs, so please reiterate them if ingestion time is not working for you and why. > Why TimeEvictor of Keyed Windows evictor do not support ProcessingTime of > TimeCharacteristic > > > Key: FLINK-22300 > URL: https://issues.apache.org/jira/browse/FLINK-22300 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.11.3 >Reporter: Bo Huang >Priority: Minor > Labels: auto-deprioritized-major, stale-minor > > StreamExecutionEnvironment.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime). > The data after Windows process can not be evicted by TimeEvictor of Keyed > Windows Beause TimestampedValue have no timestamp value. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-22300) Why TimeEvictor of Keyed Windows evictor do not support ProcessingTime of TimeCharacteristic
[ https://issues.apache.org/jira/browse/FLINK-22300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17448954#comment-17448954 ] Paul Lin commented on FLINK-22300: -- Any follow-ups? I would be happy to take this issue if we reach a consensus on the solution. [~trohrmann] [~arvid] > Why TimeEvictor of Keyed Windows evictor do not support ProcessingTime of > TimeCharacteristic > > > Key: FLINK-22300 > URL: https://issues.apache.org/jira/browse/FLINK-22300 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.11.3 >Reporter: Bo Huang >Priority: Minor > Labels: auto-deprioritized-major, stale-minor > > StreamExecutionEnvironment.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime). > The data after Windows process can not be evicted by TimeEvictor of Keyed > Windows Beause TimestampedValue have no timestamp value. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-22300) Why TimeEvictor of Keyed Windows evictor do not support ProcessingTime of TimeCharacteristic
[ https://issues.apache.org/jira/browse/FLINK-22300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17322668#comment-17322668 ] Till Rohrmann commented on FLINK-22300: --- Thanks for reporting this problem [~hxtog]. cc [~AHeise] for future reference. > Why TimeEvictor of Keyed Windows evictor do not support ProcessingTime of > TimeCharacteristic > > > Key: FLINK-22300 > URL: https://issues.apache.org/jira/browse/FLINK-22300 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.11.3 >Reporter: Bo Huang >Priority: Major > > StreamExecutionEnvironment.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime). > The data after Windows process can not be evicted by TimeEvictor of Keyed > Windows Beause TimestampedValue have no timestamp value. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-22300) Why TimeEvictor of Keyed Windows evictor do not support ProcessingTime of TimeCharacteristic
[ https://issues.apache.org/jira/browse/FLINK-22300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17322660#comment-17322660 ] Paul Lin commented on FLINK-22300: -- There's a workaround for this problem: extract timestamps and watermarks right before the processing time window, but it may affect accuracy if there's some latency. To fully solve the problem, we may need to attach processing timestamps to the records in the window assigner. > Why TimeEvictor of Keyed Windows evictor do not support ProcessingTime of > TimeCharacteristic > > > Key: FLINK-22300 > URL: https://issues.apache.org/jira/browse/FLINK-22300 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.11.3 >Reporter: Bo Huang >Priority: Major > > StreamExecutionEnvironment.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime). > The data after Windows process can not be evicted by TimeEvictor of Keyed > Windows Beause TimestampedValue have no timestamp value. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-22300) Why TimeEvictor of Keyed Windows evictor do not support ProcessingTime of TimeCharacteristic
[ https://issues.apache.org/jira/browse/FLINK-22300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17322652#comment-17322652 ] Paul Lin commented on FLINK-22300: -- It looks like a bug to me. TimeEvictor would try to extract timestamps from StreamRecord, but with processing time characteristics there would be no timestamp. cc [~chesnay] [~trohrmann] > Why TimeEvictor of Keyed Windows evictor do not support ProcessingTime of > TimeCharacteristic > > > Key: FLINK-22300 > URL: https://issues.apache.org/jira/browse/FLINK-22300 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.11.3 >Reporter: Bo Huang >Priority: Major > > StreamExecutionEnvironment.setStreamTimeCharacteristic(TimeCharacteristic.ProcessingTime). > The data after Windows process can not be evicted by TimeEvictor of Keyed > Windows Beause TimestampedValue have no timestamp value. -- This message was sent by Atlassian Jira (v8.3.4#803005)
