[ 
https://issues.apache.org/jira/browse/FLINK-41508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated FLINK-41508:
-----------------------------------
    Labels: pull-request-available  (was: )

> Support multiple bounded RANGE OVER windows with shared state in streaming SQL
> ------------------------------------------------------------------------------
>
>                 Key: FLINK-41508
>                 URL: https://issues.apache.org/jira/browse/FLINK-41508
>             Project: Flink
>          Issue Type: New Feature
>          Components: Table SQL / Planner
>            Reporter: Junfan Zhang
>            Priority: Major
>              Labels: pull-request-available
>
> Streaming SQL rejects a query that computes aggregates over several OVER 
> windows with different ranges, which is a common pattern in real-time ML 
> features engineering:
>  
> {code:java}
> SELECT 
>   entity_id, 
>   event_time, 
>   SUM(v) OVER w10, 
>   SUM(v) OVER w20, 
>   SUM(v) OVER w30 
> FROM events WINDOW 
>   w10 AS (PARTITION BY entity_id ORDER BY event_time RANGE BETWEEN INTERVAL 
> '10' MINUTE PRECEDING AND CURRENT ROW), 
>   w20 AS (PARTITION BY entity_id ORDER BY event_time RANGE BETWEEN INTERVAL 
> '20' MINUTE PRECEDING AND CURRENT ROW), 
>   w30 AS (PARTITION BY entity_id ORDER BY event_time RANGE BETWEEN INTERVAL 
> '30' MINUTE PRECEDING AND CURRENT ROW);{code}
>  
> It fails with "Over Agg: Unsupported use of OVER windows. All aggregates must 
> be computed on the same window." Batch mode supports it.
>  
> The workaround is one nested subquery per window. Each level is a separate 
> operator that stores its own copy of the input rows, so state grows with the 
> sum of all ranges (10 + 20 + 30 = 60 minutes of rows instead of 30).
>  
> Proposal: when all windows of one OVER node are bounded event-time RANGE 
> windows with the same PARTITION BY and ORDER BY, evaluate them in one 
> operator that:
> - stores the input rows once, kept for the largest range;
> - keeps one accumulator per window and retracts a row from a window when the 
> row leaves that window's range.
>  
> Row state becomes max(range_i) instead of sum(range_i). Single-window queries 
> keep their current operator and state layout; other combinations (ROWS, 
> proctime, unbounded, Python UDAFs) keep failing with a clearer error message.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to