Hi Martijn,
Thanks for your insightful feedback and careful review.
Your point about avoiding the mixing of physical concerns with
logical semantics makes perfect sense, and it prompted me to rethink
the design more thoroughly.
I would like to share an updated direction below and see whether this
aligns better with your expectations.
1. Original Proposal — Withdrawn
I initially proposed extending the existing TUMBLE window with an
optional STAGGER parameter, inspired by the existing DataStream
WindowStagger, which shifts window boundaries.
However, I agree with your analysis that doing so in SQL would
silently break the deterministic alignment contract of TUMBLE.
Therefore, I would like to withdraw this part of the proposal.
2. Hints and PTF — Deferred for Now
- Regarding Hints
I agree that a hint is probably not the right abstraction here. Staggering
changes the resulting window boundaries, while hints in
Flink are generally treated as plan-intervention mechanisms that do
not alter query semantics.
In addition, there is currently no precedent for window-related hints
in Flink SQL.
- Regarding PTF (Process Table Functions)
I agree that PTF could ultimately become a powerful extension point
for custom or user-defined windows.
However, building a comprehensive PTF-based windowing framework is
itself a substantial design effort and likely deserves a dedicated
discussion.
To keep the scope of this FLIP manageable, I would prefer to leave
PTF integration as future work for now.
------------------------------
3. Revised Proposal — Introduce a New TVF:STAGGER_TUMBLE
Since staggering fundamentally changes the window definition, I now
believe it should be treated as a logical semantic change rather than
a pure physical optimization.
Therefore, instead of modifying TUMBLE, the cleaner approach would
be to introduce a separate TVF with an explicit contract:
STAGGER_TUMBLE(
TABLE data,
DESCRIPTOR(timecol),
size,
stagger_strategy
)
-- stagger_strategy:
-- 'RANDOM'
-- 'NATURAL'
-- 'KEY_BASED'
For KEY_BASED, the requirement of a keyed context (for example,
Window Aggregation with GROUP BY) would be validated at compile
time.
Key properties of this approach:
-
*Zero impact on TUMBLE*
The semantic contract of the existing TUMBLE TVF remains fully
preserved.
-
*Explicit semantics*
STAGGER_TUMBLE would define its own semantics explicitly,
including that window boundaries may vary depending on the selected
stagger strategy.
------------------------------
4. Future Work
A potentially cleaner long-term direction may be to separate:
-
logical window boundary assignment, and
-
physical emission scheduling
In other words, preserving perfectly aligned window boundaries while
staggering only the emission timing.
That would constitute a true physical optimization without changing
query results.
This could potentially evolve into an optional parameter such as
shift_window_boundary in STAGGER_TUMBLE, and can be explored in a
follow-up FLIP.
------------------------------
Does this revised direction address your core concerns?
I would also greatly appreciate feedback from others on the mailing
list.
If there is general consensus around this direction, I will update
the FLIP document accordingly. Otherwise, I am happy to continue
iterating on the design.
Best regards,
Zihao
Martijn Visser <[email protected]> 于2026年5月21日周四 01:05写道：
> Hi Zihao,
>
> Thanks for the FLIP. I am worried that the proposal is mixing physical
> concerns (the downstream bursts of data) into logical semantics. I
> think a more natural escape hatch are hints. I also think that
> KEY_BASED is not really a physical optimization anyway, since it
> shifts window_start / window_end values in the output and therefore
> changes the result set. That makes it a poor fit for both a TVF
> argument and a hint, and probably a better fit for a PTF where the
> user explicitly owns the boundary assignment function.
>
> Looking forward to your thoughts.
>
> Best regards,
>
> Martijn
>
> Op wo 20 mei 2026 om 14:32 schreef rocxing <[email protected]>:
> >
> > Hi Zihao and all,
> >
> >
> > Thanks a lot for this practical proposal.
> > This is a valuable feature for Flink SQL users, and we have also
> encountered exactly the same pain points in our production environments.
> > Furthermore, the KEY_BASED deterministic stagger strategy is a good way
> to eliminate non-determinism problems.
> >
> >
> > Best regards,
> > Pengxiang Wang
>
