Hi everyone,
the ProcessTableFunction (PTF) feature has been well received by the
Flink community and its adoption is increasing. Since FLIP-440 [1]
introduced a lot of new API and new concepts, some design decisions need
smaller adjustments along late data handling and lazy state access.
Also, talking to community members at Current and Flink Forward
conferences has shown that broadcast state is crucial to bridge the gap
to DataStream API applications for broadcast joining and rule-based logic.
I would like to propose FLIP-562: Improve ProcessTableFunctions for late
data handling and state access" [2].
This FLIP proposes 3 important PTF improvements:
1) Don’t drop late data in ProcessFunction as data-loss is usually not
intended; similar to DataStream API’s ProcessFunction
2) Introduce ValueView to enable a “supplier”-pattern for state access;
similar to MapView and ListView
3) Introduce BROADCAST_SEMANTIC_TABLE as a new kind of argument to PTFs
Regarding forward compatibility, all proposed items can be made
available in batch mode eventually for a unified experience. From my
point of view, these remaining adjustments should make PTF fully
production ready, I don't expect any major additions in the mid-term.
Looking forward to your feedback.
Thanks,
Timo
[1] https://cwiki.apache.org/confluence/x/pQnPEQ
[2] https://cwiki.apache.org/confluence/x/qIo8G
