Timo Walther created FLINK-23426:
------------------------------------
Summary: Support changelog processing in batch mode
Key: FLINK-23426
URL: https://issues.apache.org/jira/browse/FLINK-23426
Project: Flink
Issue Type: Sub-task
Components: Table SQL / API
Reporter: Timo Walther
The DataStream API can execute arbitrary DataStream programs when running in
batch mode. However, this is not the case for the Table API batch mode. E.g. a
source with non-insert only changes is not supported and updates/deletes cannot
be emitted.
In theory, we could make this work by running the "stream mode" of the planner
(CDC transformations) on top of the "batch mode" of DataStream API (specialized
state backend, sorted inputs). It is up for discussion if and how we expose
such functionality.
If we don't allow enabling incremental updates, we can also add a special batch
operator that materializes the incoming changes for a batch pipeline. However,
it would require "complete" CDC logs (i.e. no missing UPDATE_AFTER).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
