fightBoxing opened a new pull request, #15282:
URL: https://github.com/apache/iceberg/pull/15282
This PR adds CDC support for Flink Iceberg source, allowing users to read
changelog data with proper RowKind (INSERT, DELETE, UPDATE_BEFORE,
UPDATE_AFTER).
Key changes:
- Add StreamingReadMode enum (APPEND_ONLY, CHANGELOG)
- Add ChangelogDataIterator for iterating changelog scan tasks
- Add RowDataChangelogScanTaskReader for reading changelog data
- Add ChangelogScanSplit for CDC split handling
- Add ChangelogRowDataReaderFunction for reader function support
- Modify FlinkReadOptions/FlinkReadConf to support streaming-read-mode config
- Modify ScanContext to support changelog scan mode
- Modify FlinkSplitPlanner to plan changelog scan tasks
- Modify ContinuousSplitPlannerImpl to support CDC mode
- Modify IcebergSource to support streamingReadMode builder method
- Modify IcebergTableSource to support CDC ChangelogMode
- Add integration tests for CDC streaming read
Usage:
- Java API:
IcebergSource.forRowData().streamingReadMode(StreamingReadMode.CHANGELOG)
- SQL: SELECT * FROM table /*+ OPTIONS('streaming-read-mode' = 'CHANGELOG')
*/
Supported Flink versions: v1.20, v2.0, v2.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
