Fizzabid opened a new pull request, #17899: URL: https://github.com/apache/iceberg/pull/17899
Adds experimental modes to the kafka-connect sink for CDC pipelines fed by Debezium. All controlled by connector config; defaults preserve upstream behavior. Upsert (equality-delete backed) — enabled by `iceberg.tables.upsert-mode-enabled=true` plus per-table `iceberg.tables.<name>.id-columns` (or global `default-id-columns`). New `RecordDeltaWriter` extends Iceberg core's `BaseTaskWriter` with per-partition `BaseEqualityDeltaWriter` instances. `IcebergWriter.write` reads `iceberg.tables.cdc-field` (default `_cdc.op`) and routes I/C/R to insertRow, U to updateRow, D to deleteRow. `RecordUtils.createTableWriter` selects the delta writer when both the flag and id-columns are set. Source-timezone shift — `iceberg.tables.source-timezone=<zone>` treats Debezium's naive DATETIME/DATETIME2 columns (mislabeled as UTC by the JDBC driver) as the given IANA zone and rewrites incoming epoch millis to the corrected UTC instant via recover-digits → relabel-as-source-zone → back-to-instant. `RecordConverter.correctForSourceTimezone` is invoked from both the timestamptz and naive Number/Date branches. Naive storage — `iceberg.tables.store-naive-timestamps=true` forces `SchemaUtils.toIcebergType` to emit `TimestampType.withoutZone()` for Debezium Connect.Timestamp columns. Combined with `source-timezone`, the new `RecordConverter.millisToNaive` stores the source-zone wall-clock digits as a naive `timestamp` (matches SQL Server display; ClickHouse IcebergS3 auto-discovers `DateTime64(6)` without a zone attribute). Files touched: - IcebergSinkConfig: new configs and getters - SchemaUtils: naive vs zoned type inference - RecordConverter: shift and naive value conversion - IcebergWriter, RecordUtils: upsert delta-writer routing - RecordDeltaWriter: new equality-delete writer -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
