laskoviymishka commented on issue #17542: URL: https://github.com/apache/iceberg/issues/17542#issuecomment-5239461845
Thanks for putting this together, and for validating it end to end. I'd split this in two, because the halves have opposite answers. **The silent no-op is a bug**, file it separately. If `iceberg.tables.cdc-field` / `upsert-mode-enabled` are still accepted but do nothing, that should fail loud, not silently duplicate rows. Worth fixing regardless of the rest. **Reviving eq-delete-backed upsert as a durable mode is the part I'd push back on.** The format is steering equality deletes toward a short-lived write-time buffer that converts to deletion vectors, not a resting state you accumulate, see [[DISCUSS] Deprecate Equality Deletes](https://lists.apache.org/thread/z0gvco6hn2bpgngvk4h6xqrnw8b32sw6) (read-path cost is unpredictable; CDC/row-lineage are ~impossible with eq-deletes in the table) and @mxm's [[DISCUSS] Flink: Equality delete → DV conversion](https://lists.apache.org/thread/gldqrycmo24r0vo77wzcj9s9b04rtvy7), which is the actual replacement — stage eq-deletes, resolve them against a keyed index, commit DVs — now merged as the [`ConvertEqualityDeletes`](https://github.com/apache/iceberg/pull/17142) maintenance task ([docs](https://github.com/apache/iceberg/pull/17113)). The endgame there is deprecating eq-deletes in V4 once there's index support. So a sink that makes eq-delete upsert first-class invests in exactly the mechanism the format is demoting. The durable target is the same shape as @mxm's work: emit the delete intent → resolve against a key→position index → land DVs. The hard part is the index, and that's a cross-engine concern being designed on the dev list. Suggested split: 1. Land the config silent-failure fix as its own bug. 2. Take the upsert/delete design to the dev list framed against the eq-delete→DV work, so Kafka Connect and Flink don't diverge on CDC semantics. Your op-code / tombstone / partition-migration questions all resolve differently if the backing is DV-via-index rather than eq-delete files. Been doing CDC-into-Iceberg work; the index/DV split is where the north-star heading. More generally such questions better to resolve via mail-list, so if you can kick-off the thread -- it would be super helpful. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
