abrarsher23 commented on issue #18350: URL: https://github.com/apache/iceberg/issues/18350#issuecomment-5984145273
Update: our fork now has a fix for Case 2 and for the first finding in my follow-up comment above: https://github.com/Affirm/iceberg/pull/20. It's stacked on https://github.com/Affirm/iceberg/pull/17, the fork's fix for #18262. - **Case 2 (key type promotion).** A writer whose key type is a promotion of the tracker's key type uses the tracker through a converting view. That covers `int`/`long`, `float`/`double` and same-scale decimal, in either direction. Keys that don't fit the narrower type, such as a `long` above the `int` range, go into a small second map keyed by the wide type. When the key types are equal, the writer gets the tracker itself, so the usual path does no conversion. - **Writers after a promotion.** Trackers are grouped by key type, so later writers with the same key type share with each other, not only with the first writer. - **Not covered there:** Cases 1, 3 and 4, and the partition case, which is blocked by #18355. **Testing.** - Unit tests in `core`, `data` and both Flink modules, covering every changed line and branch. - An end-to-end run of the real sink fed by a live CDC stream, with an `int` key promoted to `long` mid-checkpoint. Every key ended up visible once. Reviews are very welcome, especially on whether this direction would work upstream. If it would, I'm happy to port it as a PR here on top of whichever #18262 fix lands. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
