abrarsher23 commented on issue #18350: URL: https://github.com/apache/iceberg/issues/18350#issuecomment-5964508104
Three more findings after implementing the Case 2 direction above and testing it. All three concern key and partition promotion. **1. Writers after a promotion don't share with each other.** If the first writer's key type stays the reference for the whole checkpoint, this sequence still duplicates: insert `id=9` (`int`), promote `id` to `long`, update `id=9`, add a column, update `id=9`. The last two writers both have `long` keys, but each is compared with the original `int` key, so they get separate trackers. Fix approach: group trackers by key type. A writer joins the first group whose key type it can share, otherwise it opens a group, which later writers of the same type join. **2. A promoted key's old and new records can go to different subtasks.** With equality-field distribution (and unpartitioned hash distribution), the key hash comes from `Objects::hashCode` per field. `Integer.hashCode` and `Long.hashCode` differ for negative values, and `Float.hashCode` and `Double.hashCode` differ for nearly every value. So the records of one key written before and after the promotion can reach different writers, and no tracker can be shared. In a probe against `HashKeyGenerator` on `flink/v2.0` and `v2.1` (one `int` equality field, write parallelism 4), all 50 keys in -50..-1 split, and none of 0..50 did. I have not checked `main`. Fix approach: hash promoted primitive keys in their widest type (`int` as `long`, `float` as `double`). Non-negative `int` keys keep their hash, so their routing doesn't change. **3. The non-key partition column case is blocked by a separate bug.** If an identity or truncate partition column is promoted, a checkpoint with `Integer` and `Long` partition values of one spec fails to commit, even with no re-write, so a shared tracker can't help. That is a commit failure in core, filed as #18355. This also means the "normalise identity-partition keys the same way" part of the Case 2 direction doesn't help until that is fixed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
