pvary commented on PR #18517:
URL: https://github.com/apache/iceberg/pull/18517#issuecomment-6339261509
I talked with @mxm offline. We agreed that the current change only solves
part of the problem, and discussed how much effort and complexity we want to
put into solving this issue.
@adityaksolves: How did you run into this issue? Could you share the details?
With the help of my AI friend, we found a few more cases where the PR still
writes to the wrong field id. In each case, the record still has the old shape,
written before the column changes:
| # | Table changes | Record | PR resolves to | Current field id | Effect
with the PR | With the proposal below |
|---|---|---|---|---|---|---|
| A | add `c`, drop `c`, add `d`, re-add `c` | `{id, data, c}` | schema 1,
`c` id=3 | `c` id=5 | `c` is read as `null` | current schema, `c` id=5 |
| B | add `c int`, drop `c`, re-add `c long` | `c` as `int` | schema 1, `c`
id=3 | `c` id=4 | `c` is read as `null` | current schema, `c` id=4 (int→long) |
| C | rename `data` → `legacy`, add new `data` | `{id, data}` | schema 0,
`data` id=2 | `data` id=3 | values end up in `legacy` | current schema, `data`
id=3 |
| D | drop `myfield string`, re-add as `double` | `myfield` as `string` |
schema 1 (old) | `myfield` id=3 | value isn't visible, job keeps running |
unchanged: falls back to the old schema |
In A–C the current schema could take the record with a data conversion, but
an older schema that matches exactly wins and writes to a retired or unrelated
field id. D is @mxm's case, where falling back to the old schema is the
intended lenient behavior.
If we want to handle every case where a column name reappears, we need a
description that is easy for users to understand. One possibility:
> The Dynamic Sink matches each record to the table schema by column name.
It writes with the current schema whenever it can, converting values if needed,
and falls back to an older schema only when the record doesn't fit the current
one. Values written with an older schema may not be visible in columns that
have changed since. The table schema is only updated when the record fits no
existing schema.
The difference from the current PR is that this prefers a schema where every
column of the record stays visible in the table, even if it needs
`DATA_CONVERSION_NEEDED`, over an exact match with an older schema whose column
names now point to different field ids.
The implementation could look like this:
https://github.com/pvary/iceberg/commit/f322d30a1eb5a9858c8b781bbb71759d9e9cb94c
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
