RussellSpitzer commented on PR #16425:
URL: https://github.com/apache/iceberg/pull/16425#issuecomment-6779153428
From the sync today, Ryan wanted to loosen the rules from the ones I posted
earlier. Please take a look, I spent a lot of time on this and Promise my human
eyes and brain thought a lot about this although I did use LLM assistance on
the wording and images.
The goal is to allow more general column replacements by making writers
avoid ambiguous situations. Unlike my earlier rules, this requires writers to
be more aware of exactly which values they are overwriting, but it allows more
generic replacement.
The key rule we are trying to respect is:
"Listing a nested field does not change whether its parent struct is null.
To change that, the column file must list the parent struct."
Going through the rules, they change a bit (see 3):
1. A column file must not list a field nested in a list or map.
2. A column file stores the values of the fields listed in its `field_ids`.
Readers must ignore fields in a column file that are not listed in its
`field_ids`.
3. An added column file must list:
- each field whose value changes.
- each struct that changes between null and non-null.
- each field directly nested in a struct that has changed from null to
non-null.
A field nested in a struct that is null after the update does not need to
be listed.
4. When a writer adds a column file, the writer must remove each listed
field from the column files of the previous entry. The writer must remove a
column file that has no remaining fields.
5. The reader must read a field from the column file that lists it. If no
column file lists the field, the reader must read it from the base file, using
Column Projection for a data file.
6. If a struct is null in a row, each field nested in that struct is null in
that row.
**Why the rules are required**
```
2: customer optional struct
3: name string
4: address optional struct
5: street string
6: city string
8: tags list<9: string>
```
Column files are written as CF-# [field IDs listed].
**Rule 1 - No fields inside lists or maps**
<img width="1648" height="524" alt="image"
src="https://github.com/user-attachments/assets/80234361-5aa5-4de9-bd7c-d06efd4e186d";
/>
`SET tags[0] = 'x'` - a writer cannot write CF-1 [9] with only element 0.
We don't want any rules about how to combine lists or maps across files, so
the writer must write CF-1 [8] with the entire contents of the list. The same
applies to maps
**Rule 2 - Ignore unlisted fields**
This lets us replace fields without rewriting the column files we already
wrote.
For example:
If we have CF-1 [3, 4, 5, 6] and only want to update field 3, we can write
CF-2 [3] and change the entry for CF-1 to [4, 5, 6].
CF-1 still physically contains field 3, and readers must ignore it.
<img width="1648" height="904" alt="image"
src="https://github.com/user-attachments/assets/414828e2-c8ab-44dd-a929-4114ffab5b4a";
/>
It also covers fields a file format needs in order to store a listed field.
For example, CF-1 [5] has to store `customer` and `address` on the path to
`street`. Readers must not take the null state of `customer` or `address` from
CF-1.
**Rule 3 - No implicit change to a parent struct's null state**
Note that this rule is explicitly *row based* if no rows violate the rule
then nothing has to be done other than listing the field being modified.
Start with `customer` null in row 0.
User calls `SET name = 'andrew'`
Creating CF-1 [3] is *not allowed*.
<img width="1648" height="482" alt="image"
src="https://github.com/user-attachments/assets/1360446a-e18b-4d11-be31-36aa3dd27e2a";
/>
The reader takes `customer` (field 2) from the base file, where it is null,
so rule 6 hides the new `name`. The update changed whether `customer` is null,
so `customer` must be listed.
Instead the writer must create:
CF-1 [2, 3, 4]
<img width="1648" height="776" alt="image"
src="https://github.com/user-attachments/assets/4de33962-605c-43b3-99c5-dea25b766aee";
/>
`name` and `address` are listed because they are directly nested in a struct
that changed from null to non-null. Here is why that is needed:
Start with `address = {street: '1 Elm', city: 'Austin'}` in the base file.
`SET address = null` - CF-1 [4]. `street` and `city` do not need to be
listed, because their parent is null after the update. Rule 6 hides their old
values.
Then `SET street = '2 Oak'` - `address` becomes non-null again. CF-2 [4, 5]
is *not allowed*: `city` is no longer hidden, so the reader would return
`'Austin'` from the base file instead of null.
<img width="1648" height="904" alt="image"
src="https://github.com/user-attachments/assets/3b2a64a6-579c-47aa-845b-00d7fdfd1d30";
/>
The writer must create CF-2 [4, 5, 6] with `city = null`.
<img width="1648" height="776" alt="image"
src="https://github.com/user-attachments/assets/2bcc1ca6-e793-4dc8-8392-4f8258b930fc";
/>
**Rule 4 - Remove listed fields from older column files**
Continuing the example, after CF-2 [4, 5, 6] is written, field 4 is removed
from CF-1. CF-1 has no fields left, so it is removed from the entry.
<img width="1648" height="658" alt="image"
src="https://github.com/user-attachments/assets/79831e33-7524-4516-8f2a-67b73596226b";
/>
Without this, `address` would be listed in both CF-1 and CF-2, and the
reader would have no way to pick one.
<img width="1648" height="648" alt="image"
src="https://github.com/user-attachments/assets/e59c3275-c810-4ead-9f13-f466f0a801a9";
/>
**Rule 5 - Read each field from the column file that lists it**
With CF-1 [3], the reader takes `name` from CF-1 and everything else from
the base file, including the null state of `customer`. The reader never needs
to look at history. Each field has exactly one source.
**Rule 6 - A null struct makes every field nested in it null**
Start with `address` null in row 0 in the base file. Then `SET street = '1
Elm'` for row 1 writes CF-1 [5]. For row 0, CF-1 may store any value for
`street` (the spec only says writers *should* store null). The reader must
still return null, because `address` is null in that row.
<img width="1648" height="714" alt="image"
src="https://github.com/user-attachments/assets/76ae4788-3015-4896-bd78-8e2e6b9e5dda";
/>
Writers cannot cover every case on their own. If `zip` is added to `address`
with `initial-default = '00000'` while `address` is null in some rows, no file
has `zip`, and Column Projection returns `'00000'`. Only the reader's null
check returns the correct null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
