rdblue commented on a change in pull request #1499:
URL: https://github.com/apache/iceberg/pull/1499#discussion_r496271581
##########
File path: site/docs/spec.md
##########
@@ -138,19 +164,41 @@ Valid type promotions are:
* `float` to `double`
* `decimal(P, S)` to `decimal(P', S)` if `P' > P` -- widen the precision of
decimal types.
-Any struct, including a top-level schema, can evolve through deleting fields,
adding new fields, renaming existing fields, or promoting a primitive using the
valid type promotions. Adding a new field assigns a new ID for that field and
for any nested fields. Renaming an existing field must change the name, but not
the field ID. Deleting a field removes it from the current schema. Field
deletion cannot be rolled back unless the field was nullable or if the current
snapshot has not changed.
+Any struct, including a top-level schema, can evolve through deleting fields,
adding new fields, renaming existing fields, reordering existing fields, or
promoting a primitive using the valid type promotions. Adding a new field
assigns a new ID for that field and for any nested fields. Renaming an existing
field must change the name, but not the field ID. Deleting a field removes it
from the current schema. Field deletion cannot be rolled back unless the field
was nullable or if the current snapshot has not changed.
Grouping a subset of a struct’s fields into a nested struct is **not**
allowed, nor is moving fields from a nested struct into its immediate parent
struct (`struct<a, b, c> ↔ struct<a, struct<b, c>>`). Evolving primitive types
to structs is **not** allowed, nor is evolving a single-field struct to a
primitive (`map<string, int> ↔ map<string, struct<int>>`).
+#### Column Projection
+
+Columns in Iceberg data files are selected by field id. Column names and order
may change between after a data file is written, and projection must be done
using field ids. If a field id is missing from a data file, its value for each
row should be `null`.
+
+For example, a file may be written with schema `1: a int, 2: b string, 3: c
double` and read using projection schema `3: measurement, 2: name, 4: a`. This
must select file columns `c` (renamed to `measurement`), `b` (now called
`name`), and a column of `null` values called `a`; in that order.
+
+
+#### Reserved Field IDs
+
+Iceberg tables must not use field ids greater than 2147483447
(`Integer.MAX_VALUE - 200`). This id range is reserved for metadata columns
that can be used in user data schemas, like the `_file` column that holds the
file path in which a row was stored.
+
+The set of metadata columns is:
+
+| Field id, name | Type | Description |
+|-----------------------------|---------------|-------------|
+| **`2147483646 _file`** | `string` | Path of the file in which a
row is stored |
+| **`2147483645 _pos`** | `long` | Ordinal position of a row in
the source data file |
+| **`2147483546 file_path`** | `string` | Path of a file, used in
position-based delete files |
+| **`2147483545 pos`** | `long` | Ordinal position of a row,
used in position-based delete files |
+| **`2147483544 row`** | `struct<...>` | Deleted row values, used in
position-based delete files |
Review comment:
No, equality delete files put all columns at the top level so there is
no need for a "row" field. This is only used in position-based delete files.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
