Hi everyone,
One of the remaining open questions for v4 metadata is how we will assign
table field IDs for values that are not written into the table. I want to
propose a solution that I think is going to be flexible, while not
introducing a lot of churn in the table or REST specs.
Columnar field stats are written into metadata using a simple function from
table field ID to metadata field ID. We want to reuse what we already have
working for table fields and keep the spec simple. That means we need a way
to assign a table field ID to a non-materialized column so that we can
track its stats. There are a few cases we’ve identified:
- Partition field output for non-monotonic functions, like bucket(1024,
id)
- Clustering expressions, like to_lower_case(last_name)
- Collation sequence lower and upper bounds
We also discussed a new case this morning in the index sync: we need a
field ID for a derived value used to organize an index, like hilbert(col_a,
col_b), because we intend to use table field IDs in index schemas.
Initially, I suggested that we keep a table of expressions and assign each
one a field ID. But as we started thinking about the use cases where we
need expressions it became clear that denormalizing *all* expressions was
adding a lot of complexity for little benefit. For example, CHECK
constraints won’t have reusable expressions and it makes little sense to
create them in two parts (expression and constraint). Similarly, it is
awkward to model a collation sequence as an expression, and we don’t need
to rebuild partition specs just to assign field IDs. However, we also don’t
want to just embed table field IDs in every one of these structures.
My proposal is to directly model what we want: one table of fields that are
not materialized in the table, but are assigned IDs for stats or other
purposes. This would take a few forms:
- Partition output value: {"field-id": 102, "type": "partition-value",
"partition-field-id": 1000}
- Collation sequence: {"field-id": 103, "type": "collation-bounds",
"collation-seq-id": 1}
- Value expression results: {"field-id": 104, "type": "expr-value",
"data-type": "long", "expr": <hilbert(col_a, col_b) expr>}
This representation leaves existing structures alone and is a single place
outside of schema to allocate table field IDs. This can be expanded with
new types later when we want to add new structures, like a cluster-by spec.
I think this is a fairly clean way to move forward and solve two challenges
that we’re currently hitting. We'll discuss this in the next v4 metadata
sync, but in the meantime please reply with feedback if you have an opinion.
Thanks,
Ryan
