JanKaul commented on code in PR #11041: URL: https://github.com/apache/iceberg/pull/11041#discussion_r4613261161
########## format/view-spec.md: ########## @@ -160,7 +179,95 @@ Each entry in `version-log` is a struct with the following fields: | _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) | | _required_ | `version-id` | ID that `current-version-id` was set to | -## Appendix A: An Example +#### Storage Table Identifier + +The table identifier for the storage table that stores the precomputed results. + +| Requirement | Field name | Description | +|-------------|----------------|-------------| +| _required_ | `namespace` | A list of strings for namespace levels | +| _required_ | `name` | A string specifying the name of the table | +| _optional_ | `catalog` | A string specifying the name of the catalog. When `null` or not set, the catalog in which the materialized view is stored is used | + +### Storage table metadata + +This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views. +The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data. + +| Requirement | Field name | Description | +|-------------|-----------------|-------------| +| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string | + +Snapshots written by a refresh operation must set `refresh-state`. + +#### Refresh state + +Query engines maintain the precomputed data in the storage table by executing the materialized view’s query. During a refresh, the engine records the state of the view’s dependencies in the refresh state. Which dependencies are available depends on how source materialized views are treated, and the writer chooses which of them to record: all, a subset, or none. See [Appendix B](#appendix-b-example-freshness-criteria-and-recorded-dependencies) for examples. Each recorded dependency appears in `source-states` as a table entry or a view entry. A source materialized view may be recorded as a view entry for its definition and, optionally, a table entry for its storage table. + +The refresh state has the following fields: + +| Requirement | Field name | Description | +|-------------|------------------------------|-------------| +| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started | +| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed | +| _required_ | `source-states` | A list of [source state](#source-state) records capturing the dependencies the writer chose to record; may be empty | + +When writing the refresh state, writers may leave source-states empty (e.g., when sources are non-Iceberg tables or a mechanism outside this spec determines whether the storage table reflects the result of the view query). + +##### Source state + +Source state records capture the state of objects referenced by a materialized view. Each record has a `type` field that determines its form: + +| Type | Description | +|---------|-------------| +| `table` | An Iceberg table — either a source table in the dependency graph, or the storage table of a source materialized view | +| `view` | An Iceberg view in the dependency graph | + +A source materialized view may be recorded as a `view` entry referencing its view metadata and one or more `table` entries referencing its storage table or other source tables. These source table entries might be determined by recursively expanding its own dependencies. + +##### Source table state + +A source table record captures the state of a source table (including a source materialized view's storage table) at the time of the last refresh operation. + +| Requirement | Field name | Description | +|-------------|---------------|-------------| +| _required_ | `type` | A string that must be set to `table` | +| _required_ | `name` | A string specifying the name of the source table | +| _required_ | `namespace` | A list of strings for namespace levels | +| _required_ | `catalog` | A string specifying the name of the catalog. | +| _required_ | `uuid` | The uuid of the source table | +| _optional_ | `snapshot-id` | The snapshot-id of the source table that was read during the refresh operation. Omitted when the source table had no snapshots at refresh time (e.g. a newly created, empty table) | +| _optional_ | `ref` | Branch or tag name of the source table being referenced in the view query | + +When `ref` is set, the source table is tracked against that branch or tag. When `ref` is not set, the source table is pinned to the recorded `snapshot-id` (e.g. `FOR VERSION AS OF <snapshot-id>`). Writers that read a branch or tag should record its name in `ref`. Review Comment: Applied, thanks. With ref now required whenever a branch or tag is read (including implicit main), ref is only unset for time travel to a specific snapshot, which implies the snapshot exists, so snapshot-id is always present in that case. An empty source table is always read through a branch and recorded with ref and no snapshot-id. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
