stevenzwu commented on code in PR #11041: URL: https://github.com/apache/iceberg/pull/11041#discussion_r4589267908
########## format/view-spec.md: ########## @@ -160,7 +179,95 @@ Each entry in `version-log` is a struct with the following fields: | _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) | | _required_ | `version-id` | ID that `current-version-id` was set to | -## Appendix A: An Example +#### Storage Table Identifier + +The table identifier for the storage table that stores the precomputed results. + +| Requirement | Field name | Description | +|-------------|----------------|-------------| +| _required_ | `namespace` | A list of strings for namespace levels | +| _required_ | `name` | A string specifying the name of the table | +| _optional_ | `catalog` | A string specifying the name of the catalog. When `null` or not set, the catalog in which the materialized view is stored is used | + +### Storage table metadata + +This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views. +The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data. + +| Requirement | Field name | Description | +|-------------|-----------------|-------------| +| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string | + +Snapshots written by a refresh operation must set `refresh-state`. + +#### Refresh state + +Query engines maintain the precomputed data in the storage table by executing the materialized view’s query. During a refresh, the engine records the state of the view’s dependencies in the refresh state. Which dependencies are available depends on how source materialized views are treated, and the writer chooses which of them to record: all, a subset, or none. See [Appendix B](#appendix-b-example-freshness-criteria-and-recorded-dependencies) for examples. Each recorded dependency appears in `source-states` as a table entry or a view entry. A source materialized view may be recorded as a view entry for its definition and, optionally, a table entry for its storage table. + +The refresh state has the following fields: + +| Requirement | Field name | Description | +|-------------|------------------------------|-------------| +| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started | +| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed | +| _required_ | `source-states` | A list of [source state](#source-state) records capturing the dependencies the writer chose to record; may be empty | + +When writing the refresh state, writers may leave source-states empty (e.g., when sources are non-Iceberg tables or a mechanism outside this spec determines whether the storage table reflects the result of the view query). + +##### Source state + +Source state records capture the state of objects referenced by a materialized view. Each record has a `type` field that determines its form: + +| Type | Description | +|---------|-------------| +| `table` | An Iceberg table — either a source table in the dependency graph, or the storage table of a source materialized view | +| `view` | An Iceberg view in the dependency graph | + +A source materialized view may be recorded as a `view` entry referencing its view metadata and one or more `table` entries referencing its storage table or other source tables. These source table entries might be determined by recursively expanding its own dependencies. Review Comment: > A source materialized view may be recorded as a `view` entry referencing its view metadata and one or more `table` entries referencing its storage table or other source tables. We seem to combine two things here: how a source MV itself is recorded, and the recursive expansion of its dependencies. I know `other source tables` is to allow recursive expansion, but the current wording would exclude nested view entities. "One or more `table` entries" also implies the storage table entry is required, while the Refresh state section says it is optional, and the Appendix B example records `event_agg_mv` as a view entry only. The Refresh state section already describes how a source MV is recorded, and the type table says "in the dependency graph", which includes nested dependencies. So this sentence could be removed. If we want to call out expansion explicitly, maybe: > A source materialized view may be recorded as a `view` entry for its definition and, optionally, a `table` entry for its storage table. If a writer expands a source materialized view's own dependencies, each expanded dependency is recorded as a `table` or `view` entry like any other dependency. ########## format/view-spec.md: ########## @@ -160,7 +179,95 @@ Each entry in `version-log` is a struct with the following fields: | _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) | | _required_ | `version-id` | ID that `current-version-id` was set to | -## Appendix A: An Example +#### Storage Table Identifier + +The table identifier for the storage table that stores the precomputed results. + +| Requirement | Field name | Description | +|-------------|----------------|-------------| +| _required_ | `namespace` | A list of strings for namespace levels | +| _required_ | `name` | A string specifying the name of the table | +| _optional_ | `catalog` | A string specifying the name of the catalog. When `null` or not set, the catalog in which the materialized view is stored is used | + +### Storage table metadata + +This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views. +The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data. + +| Requirement | Field name | Description | +|-------------|-----------------|-------------| +| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string | + +Snapshots written by a refresh operation must set `refresh-state`. + +#### Refresh state + +Query engines maintain the precomputed data in the storage table by executing the materialized view’s query. During a refresh, the engine records the state of the view’s dependencies in the refresh state. Which dependencies are available depends on how source materialized views are treated, and the writer chooses which of them to record: all, a subset, or none. See [Appendix B](#appendix-b-example-freshness-criteria-and-recorded-dependencies) for examples. Each recorded dependency appears in `source-states` as a table entry or a view entry. A source materialized view may be recorded as a view entry for its definition and, optionally, a table entry for its storage table. + +The refresh state has the following fields: + +| Requirement | Field name | Description | +|-------------|------------------------------|-------------| +| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started | +| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed | +| _required_ | `source-states` | A list of [source state](#source-state) records capturing the dependencies the writer chose to record; may be empty | + +When writing the refresh state, writers may leave source-states empty (e.g., when sources are non-Iceberg tables or a mechanism outside this spec determines whether the storage table reflects the result of the view query). + +##### Source state + +Source state records capture the state of objects referenced by a materialized view. Each record has a `type` field that determines its form: + +| Type | Description | +|---------|-------------| +| `table` | An Iceberg table — either a source table in the dependency graph, or the storage table of a source materialized view | +| `view` | An Iceberg view in the dependency graph | + +A source materialized view may be recorded as a `view` entry referencing its view metadata and one or more `table` entries referencing its storage table or other source tables. These source table entries might be determined by recursively expanding its own dependencies. + +##### Source table state + +A source table record captures the state of a source table (including a source materialized view's storage table) at the time of the last refresh operation. + +| Requirement | Field name | Description | +|-------------|---------------|-------------| +| _required_ | `type` | A string that must be set to `table` | +| _required_ | `name` | A string specifying the name of the source table | +| _required_ | `namespace` | A list of strings for namespace levels | +| _required_ | `catalog` | A string specifying the name of the catalog. | +| _required_ | `uuid` | The uuid of the source table | +| _optional_ | `snapshot-id` | The snapshot-id of the source table that was read during the refresh operation. Omitted when the source table had no snapshots at refresh time (e.g. a newly created, empty table) | +| _optional_ | `ref` | Branch or tag name of the source table being referenced in the view query | + +When `ref` is set, the source table is tracked against that branch or tag. When `ref` is not set, the source table is pinned to the recorded `snapshot-id` (e.g. `FOR VERSION AS OF <snapshot-id>`). Writers that read a branch or tag should record its name in `ref`. + +##### Source view state + +A source view record captures the state of a source view at the time of the last refresh operation. + +| Requirement | Field name | Description | +|-------------|--------------|-------------| +| _required_ | `type` | A string that must be set to `view` | +| _required_ | `name` | A string specifying the name of the source view | +| _required_ | `namespace` | A list of strings for namespace levels | +| _required_ | `catalog` | A string specifying the name of the catalog. | +| _required_ | `uuid` | The uuid of the source view | +| _required_ | `version-id` | The version-id of the source view that was read during the refresh operation | + +#### Storage table creation and configuration + +When processing a `CREATE MATERIALIZED VIEW` statement, query engines should: + +1. Create the storage table as a regular Iceberg table with any specified configurations (partitioning, sort order, compression, etc.). Review Comment: nit: drop the word `any`? ########## format/view-spec.md: ########## @@ -42,12 +42,28 @@ An atomic swap of one view metadata file for another provides the basis for maki Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location. +### Materialized Views + +Materialized views are a type of view with precomputed results from the view query stored as a table. +When queried, engines may return the precomputed data for the materialized views, shifting the cost of query execution to the precomputation step. + +Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, the "storage-table", which stores the precomputed data. +Materialized View metadata is a superset of View metadata with an additional pointer to the storage table. The storage table is an Iceberg table with additional materialized view refresh state metadata. +This addition is backward compatible: a reader that is unaware of materialized views sees the unknown `storage-table` field, ignores it, and evaluates the view query directly. +Refresh metadata contains information about the "source tables", "source views", and/or "source materialized views", which are the tables/views/materialized views used in the computation of the query results of the materialized view. Review Comment: nit: `and/or` -> `and` ########## format/view-spec.md: ########## @@ -160,7 +179,95 @@ Each entry in `version-log` is a struct with the following fields: | _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) | | _required_ | `version-id` | ID that `current-version-id` was set to | -## Appendix A: An Example +#### Storage Table Identifier + +The table identifier for the storage table that stores the precomputed results. + +| Requirement | Field name | Description | +|-------------|----------------|-------------| +| _required_ | `namespace` | A list of strings for namespace levels | +| _required_ | `name` | A string specifying the name of the table | +| _optional_ | `catalog` | A string specifying the name of the catalog. When `null` or not set, the catalog in which the materialized view is stored is used | + +### Storage table metadata + +This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views. +The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data. + +| Requirement | Field name | Description | +|-------------|-----------------|-------------| +| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string | + +Snapshots written by a refresh operation must set `refresh-state`. Review Comment: After storage table maintenance (e.g., a compaction that commits a `replace` snapshot), the current snapshot has no `refresh-state`. Should readers use the most recent snapshot with `refresh-state` in the ancestry of the storage table's `main` branch? And does a later snapshot that changes data (anything other than `replace`) invalidate that state? What do you think of sth below? is the 2nd sentence over specified? > The refresh state of a storage table is the `refresh-state` of the most recent snapshot in the history of its `main` branch that has one. If a later snapshot in that history changes the table's data (any operation other than `replace`), the storage table no longer reflects that refresh state. ########## format/view-spec.md: ########## @@ -160,7 +179,95 @@ Each entry in `version-log` is a struct with the following fields: | _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) | | _required_ | `version-id` | ID that `current-version-id` was set to | -## Appendix A: An Example +#### Storage Table Identifier + +The table identifier for the storage table that stores the precomputed results. + +| Requirement | Field name | Description | +|-------------|----------------|-------------| +| _required_ | `namespace` | A list of strings for namespace levels | +| _required_ | `name` | A string specifying the name of the table | +| _optional_ | `catalog` | A string specifying the name of the catalog. When `null` or not set, the catalog in which the materialized view is stored is used | + +### Storage table metadata + +This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views. +The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data. + +| Requirement | Field name | Description | +|-------------|-----------------|-------------| +| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string | + +Snapshots written by a refresh operation must set `refresh-state`. + +#### Refresh state + +Query engines maintain the precomputed data in the storage table by executing the materialized view’s query. During a refresh, the engine records the state of the view’s dependencies in the refresh state. Which dependencies are available depends on how source materialized views are treated, and the writer chooses which of them to record: all, a subset, or none. See [Appendix B](#appendix-b-example-freshness-criteria-and-recorded-dependencies) for examples. Each recorded dependency appears in `source-states` as a table entry or a view entry. A source materialized view may be recorded as a view entry for its definition and, optionally, a table entry for its storage table. + +The refresh state has the following fields: + +| Requirement | Field name | Description | +|-------------|------------------------------|-------------| +| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started | Review Comment: nit: add the unit, matching `timestamp-ms` elsewhere in the spec. ```suggestion | _required_ | `refresh-start-timestamp-ms` | Timestamp when the refresh operation was started (ms from epoch) | ``` ########## format/view-spec.md: ########## @@ -160,7 +179,95 @@ Each entry in `version-log` is a struct with the following fields: | _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) | | _required_ | `version-id` | ID that `current-version-id` was set to | -## Appendix A: An Example +#### Storage Table Identifier + +The table identifier for the storage table that stores the precomputed results. + +| Requirement | Field name | Description | +|-------------|----------------|-------------| +| _required_ | `namespace` | A list of strings for namespace levels | +| _required_ | `name` | A string specifying the name of the table | +| _optional_ | `catalog` | A string specifying the name of the catalog. When `null` or not set, the catalog in which the materialized view is stored is used | + +### Storage table metadata + +This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views. +The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data. + +| Requirement | Field name | Description | +|-------------|-----------------|-------------| +| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string | + +Snapshots written by a refresh operation must set `refresh-state`. + +#### Refresh state + +Query engines maintain the precomputed data in the storage table by executing the materialized view’s query. During a refresh, the engine records the state of the view’s dependencies in the refresh state. Which dependencies are available depends on how source materialized views are treated, and the writer chooses which of them to record: all, a subset, or none. See [Appendix B](#appendix-b-example-freshness-criteria-and-recorded-dependencies) for examples. Each recorded dependency appears in `source-states` as a table entry or a view entry. A source materialized view may be recorded as a view entry for its definition and, optionally, a table entry for its storage table. + +The refresh state has the following fields: + +| Requirement | Field name | Description | +|-------------|------------------------------|-------------| +| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started | +| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed | +| _required_ | `source-states` | A list of [source state](#source-state) records capturing the dependencies the writer chose to record; may be empty | + +When writing the refresh state, writers may leave source-states empty (e.g., when sources are non-Iceberg tables or a mechanism outside this spec determines whether the storage table reflects the result of the view query). Review Comment: An empty or partial `source-states` looks the same whether the writer's criteria didn't need sources or the writer couldn't record them. Freshness is intentionally left to engines. Maybe we can update this paragraphs so that it states what readers can't infer from the refresh state. ```suggestion `source-states` may be empty or cover only some of the view query's dependencies (e.g., when sources are non-Iceberg tables or a mechanism outside this spec determines whether the storage table reflects the result of the view query). A dependency that is not recorded carries no information about its state. ``` ########## format/view-spec.md: ########## @@ -160,7 +179,95 @@ Each entry in `version-log` is a struct with the following fields: | _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) | | _required_ | `version-id` | ID that `current-version-id` was set to | -## Appendix A: An Example +#### Storage Table Identifier + +The table identifier for the storage table that stores the precomputed results. + +| Requirement | Field name | Description | +|-------------|----------------|-------------| +| _required_ | `namespace` | A list of strings for namespace levels | +| _required_ | `name` | A string specifying the name of the table | +| _optional_ | `catalog` | A string specifying the name of the catalog. When `null` or not set, the catalog in which the materialized view is stored is used | + +### Storage table metadata + +This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views. +The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data. + +| Requirement | Field name | Description | +|-------------|-----------------|-------------| +| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string | + +Snapshots written by a refresh operation must set `refresh-state`. + +#### Refresh state + +Query engines maintain the precomputed data in the storage table by executing the materialized view’s query. During a refresh, the engine records the state of the view’s dependencies in the refresh state. Which dependencies are available depends on how source materialized views are treated, and the writer chooses which of them to record: all, a subset, or none. See [Appendix B](#appendix-b-example-freshness-criteria-and-recorded-dependencies) for examples. Each recorded dependency appears in `source-states` as a table entry or a view entry. A source materialized view may be recorded as a view entry for its definition and, optionally, a table entry for its storage table. + +The refresh state has the following fields: + +| Requirement | Field name | Description | +|-------------|------------------------------|-------------| +| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started | +| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed | +| _required_ | `source-states` | A list of [source state](#source-state) records capturing the dependencies the writer chose to record; may be empty | + +When writing the refresh state, writers may leave source-states empty (e.g., when sources are non-Iceberg tables or a mechanism outside this spec determines whether the storage table reflects the result of the view query). + +##### Source state + +Source state records capture the state of objects referenced by a materialized view. Each record has a `type` field that determines its form: + +| Type | Description | +|---------|-------------| +| `table` | An Iceberg table — either a source table in the dependency graph, or the storage table of a source materialized view | +| `view` | An Iceberg view in the dependency graph | + +A source materialized view may be recorded as a `view` entry referencing its view metadata and one or more `table` entries referencing its storage table or other source tables. These source table entries might be determined by recursively expanding its own dependencies. + +##### Source table state + +A source table record captures the state of a source table (including a source materialized view's storage table) at the time of the last refresh operation. + +| Requirement | Field name | Description | +|-------------|---------------|-------------| +| _required_ | `type` | A string that must be set to `table` | +| _required_ | `name` | A string specifying the name of the source table | +| _required_ | `namespace` | A list of strings for namespace levels | +| _required_ | `catalog` | A string specifying the name of the catalog. | +| _required_ | `uuid` | The uuid of the source table | +| _optional_ | `snapshot-id` | The snapshot-id of the source table that was read during the refresh operation. Omitted when the source table had no snapshots at refresh time (e.g. a newly created, empty table) | +| _optional_ | `ref` | Branch or tag name of the source table being referenced in the view query | + +When `ref` is set, the source table is tracked against that branch or tag. When `ref` is not set, the source table is pinned to the recorded `snapshot-id` (e.g. `FOR VERSION AS OF <snapshot-id>`). Writers that read a branch or tag should record its name in `ref`. Review Comment: If the view query reads the source table without a branch or time-travel clause, it reads `main` implicitly. Should the writer record `ref: main` in that case? Otherwise the entry is treated as pinned to `snapshot-id`, and later commits to the source table would never show up as changes. Since omitting `ref` changes the meaning this much, I am wondering if this should be a `must`. Also, if `ref` is not set and `snapshot-id` is omitted (empty source table), what is the entry pinned to? Maybe sth like: ```suggestion When `ref` is set, the source table is tracked against that branch or tag. When `ref` is not set, the source table is pinned to the recorded `snapshot-id` (e.g. `FOR VERSION AS OF <snapshot-id>`). Writers must record the branch or tag name in `ref` when the source table is read by branch or tag, including the implicit `main` branch when the query does not specify one. ``` ########## format/view-spec.md: ########## @@ -82,9 +97,12 @@ Each version in `versions` is a struct with the following fields: | _required_ | `representations` | A list of [representations](#representations) for the view definition | | _optional_ | `default-catalog` | Catalog name to use when a reference in the SELECT does not contain a catalog | | _required_ | `default-namespace` | Namespace to use when a reference in the SELECT is a single identifier | +| _optional_ | `storage-table` | A [storage table identifier](#storage-table-identifier) of the storage table | Review Comment: Just to close the loop. @bennychow laid out the case for per-version storage tables in https://github.com/apache/iceberg/pull/11041#discussion_r2809831576: a view can gain, drop, or swap its storage table across versions, and a full refresh can write a new storage table and commit it with a new view version instead of truncating the old one. The Feb 19 sync noted that no engine we know of supports DDL to convert a logical view to an MV. So the full-refresh case seems to be the main argument for keeping `storage-table` in `Version`. I am fine with keeping the reference inside the version struct. ########## format/view-spec.md: ########## @@ -160,7 +179,95 @@ Each entry in `version-log` is a struct with the following fields: | _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) | | _required_ | `version-id` | ID that `current-version-id` was set to | -## Appendix A: An Example +#### Storage Table Identifier + +The table identifier for the storage table that stores the precomputed results. + +| Requirement | Field name | Description | +|-------------|----------------|-------------| +| _required_ | `namespace` | A list of strings for namespace levels | +| _required_ | `name` | A string specifying the name of the table | +| _optional_ | `catalog` | A string specifying the name of the catalog. When `null` or not set, the catalog in which the materialized view is stored is used | + +### Storage table metadata + +This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views. +The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data. + +| Requirement | Field name | Description | +|-------------|-----------------|-------------| +| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string | + +Snapshots written by a refresh operation must set `refresh-state`. + +#### Refresh state + +Query engines maintain the precomputed data in the storage table by executing the materialized view’s query. During a refresh, the engine records the state of the view’s dependencies in the refresh state. Which dependencies are available depends on how source materialized views are treated, and the writer chooses which of them to record: all, a subset, or none. See [Appendix B](#appendix-b-example-freshness-criteria-and-recorded-dependencies) for examples. Each recorded dependency appears in `source-states` as a table entry or a view entry. A source materialized view may be recorded as a view entry for its definition and, optionally, a table entry for its storage table. + +The refresh state has the following fields: + +| Requirement | Field name | Description | +|-------------|------------------------------|-------------| +| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started | +| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed | +| _required_ | `source-states` | A list of [source state](#source-state) records capturing the dependencies the writer chose to record; may be empty | + +When writing the refresh state, writers may leave source-states empty (e.g., when sources are non-Iceberg tables or a mechanism outside this spec determines whether the storage table reflects the result of the view query). + +##### Source state + +Source state records capture the state of objects referenced by a materialized view. Each record has a `type` field that determines its form: + +| Type | Description | +|---------|-------------| +| `table` | An Iceberg table — either a source table in the dependency graph, or the storage table of a source materialized view | +| `view` | An Iceberg view in the dependency graph | + +A source materialized view may be recorded as a `view` entry referencing its view metadata and one or more `table` entries referencing its storage table or other source tables. These source table entries might be determined by recursively expanding its own dependencies. + +##### Source table state + +A source table record captures the state of a source table (including a source materialized view's storage table) at the time of the last refresh operation. Review Comment: nit: "the last refresh operation" reads as the most recent refresh, but a reader may be looking at an older storage table snapshot. Same for the source view state description. ```suggestion A source table record captures the state of a source table (including a source materialized view's storage table) at the time of the refresh operation that wrote this refresh state. ``` ########## format/view-spec.md: ########## @@ -160,7 +179,95 @@ Each entry in `version-log` is a struct with the following fields: | _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) | | _required_ | `version-id` | ID that `current-version-id` was set to | -## Appendix A: An Example +#### Storage Table Identifier + +The table identifier for the storage table that stores the precomputed results. + +| Requirement | Field name | Description | +|-------------|----------------|-------------| +| _required_ | `namespace` | A list of strings for namespace levels | +| _required_ | `name` | A string specifying the name of the table | +| _optional_ | `catalog` | A string specifying the name of the catalog. When `null` or not set, the catalog in which the materialized view is stored is used | + +### Storage table metadata + +This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views. +The `refresh-state` property is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of a storage table snapshot to provide information about the state of the precomputed data. + +| Requirement | Field name | Description | +|-------------|-----------------|-------------| +| _optional_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string | + +Snapshots written by a refresh operation must set `refresh-state`. + +#### Refresh state + +Query engines maintain the precomputed data in the storage table by executing the materialized view’s query. During a refresh, the engine records the state of the view’s dependencies in the refresh state. Which dependencies are available depends on how source materialized views are treated, and the writer chooses which of them to record: all, a subset, or none. See [Appendix B](#appendix-b-example-freshness-criteria-and-recorded-dependencies) for examples. Each recorded dependency appears in `source-states` as a table entry or a view entry. A source materialized view may be recorded as a view entry for its definition and, optionally, a table entry for its storage table. Review Comment: nit: "Which dependencies are available depends on how source materialized views are treated" is hard to act on: it's unclear what "available" means here. The rest of the sentence already says the writer chooses what to record. Could we drop that clause? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
