osscm commented on code in PR #11041: URL: https://github.com/apache/iceberg/pull/11041#discussion_r4126766892
########## format/view-spec.md: ########## @@ -160,6 +178,108 @@ Each entry in `version-log` is a struct with the following fields: | _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) | | _required_ | `version-id` | ID that `current-version-id` was set to | +#### Storage Table Identifier + +The table identifier for the storage table that stores the precomputed results. + +| Requirement | Field name | Description | +|-------------|----------------|-------------| +| _required_ | `namespace` | A list of strings for namespace levels | +| _required_ | `name` | A string specifying the name of the table | + +### Storage table metadata + +This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views. +The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table. + +| Requirement | Field name | Description | +|-------------|-----------------|-------------| +| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string | + +#### Freshness + +A materialized view's precomputed data becomes stale as the tables and views referenced in its query definition change over time. Freshness determines whether the precomputed data accurately represents the logical query definition at the current state of its dependencies. + +Different systems define freshness differently, based on how much of the dependency graph must be current. Some require the entire query tree to be fully up to date, while others only require direct children or allow bounded staleness at leaf nodes. As a result, "fresh" can mean strict end-to-end consistency, acceptable lag, or policy/version compliance. + +A materialized view is considered fresh when its precomputed data meets the freshness criteria defined by the consumer's evaluation policy. When these criteria are not met, the materialized view is considered stale. + +#### Refresh state + +The refresh state record captures the unique dependencies in the materialized view's dependency graph. These dependencies include source Iceberg tables, views, and nested materialized views that allow a consumer to determine the freshness of the materialized view. + +**Producer responsibilities:** +- The producer of the storage table must provide a sufficient list of source states so that consumers can determine freshness according to the producer's interpretation. +- The source states list may be empty if the source state cannot be determined for all objects (for example, for non-Iceberg tables). + +**Consumer evaluation:** +- The consumer must at least perform a coarse-grained evaluation based on `refresh-start-timestamp-ms` and `max-staleness-ms`. A materialized view is fresh if `refresh-start-timestamp-ms` is within the window `[now - max-staleness-ms, now]`. +- The consumer may additionally compare the `source-states` list against the states loaded from the catalog. If this evaluation determines the materialized view is fresh, it overrides the coarse-grained evaluation result. +- The consumer may parse the view definition to implement a more sophisticated policy. +- When a materialized view is considered stale, the consumer can fail, refresh inline, or treat the materialized view as a logical view. The consumer must not consume from the storage table when the materialized view is stale. + +The refresh state has the following fields: + +| Requirement | Field name | Description | +|-------------|----------------|-------------| +| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed | +| _required_ | `source-states` | A list of [source states](#source-state) records | +| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started | + +#### Source state + +Materialized views can reference source objects of different types, such as Iceberg tables, view, and materialized views. Source state records have a common field `type` that determines the form, which can be one of the following: + +* `table`: An Iceberg table +* `view`: An Iceberg view +* `materialized-view`: An Iceberg materialized view + +The metadata fields for each type are defined below: + +#### Source table state + +A source table record captures the state of a source table (including source MV's storage table) at the time of the last refresh operation. + +| Requirement | Field name | Description | +|-------------|----------------|-------------| +| _required_ | `type` | A string that must be set to `table` | Review Comment: Following up on this since it doesn't look like it made it into the spec text yet (I don't see anything about snapshot expiration/retention under `source-table-state`) — apologies if this was actually settled elsewhere and I missed it. The scenario I keep getting stuck on: once a source table's retention/expiration policy removes the snapshot that was recorded in `source-table-state.snapshot-id`, the recorded reference just points at nothing anymore. At that point there's no way to compare "recorded snapshot" vs. "current snapshot" to decide whether the source actually changed — the comparison itself becomes undefined, not just "stale." For what it's worth, this isn't just a theoretical corner case for engines doing snapshot-based incremental tracking — Iceberg's own `IncrementalAppendScan` throws when the snapshot range crosses a non-append operation or an expired/GC'd ancestor (the "is not a parent ancestor" case), and at least one engine's answer to hitting that is to treat the incremental basis as unrecoverable rather than merely stale, and require a full rebuild rather than attempt any fallback comparison. Is the intent here that a missing/expired snapshot reference should always be treated as stale (safe, but can trigger unnecessary full refreshes on any table with a live expiration policy), or is there some other fallback in mind — e.g. keying off something more durable than the raw `snapshot-id` (the sequence-number idea mentioned earlier in this thread)? Trying to understand whether "reference no longer resolves" is meant to be a distinct case from "reference resolves but doesn't match," or if I'm overthinking this. ########## format/view-spec.md: ########## @@ -160,6 +177,71 @@ Each entry in `version-log` is a struct with the following fields: | _required_ | `timestamp-ms` | Timestamp when the view's `current-version-id` was updated (ms from epoch) | | _required_ | `version-id` | ID that `current-version-id` was set to | +#### Storage Table Identifier + +The table identifier for the storage table that stores the precomputed results. + +| Requirement | Field name | Description | +|-------------|----------------|-------------| +| _required_ | `namespace` | A list of strings for namespace levels | +| _required_ | `name` | A string specifying the name of the table | + +### Storage table metadata + +This section describes additional metadata for the storage table that supplements the regular table metadata and is required for materialized views. +The property "refresh-state" is set on the [snapshot summary](https://iceberg.apache.org/spec/#snapshots) property of every storage table snapshot to determine the freshness of the precomputed data of the storage table. + +| Requirement | Field name | Description | +|-------------|-----------------|-------------| +| _required_ | `refresh-state` | A [refresh state](#refresh-state) record stored as a JSON-encoded string | + +#### Refresh state + +The refresh state record captures the state of source tables, views, and materialized views at refresh time. + +* Source view states are stored in `source-view-states`. It includes indirect references — views nested within other views (excluding MVs). +* Source table states are stored in `source-table-states`. It includes indirect references - tables nested within other views (excluding MVs). + +For directly referenced source materialized views, both the source view and its storage table are included in the refresh state. Indirect references (views or tables) from source materialized views are excluded in the refresh-state. During read time, a query engine recursively expands the query tree to determine freshness if it chooses to enforce recursive evaluation semantic. + +The refresh state has the following fields: + +| Requirement | Field name | Description | +|-------------|----------------|-------------| +| _required_ | `view-version-id` | The `version-id` of the materialized view when the refresh operation was performed | +| _required_ | `source-table-states` | A list of [source table](#source-table) records for tables directly or indirectly referenced through common views, plus storage tables of directly referenced source materialized views | +| _required_ | `source-view-states` | A list of [source view](#source-view) records for all views (including materialized views) that are directly referenced, plus common views indirectly referenced through other common views | +| _required_ | `refresh-start-timestamp-ms` | A timestamp of when the refresh operation was started | + +#### Source table + +A source table record captures the state of a source table (including source MV's storage table) at the time of the last refresh operation. + +| Requirement | Field name | Description | +|-------------|----------------|-------------| +| _required_ | `uuid` | The uuid of the source table | +| _required_ | `snapshot-id` | Snapshot-id of when the last refresh operation was performed | +| _optional_ | `ref` | Branch name of the source table being referenced in the view query | + +When `ref` is `null` or not set, it defaults to "main". + +#### Source view + +A source view record captures the state of a source view at the time of the last refresh operation. + +| Requirement | Field name | Description | +|-------------|----------------|-------------| +| _required_ | `uuid` | The uuid of the source view | +| _required_ | `version-id` | Version-id of when the last refresh operation was performed | + +#### Status Interpretation + +During read time, a materialized view (storage table) can be interpreted as "fresh", "stale" or "invalid", depending on the following situations: + +* **invalid** -- The current `version_id` of the materialized view does not match the `view-version-id` recorded in its refresh state. A read operation cannot proceed using the materialized view's data. Review Comment: Related to this thread's point that fresh/stale/invalid might be too generic — one thing that seems to be missing entirely: the vocabulary implicitly assumes any non-fresh materialized view can always be recovered by just running a refresh. At least one engine's MV implementation has a state that doesn't fit that assumption — a materialized view can become permanently unrefreshable, not just stale, when its own definition no longer matches its dependencies (e.g. a base table's column was dropped, renamed, or had an incompatible type change, or a base table was dropped entirely). In that state, attempting a refresh fails outright, and the view has to be explicitly redefined/reactivated by a person before any refresh can be attempted again. Is that kind of state in scope for this spec, or intentionally left as an engine-internal concern with no interchange representation? If a consumer sees a materialized view that's "invalid" per the spec's current definition, is there a way to distinguish "just needs a refresh" from "structurally broken, refreshing won't help"? Or is that considered out of scope for v1 since it's orthogonal to the storage-table/refresh-state model? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
