stevenzwu commented on code in PR #11041: URL: https://github.com/apache/iceberg/pull/11041#discussion_r4635014360
########## format/view-spec.md: ########## @@ -42,12 +42,28 @@ An atomic swap of one view metadata file for another provides the basis for maki Writers create view metadata files optimistically, assuming that the current metadata location will not be changed before the writer's commit. Once a writer has created an update, it commits by swapping the view's metadata file pointer from the base location to the new location. +### Materialized Views + +Materialized views are a type of view with precomputed results from the view query stored as a table. +When queried, engines may return the precomputed data for the materialized views, shifting the cost of query execution to the precomputation step. + +Iceberg materialized views are implemented as a combination of an Iceberg view and an underlying Iceberg table, the "storage-table", which stores the precomputed data. +Materialized View metadata is a superset of View metadata with an additional pointer to the storage table. The storage table is an Iceberg table with additional materialized view refresh state metadata. +This addition is backward compatible: a reader that is unaware of materialized views sees the unknown `storage-table` field, ignores it, and evaluates the view query directly. Review Comment: You're right that an MV-unaware writer on a non-REST catalog would silently drop `storage-table` when it rewrites the view metadata. But this isn't a new situation for Iceberg. The table spec only bumps the format version when older readers would read newer features incorrectly, and we have added semantically meaningful optional fields within existing table versions before. `refs` was added to the v2 spec in PR #3425, after v2 was adopted, without a bump. Older writers can drop all branches and tags on commit. Bumping the view format version would also remove the graceful fallback for old readers: they would fail on an MV instead of evaluating the view query. But this is probably a rather minor point. So I'd lean toward keeping `format-version` 1 and adding a writer requirement, following the table spec's rule for [unknown partition transforms](https://iceberg.apache.org/spec/#partitioning): > Writers are not allowed to commit data using a partition spec that contains a field with an unknown transform. Something like this, which also scopes the compatibility claim to readers since already-released writers can still drop the field: ```suggestion This addition is backward compatible for readers: a reader that is unaware of materialized views sees the unknown `storage-table` field, ignores it, and evaluates the view query directly. Writers must not drop `storage-table` from existing view versions. Implementations that do not support materialized views must not commit changes to a view that has `storage-table` set. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
