stevenzwu commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r4635014360


##########
format/view-spec.md:
##########
@@ -42,12 +42,28 @@ An atomic swap of one view metadata file for another 
provides the basis for maki
 
 Writers create view metadata files optimistically, assuming that the current 
metadata location will not be changed before the writer's commit. Once a writer 
has created an update, it commits by swapping the view's metadata file pointer 
from the base location to the new location.
 
+### Materialized Views
+
+Materialized views are a type of view with precomputed results from the view 
query stored as a table.
+When queried, engines may return the precomputed data for the materialized 
views, shifting the cost of query execution to the precomputation step.
+
+Iceberg materialized views are implemented as a combination of an Iceberg view 
and an underlying Iceberg table, the "storage-table", which stores the 
precomputed data.
+Materialized View metadata is a superset of View metadata with an additional 
pointer to the storage table. The storage table is an Iceberg table with 
additional materialized view refresh state metadata.
+This addition is backward compatible: a reader that is unaware of materialized 
views sees the unknown `storage-table` field, ignores it, and evaluates the 
view query directly.

Review Comment:
   You're right that an MV-unaware writer on a non-REST catalog would silently 
drop `storage-table` when it rewrites the view metadata.
   
   But this isn't a new situation for Iceberg. The table spec only bumps the 
format version when older readers would read newer features incorrectly, and we 
have added semantically meaningful optional fields within existing table 
versions before. `refs` was added to the v2 spec in PR #3425, after v2 was 
adopted, without a bump. Older writers can drop all branches and tags on commit.
   
   Bumping the view format version would also remove the graceful fallback for 
old readers: they would fail on an MV instead of evaluating the view query. But 
this is probably a rather minor point.
   
   So I'd lean toward keeping `format-version` 1 and adding a writer 
requirement, following the table spec's rule for [unknown partition 
transforms](https://iceberg.apache.org/spec/#partitioning):
   
   > Writers are not allowed to commit data using a partition spec that 
contains a field with an unknown transform.
   
   Something like this, which also scopes the compatibility claim to readers 
since already-released writers can still drop the field:
   
   ```suggestion
   This addition is backward compatible for readers: a reader that is unaware 
of materialized views sees the unknown `storage-table` field, ignores it, and 
evaluates the view query directly.
   Writers must not drop `storage-table` from existing view versions. 
Implementations that do not support materialized views must not commit changes 
to a view that has `storage-table` set.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to