findinpath commented on code in PR #11041:
URL: https://github.com/apache/iceberg/pull/11041#discussion_r4283412678
##########
format/view-spec.md:
##########
@@ -322,3 +430,206 @@
s3://bucket/warehouse/default.db/event_agg/metadata/00002-(uuid).metadata.json
} ]
}
```
+
+### Materialized View Example
+
+Imagine the following operation, which creates a materialized view that
precomputes daily event counts:
+
+```sql
+USE prod.default
+```
+
+```sql
+CREATE MATERIALIZED VIEW event_agg_mv (
+ event_count COMMENT 'Count of events',
+ event_date)
+COMMENT 'Precomputed daily event counts'
+AS
+SELECT
+ COUNT(1), CAST(event_ts AS DATE)
+FROM events
+GROUP BY 2
+```
+
+The materialized view metadata JSON file looks as follows:
+
+```
+s3://bucket/warehouse/default.db/event_agg_mv/metadata/00001-(uuid).metadata.json
+```
+
+```json
+{
+ "view-uuid": "b2a12651-3038-4a72-8a31-5027ab84da35",
+ "format-version" : 1,
+ "location" : "s3://bucket/warehouse/default.db/event_agg_mv",
+ "current-version-id" : 1,
+ "properties" : {
+ "comment" : "Precomputed daily event counts"
+ },
+ "versions" : [ {
+ "version-id" : 1,
+ "timestamp-ms" : 1573518431292,
+ "schema-id" : 1,
+ "default-catalog" : "prod",
+ "default-namespace" : [ "default" ],
+ "summary" : {
+ "engine-name" : "Spark",
+ "engine-version" : "3.4.1"
+ },
+ "representations" : [ {
+ "type" : "sql",
+ "sql" : "SELECT\n COUNT(1), CAST(event_ts AS DATE)\nFROM
events\nGROUP BY 2",
+ "dialect" : "spark"
+ } ],
+ "storage-table" : {
+ "namespace" : [ "default" ],
+ "name" : "event_agg_mv__storage"
+ }
+ } ],
+ "schemas": [ {
+ "schema-id": 1,
+ "type" : "struct",
+ "fields" : [ {
+ "id" : 1,
+ "name" : "event_count",
+ "required" : false,
+ "type" : "int",
+ "doc" : "Count of events"
+ }, {
+ "id" : 2,
+ "name" : "event_date",
+ "required" : false,
+ "type" : "date"
+ } ]
+ } ],
+ "version-log" : [ {
+ "timestamp-ms" : 1573518431292,
+ "version-id" : 1
+ } ]
+}
+```
+
+After a refresh operation, the storage table's snapshot summary contains the
`refresh-state` property.
+The following is an example of the `refresh-state` JSON value stored in the
snapshot summary of the storage table:
+
+```json
+{
+ "view-version-id" : 1,
+ "refresh-start-timestamp-ms" : 1573518435000,
+ "source-states" : [ {
+ "type" : "table",
+ "catalog" : "prod",
+ "namespace" : [ "default" ],
+ "name" : "events",
+ "uuid" : "d4a10b5c-1e8a-4b72-9d67-3f4a8c9e1b2d",
+ "snapshot-id" : 6148331192489823102
+ } ]
+}
+```
+
+## Appendix B: Example strategies for selecting dependencies
+
+A writer's choice of what to record varies along two independent dimensions:
+
+* Query evaluation — how the writer evaluates the materialized view’s query,
in particular whether the result of nested materialized views is determined by
expanding their own dependencies or consuming directly from the storage table.
This determines the set of dependencies that could be recorded.
+* Coverage — which of those dependencies are actually recorded; a writer may
record all, a subset, or none.
+
+The strategies below illustrate common combinations against the same shared
query. Strategies 2 and 3 differ along structure; Strategies 1, 4, and 5
illustrate coverage choices.
+
+### Shared query
+
+* `A` (the materialized view being refreshed): `SELECT B.customer_id,
C.category, SUM(B.amount) AS total FROM B JOIN C ON B.product_id = C.product_id
GROUP BY B.customer_id, C.category`
+* `B` (regular view): `SELECT E.customer_id, E.product_id, E.amount FROM E
JOIN D ON E.customer_id = D.customer_id`
+* `C` (materialized view): `SELECT F.product_id, G.category FROM F JOIN G ON
F.category_id = G.category_id`
+* `D` (materialized view): `SELECT customer_id FROM H WHERE status = 'active'`
+* `E` (orders), `F` (products), `G` (categories), `H` (customers): source
Iceberg tables
+
+### Strategy 1: Empty refresh state (recency only)
+
+The producer leaves `source-states` empty and relies entirely on
`refresh-start-timestamp-ms`. Readers reuse the storage table based on a
recency policy alone.
+
+`A`'s refresh state:
+
+```json
+{
+ "view-version-id": 1,
+ "refresh-start-timestamp-ms": 1573518435000,
+ "source-states": []
+}
+```
+
+### Strategy 2: Treat nested materialized views as tables
+
+Same as Strategy 1, but the query reads from materialized views. The producer
stops at each MV boundary and records the MV's storage table snapshot ID and
view version ID. No expansion beyond the MV.
+
+`C` and `D` are materialized views, treated as tables.
+
+```
+A [MV — being refreshed]
+├── B [VIEW] <-- recorded in A: version-id: 5
+│ ├── E [TABLE] <-- recorded in A: snapshot-id: 101
+│ └── D [MV] <-- recorded in A: storage-table
snapshot-id: 14, version-id: 9
+│ ┄┄┄┄┄┄ recursive boundary ┄┄┄┄┄┄
+│ └── H [TABLE] (D's dependency; verified via D's
refresh-state)
+└── C [MV] <-- recorded in A: storage-table
snapshot-id: 12, version-id: 7
+ ┄┄┄┄┄┄ recursive boundary ┄┄┄┄┄┄
+ ├── F [TABLE] (C's dependency; verified via C's
refresh-state)
+ └── G [TABLE] (C's dependency; verified via C's
refresh-state)
+```
+
+`F`, `G`, and `H` do not appear in `A`'s `source-states`; they belong to `C`
and `D`'s dependency graphs.
+
+### Strategy 3: Treat nested materialized views as views
+
+Same as Strategy 1, but the query reads from materialized views. The producer
treats each materialized view as a regular view: expand through the MV's view
definition and record the underlying tables and views. The MV's storage table
snapshot ID is **not** recorded.
Review Comment:
> but the query reads from materialized views
but the query expands recursively the view queries corresponding to the
materialized views.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
