Thanks, Ryan! I agree, and it maps well onto how the implementation is split into two components. The first is in Iceberg-core: it owns the materialized-view metadata and the parsing/recording of the source states (what the result was computed from) and performs the basic freshness checks. The second extends Iceberg's Spark implementation with the engine-facing definition of a materialized view: read the storage table when it's fresh, and evaluate the view query when it's stale.
Also, a quick update on that second (Spark) component: I've gone with the DSv2 catalog-based routing (#9830 <https://github.com/apache/iceberg/pull/9830>) and updated it to use Spark 4.2's new loadRelation API. A single loadRelation call now returns the storage table when the MV is fresh and the view when it's stale, which collapses the pre-4.2 indirection (loadView throwing / loadTable returning a wrapper) into one path. Thanks, Walaa. On Tue, Jul 14, 2026 at 3:21 PM Ryan Blue <[email protected]> wrote: > I think that we should start with an implementation in Iceberg that > performs basic checks. That can be used for a variety of contexts, > including direct API usage without an engine (MV use without an engine to > run the view is a plus). This would probably be used in the Spark > integration to provide MV support without needing Spark APIs for it yet. > > We would also want to have a way for engines to make better decisions and > do things like join in newer information on top of what is in the > materialized table. That depends on engine capabilities and logic. That > doesn't exist yet so I think starting with the simple case (replace a view > with a table read) and allowing engines to add complexity over time is > probably the right way to go. > > Ryan > > On Mon, Jun 29, 2026 at 12:50 PM Walaa Eldin Moustafa < > [email protected]> wrote: > >> Hi all (cross-posting to Iceberg and Spark dev lists), >> >> I'd like to gather input from both communities on an architectural >> decision for the Spark implementation of Iceberg Materialized Views. >> >> *Background* >> >> The Iceberg community has been working on a Materialized View >> specification (https://github.com/apache/iceberg/pull/11041), and the >> design is now directionally aligned among contributors active on the spec >> PR. We are now devising the Spark implementation as the first end-to-end >> realization of the spec. A core engine concern is read-time routing: when a >> query references an MV, the engine must decide whether to read the >> precomputed storage table (when the MV is fresh) or evaluate the view query >> (when stale). Both architectures have been implemented; the PRs and >> trade-offs are laid out below. >> >> >> *1. DSv2 catalog-based routing: * >> https://github.com/apache/iceberg/pull/9830 >> >> The catalog itself decides routing. SparkCatalog.loadTable returns a >> wrapper that reads from the storage table when fresh; loadView indicates >> "use loadTable instead" when fresh. >> >> Pros: >> - Routing decision lives at the catalog boundary, where the View ↔ >> storage Table relationship is already modeled. >> - Aligns with Spark 4.2's new RelationCatalog.loadRelation API [1], which >> lets a catalog return either a Table or View for the same identifier. MV >> routing fits this model naturally, and the current pre-4.2 indirection >> (loadView throws / loadTable returns a wrapper) can collapse to a single >> loadRelation call. >> >> Cons: >> - Freshness evaluation runs inside the view catalog. When MV dependencies >> span multiple catalogs, loading dependency metadata from another catalog >> doesn't compose cleanly inside one catalog's load path. >> - Pre-4.2, routing requires the indirection above. >> >> >> *2. Catalyst analyzer-based routing:* >> https://github.com/wmoustafa/iceberg-1/pull/2 >> >> A new Catalyst rule (ResolveMaterializedViews) rewrites >> UnresolvedRelation for a fresh MV to the storage table identifier; Spark's >> normal table resolution loads it. Stale MVs fall through to the existing >> view-expansion rule. >> >> Pros: >> - Sits above the catalog layer, so freshness checks that need to load >> dependency metadata from other catalogs are a natural fit. The analyzer >> rule has access to the full catalog manager. >> - Works across Spark 3.x and 4.x without depending on a specific DSv2 API >> shape. >> >> Cons: >> - Doesn't leverage loadRelation where DSv2 is evolving. >> >> *The crux of the trade-off:* the DSv2 approach aligns with the direction >> of Spark's DSv2 surface (especially 4.2's loadRelation), while the analyzer >> approach accommodates cross-catalog freshness checks more naturally. >> >> Feedback welcome from either community, particularly anyone who has >> thought about MVs, or planned loadRelation-based integration patterns. >> >> Thanks, >> Walaa. >> >> [1] >> https://github.com/apache/spark/blob/e754420f1c43423cb865adcc840e1d3111f3ef3b/sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/RelationCatalog.java#L123-L135 >> >
