> All of these approaches are aligned in one, specific way: the storage table is an iceberg table.
I do not think that is true. I think people are aligned that we would like to re-use the Iceberg table metadata defined in the Iceberg table spec to express the data in MV, but I don't think it goes that far to say it must be an Iceberg table. Once you have that mindset, then of course option 1 (separate table and view) is the only option. > I don't think that is necessary and it significantly increases the complexity. And can you quantify what you mean by "significantly increases the complexity"? Seems like a lot of concerns are coming from the tradeoff with complexity. We probably all agree that using option 7 (a completely new metadata type) is a lot of work from scratch, that is why it is not favored. However, my understanding is that as long as we re-use the view and table metadata, then the majority of the existing logic can be reused. I think what we have gone through in Slack to draft the rough Java API shape helps here, because people can estimate the amount of effort required to implement it. And I don't think they are **significantly** more complex to implement. Could you elaborate more about the complexity that you imagine? -Jack On Fri, Mar 1, 2024 at 8:57 AM Daniel Weeks <daniel.c.we...@gmail.com> wrote: > I feel I've been most vocal about pushing back against options 2+ (or > Ryan's categories of combined table/view, or new metadata type), so I'll > try to expand on my reasoning. > > I understand the appeal of creating a design where we encapsulate the > view/storage from both a structural and performance standpoint, but I don't > think that is necessary and it significantly increases the complexity. > > All of these approaches are aligned in one, specific way: the storage > table is an iceberg table. > > Because of this, all the behaviors and requirements still apply to these > tables. They need to be maintained (snapshot cleanup, orphan files), in > cases need to be optimized (compaction, manifest rewrites), they need to be > able to be inspected (this will be even more important with MV since > staleness can produce different results and questions will arise about what > state the storage table was in). There may be cases where the tables need > to be managed directly. > > Anywhere we deviate from the existing constructs/commit/access for tables, > we will ultimately have to then unwrap to re-expose the underlying Iceberg > behavior. This creates unnecessary complexity in the library/API layer, > which are not the primary interface users will have with materialized views > where an engine is almost entirely necessary to interact with the dataset. > > As to the performance concerns around option 1, I think we're overstating > the downsides. It really comes down to how many metadata loads are > necessary and evaluating freshness would likely be the real bottleneck as > it involves potentially loading many tables. All of the options are on the > same order of performance for the metadata and table loads. > > As to the visibility of tables and whether they're registered in the > catalog, I think registering in the catalog is the right approach so that > the tables are still addressable for maintenance/etc. The visibility of > the storage table is a catalog implementation decision and shouldn't be a > requirement of the MV spec (I can see cases for both and it isn't necessary > to dictate a behavior). > > I'm still strongly in favor of Option 1 (separate table and view) for > these reasons. > > -Dan > > > > On Thu, Feb 29, 2024 at 11:07 PM Jack Ye <yezhao...@gmail.com> wrote: > >> > Jack, it sounds like you’re the proponent of a combined table and view >> (rather than a new metadata spec for a materialized view). What is the main >> motivation? It seems like you’re convinced of that approach, but I don’t >> understand the advantage it brings. >> >> Sorry I have to make a Google Sheet to capture all the options we have >> discussed so far, I wanted to use the existing Google Doc, but it has >> really bad table/sheet support... >> >> >> https://docs.google.com/spreadsheets/d/1a0tlyh8f2ft2SepE7H3bgoY2A0q5IILgzAsJMnwjTBs/edit#gid=0 >> >> I have listed all the options, with how they are implemented and some >> important considerations we have discussed so far. Note that: >> 1. This sheet currently excludes the lineage information, which we can >> discuss more later after the current topic is resolved. >> 2. I removed the considerations for REST integration since from the other >> thread we have clarified that they should be considered completely >> separately. >> >> *Why I come as a proponent of having a new MV object with table and view >> metadata file pointer* >> >> In my sheet, there are 3 options that do not have major problems: >> Option 2: Add storage table metadata file pointer in view object >> Option 5: New MV object with table and view metadata file pointer >> Option 6: New MV spec with table and view metadata >> >> I originally excluded option 2 because I think it does not align with the >> REST spec, but after the other discussion thread about "Inconsistency >> between REST spec and table/view spec", I think my original concern no >> longer holds true so now I put it back. And based on my personal >> preference that MV is an independent object that should be separated from >> view and table, plus the fact that option 5 is probably less work than >> option 6 for implementation, that is how I come as a proponent of option 5 >> at this moment. >> >> >> *Regarding Ryan's evaluation framework* >> >> I think we need to reconcile this sheet with Ryan's evaluation framework. >> That framework categorization puts option 2, 3, 4, 5, 6 all under the same >> category of "A combination of a view and a table" and concludes that >> they don't have any advantage for the same set of reasons. But those >> reasons are not really convincing to me so let's talk about them in more >> detail. >> >> (1) You said "I don’t see a reason why a combined view and table is >> advantageous" as "this would cause unnecessary dependence between the view >> and table in catalogs." What dependency exactly do you mean here? And why >> is that unnecessary, given there has to be some sort of dependency anyway >> unless we go with option 5 or 6? >> >> (2) You said "I guess there’s an argument that you could load both table >> and view metadata locations at the same time. That hardly seems worth the >> trouble". I disagree with that. Catalog interaction performance is critical >> to at least everyone working in EMR and Athena, and MV itself as an >> acceleration approach needs to be as fast as possible. >> >> I have put 3 key operations in the doc that I think matters for MV during >> interactions with engine: >> 1. refreshes storage table >> 2. get the storage table of the MV >> 3. if stale, get the view SQL >> >> And option 1 clearly falls short with 4 sequential steps required to load >> a storage table. You mentioned "recent issues with adding views to the JDBC >> catalog" in this topic, could you explain a bit more? >> >> (3) You said "I also think that once we decide on structure, we can make >> it possible for REST catalog implementations to do smart things, in a way >> that doesn’t put additional requirements on the underlying catalog store." >> If REST is fully compatible with Iceberg spec then I have no problem with >> this statement. However, as we discussed in the other thread, it is not the >> case. In the current state, I think the sequence of action should be to >> evolve the Iceberg table/view spec (or add a MV spec) first, and then think >> about how REST can incorporate it or do smart things that are not Iceberg >> spec compliant. Do you agree with that? >> >> (4) You said the table identifier pointer "is a problem we need to solve >> generally because a materialized table needs to be able to track the >> upstream state of tables that were used". I don't think that is a reason to >> choose to use a table identifier pointer for a storage table. The issue is >> not about using a table identifier pointer. It is about exposing the >> storage table as a separate entity in the catalog, which is what people do >> not like and is already discussed in length in Jan's question 3 (also >> linked in the sheet). I agree with that statement, because without a REST >> implementation that can magically hide the storage table, this model adds >> additional burden regarding compliance and data governance for any other >> non-REST catalog implementations that are compliant to the Iceberg spec. >> Many mechanisms need to be built in a catalog to hide, protect, maintain, >> recycle the storage table, that can be avoided by using other approaches. I >> think we should reach a consensus about that and discuss further if you do >> not agree. >> >> Best, >> Jack Ye >> >> On Thu, Feb 29, 2024 at 10:53 PM Jan Kaul <jank...@mailbox.org.invalid> >> wrote: >> >>> Hi Ryan, we actually discussed your categories in this question >>> <https://docs.google.com/document/d/1UnhldHhe3Grz8JBngwXPA6ZZord1xMedY5ukEhZYF-A/edit?pli=1#heading=h.y70rtfhi9qxi>. >>> Where your categories correspond to the following designs: >>> >>> - Separate table and view => Design 1 >>> - Combination of view and table => Design 2 >>> - A new metadata type => Design 4 >>> >>> Jan >>> On 01.03.24 00:03, Ryan Blue wrote: >>> >>> Looks like it wasn’t clear what I meant for the 3 categories, so I’ll be >>> more specific: >>> >>> - *Separate table and view*: this option is to have the objects that >>> we have today, with extra metadata. Commit processes are separate: >>> committing to the table doesn’t alter the view and committing to the view >>> doesn’t change the table. However, changing the view can make it so the >>> table is no longer useful as a materialization. >>> - *A combination of a view and a table*: in this option, the table >>> metadata and view metadata are the same as the first option. The >>> difference >>> is that the commit process combines them, either by embedding a table >>> metadata location in view metadata or by tracking both in the same >>> catalog >>> reference. >>> - *A new metadata type*: this option is where we define a new >>> metadata object that has view attributes, like SQL representations, along >>> with table attributes, like partition specs and snapshots. >>> >>> Hopefully this is clear because I think much of the confusion is caused >>> by different definitions. >>> >>> The LoadTableResponse having optional metadata-location field implies >>> that the object in the catalog no longer needs to hold a metadata file >>> pointer >>> >>> The REST protocol has not removed the requirement for a metadata file, >>> so I’m going to keep focused on the MV design options. >>> >>> When we say a MV can be a “new metadata type”, it does not mean it needs >>> to define a completely brand new structure of the metadata content >>> >>> I’m making a distinction between separate metadata files for the table >>> and the view and a combined metadata object, as above. >>> >>> We can define an “Iceberg MV” to be an object in a catalog, which has 1 >>> table metadata file pointer, and 1 view metadata file pointer >>> >>> This is the option I am referring to as a “combination of a view and a >>> table”. >>> >>> So to review my initial email, I don’t see a reason why a combined view >>> and table is advantageous, either implemented by having a catalog reference >>> with two metadata locations or embedding a table metadata location in view >>> metadata. This would cause unnecessary dependence between the view and >>> table in catalogs. I guess there’s an argument that you could load both >>> table and view metadata locations at the same time. That hardly seems worth >>> the trouble given the recent issues with adding views to the JDBC catalog. >>> >>> I also think that once we decide on structure, we can make it possible >>> for REST catalog implementations to do smart things, in a way that doesn’t >>> put additional requirements on the underlying catalog store. For instance, >>> we could specify how to send additional objects in a LoadViewResult, in >>> case the catalog wants to pre-fetch table metadata. I think these >>> optimizations are a later addition, after we define the relationship >>> between views and tables. >>> >>> Jack, it sounds like you’re the proponent of a combined table and view >>> (rather than a new metadata spec for a materialized view). What is the main >>> motivation? It seems like you’re convinced of that approach, but I don’t >>> understand the advantage it brings. >>> >>> Ryan >>> >>> On Thu, Feb 29, 2024 at 12:26 PM Szehon Ho <szehon.apa...@gmail.com> >>> wrote: >>> >>>> Hi >>>> >>>> Yes I mostly agree with the assessment. To clarify a few minor points. >>>> >>>> is a materialized view a view and a separate table, a combination of >>>>> the two (i.e. commits are combined), or a new metadata type? >>>> >>>> >>>> For 'new metadata type', I consider mostly Jack's initial proposal of a >>>> new Catalog MV object that has two references (ViewMetadata + >>>> TableMetadata). >>>> >>>> The arguments that I see for a combined materialized view object are: >>>>> >>>>> - Regular views are separate, rather than being tables with SQL >>>>> and no data so it would be inconsistent (“Iceberg view is just a table >>>>> with >>>>> no data but with representations defined. But we did not do that.”) >>>>> >>>>> >>>>> - Materialized views are different objects in DDL >>>>> >>>>> >>>>> - Tables may be a superset of functionality needed for >>>>> materialized views >>>>> >>>>> >>>>> - Tables are not typically exposed to end users — but this isn’t >>>>> required by the separate view and table option >>>>> >>>>> For completeness, there seem to be a few additional ones (mentioned in >>>> the Slack and above messages). >>>> >>>> - Lack of spec change (to ViewMetadata). But as Jack says it is a >>>> spec change (ie, to catalogs) >>>> - A single call to get the View's StorageTable (versus two calls) >>>> - A more natural API, no opportunity for user to call >>>> Catalog.dropTable() and renameTable() on storage table >>>> >>>> >>>> *Thoughts: *I think the long discussion sessions we had on Slack >>>> was fruitful for me, as seeing the API clarified some things. >>>> >>>> I was initially more in favor of MV being a new metadata type >>>> (TableMetadata + ViewMetadata). But seeing most of the MV operations end >>>> up being ViewCatalog or Catalog operations, I am starting to think API-wise >>>> that it may not align with the new metadata type (unless we define >>>> MVCatalog and /MV REST endpoints, which then are boilerplate wrappers). >>>> >>>> Initially one question I had for option 'a view and a separate table', >>>> was how to make this table reference (metadata.json or catalog reference). >>>> In the previous option, we had a precedent of Catalog references to >>>> Metadata, but not pointers between Metadatas. I initially saw the proposed >>>> Catalog's TableIdentifier pointer as 'polluting' catalog concerns in >>>> ViewMetadata. (I saw Catalog and ViewCatalog as a layer above >>>> TableMetadata and ViewMetadata). But I think Dan in the Slack made a fair >>>> point that ViewMetadata already is tightly bound with a Catalog. In this >>>> case, I think this approach does have its merits as well in aligning >>>> Catalog API's with the metadata. >>>> >>>> Thanks >>>> Szehon >>>> >>>> >>>> >>>> On Thu, Feb 29, 2024 at 5:45 AM Jan Kaul <jank...@mailbox.org.invalid> >>>> <jank...@mailbox.org.invalid> wrote: >>>> >>>>> Hi all, >>>>> >>>>> I would like to provide my perspective on the question of what a >>>>> materialized view is and elaborate on Jack's recent proposal to view a >>>>> materialized view as a catalog concept. >>>>> >>>>> Firstly, let's look at the role of the catalog. Every entity in the >>>>> catalog has a *unique identifier*, and the catalog provides methods >>>>> to create, load, and update these entities. An important thing to note is >>>>> that the catalog methods exhibit two different behaviors: the *create >>>>> and load methods deal with the entire entity*, while the *update(commit) >>>>> method only deals with partial changes* to the entities. >>>>> >>>>> In the context of our current discussion, materialized view (MV) >>>>> metadata is a union of view and table metadata. The fact that the update >>>>> method deals only with partial changes, enables us to *reuse the >>>>> existing methods for updating tables and views*. For updates we don't >>>>> have to define what constitutes an entire materialized view. Changes to a >>>>> materialized view targeting the properties related to the view metadata >>>>> could use the update(commit) view method. Similarly, changes targeting the >>>>> properties related to the table metadata could use the update(commit) >>>>> table >>>>> method. This is great news because we don't have to redefine view and >>>>> table >>>>> commits (requirements, updates). >>>>> This is shown in the fact that Jack uses the same operation to update >>>>> the storage table for Option 1 and 3: >>>>> >>>>> // REST: POST /namespaces/db1/tables/mv1?materializedView=true >>>>> // non-REST: update JSON files at table_metadata_location >>>>> storageTable.newAppend().appendFile(...).commit(); >>>>> >>>>> The open question is *whether the create and load methods should >>>>> treat the properties that constitute the MV metadata as two entities (View >>>>> + Table) or one entity (new MV object)*. This is all part of Jack's >>>>> proposal, where Option 1 proposes a new MV object, and Option 3 proposes >>>>> two separate entities. The advantage of Option 1 is that it doesn't >>>>> require >>>>> two operations to load the metadata. On the other hand, the advantage of >>>>> Option 3 is that no new operations or catalogs have to be defined. >>>>> >>>>> In my opinion, defining a new representation for materialized views >>>>> (Option 1) is generally the cleaner solution. However, I see a path where >>>>> we could first introduce Option 3 and still have the possibility to >>>>> transition to Option 1 if needed. The great thing about Option 3 is that >>>>> it >>>>> only requires minor changes to the current spec and is mostly >>>>> implementation detail. >>>>> >>>>> Therefore I would propose small additions to Jacks Option 3 that only >>>>> introduce changes to the spec that are not specific to materialized views. >>>>> The idea is to introduce boolean properties to be set on the creation of >>>>> the view and the storage table that indicate that they belong to a >>>>> materialized view. The view property "materialized" is set to "true" for a >>>>> MV and "false" for a regular view. And the table property "storage_table" >>>>> is set to "true" for a storage table and "false" for a regular table. The >>>>> absence of these properties indicates a regular view or table. >>>>> >>>>> ViewCatalog viewCatalog = (ViewCatalog) catalog; >>>>> >>>>> // REST: GET /namespaces/db1/views/mv1 >>>>> // non-REST: load JSON file at metadata_location >>>>> View mv = viewCatalog.loadView(TableIdentifier.of("db1", "mv1")); >>>>> >>>>> // REST: GET /namespaces/db1/tables/mv1 >>>>> // non-REST: load JSON file at table_metadata_location if present >>>>> Table storageTable = view.storageTable(); >>>>> >>>>> // REST: POST /namespaces/db1/tables/mv1 >>>>> // non-REST: update JSON file at table_metadata_location >>>>> storageTable.newAppend().appendFile(...).commit(); >>>>> >>>>> We could then introduce a new requirement for views and tables called >>>>> "AssertProperty" which could make sure to only perform updates that are >>>>> inline with materialized views. The additional requirement can be seen as >>>>> a >>>>> general extension which does not need to be changed if we decide to got >>>>> with Option 1 in the future. >>>>> >>>>> Let me know what you think. >>>>> >>>>> Best wishes, >>>>> >>>>> Jan >>>>> On 29.02.24 04:09, Walaa Eldin Moustafa wrote: >>>>> >>>>> Thanks Ryan for the insights. I agree that reusing existing metadata >>>>> definitions and minimizing spec changes are very important. This also >>>>> minimizes spec drift (between materialized views and views spec, and >>>>> between materialized views and tables spec), and simplifies the >>>>> implementation. >>>>> >>>>> In an effort to take the discussion forward with concrete design >>>>> options based on an end-to-end implementation, I have prototyped the >>>>> implementation (and added Spark support) in this PR >>>>> https://github.com/apache/iceberg/pull/9830. I hope it helps us reach >>>>> convergence faster. More details about some of the design options are >>>>> discussed in the description of the PR. >>>>> >>>>> Thanks, >>>>> Walaa. >>>>> >>>>> >>>>> On Wed, Feb 28, 2024 at 6:20 PM Ryan Blue <b...@tabular.io> wrote: >>>>> >>>>>> I mean separate table and view metadata that is somehow combined >>>>>> through a commit process. For instance, keeping a pointer to a table >>>>>> metadata file in a view metadata file or combining commits to reference >>>>>> both. I don't see the value in either option. >>>>>> >>>>>> On Wed, Feb 28, 2024 at 5:05 PM Jack Ye <yezhao...@gmail.com> wrote: >>>>>> >>>>>>> Thanks Ryan for the help to trace back to the root question! Just a >>>>>>> clarification question regarding your reply before I reply further: what >>>>>>> exactly does the option "a combination of the two (i.e. commits are >>>>>>> combined)" mean? How is that different from "a new metadata type"? >>>>>>> >>>>>>> -Jack >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Feb 28, 2024 at 2:10 PM Ryan Blue <b...@tabular.io> wrote: >>>>>>> >>>>>>>> I’m catching up on this conversation, so hopefully I can bring a >>>>>>>> fresh perspective. >>>>>>>> >>>>>>>> Jack already pointed out that we need to start from the basics and >>>>>>>> I agree with that. Let’s remove voting at this point. Right now is the >>>>>>>> time >>>>>>>> for discussing trade-offs, not lining up and taking sides. I realize >>>>>>>> that >>>>>>>> wasn’t the intent with adding a vote, but that’s almost always the >>>>>>>> result. >>>>>>>> It’s too easy to use it as a stand-in for consensus and move on >>>>>>>> prematurely. I get the impression from the swirl in Slack that >>>>>>>> discussion >>>>>>>> has moved ahead of agreement. >>>>>>>> >>>>>>>> We’re still at the most basic question: is a materialized view a >>>>>>>> view and a separate table, a combination of the two (i.e. commits are >>>>>>>> combined), or a new metadata type? >>>>>>>> >>>>>>>> For now, I’m ignoring whether the “separate table” is some kind of >>>>>>>> “system table” (meaning hidden?) or if it is exposed in the catalog. >>>>>>>> That’s >>>>>>>> a later choice (already pointed out) and, I suspect, it should be >>>>>>>> delegated >>>>>>>> to catalog implementations. >>>>>>>> >>>>>>>> To simplify this a little, I think that we can eliminate the option >>>>>>>> to combine table and view commits. I don’t think there is a reason to >>>>>>>> combine the two. If separate, a table would track the view version used >>>>>>>> along with freshness information for referenced tables. If the table is >>>>>>>> automatically skipped when the version no longer matches the view, >>>>>>>> then no >>>>>>>> action needs to happen when a view definition changes. Similarly, the >>>>>>>> table >>>>>>>> can be updated independently without needing to also swap view >>>>>>>> metadata. >>>>>>>> This also aligns with the idea from the original doc that there can be >>>>>>>> multiple materialization tables for a view. Each should operate >>>>>>>> independently unless I’m missing something >>>>>>>> >>>>>>>> I don’t think the last paragraph’s conclusion is contentious so >>>>>>>> I’ll move on, but please stop here and reply if you disagree! >>>>>>>> >>>>>>>> That leaves the main two options, a view and a separate table >>>>>>>> linked by metadata, or, combined materialized view metadata. >>>>>>>> >>>>>>>> As the doc notes, the separate view and table option is simpler >>>>>>>> because it reuses existing metadata definitions and falls back to >>>>>>>> simple >>>>>>>> views. That is a significantly smaller spec and small is very, very >>>>>>>> important when it comes to specs. I think that the argument for a new >>>>>>>> definition of a materialized view needs to overcome this disadvantage. >>>>>>>> >>>>>>>> The arguments that I see for a combined materialized view object >>>>>>>> are: >>>>>>>> >>>>>>>> - Regular views are separate, rather than being tables with SQL >>>>>>>> and no data so it would be inconsistent (“Iceberg view is just a >>>>>>>> table with >>>>>>>> no data but with representations defined. But we did not do that.”) >>>>>>>> - Materialized views are different objects in DDL >>>>>>>> - Tables may be a superset of functionality needed for >>>>>>>> materialized views >>>>>>>> - Tables are not typically exposed to end users — but this >>>>>>>> isn’t required by the separate view and table option >>>>>>>> >>>>>>>> Am I missing any arguments for combined metadata? >>>>>>>> >>>>>>>> Ryan >>>>>>>> -- >>>>>>>> Ryan Blue >>>>>>>> Tabular >>>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> Ryan Blue >>>>>> Tabular >>>>>> >>>>> >>> >>> -- >>> Ryan Blue >>> Tabular >>> >>>
