+1 for the approach given it reduces the work. On this, as it exposes storage tables to user catalog, I was mainly thinking we should have a common suffix/naming pattern for storage table across catalog. The netflix approach sounds good to me.
Hope we can continue the proposal, as there's still decisions on how to standardize other metadata like how MV lineages. Thanks, Szehon On Fri, Apr 19, 2024 at 6:17 PM John Zhuge <jzh...@apache.org> wrote: > +1 on separate view and table metadata > > I'd like to share our experience of such a design at Netflix for years. > The changes to the view spec are minimal and there are no changes to the > Iceberg table metadata other than tracking an additional table property for > capturing freshness. The storage tables have a specific suffix and a naming > pattern. It is convenient to use existing toolings on these tables. We have > not encountered any fundamental issues with this modeling. > > On Fri, Apr 19, 2024 at 5:49 AM Renjie Liu <liurenjie2...@gmail.com> > wrote: > >> +1 for this proposal. >> >> On Fri, Apr 19, 2024 at 3:40 PM Ajantha Bhat <ajanthab...@gmail.com> >> wrote: >> >>> +1 for the proposal. >>> >>> - Ajantha >>> >>> On Fri, Apr 19, 2024 at 7:29 AM Benny Chow <btc...@gmail.com> wrote: >>> >>>> +1 for separate view and table objects. Walaa's Spark >>>> implementation demonstrates how little change it takes on the Iceberg APIs >>>> to start sharing MVs between engines. >>>> >>>> Thanks >>>> Benny >>>> >>>> On Thu, Apr 18, 2024 at 9:52 AM Walaa Eldin Moustafa < >>>> wa.moust...@gmail.com> wrote: >>>> >>>>> Hi everyone, >>>>> >>>>> I would like to make a proposal for issue [1] to support materialized >>>>> views in Iceberg. The support leverages two separate objects, an Iceberg >>>>> view and an Iceberg table to implement materialized views. Each object >>>>> retains relevant metadata to support the MV operations. An initial design, >>>>> which we can refine, is detailed in the description section of this PR >>>>> [2]. >>>>> >>>>> This proposal is the outcome of extensive community discussions in >>>>> various forums [3, 4, 5, 6, 7]. >>>>> >>>>> Please respond with your recommendation: >>>>> +1 if you support moving forward with the two separate objects model. >>>>> 0 if you are neutral. >>>>> -1 if you disagree with the two separate objects model. >>>>> >>>>> Thanks, >>>>> Walaa. >>>>> >>>>> [1] https://github.com/apache/iceberg/issues/10043 >>>>> [2] https://github.com/apache/iceberg/pull/9830 >>>>> [3] >>>>> https://docs.google.com/document/d/1zg0wQ5bVKTckf7-K_cdwF4mlRi6sixLcyEh6jErpGYY >>>>> [4] https://github.com/apache/iceberg/issues/6420 >>>>> [5] >>>>> https://docs.google.com/document/d/1UnhldHhe3Grz8JBngwXPA6ZZord1xMedY5ukEhZYF >>>>> [6] https://lists.apache.org/thread/tb3wcs7czjvjbq9y1qtr87g9s95ky5zh >>>>> [7] https://lists.apache.org/thread/l6cvrp4r1001k08cy2ypybzy2kgxpt1y >>>>> >>>> > > -- > John Zhuge >
