correction for a typo. Prashanth brought up another scenario of compaction/rewrite where a new snapshot was added *with* actual data change --> Prashanth brought up another scenario of compaction/rewrite where a new snapshot was added *without* actual data change
On Wed, Oct 8, 2025 at 2:12 PM Steven Wu <[email protected]> wrote: > Hi, > > Thanks everyone for joining the MV discussion meeting. We will continue to > have the recurring sync meeting on Wednesday 9 am (Pacific) every 3 weeks > until we get to the finish line where Jan's MV spec PR [1] is merged. I > have scheduled our next meeting on Oct 29 in the Iceberg dev events > calendar. > > Here is the video recording for today's meeting. > > https://drive.google.com/file/d/1-nfhBPDWLoAFDu5cKP0rwLd_30HB6byR/view?usp=sharing > > We mostly discussed freshness evaluation. Here is the meeting summary. > > 1. For tracking the refresh state for the source MV [2], the consensus > is option 2 (treating source MV as a materialized table) which would give > engines the flexibility on freshness determination (recursive beyond source > MV or not). > 2. Earlier design doc [3] discussed max staleness config. But it > wasn't reflected in the spec PR. The general opinion is to add the config > to the spec PR. The open question is whether the ` > materialization.max-staleness-ms` config should be added to the view > metadata or the storage table metadata. Either can work. We just need to > decide which makes a little better fit. > 3. Prashanth brought up schema change with default value and how it > may affect the MV refresh state (for SQL representation with select *). Jan > mentioned that snapshot contains schema id when the snapshot was created. > Engine can compare the snapshot schema id to the source table schema id > during freshness evaluation. There is no need for additional schema info in > refresh-state tracking in the storage table. > 4. Prashanth brought up another scenario of compaction/rewrite where a > new snapshot was added with actual data change. The general take is that > the engine can optimize and decide that MV is fresh as the new snapshot > doesn't have any data change. > > > We can add some clarifications in the spec PR for freshness evaluation > based on the above discussions. > > [1] https://github.com/apache/iceberg/pull/11041 > [2] > https://docs.google.com/document/d/1_StBW5hCQhumhIvgbdsHjyW0ED3dWMkjtNzyPp9Sfr8/edit?tab=t.0 > [3] > https://docs.google.com/document/d/1UnhldHhe3Grz8JBngwXPA6ZZord1xMedY5ukEhZYF-A/edit?tab=t.0#heading=h.3wigecex0zls > > > > > On Thu, Sep 25, 2025 at 9:27 AM Steven Wu <[email protected]> wrote: > >> Hi all, >> >> Iceberg materialized view has been discussed in the community for a long >> time. Thanks Jan Kaul for driving the discussion and the spec PR. It has >> been stalled for a long time due to lack of consensus on 1 or 2 topics. In >> Wed's Iceberg community sync meeting, Talat brought up the question on how >> to move forward and if we can have a dedicated meeting for MV. >> >> I have set up a meeting on *Oct 8 (9-10 am Pacific)*. If you subscribe >> to the "Iceberg Dev Events" calendar, you should be able to see it. If >> not, here is the link: https://meet.google.com/nfe-guyq-pqf >> >> We are going to discuss >> * remaining open questions >> * unresolved concerns >> * the next step and hopefully some consensus on moving forward >> >> MV spec PR is up to date. Jan has incorporated recent feedback. This >> should be the base of the discussion. >> https://github.com/apache/iceberg/pull/11041 >> <https://www.google.com/url?q=https://github.com/apache/iceberg/pull/11041&sa=D&source=calendar&usd=2&usg=AOvVaw3w0TjRpwbC17AGzmxZmElM> >> >> Dev discussion thread (a long-running thread started by Jan). >> https://lists.apache.org/thread/y1vlpzbn2x7xookjkffcl08zzyofk5hf >> <https://www.google.com/url?q=https://lists.apache.org/thread/y1vlpzbn2x7xookjkffcl08zzyofk5hf&sa=D&source=calendar&usd=2&usg=AOvVaw0fotlsrnRBOb820mA5JRyB> >> >> The mail archive has broken lineage and doesn't show all replies. Email >> subject is "*[DISCUSS] Iceberg Materialzied Views*". >> >> Thanks, >> Steven >> >>
