Hi, Thanks everyone for joining the MV discussion meeting. We will continue to have the recurring sync meeting on Wednesday 9 am (Pacific) every 3 weeks until we get to the finish line where Jan's MV spec PR [1] is merged. I have scheduled our next meeting on Oct 29 in the Iceberg dev events calendar.
Here is the video recording for today's meeting. https://drive.google.com/file/d/1-nfhBPDWLoAFDu5cKP0rwLd_30HB6byR/view?usp=sharing We mostly discussed freshness evaluation. Here is the meeting summary. 1. For tracking the refresh state for the source MV [2], the consensus is option 2 (treating source MV as a materialized table) which would give engines the flexibility on freshness determination (recursive beyond source MV or not). 2. Earlier design doc [3] discussed max staleness config. But it wasn't reflected in the spec PR. The general opinion is to add the config to the spec PR. The open question is whether the ` materialization.max-staleness-ms` config should be added to the view metadata or the storage table metadata. Either can work. We just need to decide which makes a little better fit. 3. Prashanth brought up schema change with default value and how it may affect the MV refresh state (for SQL representation with select *). Jan mentioned that snapshot contains schema id when the snapshot was created. Engine can compare the snapshot schema id to the source table schema id during freshness evaluation. There is no need for additional schema info in refresh-state tracking in the storage table. 4. Prashanth brought up another scenario of compaction/rewrite where a new snapshot was added with actual data change. The general take is that the engine can optimize and decide that MV is fresh as the new snapshot doesn't have any data change. We can add some clarifications in the spec PR for freshness evaluation based on the above discussions. [1] https://github.com/apache/iceberg/pull/11041 [2] https://docs.google.com/document/d/1_StBW5hCQhumhIvgbdsHjyW0ED3dWMkjtNzyPp9Sfr8/edit?tab=t.0 [3] https://docs.google.com/document/d/1UnhldHhe3Grz8JBngwXPA6ZZord1xMedY5ukEhZYF-A/edit?tab=t.0#heading=h.3wigecex0zls On Thu, Sep 25, 2025 at 9:27 AM Steven Wu <stevenz...@gmail.com> wrote: > Hi all, > > Iceberg materialized view has been discussed in the community for a long > time. Thanks Jan Kaul for driving the discussion and the spec PR. It has > been stalled for a long time due to lack of consensus on 1 or 2 topics. In > Wed's Iceberg community sync meeting, Talat brought up the question on how > to move forward and if we can have a dedicated meeting for MV. > > I have set up a meeting on *Oct 8 (9-10 am Pacific)*. If you subscribe to > the "Iceberg Dev Events" calendar, you should be able to see it. If not, > here is the link: https://meet.google.com/nfe-guyq-pqf > > We are going to discuss > * remaining open questions > * unresolved concerns > * the next step and hopefully some consensus on moving forward > > MV spec PR is up to date. Jan has incorporated recent feedback. This > should be the base of the discussion. > https://github.com/apache/iceberg/pull/11041 > <https://www.google.com/url?q=https://github.com/apache/iceberg/pull/11041&sa=D&source=calendar&usd=2&usg=AOvVaw3w0TjRpwbC17AGzmxZmElM> > > Dev discussion thread (a long-running thread started by Jan). > https://lists.apache.org/thread/y1vlpzbn2x7xookjkffcl08zzyofk5hf > <https://www.google.com/url?q=https://lists.apache.org/thread/y1vlpzbn2x7xookjkffcl08zzyofk5hf&sa=D&source=calendar&usd=2&usg=AOvVaw0fotlsrnRBOb820mA5JRyB> > > The mail archive has broken lineage and doesn't show all replies. Email > subject is "*[DISCUSS] Iceberg Materialzied Views*". > > Thanks, > Steven > >
