I think Fokko's OOO. Should we help with that PR? On Fri, Jul 25, 2025 at 9:38 AM Eduard Tudenhöfner <etudenhoef...@apache.org> wrote:
> I believe we also wanted to get in at least the read path for UnknownType. > Fokko has a WIP PR <https://github.com/apache/iceberg/pull/13445> for > that. > > On Fri, Jul 25, 2025 at 6:13 PM Steven Wu <stevenz...@gmail.com> wrote: > >> 3. Spark: fix data frame join based on different versions of the same >> table that may lead to weird results. Anton is working on a fix. It >> requires a small behavior change (table state may be stale up to refresh >> interval). Hence it is better to include it in the 1.10.0 release where >> Spark 4.0 is first supported. >> 4. Variant support in core and Spark 4.0. Ryan thinks this is very close >> and will prioritize the review. >> >> We still have the above two issues pending. 3 doesn't have a PR yet. PR >> for 4 is not associated with the milestone yet. >> >> On Fri, Jul 25, 2025 at 9:02 AM Kevin Liu <kevinjq...@apache.org> wrote: >> >>> Thanks everyone for the review. The 2 PRs are both merged. >>> Looks like there's only 1 PR left in the 1.10 milestone >>> <https://github.com/apache/iceberg/milestone/54> :) >>> >>> Best, >>> Kevin Liu >>> >>> On Thu, Jul 24, 2025 at 7:44 PM Manu Zhang <owenzhang1...@gmail.com> >>> wrote: >>> >>>> Thanks Kevin. The first change is not in the versioned doc so it can be >>>> released anytime. >>>> >>>> Regards, >>>> Manu >>>> >>>> On Fri, Jul 25, 2025 at 4:21 AM Kevin Liu <kevinjq...@apache.org> >>>> wrote: >>>> >>>>> The 3 PRs above are merged. Thanks everyone for the review. >>>>> >>>>> I've added 2 more PRs to the 1.10 milestone. These are both >>>>> nice-to-haves. >>>>> - docs: add subpage for REST Catalog Spec in "Specification" #13521 >>>>> <https://github.com/apache/iceberg/pull/13521> >>>>> - REST-Fixture: Ensure strict mode on jdbc catalog for rest fixture >>>>> #13599 <https://github.com/apache/iceberg/pull/13599> >>>>> >>>>> The first one changes the link for "REST Catalog Spec" on the left nav >>>>> of https://iceberg.apache.org/spec/ from the swagger.io link to a >>>>> dedicated page for IRC. >>>>> The second one fixes the default behavior of `iceberg-rest-fixture` >>>>> image to align with the general expectation when creating a table in a >>>>> catalog. >>>>> >>>>> Please take a look. I would like to have both of these as part of the >>>>> 1.10 release. >>>>> >>>>> Best, >>>>> Kevin Liu >>>>> >>>>> >>>>> On Wed, Jul 23, 2025 at 1:31 PM Kevin Liu <kevinjq...@apache.org> >>>>> wrote: >>>>> >>>>>> Here are the 3 PRs to add corresponding tests. >>>>>> https://github.com/apache/iceberg/pull/13648 >>>>>> https://github.com/apache/iceberg/pull/13649 >>>>>> https://github.com/apache/iceberg/pull/13650 >>>>>> >>>>>> I've tagged them with the 1.10 milestone, waiting for CI to complete >>>>>> :) >>>>>> >>>>>> Best, >>>>>> Kevin Liu >>>>>> >>>>>> On Wed, Jul 23, 2025 at 1:08 PM Steven Wu <stevenz...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Kevin, thanks for checking that. I will take a look at your backport >>>>>>> PRs. Can you add them to the 1.10.0 milestone? >>>>>>> >>>>>>> On Wed, Jul 23, 2025 at 12:27 PM Kevin Liu <kevinjq...@apache.org> >>>>>>> wrote: >>>>>>> >>>>>>>> Thanks again for driving this Steven! We're very close!! >>>>>>>> >>>>>>>> As mentioned in the community sync today, I wanted to verify >>>>>>>> feature parity between Spark 3.5 and Spark 4.0 for this release. >>>>>>>> I was able to verify that Spark 3.5 and Spark 4.0 have feature >>>>>>>> parity for this upcoming release. More details in the other devlist >>>>>>>> thread >>>>>>>> https://lists.apache.org/thread/7x7xcm3y87y81c4grq4nn9gdjd4jm05f >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Kevin Liu >>>>>>>> >>>>>>>> On Wed, Jul 23, 2025 at 12:17 PM Steven Wu <stevenz...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Another update on the release. >>>>>>>>> >>>>>>>>> The existing blocker PRs are almost done. >>>>>>>>> >>>>>>>>> During today's community sync, we identified the following >>>>>>>>> issues/PRs to be included in the 1.10.0 release. >>>>>>>>> >>>>>>>>> 1. backport of PR 13100 to the main branch. I have created a >>>>>>>>> cherry-pick >>>>>>>>> PR <https://github.com/apache/iceberg/pull/13647> for that. >>>>>>>>> There is a one line difference compared to the original PR due to >>>>>>>>> the >>>>>>>>> removal of the deprecated RemoveSnapshot class in main branch for >>>>>>>>> 1.10.0 >>>>>>>>> target. Amogh has suggested using RemoveSnapshots with a single >>>>>>>>> snapshot >>>>>>>>> id, which should be supported by all REST catalog servers. >>>>>>>>> 2. Flink compaction doesn't support row lineage. Fail the >>>>>>>>> compaction for V3 tables. I created a PR >>>>>>>>> <https://github.com/apache/iceberg/pull/13646> for that. Will >>>>>>>>> backport after it is merged. >>>>>>>>> 3. Spark: fix data frame join based on different versions of >>>>>>>>> the same table that may lead to weird results. Anton is working on >>>>>>>>> a fix. >>>>>>>>> It requires a small behavior change (table state may be stale up >>>>>>>>> to refresh >>>>>>>>> interval). Hence it is better to include it in the 1.10.0 release >>>>>>>>> where >>>>>>>>> Spark 4.0 is first supported. >>>>>>>>> 4. Variant support in core and Spark 4.0. Ryan thinks this is >>>>>>>>> very close and will prioritize the review. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> steven >>>>>>>>> >>>>>>>>> The 1.10.0 milestone can be found here. >>>>>>>>> https://github.com/apache/iceberg/milestone/54 >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Jul 16, 2025 at 9:15 AM Steven Wu <stevenz...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Ajantha/Robin, thanks for the note. We can include the PR in the >>>>>>>>>> 1.10.0 milestone. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Jul 16, 2025 at 3:20 AM Robin Moffatt >>>>>>>>>> <ro...@confluent.io.invalid> wrote: >>>>>>>>>> >>>>>>>>>>> Thanks Ajantha. Just to confirm, from a Confluent point of view, >>>>>>>>>>> we will not be able to publish the connector on Confluent Hub until >>>>>>>>>>> this >>>>>>>>>>> CVE[1] is fixed. >>>>>>>>>>> Since we would not publish a snapshot build, if the fix doesn't >>>>>>>>>>> make it into 1.10 then we'd have to wait for 1.11 (or a dot release >>>>>>>>>>> of >>>>>>>>>>> 1.10) to be able to include the connector on Confluent Hub. >>>>>>>>>>> >>>>>>>>>>> Thanks, Robin. >>>>>>>>>>> >>>>>>>>>>> [1] >>>>>>>>>>> https://github.com/apache/iceberg/issues/10745#issuecomment-3074300861 >>>>>>>>>>> >>>>>>>>>>> On Wed, 16 Jul 2025 at 04:03, Ajantha Bhat < >>>>>>>>>>> ajanthab...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> I have approached Confluent people >>>>>>>>>>>> <https://github.com/apache/iceberg/issues/10745#issuecomment-3058281281> >>>>>>>>>>>> to help us publish the OSS Kafka Connect Iceberg sink plugin. >>>>>>>>>>>> It seems we have a CVE from dependency that blocks us from >>>>>>>>>>>> publishing the plugin. >>>>>>>>>>>> >>>>>>>>>>>> Please include the below PR for 1.10.0 release which fixes >>>>>>>>>>>> that. >>>>>>>>>>>> https://github.com/apache/iceberg/pull/13561 >>>>>>>>>>>> >>>>>>>>>>>> - Ajantha >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Jul 15, 2025 at 10:48 AM Steven Wu < >>>>>>>>>>>> stevenz...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> > Engines may model operations as deleting/inserting rows or >>>>>>>>>>>>> as modifications to rows that preserve row ids. >>>>>>>>>>>>> >>>>>>>>>>>>> Manu, I agree this sentence probably lacks some context. The >>>>>>>>>>>>> first half (as deleting/inserting rows) is probably about the >>>>>>>>>>>>> row lineage handling with equality deletes, which is described in >>>>>>>>>>>>> another >>>>>>>>>>>>> place. >>>>>>>>>>>>> >>>>>>>>>>>>> "Row lineage does not track lineage for rows updated via Equality >>>>>>>>>>>>> Deletes >>>>>>>>>>>>> <https://iceberg.apache.org/spec/#equality-delete-files>, >>>>>>>>>>>>> because engines using equality deletes avoid reading existing >>>>>>>>>>>>> data before >>>>>>>>>>>>> writing changes and can't provide the original row ID for the new >>>>>>>>>>>>> rows. >>>>>>>>>>>>> These updates are always treated as if the existing row was >>>>>>>>>>>>> completely >>>>>>>>>>>>> removed and a unique new row was added." >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Jul 14, 2025 at 5:49 PM Manu Zhang < >>>>>>>>>>>>> owenzhang1...@gmail.com> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks Steven, I missed that part but the following sentence >>>>>>>>>>>>>> is a bit hard to understand (maybe just me) >>>>>>>>>>>>>> >>>>>>>>>>>>>> Engines may model operations as deleting/inserting rows or as >>>>>>>>>>>>>> modifications to rows that preserve row ids. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Can you please help to explain? >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Steven Wu <stevenz...@gmail.com>于2025年7月15日 周二04:41写道： >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Manu >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The spec already covers the row lineage carry over (for >>>>>>>>>>>>>>> replace) >>>>>>>>>>>>>>> https://iceberg.apache.org/spec/#row-lineage >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> "When an existing row is moved to a different data file for >>>>>>>>>>>>>>> any reason, writers should write _row_id and >>>>>>>>>>>>>>> _last_updated_sequence_number according to the following >>>>>>>>>>>>>>> rules:" >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> Steven >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Mon, Jul 14, 2025 at 1:38 PM Steven Wu < >>>>>>>>>>>>>>> stevenz...@gmail.com> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> another update on the release. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> We have one open PR left for the 1.10.0 milestone >>>>>>>>>>>>>>>> <https://github.com/apache/iceberg/milestone/54> (with 25 >>>>>>>>>>>>>>>> closed PRs). Amogh is actively working on the last blocker PR. >>>>>>>>>>>>>>>> Spark 4.0: Preserve row lineage information on compaction >>>>>>>>>>>>>>>> <https://github.com/apache/iceberg/pull/13555> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I will publish a release candidate after the above blocker >>>>>>>>>>>>>>>> is merged and backported. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> Steven >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Mon, Jul 7, 2025 at 11:56 PM Manu Zhang < >>>>>>>>>>>>>>>> owenzhang1...@gmail.com> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi Amogh, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Is it defined in the table spec that "replace" operation >>>>>>>>>>>>>>>>> should carry over existing lineage info insteading of >>>>>>>>>>>>>>>>> assigning new IDs? If >>>>>>>>>>>>>>>>> not, we'd better firstly define it in spec because all >>>>>>>>>>>>>>>>> engines and >>>>>>>>>>>>>>>>> implementations need to follow it. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Tue, Jul 8, 2025 at 11:44 AM Amogh Jahagirdar < >>>>>>>>>>>>>>>>> 2am...@gmail.com> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> One other area I think we need to make sure works with >>>>>>>>>>>>>>>>>> row lineage before release is data file compaction. At >>>>>>>>>>>>>>>>>> the moment, >>>>>>>>>>>>>>>>>> <https://github.com/apache/iceberg/blob/main/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/SparkBinPackFileRewriteRunner.java#L44> >>>>>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>>>> looks like compaction will read the records from the data >>>>>>>>>>>>>>>>>> files without >>>>>>>>>>>>>>>>>> projecting the lineage fields. What this means is that on >>>>>>>>>>>>>>>>>> write of the new >>>>>>>>>>>>>>>>>> compacted data files we'd be losing the lineage information. >>>>>>>>>>>>>>>>>> There's no >>>>>>>>>>>>>>>>>> data change in a compaction but we do need to make sure the >>>>>>>>>>>>>>>>>> lineage info >>>>>>>>>>>>>>>>>> from carried over records is materialized in the newly >>>>>>>>>>>>>>>>>> compacted files so >>>>>>>>>>>>>>>>>> they don't get new IDs or inherit the new file sequence >>>>>>>>>>>>>>>>>> number. I'm working >>>>>>>>>>>>>>>>>> on addressing this as well, but I'd call this out as a >>>>>>>>>>>>>>>>>> blocker as well. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> *Robin Moffatt* >>>>>>>>>>> *Sr. Principal Advisor, Streaming Data Technologies* >>>>>>>>>>> >>>>>>>>>>
