Hi all, Wonder if we made some progress or if we should fall back on the spark 3 like compatibility without leveraging new feature just to enable the community to upgrade Apache Spark?
Thanks in advance, Romain Manni-Bucau @rmannibucau <https://x.com/rmannibucau> | .NET Blog <https://dotnetbirdie.github.io/> | Blog <https://rmannibucau.github.io/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> | LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book <https://www.packtpub.com/en-us/product/java-ee-8-high-performance-9781788473064> Javaccino founder (Java/.NET service - contact via linkedin) Le mer. 4 févr. 2026 à 19:59, Anton Okolnychyi <[email protected]> a écrit : > Will share the WIP PR for the new version framework by the end of today. I > will have to then split it into manageable pieces and iterate on still > failing tests. > > пн, 2 лют. 2026 р. о 15:20 Aihua Xu <[email protected]> пише: > >> Hi Anton, >> >> I want to keep track of this work item for the Iceberg 1.11.0 release. >> Can you share the status for the work? >> >> Thanks, >> Aihua >> >> On Mon, Jan 26, 2026 at 2:55 PM Romain Manni-Bucau <[email protected]> >> wrote: >> >>> Hi Anton, >>> >>> Didn't want to say the work done was 1h, I literally meant the fix to >>> make it work is 1h of work. >>> I'm not sure I get your opportunity point since deprecating/reworking >>> fundamentally something might need a new minor of iceberg for end users >>> IMHO. >>> >>> That said happy to test a PR on my use cases (mainly maintenance and SQL >>> as of today) once you think it is worth it too. >>> >>> Romain Manni-Bucau >>> @rmannibucau <https://x.com/rmannibucau> | .NET Blog >>> <https://dotnetbirdie.github.io/> | Blog >>> <https://rmannibucau.github.io/> | Old Blog >>> <http://rmannibucau.wordpress.com> | Github >>> <https://github.com/rmannibucau> | LinkedIn >>> <https://www.linkedin.com/in/rmannibucau> | Book >>> <https://www.packtpub.com/en-us/product/java-ee-8-high-performance-9781788473064> >>> Javaccino founder (Java/.NET service - contact via linkedin) >>> >>> >>> Le lun. 26 janv. 2026 à 23:27, Anton Okolnychyi <[email protected]> >>> a écrit : >>> >>>> What may seem like 1 hr of work is actually quite complicated. I am >>>> sure Manu dedicated a lot of time in setting the new module and validating >>>> all failed tests, not to mention fixes that were made after testing Spark >>>> RCs. >>>> >>>> Anyway, a new Spark release is always a rare opportunity to >>>> deprecate/improve something fundamentally. In Spark 4.1, one of these >>>> things is the new version framework in DSv2 that requires us to rethink a >>>> large chunk of the connector. We shouldn't rush this. We will see >>>> pre-requisite PRs this week. Help with reviews is always appreciated. >>>> >>>> сб, 17 січ. 2026 р. о 01:02 Romain Manni-Bucau <[email protected]> >>>> пише: >>>> >>>>> Hi JB, >>>>> >>>>> As said it is 100% an user standpoint and since enbling it is >>>>> literally 1h I think it is worth if next week is missed but still happy to >>>>> avoir. >>>>> >>>>> >>>>> Romain Manni-Bucau >>>>> @rmannibucau <https://x.com/rmannibucau> | .NET Blog >>>>> <https://dotnetbirdie.github.io/> | Blog >>>>> <https://rmannibucau.github.io/> | Old Blog >>>>> <http://rmannibucau.wordpress.com> | Github >>>>> <https://github.com/rmannibucau> | LinkedIn >>>>> <https://www.linkedin.com/in/rmannibucau> | Book >>>>> <https://www.packtpub.com/en-us/product/java-ee-8-high-performance-9781788473064> >>>>> Javaccino founder (Java/.NET service - contact via linkedin) >>>>> >>>>> Le sam. 17 janv. 2026, 06:21, Jean-Baptiste Onofré <[email protected]> >>>>> a écrit : >>>>> >>>>>> Hi Romain >>>>>> >>>>>> I'm not in favor of 1.10.2 with Spark 4.1 support. >>>>>> >>>>>> As I think we are pretty close to having the cleanup/enhancement done >>>>>> on main, by chance, it will be included in 1.11.0, so worth to wait for >>>>>> it. >>>>>> >>>>>> Regards >>>>>> JB >>>>>> >>>>>> On Fri, Jan 16, 2026 at 10:06 PM Romain Manni-Bucau < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> @Steven sure, my point was just that delaying Spark 4.1 support to >>>>>>> get new feature can not be the best so branch latest 1.10 tag and just >>>>>>> fixing the 2-3 compilation issues to get a 1.10.2 and Spark 4.1 working >>>>>>> can >>>>>>> be an intermediate step. >>>>>>> >>>>>>> let's see next week what Anton comes back with :) >>>>>>> >>>>>>> Romain Manni-Bucau >>>>>>> @rmannibucau <https://x.com/rmannibucau> | .NET Blog >>>>>>> <https://dotnetbirdie.github.io/> | Blog >>>>>>> <https://rmannibucau.github.io/> | Old Blog >>>>>>> <http://rmannibucau.wordpress.com> | Github >>>>>>> <https://github.com/rmannibucau> | LinkedIn >>>>>>> <https://www.linkedin.com/in/rmannibucau> | Book >>>>>>> <https://www.packtpub.com/en-us/product/java-ee-8-high-performance-9781788473064> >>>>>>> Javaccino founder (Java/.NET service - contact via linkedin) >>>>>>> >>>>>>> >>>>>>> Le ven. 16 janv. 2026 à 20:28, Steven Wu <[email protected]> a >>>>>>> écrit : >>>>>>> >>>>>>>> Romain, patch releases (like 1.10.2) are only for bug fixes. New >>>>>>>> features (like Spark 4.1) would have to happen in minor version >>>>>>>> releases >>>>>>>> (like 1.11.0) >>>>>>>> >>>>>>>> On Mon, Jan 12, 2026 at 9:22 AM Romain Manni-Bucau < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> Ok, good news Anton, if it is longer than expected (hoping not) >>>>>>>>> maybe a 1.10.2 with the new module be a compromise, impatient to get >>>>>>>>> your >>>>>>>>> feedback next week. >>>>>>>>> >>>>>>>>> Thanks for the hard work. >>>>>>>>> >>>>>>>>> Romain Manni-Bucau >>>>>>>>> @rmannibucau <https://x.com/rmannibucau> | .NET Blog >>>>>>>>> <https://dotnetbirdie.github.io/> | Blog >>>>>>>>> <https://rmannibucau.github.io/> | Old Blog >>>>>>>>> <http://rmannibucau.wordpress.com> | Github >>>>>>>>> <https://github.com/rmannibucau> | LinkedIn >>>>>>>>> <https://www.linkedin.com/in/rmannibucau> | Book >>>>>>>>> <https://www.packtpub.com/en-us/product/java-ee-8-high-performance-9781788473064> >>>>>>>>> Javaccino founder (Java/.NET service - contact via linkedin) >>>>>>>>> >>>>>>>>> >>>>>>>>> Le lun. 12 janv. 2026 à 17:54, Anton Okolnychyi < >>>>>>>>> [email protected]> a écrit : >>>>>>>>> >>>>>>>>>> We already have a module for Spark 4.1 in main. That said, Spark >>>>>>>>>> 4.1 has introduced substantial improvements in table lifecycle that >>>>>>>>>> should >>>>>>>>>> let us remove some legacy hacks that may cause data correctness >>>>>>>>>> issues. >>>>>>>>>> Given that this work may result in some properties being no longer >>>>>>>>>> supported, we can do that only in the first Iceberg release for >>>>>>>>>> Spark 4.1. >>>>>>>>>> I would strongly oppose releasing support for Spark 4.1 prior to >>>>>>>>>> embracing >>>>>>>>>> the new version framework in DSv2. >>>>>>>>>> >>>>>>>>>> Good news, it still shouldn't take too long. I am working on a >>>>>>>>>> prototype as we speak. It should be available next week. >>>>>>>>>> >>>>>>>>>> - Anton >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> пн, 12 січ. 2026 р. о 07:33 Romain Manni-Bucau < >>>>>>>>>> [email protected]> пише: >>>>>>>>>> >>>>>>>>>>> Hi everyone, >>>>>>>>>>> >>>>>>>>>>> I wonder if you do have some plan to release Iceberg 1.11 (I >>>>>>>>>>> assume next one?) with Spark 4.1 support. >>>>>>>>>>> The datasource v2 API breakage was very localized but is just >>>>>>>>>>> blocking to upgrade and some spark 4.1 features are game changers >>>>>>>>>>> (like the >>>>>>>>>>> SQL enhancement) so I'm really looking forward being able to move >>>>>>>>>>> on it >>>>>>>>>>> with iceberg support. >>>>>>>>>>> If you need some help please let me know. >>>>>>>>>>> >>>>>>>>>>> Romain Manni-Bucau >>>>>>>>>>> @rmannibucau <https://x.com/rmannibucau> | .NET Blog >>>>>>>>>>> <https://dotnetbirdie.github.io/> | Blog >>>>>>>>>>> <https://rmannibucau.github.io/> | Old Blog >>>>>>>>>>> <http://rmannibucau.wordpress.com> | Github >>>>>>>>>>> <https://github.com/rmannibucau> | LinkedIn >>>>>>>>>>> <https://www.linkedin.com/in/rmannibucau> | Book >>>>>>>>>>> <https://www.packtpub.com/en-us/product/java-ee-8-high-performance-9781788473064> >>>>>>>>>>> Javaccino founder (Java/.NET service - contact via linkedin) >>>>>>>>>>> >>>>>>>>>>
