Wenchen and I are chatting about TableViewCatalog offline right now. We will try one idea but it is not clear if it is going to work yet.
I also found that the column ID work that we planned to ship in 4.2 has some issues w.r.t to nested columns. If released, we will have to maintain this behavior contract / API forever. I submitted SPARK-57544 for this and will share a WIP PR later today. - Anton чт, 18 черв. 2026 р. о 15:07 huaxin gao <[email protected]> пише: > Thanks Anton. I’ll wait for your update before cutting RC4. Please let me > know if you find anything that should block the release. > > Best, > > Huaxin > > On Wed, Jun 17, 2026 at 4:24 PM Anton Okolnychyi <[email protected]> > wrote: > >> After reviewing Iceberg's PR for integrating Spark 4.2, I have some >> questions about the recently added TableViewCatalog. I am trying to >> understand if it changes the contract / behavior for connectors. Will >> report back here tomorrow. No new issues detected so far. >> >> вт, 16 черв. 2026 р. о 17:08 huaxin gao <[email protected]> пише: >> >>> Hi all, >>> >>> Thanks Wenchen for reviewing RC3 and raising the API doc issue. >>> >>> Given the -1, I will cancel RC3 and prepare RC4 after the API doc fix >>> is merged and backported to branch-4.2. I plan to cut RC4 on Thursday. >>> >>> Thanks, >>> Huaxin >>> >>> On Tue, Jun 16, 2026 at 4:20 PM Holden Karau <[email protected]> >>> wrote: >>> >>>> Also it's Data+AI summit so I suspect a lot of folks are busy with that >>>> right now. >>>> >>>> On Tue, Jun 16, 2026 at 4:05 PM Wenchen Fan <[email protected]> >>>> wrote: >>>> >>>>> No open bug report for 4.2 so far, but I found an issue in the API >>>>> doc: https://github.com/apache/spark/pull/56551 . The generated proto >>>>> java classes and the new internal Types Framework ops classes are leaked >>>>> into the Scala API doc. >>>>> >>>>> Given we claim APIs in the API doc as public, I'm -1 to avoid >>>>> releasing unexpected public APIs. >>>>> >>>>> On Mon, Jun 15, 2026 at 6:38 PM huaxin gao <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> The RC3 voting deadline has passed, but we have not received any >>>>>> votes yet. >>>>>> >>>>>> Since the previous 72-hour voting period included a weekend, I’ll >>>>>> keep the vote open and extend it for another 48 hours. Could community >>>>>> members please help review RC3 and vote when you have a chance? Binding >>>>>> votes from PMC members are especially needed to close the release vote. >>>>>> >>>>>> Also, SPARK-57452 <https://issues.apache.org/jira/browse/SPARK-57452> >>>>>> was raised to audit the 4.2 migration guide for potentially missing >>>>>> behavior-change notes, and it is currently marked as a blocker. Could the >>>>>> owners of the changes listed in SPARK-57452 please review the >>>>>> corresponding >>>>>> items and confirm whether any of them should block RC3? If you are aware >>>>>> of >>>>>> any other 4.2 behavior changes that require migration-guide updates, >>>>>> please >>>>>> also flag them. >>>>>> >>>>>> If you believe SPARK-57452 should block the release, please vote -1 >>>>>> and share the specific items that must be fixed before 4.2.0. Otherwise, >>>>>> please help verify RC3 and vote. >>>>>> >>>>>> Thanks, >>>>>> Huaxin >>>>>> >>>>>> On Fri, Jun 12, 2026 at 5:14 PM <[email protected]> wrote: >>>>>> >>>>>>> Please vote on releasing the following candidate as Apache Spark >>>>>>> version 4.2.0. >>>>>>> >>>>>>> The vote is open until Mon, 15 Jun 2026 18:14:21 PDT and passes if a >>>>>>> majority +1 PMC votes are cast, with >>>>>>> a minimum of 3 +1 votes. >>>>>>> >>>>>>> [ ] +1 Release this package as Apache Spark 4.2.0 >>>>>>> [ ] -1 Do not release this package because ... >>>>>>> >>>>>>> To learn more about Apache Spark, please see >>>>>>> https://spark.apache.org/ >>>>>>> >>>>>>> The tag to be voted on is v4.2.0-rc3 (commit 560dc9d3c95): >>>>>>> https://github.com/apache/spark/tree/v4.2.0-rc3 >>>>>>> >>>>>>> The release files, including signatures, digests, etc. can be found >>>>>>> at: >>>>>>> https://dist.apache.org/repos/dist/dev/spark/v4.2.0-rc3-bin/ >>>>>>> >>>>>>> Signatures used for Spark RCs can be found in this file: >>>>>>> https://downloads.apache.org/spark/KEYS >>>>>>> >>>>>>> The staging repository for this release can be found at: >>>>>>> >>>>>>> https://repository.apache.org/content/repositories/orgapachespark-1523/ >>>>>>> >>>>>>> The documentation corresponding to this release can be found at: >>>>>>> https://dist.apache.org/repos/dist/dev/spark/v4.2.0-rc3-docs/ >>>>>>> >>>>>>> The list of bug fixes going into 4.2.0 can be found at the following >>>>>>> URL: >>>>>>> https://issues.apache.org/jira/projects/SPARK/versions/12356380 >>>>>>> >>>>>>> FAQ >>>>>>> >>>>>>> ========================= >>>>>>> How can I help test this release? >>>>>>> ========================= >>>>>>> >>>>>>> If you are a Spark user, you can help us test this release by taking >>>>>>> an existing Spark workload and running on this release candidate, >>>>>>> then >>>>>>> reporting any regressions. >>>>>>> >>>>>>> If you're working in PySpark you can set up a virtual env and install >>>>>>> the current RC via "pip install >>>>>>> https://dist.apache.org/repos/dist/dev/spark/v4.2.0-rc3-bin/pyspark-4.2.0.tar.gz >>>>>>> " >>>>>>> and see if anything important breaks. >>>>>>> In the Java/Scala, you can add the staging repository to your >>>>>>> project's resolvers and test >>>>>>> with the RC (make sure to clean up the artifact cache before/after so >>>>>>> you don't end up building with an out of date RC going forward). >>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> To unsubscribe e-mail: [email protected] >>>>>>> >>>>>>> >>>> >>>> -- >>>> Twitter: https://twitter.com/holdenkarau >>>> Fight Health Insurance: https://www.fighthealthinsurance.com/ >>>> <https://www.fighthealthinsurance.com/?q=hk_email> >>>> Books (Learning Spark, High Performance Spark, etc.): >>>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>>> Pronouns: she/her >>>> >>>
