FYI, SPARK-57774 https://github.com/apache/spark/pull/56890 has been landed in branch-4.2, looking forward to RC5.
Thanks, Cheng Pan > On Jul 1, 2026, at 08:40, huaxin gao <[email protected]> wrote: > > Hi all, > > > The WKB fix has been merged and backported to branch-4.2, so I'll cut RC5 > soon. > > > As discussed, the log4j version upgrade is not a blocker and won't hold RC5. > The DriverLogger workaround (SPARK-57774 > https://github.com/apache/spark/pull/56890) is a small Spark-side fix, so I'm > OK to include it in RC5 unless there are any objections. > > Thanks, > > Huaxin > > > On Mon, Jun 29, 2026 at 11:14 PM Cheng Pan <[email protected] > <mailto:[email protected]>> wrote: >> After taking a closer look, I think at least the Spark built-in DriverLogger >> are affected - it sets ignoreExceptions=false, which means that log4j >> internal exceptions will propagate to the caller and crash the call site, >> that's why we have observed UT failed in GHA occasionally. For other logging >> places, it depends on whether user to set ignoreExceptions=false (default is >> true, safe, no exception propagation). >> >> I opened SPARK-57774 https://github.com/apache/spark/pull/56890 to fix the >> DriverLogger ignoreExceptions issue. >> >> Thanks, >> Cheng Pan >> >> >> >>> On Jun 30, 2026, at 12:59, Yang Jie <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> Will this NPE make Spark 4.2 completely unusable? If not, dependency >>> upgrades theoretically should not be backported to the 4.2.x branch, as >>> third-party dependencies for the entire 4.x line have been frozen, correct? >>> >>> Jie Yang >>> >>> On 2026/06/30 02:35:43 huaxin gao wrote: >>>> Thanks Cheng for flagging this! >>>> >>>> I prefer not to delay RC5 (which is for the WKB security fix) just to wait >>>> for an outside dependency. Also, 2.26.1 just started voting, so we can't >>>> use it until it is on Maven Central (about 4-5 days). So I will cut RC5 >>>> with the current log4j. If RC5 fails, we can include 2.26.1, otherwise >>>> let's include it in 4.2.1. >>>> >>>> Thanks, >>>> >>>> Huaxin >>>> >>>> On Mon, Jun 29, 2026 at 6:57 PM Cheng Pan <[email protected] >>>> <mailto:[email protected]>> wrote: >>>> >>>>> Can we include the log4j 2.26.1 upgrading in next RC? >>>>> >>>>> TL:DR, the current log4j2 2.25.x used by branch-4.2 may throw NPE on >>>>> logging, downgrading to 2.24.x can fix the issue but bring some CVEs back, >>>>> log4j2 2.26.1 is in the voting phase and likely available soon. See full >>>>> context at [1] >>>>> >>>>> [1] https://github.com/apache/spark/pull/51719#issuecomment-3341344974 >>>>> >>>>> Thanks, >>>>> Cheng Pan >>>>> >>>>> >>>>> >>>>> On Jun 30, 2026, at 08:52, huaxin gao <[email protected] >>>>> <mailto:[email protected]>> wrote: >>>>> >>>>> Thanks Szehon for reporting the problem. I'm failing RC4, and will roll >>>>> RC5 once the fix <https://github.com/apache/spark/pull/56875> is merged >>>>> and backported to branch-4.2. >>>>> >>>>> Thanks to everyone who tested RC4! >>>>> >>>>> Huaxin >>>>> >>>>> On Mon, Jun 29, 2026 at 5:14 PM Szehon Ho <[email protected] >>>>> <mailto:[email protected]>> wrote: >>>>> >>>>>> I made a fix for this, it would be great to get it into the release. >>>>>> >>>>>> https://github.com/apache/spark/pull/56875 >>>>>> >>>>>> Thanks Huaxin! >>>>>> Szehon >>>>>> >>>>>> >>>>>> On Mon, Jun 29, 2026 at 4:54 PM Szehon Ho <[email protected] >>>>>> <mailto:[email protected]>> >>>>>> wrote: >>>>>> >>>>>>> Sorry I was informed of a potential vulnerability/problem in new fromWKB >>>>>>> in Spark 4.2, looking now at it. >>>>>>> >>>>>>> Thanks >>>>>>> Szehon >>>>>>> >>>>>>> On Mon, Jun 29, 2026 at 1:05 PM Allison Wang <[email protected] >>>>>>> <mailto:[email protected]>> >>>>>>> wrote: >>>>>>> >>>>>>>> +1 >>>>>>>> >>>>>>>> On Mon, Jun 29, 2026 at 10:34 AM Max Gekk <[email protected] >>>>>>>> <mailto:[email protected]>> wrote: >>>>>>>> >>>>>>>>> +1 >>>>>>>>> >>>>>>>>> On Mon, Jun 29, 2026 at 6:32 PM Xiao Li <[email protected] >>>>>>>>> <mailto:[email protected]>> wrote: >>>>>>>>>> >>>>>>>>>> +1 (binding) >>>>>>>>>> >>>>>>>>>> Verified on Linux (Ubuntu, JDK 17, Python 3.11): >>>>>>>>>> >>>>>>>>>> - Signatures: every artifact (source, the three binary >>>>>>>>> distributions, the pyspark/pyspark_client/pyspark_connect tarballs, >>>>>>>>> and >>>>>>>>> SparkR) has a good GPG signature from the RM key >>>>>>>>> 709226B910E0F10917123B6259B586ADA5A538D1, which is in KEYS. >>>>>>>>>> - Checksums: SHA512 matches for all artifacts.- Tag/commit: >>>>>>>>> v4.2.0-rc4 resolves to f92a807c06b, and the RELEASE metadata embedded >>>>>>>>> in >>>>>>>>> the binary distributions records the same git revision. >>>>>>>>>> - Source build: compiled and packaged spark-core and its module >>>>>>>>> dependencies from the source tarball with ./build/mvn on JDK 17 (BUILD >>>>>>>>> SUCCESS). >>>>>>>>>> - Binary distribution: ran SparkPi and a spark-shell Scala job >>>>>>>>> (range aggregation + Spark SQL) — results correct. >>>>>>>>>> - PySpark: pip-installed pyspark-4.2.0.tar.gz and ran DataFrame, >>>>>>>>> Spark SQL, and a Python UDF in local mode — results correct. >>>>>>>>>> >>>>>>>>>> Thanks Huaxin for driving the release! >>>>>>>>>> >>>>>>>>>> Xiao >>>>>>>>>> >>>>>>>>>> Uroš Bojanić <[email protected] <mailto:[email protected]>> >>>>>>>>>> 于2026年6月29日周一 09:12写道： >>>>>>>>>>> >>>>>>>>>>> +1 (non-binding) >>>>>>>>>>> >>>>>>>>>>> verified RC4 (tag v4.2.0-rc4, commit f92a807c06b) on macOS/arm64. >>>>>>>>>>> >>>>>>>>>>> - Good GPG signatures and SHA512 sums on all artifacts (source, >>>>>>>>> three binaries, the PySpark/SparkR tarballs) against KEYS. >>>>>>>>>>> - Built the full source tree from the tag with -Phive >>>>>>>>> -Phive-thriftserver, got a clean BUILD SUCCESS across all 39 modules. >>>>>>>>>>> - Ran quick smoke tests across every API surface: Java, Scala, SQL, >>>>>>>>> and PySpark job from the bundled distro; all look good. >>>>>>>>>>> - Sanity-checked the binary dist (RELEASE, LICENSE/NOTICE/licenses >>>>>>>>> all present) and ran dev/check-license; RAT passes. >>>>>>>>>>> - Diffed the docs against 4.1.0 and analyzed the changes (new >>>>>>>>> pages, migration guides and version refs); all look good. >>>>>>>>>>> >>>>>>>>>>> Thank you Huaxin Gao! >>>>>>>>>>> >>>>>>>>>>> On 2026/06/27 00:21:32 [email protected] >>>>>>>>>>> <mailto:[email protected]> wrote: >>>>>>>>>>>> Please vote on releasing the following candidate as Apache Spark >>>>>>>>> version 4.2.0. >>>>>>>>>>>> >>>>>>>>>>>> The vote is open until Mon, 29 Jun 2026 18:21:31 PDT and passes >>>>>>>>> if a majority +1 PMC votes are cast, with >>>>>>>>>>>> a minimum of 3 +1 votes. >>>>>>>>>>>> >>>>>>>>>>>> [ ] +1 Release this package as Apache Spark 4.2.0 >>>>>>>>>>>> [ ] -1 Do not release this package because ... >>>>>>>>>>>> >>>>>>>>>>>> To learn more about Apache Spark, please see >>>>>>>>> https://spark.apache.org/ >>>>>>>>>>>> >>>>>>>>>>>> The tag to be voted on is v4.2.0-rc4 (commit f92a807c06b): >>>>>>>>>>>> https://github.com/apache/spark/tree/v4.2.0-rc4 >>>>>>>>>>>> >>>>>>>>>>>> The release files, including signatures, digests, etc. can be >>>>>>>>> found at: >>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/spark/v4.2.0-rc4-bin/ >>>>>>>>>>>> >>>>>>>>>>>> Signatures used for Spark RCs can be found in this file: >>>>>>>>>>>> https://downloads.apache.org/spark/KEYS >>>>>>>>>>>> >>>>>>>>>>>> The staging repository for this release can be found at: >>>>>>>>>>>> >>>>>>>>> https://repository.apache.org/content/repositories/orgapachespark-1524/ >>>>>>>>>>>> >>>>>>>>>>>> The documentation corresponding to this release can be found at: >>>>>>>>>>>> https://dist.apache.org/repos/dist/dev/spark/v4.2.0-rc4-docs/ >>>>>>>>>>>> >>>>>>>>>>>> The list of bug fixes going into 4.2.0 can be found at the >>>>>>>>> following URL: >>>>>>>>>>>> https://issues.apache.org/jira/projects/SPARK/versions/12356380 >>>>>>>>>>>> >>>>>>>>>>>> FAQ >>>>>>>>>>>> >>>>>>>>>>>> ========================= >>>>>>>>>>>> How can I help test this release? >>>>>>>>>>>> ========================= >>>>>>>>>>>> >>>>>>>>>>>> If you are a Spark user, you can help us test this release by >>>>>>>>> taking >>>>>>>>>>>> an existing Spark workload and running on this release candidate, >>>>>>>>> then >>>>>>>>>>>> reporting any regressions. >>>>>>>>>>>> >>>>>>>>>>>> If you're working in PySpark you can set up a virtual env and >>>>>>>>> install >>>>>>>>>>>> the current RC via "pip install >>>>>>>>> https://dist.apache.org/repos/dist/dev/spark/v4.2.0-rc4-bin/pyspark-4.2.0.tar.gz >>>>>>>>> " >>>>>>>>>>>> and see if anything important breaks. >>>>>>>>>>>> In the Java/Scala, you can add the staging repository to your >>>>>>>>> project's resolvers and test >>>>>>>>>>>> with the RC (make sure to clean up the artifact cache >>>>>>>>> before/after so >>>>>>>>>>>> you don't end up building with an out of date RC going forward). >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>>> To unsubscribe e-mail: [email protected] >>>>>>>>>>>> <mailto:[email protected]> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>> To unsubscribe e-mail: [email protected] >>>>>>>>>>> <mailto:[email protected]> >>>>>>>>>>> >>>>>>>>> >>>>>>>>> --------------------------------------------------------------------- >>>>>>>>> To unsubscribe e-mail: [email protected] >>>>>>>>> <mailto:[email protected]> >>>>>>>>> >>>>>>>>> >>>>> >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe e-mail: [email protected] >>> <mailto:[email protected]>
