-1 I think we should complete an API audit for Spark 4.3 before release, similar to the audit performed for the previous releases
I found the following issues in RC2. 1. Accidental public API exposure ShuffleLocationResponse is public and appears in the generated ScalaDoc, although it belongs to an internal implementation and exposes the private[spark] StreamingShuffleTaskLocation type:https://github.com/apache/spark/blob/v4.3.0-rc2/core/src/main/scala/org/apache/spark/StreamingShuffleOutputTracker.scala#L128-L131 It likely should be marked private[spark] before the release. 2. APIs missing from the reference documentation - pyspark.sql.functions.collect_union is exported but absent from the PySpark SQL function reference: https://github.com/apache/spark/blob/v4.3.0-rc2/python/pyspark/sql/functions/__init__.py#L372 https://github.com/apache/spark/blob/v4.3.0-rc2/python/docs/source/reference/pyspark.sql.functions.rst - The following pandas-on-Spark APIs are missing from the window reference: - Rolling.sem - RollingGroupby.median - Expanding.sem - ExpandingGroupby.median Implementations: https://github.com/apache/spark/blob/v4.3.0-rc2/python/pyspark/pandas/window.py#L859 https://github.com/apache/spark/blob/v4.3.0-rc2/python/pyspark/pandas/window.py#L1462 https://github.com/apache/spark/blob/v4.3.0-rc2/python/pyspark/pandas/window.py#L2088 https://github.com/apache/spark/blob/v4.3.0-rc2/python/pyspark/pandas/window.py#L2611 Reference: https://github.com/apache/spark/blob/v4.3.0-rc2/python/docs/source/reference/pyspark.pandas/window.rst - Seven vector functions are absent from the SQL built-in function reference: vector_cosine_similarity, vector_inner_product, vector_l2_distance, vector_norm, vector_normalize, vector_avg, and vector_sum. APIs: https://github.com/apache/spark/blob/v4.3.0-rc2/sql/api/src/main/scala/org/apache/spark/sql/functions.scala#L17597 Reference: https://github.com/apache/spark/blob/v4.3.0-rc2/docs/sql-ref-functions-builtin.md 3. Incorrect version metadata - KMeans.setIntermediateStorageLevel says it was added in 5.0.0, but it is new in 4.3.0: https://github.com/apache/spark/blob/v4.3.0-rc2/mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala#L445 https://github.com/apache/spark/blob/v4.3.0-rc2/python/pyspark/ml/clustering.py#L926 - TimestampLTZNanosType and TimestampNTZNanosType say @since 4.2.0, but they are new in 4.3.0: https://github.com/apache/spark/blob/v4.3.0-rc2/sql/api/src/main/scala/org/apache/spark/sql/types/TimestampLTZNanosType.scala#L34 https://github.com/apache/spark/blob/v4.3.0-rc2/sql/api/src/main/scala/org/apache/spark/sql/types/TimestampNTZNanosType.scala#L34 Once released, an accidental public API becomes much harder to retract. I recommend creating a Spark 4.3 API-audit umbrella, fixing these findings, and producing RC3. Xiao Holden Karau <[email protected]> 于2026年10月14日周三 12:59写道： > Moved to +1 since it’s only a regression from 4.1 not 4.2, but I still > think we should consider if an RC3 is appropriate for this fix. > > > Twitter: https://twitter.com/holdenkarau > Fight Health Insurance: https://www.fighthealthinsurance.com/ > <https://www.fighthealthinsurance.com/?q=hk_email> > Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau > Pronouns: she/her > > On Wed, Oct 14, 2026 at 11:21 AM Holden Karau <[email protected]> > wrote: > >> -1: There's a PyUDF Arrow failure on Docker reported by Lisa (which is >> enabled by default now), and apparently the fallback doesn't function >> properly in Connect so its a bad one >> https://github.com/apache/spark/pull/59280 >> >> On Wed, Oct 14, 2026 at 2:08 AM Nicholas Chammas < >> [email protected]> wrote: >> >>> +1 >>> >>> I ran the additional distribution tests from #57645 >>> <https://github.com/apache/spark/pull/57645>. They all passed. >>> >>> >>> On Oct 13, 2026, at 4:32 AM, Wenchen Fan <[email protected]> wrote: >>> >>> +1 >>> >>> On Tue, Oct 13, 2026 at 8:31 AM <[email protected]> wrote: >>> >>>> Please vote on releasing the following candidate as Apache Spark >>>> version 4.3.0. >>>> >>>> The vote is open until Thu, 15 Oct 2026 18:30:01 PDT and passes if a >>>> majority +1 PMC votes are cast, with >>>> a minimum of 3 +1 votes. >>>> >>>> [ ] +1 Release this package as Apache Spark 4.3.0 >>>> [ ] -1 Do not release this package because ... >>>> >>>> To learn more about Apache Spark, please see https://spark.apache.org/ >>>> >>>> The tag to be voted on is v4.3.0-rc2 (commit c0fd65d3f4f): >>>> https://github.com/apache/spark/tree/v4.3.0-rc2 >>>> >>>> The release files, including signatures, digests, etc. can be found at: >>>> https://dist.apache.org/repos/dist/dev/spark/v4.3.0-rc2-bin/ >>>> >>>> Signatures used for Spark RCs can be found in this file: >>>> https://downloads.apache.org/spark/KEYS >>>> >>>> The staging repository for this release can be found at: >>>> https://repository.apache.org/content/repositories/orgapachespark-1538/ >>>> >>>> The documentation corresponding to this release can be found at: >>>> https://dist.apache.org/repos/dist/dev/spark/v4.3.0-rc2-docs/ >>>> >>>> The list of bug fixes going into 4.3.0 can be found at the following >>>> URL: >>>> https://issues.apache.org/jira/projects/SPARK/versions/12356944 >>>> >>>> FAQ >>>> >>>> ========================= >>>> How can I help test this release? >>>> ========================= >>>> >>>> If you are a Spark user, you can help us test this release by taking >>>> an existing Spark workload and running on this release candidate, then >>>> reporting any regressions. >>>> >>>> If you're working in PySpark you can set up a virtual env and install >>>> the current RC via "pip install >>>> https://dist.apache.org/repos/dist/dev/spark/v4.3.0-rc2-bin/pyspark-4.3.0.tar.gz >>>> " >>>> and see if anything important breaks. >>>> In the Java/Scala, you can add the staging repository to your project's >>>> resolvers and test >>>> with the RC (make sure to clean up the artifact cache before/after so >>>> you don't end up building with an out of date RC going forward). >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe e-mail: [email protected] >>>> >>>> >>> >> >> -- >> Twitter: https://twitter.com/holdenkarau >> Fight Health Insurance: https://www.fighthealthinsurance.com/ >> <https://www.fighthealthinsurance.com/?q=hk_email> >> Books (Learning Spark, High Performance Spark, etc.): >> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >> Pronouns: she/her >> >
