-1 (non-binding) for me as well There are several critical fixes for performance, correctness, functionality, etc. (I heard at least 3 critical fixes.) While I don't see them to be a new regression on 4.3.0, there are a meaningful number of fixes warranted to port back to the 4.3 branch. I think an accumulation of those patches warrants enough value to not ship 4.3.0 as it is.
In the meanwhile, I'm also looking at stream-stream join correctness issue ( SPARK-58904 <https://issues.apache.org/jira/browse/SPARK-58904>); that's also a sort of long standing problem, but the new join state format version (v4) introduced in 4.3.0 is impacted by this bug so I hope we fix this before releasing. It's not a blocker for 4.3.0 on usual judgement but I'll see whether I can make the fix in the short time while the API audit is happening. That said, I'll cancel the VOTE for RC2. We will have RC3 once the API audit is settled. (stream-stream join correctness issue is a best effort) On Thu, Oct 15, 2026 at 6:14 AM Xiao Li <[email protected]> wrote: > -1 > > I think we should complete an API audit for Spark 4.3 before release, similar > to the audit performed for the previous releases > > I found the following issues in RC2. > > 1. Accidental public API exposure > > ShuffleLocationResponse is public and appears in the generated ScalaDoc, > although it belongs to an internal implementation and exposes the > private[spark] StreamingShuffleTaskLocation > type:https://github.com/apache/spark/blob/v4.3.0-rc2/core/src/main/scala/org/apache/spark/StreamingShuffleOutputTracker.scala#L128-L131 > > It likely should be marked private[spark] before the release. > > 2. APIs missing from the reference documentation > > - pyspark.sql.functions.collect_union is exported but absent from the PySpark > SQL function reference: > > https://github.com/apache/spark/blob/v4.3.0-rc2/python/pyspark/sql/functions/__init__.py#L372 > > https://github.com/apache/spark/blob/v4.3.0-rc2/python/docs/source/reference/pyspark.sql.functions.rst > > - The following pandas-on-Spark APIs are missing from the window reference: > - Rolling.sem > - RollingGroupby.median > - Expanding.sem > - ExpandingGroupby.median > > Implementations: > > https://github.com/apache/spark/blob/v4.3.0-rc2/python/pyspark/pandas/window.py#L859 > > https://github.com/apache/spark/blob/v4.3.0-rc2/python/pyspark/pandas/window.py#L1462 > > https://github.com/apache/spark/blob/v4.3.0-rc2/python/pyspark/pandas/window.py#L2088 > > https://github.com/apache/spark/blob/v4.3.0-rc2/python/pyspark/pandas/window.py#L2611 > > Reference: > > https://github.com/apache/spark/blob/v4.3.0-rc2/python/docs/source/reference/pyspark.pandas/window.rst > > - Seven vector functions are absent from the SQL built-in function reference: > vector_cosine_similarity, vector_inner_product, vector_l2_distance, > vector_norm, vector_normalize, vector_avg, and vector_sum. > > APIs: > > https://github.com/apache/spark/blob/v4.3.0-rc2/sql/api/src/main/scala/org/apache/spark/sql/functions.scala#L17597 > > Reference: > > https://github.com/apache/spark/blob/v4.3.0-rc2/docs/sql-ref-functions-builtin.md > > 3. Incorrect version metadata > > - KMeans.setIntermediateStorageLevel says it was added in 5.0.0, but it is > new in 4.3.0: > > https://github.com/apache/spark/blob/v4.3.0-rc2/mllib/src/main/scala/org/apache/spark/ml/clustering/KMeans.scala#L445 > > https://github.com/apache/spark/blob/v4.3.0-rc2/python/pyspark/ml/clustering.py#L926 > > - TimestampLTZNanosType and TimestampNTZNanosType say @since 4.2.0, but they > are new in 4.3.0: > > https://github.com/apache/spark/blob/v4.3.0-rc2/sql/api/src/main/scala/org/apache/spark/sql/types/TimestampLTZNanosType.scala#L34 > > https://github.com/apache/spark/blob/v4.3.0-rc2/sql/api/src/main/scala/org/apache/spark/sql/types/TimestampNTZNanosType.scala#L34 > > Once released, an accidental public API becomes much harder to retract. I > recommend creating a Spark 4.3 API-audit umbrella, fixing these findings, and > producing RC3. > > Xiao > > > Holden Karau <[email protected]> 于2026年10月14日周三 12:59写道： > >> Moved to +1 since it’s only a regression from 4.1 not 4.2, but I still >> think we should consider if an RC3 is appropriate for this fix. >> >> >> Twitter: https://twitter.com/holdenkarau >> Fight Health Insurance: https://www.fighthealthinsurance.com/ >> <https://www.fighthealthinsurance.com/?q=hk_email> >> Books (Learning Spark, High Performance Spark, etc.): >> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >> Pronouns: she/her >> >> On Wed, Oct 14, 2026 at 11:21 AM Holden Karau <[email protected]> >> wrote: >> >>> -1: There's a PyUDF Arrow failure on Docker reported by Lisa (which is >>> enabled by default now), and apparently the fallback doesn't function >>> properly in Connect so its a bad one >>> https://github.com/apache/spark/pull/59280 >>> >>> On Wed, Oct 14, 2026 at 2:08 AM Nicholas Chammas < >>> [email protected]> wrote: >>> >>>> +1 >>>> >>>> I ran the additional distribution tests from #57645 >>>> <https://github.com/apache/spark/pull/57645>. They all passed. >>>> >>>> >>>> On Oct 13, 2026, at 4:32 AM, Wenchen Fan <[email protected]> wrote: >>>> >>>> +1 >>>> >>>> On Tue, Oct 13, 2026 at 8:31 AM <[email protected]> wrote: >>>> >>>>> Please vote on releasing the following candidate as Apache Spark >>>>> version 4.3.0. >>>>> >>>>> The vote is open until Thu, 15 Oct 2026 18:30:01 PDT and passes if a >>>>> majority +1 PMC votes are cast, with >>>>> a minimum of 3 +1 votes. >>>>> >>>>> [ ] +1 Release this package as Apache Spark 4.3.0 >>>>> [ ] -1 Do not release this package because ... >>>>> >>>>> To learn more about Apache Spark, please see https://spark.apache.org/ >>>>> >>>>> The tag to be voted on is v4.3.0-rc2 (commit c0fd65d3f4f): >>>>> https://github.com/apache/spark/tree/v4.3.0-rc2 >>>>> >>>>> The release files, including signatures, digests, etc. can be found at: >>>>> https://dist.apache.org/repos/dist/dev/spark/v4.3.0-rc2-bin/ >>>>> >>>>> Signatures used for Spark RCs can be found in this file: >>>>> https://downloads.apache.org/spark/KEYS >>>>> >>>>> The staging repository for this release can be found at: >>>>> https://repository.apache.org/content/repositories/orgapachespark-1538/ >>>>> >>>>> The documentation corresponding to this release can be found at: >>>>> https://dist.apache.org/repos/dist/dev/spark/v4.3.0-rc2-docs/ >>>>> >>>>> The list of bug fixes going into 4.3.0 can be found at the following >>>>> URL: >>>>> https://issues.apache.org/jira/projects/SPARK/versions/12356944 >>>>> >>>>> FAQ >>>>> >>>>> ========================= >>>>> How can I help test this release? >>>>> ========================= >>>>> >>>>> If you are a Spark user, you can help us test this release by taking >>>>> an existing Spark workload and running on this release candidate, then >>>>> reporting any regressions. >>>>> >>>>> If you're working in PySpark you can set up a virtual env and install >>>>> the current RC via "pip install >>>>> https://dist.apache.org/repos/dist/dev/spark/v4.3.0-rc2-bin/pyspark-4.3.0.tar.gz >>>>> " >>>>> and see if anything important breaks. >>>>> In the Java/Scala, you can add the staging repository to your >>>>> project's resolvers and test >>>>> with the RC (make sure to clean up the artifact cache before/after so >>>>> you don't end up building with an out of date RC going forward). >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe e-mail: [email protected] >>>>> >>>>> >>>> >>> >>> -- >>> Twitter: https://twitter.com/holdenkarau >>> Fight Health Insurance: https://www.fighthealthinsurance.com/ >>> <https://www.fighthealthinsurance.com/?q=hk_email> >>> Books (Learning Spark, High Performance Spark, etc.): >>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> >>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau >>> Pronouns: she/her >>> >>
