Hi, I think we need to fix a simple bug in a new metric in numSourceRows, introduced in this pr: https://github.com/apache/spark/pull/52669. The author will make one soon.
Also as you know, there is a bug being fixed now that makes Merge Into with schema evolution not work with Dataframe API: https://github.com/apache/spark/pull/53207 Thanks, Szehon On Tue, Nov 25, 2025 at 11:18 AM Sandy Ryza <[email protected]> wrote: > I also unfortunately encountered a bug in the spark-pipelines CLI > that causes it to fail when the spec file is in a different directory than > the CWD. I'm not sure whether this merits a new RC, but posting here in > case. > > JIRA: https://issues.apache.org/jira/browse/SPARK-54508 > PR: https://github.com/apache/spark/pull/53219 > > On Tue, Nov 25, 2025 at 9:47 AM Sandy Ryza <[email protected]> wrote: > >> One thing I did just notice is that the pyspark.pipelines package is not >> included in the API reference docs. Are pure docs site changes part of the >> release, or is there an ability to push them independently? >> >> PR to fix: https://github.com/apache/spark/pull/53213 >> >> On Tue, Nov 25, 2025 at 9:12 AM Kumar Chinnakali < >> [email protected]> wrote: >> >>> [email protected] >>> >>> On Tue, Nov 25, 2025 at 11:05 AM Sandy Ryza <[email protected]> wrote: >>> >>>> +1 (non-binding) >>>> >>>> Installed via `pip install >>>> https://dist.apache.org/repos/dist/dev/spark/v4.1.0-rc1-bin/pyspark-4.1.0.tar.gz` >>>> and verified functionality of pyspark and spark-pipelines. >>>> >>>> On Sun, Nov 23, 2025 at 7:40 PM <[email protected]> wrote: >>>> >>>>> Please vote on releasing the following candidate as Apache Spark >>>>> version 4.1.0. >>>>> >>>>> The vote is open until Wed, 26 Nov 2025 20:40:01 PST and passes if a >>>>> majority +1 PMC votes are cast, with >>>>> a minimum of 3 +1 votes. >>>>> >>>>> [ ] +1 Release this package as Apache Spark 4.1.0 >>>>> [ ] -1 Do not release this package because ... >>>>> >>>>> To learn more about Apache Spark, please see https://spark.apache.org/ >>>>> >>>>> The tag to be voted on is v4.1.0-rc1 (commit 2808b96a1b8): >>>>> https://github.com/apache/spark/tree/v4.1.0-rc1 >>>>> >>>>> The release files, including signatures, digests, etc. can be found at: >>>>> https://dist.apache.org/repos/dist/dev/spark/v4.1.0-rc1-bin/ >>>>> >>>>> Signatures used for Spark RCs can be found in this file: >>>>> https://downloads.apache.org/spark/KEYS >>>>> >>>>> The staging repository for this release can be found at: >>>>> https://repository.apache.org/content/repositories/orgapachespark-1506/ >>>>> >>>>> The documentation corresponding to this release can be found at: >>>>> https://dist.apache.org/repos/dist/dev/spark/v4.1.0-rc1-docs/ >>>>> >>>>> The list of bug fixes going into 4.1.0 can be found at the following >>>>> URL: >>>>> https://issues.apache.org/jira/projects/SPARK/versions/12355581 >>>>> >>>>> FAQ >>>>> >>>>> ========================= >>>>> How can I help test this release? >>>>> ========================= >>>>> >>>>> If you are a Spark user, you can help us test this release by taking >>>>> an existing Spark workload and running on this release candidate, then >>>>> reporting any regressions. >>>>> >>>>> If you're working in PySpark you can set up a virtual env and install >>>>> the current RC via "pip install >>>>> https://dist.apache.org/repos/dist/dev/spark/v4.1.0-rc1-bin/pyspark-4.1.0.tar.gz >>>>> " >>>>> and see if anything important breaks. >>>>> In the Java/Scala, you can add the staging repository to your >>>>> project's resolvers and test >>>>> with the RC (make sure to clean up the artifact cache before/after so >>>>> you don't end up building with an out of date RC going forward). >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe e-mail: [email protected] >>>>> >>>>>
