This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new ecae0ebbdef3 [SPARK-57452][DOC] Add missing 4.2 behavior changes to
migration guides
ecae0ebbdef3 is described below
commit ecae0ebbdef3dece4b45610d6fd4bf5196cdf1d5
Author: Wenchen Fan <[email protected]>
AuthorDate: Wed Jun 17 11:03:34 2026 -0700
[SPARK-57452][DOC] Add missing 4.2 behavior changes to migration guides
### What changes were proposed in this pull request?
An audit of Spark 4.2 commits (SPARK-57452) found a number of user-facing
behavior changes that were not documented in the migration guides. This PR adds
entries for the subset of those changes that **require a user to take action to
migrate**, across three guides:
PySpark (`python/docs/source/migration_guide/pyspark_upgrade.rst`):
- PyPy is no longer officially supported.
- Minimum pandas version for Spark Connect raised to 2.2.0.
- `createDataFrame` from a NumPy `ndarray` now requires PyArrow and may
infer a different schema.
- Pandas UDFs receive nullable integer columns as a pandas nullable `Int`
dtype instead of `float64`.
- `Observation.get` raises on metric collection failure instead of
returning an empty dictionary.
- `DataFrame.drop` / `Series.drop` (pandas API on Spark) raise `KeyError`
if any label is missing.
- Python Data Sources fail with `DATA_SOURCE_RETURN_SCHEMA_MISMATCH` on
schema mismatch.
- Python streaming data sources fail when reported offsets do not advance.
SQL (`docs/sql-migration-guide.md`):
- Duplicate CTE names within a `WITH` clause are detected
case-insensitively.
- `NATURAL JOIN` honors `spark.sql.caseSensitive` when choosing join
columns.
- Built-in parameterless functions take precedence over same-named UDFs.
- `SET CATALOG <name>` resolves a bare name as a session variable first.
Structured Streaming (`docs/streaming/ss-migration-guide.md`):
- Restarting from a checkpoint with missing required metadata now fails by
default.
Behavior changes that are purely cosmetic/UI, internal, additive, or
correctness fixes requiring no user action were intentionally excluded.
### Why are the changes needed?
Without these entries, users upgrading to Spark 4.2 can hit silent result
changes, new errors on previously-valid input, or dependency/environment
requirements with no documented guidance on how to migrate.
### Does this PR introduce _any_ user-facing change?
No. This is a documentation-only change.
### How was this patch tested?
Documentation-only change; verified by review of the rendered
Markdown/reStructuredText.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
Closes #56553 from cloud-fan/SPARK-57452-migration-guide.
Authored-by: Wenchen Fan <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
(cherry picked from commit 11e4a426b7567b8cb0604d90982f868123ce442e)
Signed-off-by: Wenchen Fan <[email protected]>
---
docs/sql-migration-guide.md | 5 +++++
docs/streaming/ss-migration-guide.md | 4 ++++
python/docs/source/migration_guide/pyspark_upgrade.rst | 6 ++++++
3 files changed, 15 insertions(+)
diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index 5f9a21fd16c8..ef98ab3eb884 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -35,6 +35,11 @@ license: |
- Since Spark 4.2, the virtual `system` catalog hosts the new `system.builtin`
and `system.session` namespaces. `system.builtin` exposes built-in functions
and functions injected through `SparkSessionExtensions`; `system.session`
exposes temporary views, temporary functions, and session variables created in
the current session. As a result, 2-part references like `builtin.func()` and
`session.func()` now follow a mini-path that tries the system namespace first
and the current catalog sec [...]
- Since Spark 4.2, `CREATE TEMPORARY VIEW`, `CREATE TEMPORARY FUNCTION`, and
the corresponding `DROP` statements accept the `session` and `system.session`
qualifiers on the object name (in addition to the previously supported
unqualified form); for example, `CREATE TEMPORARY VIEW system.session.v AS ...`
and `DROP TEMPORARY FUNCTION session.f` are now valid. Any other qualifier on a
temporary object is rejected with `INVALID_TEMP_OBJ_QUALIFIER`.
- Since Spark 4.2, the SQL standard `PATH` feature is available: the `SET
PATH` statement, the `current_path()` function, path-based resolution of
unqualified routines, tables, views, and session variables, and the
configurations `spark.sql.path.enabled` (default `false`) and
`spark.sql.defaultPath`. The feature is opt-in; when `spark.sql.path.enabled`
is `false`, unqualified resolution falls back to a fixed default path and `SET
PATH` is rejected with `UNSUPPORTED_FEATURE.SET_PATH_WHEN_ [...]
+- Since Spark 4.2, duplicate Common Table Expression (CTE) names within a
single `WITH` clause are detected case-insensitively at parse time, so names
that differ only in case (for example, `WITH a AS (...), A AS (...)`) are
rejected with `DUPLICATED_CTE_NAMES`. This check is always case-insensitive and
does not depend on `spark.sql.caseSensitive`. Rename the conflicting CTEs.
+- Since Spark 4.2, `NATURAL JOIN` honors the `spark.sql.caseSensitive`
configuration when determining the common columns to join on. This can change
which columns are used as join keys, and therefore the query result, when
column names differ only in case.
+- Since Spark 4.2, when a SQL UDF has a parameter whose name matches a
parameterless built-in function (`current_user`, `current_date`,
`current_time`, `current_timestamp`, `user`, `session_user`, `grouping__id`), a
bare reference to that name in the function body resolves to the built-in
function instead of the parameter, matching the documented name resolution
rules. Rename the parameter to avoid the collision, or set
`spark.sql.legacy.allowUdfParameterToShadowParameterlessFunction` to [...]
+- Since Spark 4.2, `SET CATALOG <name>` resolves a bare (unquoted) name as a
session variable first, using the variable's value as the catalog name when
such a variable exists, and otherwise treats the name as a literal catalog
name. Use a string literal (`SET CATALOG 'name'`) to force it to be interpreted
literally.
+- Since Spark 4.2, when an error occurs while collecting observed metrics,
`Observation.get` raises the underlying exception (for example,
`SparkRuntimeException` in Scala or `PySparkException` in Python) instead of
silently returning an empty result. Add error handling if your code relied on
receiving an empty result on failure.
## Upgrading from Spark SQL 4.0 to 4.1
diff --git a/docs/streaming/ss-migration-guide.md
b/docs/streaming/ss-migration-guide.md
index a0c0a397edea..210f752d9d17 100644
--- a/docs/streaming/ss-migration-guide.md
+++ b/docs/streaming/ss-migration-guide.md
@@ -23,6 +23,10 @@ Note that this migration guide describes the items specific
to Structured Stream
Many items of SQL migration can be applied when migrating Structured Streaming
to higher versions.
Please refer [Migration Guide: SQL, Datasets and
DataFrame](../sql-migration-guide.html).
+## Upgrading from Structured Streaming 4.1 to 4.2
+
+- Since Spark 4.2, restarting a streaming query from a checkpoint whose
metadata file is missing while the offset or commit logs contain data fails
with `STREAMING_CHECKPOINT_MISSING_METADATA_FILE`, instead of silently
generating a new query ID (which can duplicate data in exactly-once sinks).
Restore the metadata file or use a new checkpoint location. To restore the
previous behavior, set
`spark.sql.streaming.checkpoint.verifyMetadataExists.enabled` to `false`. (See
[SPARK-55058](https: [...]
+
## Upgrading from Structured Streaming 4.0 to 4.1
- Since Spark 4.1, AQE is supported for stateless workloads, and it could
affect the behavior of the query after upgrade (especially since AQE is turned
on by default). In general, it helps to achieve better performance including
resolution of skewed partition, but you can turn off AQE via changing
`spark.sql.adaptive.streaming.stateless.enabled` to `false` to restore the
behavior if you see regression.
diff --git a/python/docs/source/migration_guide/pyspark_upgrade.rst
b/python/docs/source/migration_guide/pyspark_upgrade.rst
index 041640b2eb7f..b257ee49b7dd 100644
--- a/python/docs/source/migration_guide/pyspark_upgrade.rst
+++ b/python/docs/source/migration_guide/pyspark_upgrade.rst
@@ -27,6 +27,12 @@ Upgrading from PySpark 4.1 to 4.2
* In Spark 4.2, columnar data exchange between PySpark and the JVM uses Apache
Arrow by default. The configuration
``spark.sql.execution.arrow.pyspark.enabled`` now defaults to true. To restore
the legacy (non-Arrow) row-based data exchange, set
``spark.sql.execution.arrow.pyspark.enabled`` to ``false``.
* In Spark 4.2, regular Python UDFs are Arrow-optimized by default. The
configuration ``spark.sql.execution.pythonUDF.arrow.enabled`` now defaults to
true. To restore the legacy behavior for Python UDF execution, set
``spark.sql.execution.pythonUDF.arrow.enabled`` to ``false``.
* In Spark 4.2, regular Python UDTFs are Arrow-optimized by default. The
configuration ``spark.sql.execution.pythonUDTF.arrow.enabled`` now defaults to
true. To restore the legacy behavior for Python UDTF execution, set
``spark.sql.execution.pythonUDTF.arrow.enabled`` to ``false``.
+* In Spark 4.2, PyPy is no longer officially supported. Run PySpark on CPython
instead.
+* In Spark 4.2, ``SparkSession.createDataFrame`` from a NumPy ``ndarray``
requires PyArrow (instead of pandas) and converts the array directly to Arrow
rather than through pandas. Install PyArrow; if you previously ran with Arrow
disabled and relied on NumPy-dtype-based schema inference, review the inferred
schema, as it now follows Arrow's type mapping.
+* In Spark 4.2, when a pandas UDF receives a nullable integer column whose
batch contains nulls, the column is delivered as a pandas nullable integer
extension dtype (``Int8``/``Int16``/``Int32``/``Int64``) instead of
``float64``. Update UDF code that assumed ``float64`` input for nullable
integer columns.
+* In Spark 4.2, ``DataFrame.drop`` and ``Series.drop`` in pandas API on Spark
raise a ``KeyError`` when any of the specified labels is missing, instead of
only when all of them are missing, matching pandas. Make sure all labels exist
before dropping, filter to existing labels, or pass ``errors="ignore"``.
+* In Spark 4.2, a Python Data Source whose returned Arrow data has column
types that do not match its declared schema fails with
``DATA_SOURCE_RETURN_SCHEMA_MISMATCH`` (column count and name mismatches
already raised this error in earlier versions). Make the data source return
data whose types match its declared schema.
+* In Spark 4.2, a ``SimpleDataSourceStreamReader`` whose ``read()`` returns a
non-empty batch without advancing the end offset past the start offset fails
with ``SIMPLE_STREAM_READER_OFFSET_DID_NOT_ADVANCE``, instead of reprocessing
the same batch and growing the prefetch cache without bound. Ensure the
returned end offset advances past the last record.
Upgrading from PySpark 4.0 to 4.1
---------------------------------
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
