Yicong-Huang opened a new pull request, #58267:
URL: https://github.com/apache/spark/pull/58267
### What changes were proposed in this pull request?
This PR removes the optional `collections.namedtuple` monkey-patch in
`python/pyspark/serializers.py` that was gated behind the
`PYSPARK_ENABLE_NAMEDTUPLE_PATCH` environment variable. It deletes the entire
`if os.environ.get("PYSPARK_ENABLE_NAMEDTUPLE_PATCH") == "1":` block
(`_restore`, `_hack_namedtuple`, `_hijack_namedtuple`, and the `__cls` cache),
collapses the `CPickleSerializer` alias so it is always
`CloudPickleSerializer`, drops the now-unused `os`, `types`, and `collections`
imports, and adds a migration-guide note. `PickleSerializer` is kept, as it is
still a public serializer exported in `__all__`.
### Why are the changes needed?
The switch was introduced in SPARK-41189 as a transitional safety net when
the built-in pickle was replaced by cloudpickle in SPARK-32079, for the rare
cases where the two disagreed on `namedtuple` pickling. It is off by default,
has no CI coverage (the variable is never set anywhere in the build or test
infrastructure), and its inline comment still refers to "Python 3.8+" even
though the minimum supported Python is now 3.11. cloudpickle has handled
`namedtuple` pickling by default for several releases, and the corresponding
reader helper was dropped from the vendored cloudpickle long ago, so the escape
hatch is dead weight.
### Does this PR introduce _any_ user-facing change?
Yes. The `PYSPARK_ENABLE_NAMEDTUPLE_PATCH` environment variable is removed.
It defaulted to off and was documented only as a legacy escape hatch in the
migration guide, so users who never set it are unaffected; users who set it to
`1` will now always get cloudpickle's default `namedtuple` handling. This is a
removal targeted at the master line (Spark 5.0.0) only and is not backported to
maintenance branches. It is documented in the migration guide. Happy to start a
dev@ thread if reviewers feel this removal warrants broader discussion.
### How was this patch tested?
Covered by the existing `pyspark.tests.test_serializers` and
`pyspark.tests.test_rdd` suites, plus manual `namedtuple` RDD/UDF round-trips;
relying on the PR's GitHub Actions run for full coverage.
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
