This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new af71469706e7 [SPARK-58991][PYTHON] Remove dead and broken
AutoSerializer class
af71469706e7 is described below
commit af71469706e79c0394bf78f1cfc9337d4b4b380b
Author: Yicong Huang <[email protected]>
AuthorDate: Wed Aug 26 19:04:43 2026 +0800
[SPARK-58991][PYTHON] Remove dead and broken AutoSerializer class
### What changes were proposed in this pull request?
Remove the dead and broken `AutoSerializer` class from
`python/pyspark/serializers.py`, along with its two references in
`python/pyspark/tests/test_serializers.py`.
### Why are the changes needed?
`AutoSerializer` is dead and broken code:
- It is not exported in `__all__` and has no production callers.
- Its `loads()` is broken under Python 3: it reads `_type = obj[0]`, which
yields an `int` rather than a `bytes` value like `b"M"`/`b"P"`, so the type
check never matches and every call raises `ValueError`.
- The only reference is a smoke test in `test_serializers.py` (an import
plus a `hash()` call).
### Does this PR introduce _any_ user-facing change?
No. The class is not part of the public API and is non-functional.
### How was this patch tested?
Existing `python/pyspark/tests/test_serializers.py` still passes after
removing the unused import and `hash()` line.
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #58275 from Yicong-Huang/remove-dead-autoserializer.
Authored-by: Yicong Huang <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
---
python/pyspark/serializers.py | 28 ----------------------------
python/pyspark/tests/test_serializers.py | 2 --
2 files changed, 30 deletions(-)
diff --git a/python/pyspark/serializers.py b/python/pyspark/serializers.py
index b89fe49c816c..9f72e771b68e 100644
--- a/python/pyspark/serializers.py
+++ b/python/pyspark/serializers.py
@@ -478,34 +478,6 @@ class MarshalSerializer(FramedSerializer):
return marshal.loads(obj)
-class AutoSerializer(FramedSerializer):
- """
- Choose marshal or pickle as serialization protocol automatically
- """
-
- def __init__(self):
- FramedSerializer.__init__(self)
- self._type = None
-
- def dumps(self, obj):
- if self._type is not None:
- return b"P" + pickle.dumps(obj, -1)
- try:
- return b"M" + marshal.dumps(obj)
- except Exception:
- self._type = b"P"
- return b"P" + pickle.dumps(obj, -1)
-
- def loads(self, obj):
- _type = obj[0]
- if _type == b"M":
- return marshal.loads(obj[1:])
- elif _type == b"P":
- return pickle.loads(obj[1:])
- else:
- raise ValueError("invalid serialization type: %s" % _type)
-
-
class CompressedSerializer(FramedSerializer):
"""
Compress the serialized data
diff --git a/python/pyspark/tests/test_serializers.py
b/python/pyspark/tests/test_serializers.py
index ec2855a4cc66..243ecc89fbb4 100644
--- a/python/pyspark/tests/test_serializers.py
+++ b/python/pyspark/tests/test_serializers.py
@@ -21,7 +21,6 @@ import unittest
from pyspark import serializers
from pyspark.serializers import (
AutoBatchedSerializer,
- AutoSerializer,
BatchedSerializer,
CartesianDeserializer,
CloudPickleSerializer,
@@ -151,7 +150,6 @@ class SerializationTestCase(unittest.TestCase):
hash(UTF8Deserializer())
hash(CPickleSerializer())
hash(MarshalSerializer())
- hash(AutoSerializer())
hash(BatchedSerializer(CPickleSerializer()))
hash(AutoBatchedSerializer(MarshalSerializer()))
hash(PairDeserializer(NoOpSerializer(), UTF8Deserializer()))
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
