Lorenzo Martini created SPARK-61253:
---------------------------------------
Summary: PySpark import fails with vendor-branded Python version
strings
Key: SPARK-61253
URL: https://issues.apache.org/jira/browse/SPARK-61253
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 5.0.0
Reporter: Lorenzo Martini
PySpark can fail during import when Python’s {{sys.version}} banner contains
additional vendor metadata, such as a {{packaged by ...}} note.
The vendored cloudpickle module calls {{platform.python_implementation()}} to
detect PyPy. That function parses the full version banner, and can raise
{{ValueError}} if the parser does not recognize the additional text—even though
identifying the interpreter does not require parsing build metadata.
A minimal reproduction on an affected runtime:
{{import sys
sys.version = sys.version.replace(" ", " | packaged by example | ", 1)
from pyspark import cloudpickle}}
This fails with {{{}ValueError: failed to parse CPython sys.version{}}}. The
expected behavior is for PySpark to import successfully regardless of vendor
branding in the version banner.
Conda-forge provides a concrete example of this dependency: its [Python
branding
patch|https://github.com/conda-forge/python-feedstock/blob/df31527ad9ce87110d7ddcc3a84d076c4b0137a7/recipe/patches/0023-Brand-conda-forge.patch#L15-L22]
explicitly extends {{{}platform._sys_version(){}}}’s regex to recognize its
own {{packaged by conda-forge}} note.
The proposed fix uses {{sys.implementation.name == "pypy"}} instead. This reads
the interpreter’s identity directly, preserving PyPy-specific behavior without
depending on version-string formatting or downstream parser patches.
The failure was reproduced on Spark’s 5.0.0 development branch. Source
inspection also confirms the same check in released versions from 3.1.1 through
4.3.0.
[PR #60244|https://github.com/apache/spark/pull/60244] includes a regression
test that modifies the banner in a fresh Python process, imports cloudpickle,
and verifies a lambda serialization round trip. The test fails before the fix
and passes afterward; all 12 selected serialization and import tests pass.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
