[
https://issues.apache.org/jira/browse/SPARK-61253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61253:
-----------------------------------
Labels: pull-request-available (was: )
> PySpark import fails with vendor-branded Python version strings
> ---------------------------------------------------------------
>
> Key: SPARK-61253
> URL: https://issues.apache.org/jira/browse/SPARK-61253
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 5.0.0
> Reporter: Lorenzo Martini
> Priority: Major
> Labels: pull-request-available
>
> PySpark can fail during import when Python’s {{sys.version}} banner contains
> additional vendor metadata, such as a {{packaged by ...}} note.
> The vendored cloudpickle module calls {{platform.python_implementation()}} to
> detect PyPy. That function parses the full version banner, and can raise
> {{ValueError}} if the parser does not recognize the additional text—even
> though identifying the interpreter does not require parsing build metadata.
> A minimal reproduction on an affected runtime:
> {{import sys
> sys.version = sys.version.replace(" ", " | packaged by example | ", 1)
> from pyspark import cloudpickle}}
> This fails with {{{}ValueError: failed to parse CPython sys.version{}}}. The
> expected behavior is for PySpark to import successfully regardless of vendor
> branding in the version banner.
> Conda-forge provides a concrete example of this dependency: its [Python
> branding
> patch|https://github.com/conda-forge/python-feedstock/blob/df31527ad9ce87110d7ddcc3a84d076c4b0137a7/recipe/patches/0023-Brand-conda-forge.patch#L15-L22]
> explicitly extends {{{}platform._sys_version(){}}}’s regex to recognize its
> own {{packaged by conda-forge}} note.
> The proposed fix uses {{sys.implementation.name == "pypy"}} instead. This
> reads the interpreter’s identity directly, preserving PyPy-specific behavior
> without depending on version-string formatting or downstream parser patches.
> The failure was reproduced on Spark’s 5.0.0 development branch. Source
> inspection also confirms the same check in released versions from 3.1.1
> through 4.3.0.
> [PR #60244|https://github.com/apache/spark/pull/60244] includes a regression
> test that modifies the banner in a fresh Python process, imports cloudpickle,
> and verifies a lambda serialization round trip. The test fails before the fix
> and passes afterward; all 12 selected serialization and import tests pass.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
