lisancao opened a new pull request, #138: URL: https://github.com/apache/spark-docker/pull/138
### What changes were proposed in this pull request? This PR installs PyArrow and pandas in the Python images by adding one line to the `HAVE_PY` block of `r-python.template`: ```dockerfile PIP_BREAK_SYSTEM_PACKAGES=1 python3 -m pip install --no-cache-dir "pyarrow>=18.0.0" "pandas>=2.2.0,<3"; \ ``` Published Dockerfiles are unchanged; this applies to the next generated versions (4.2.1, 4.3.0). Closes #137 (a SPARK JIRA will follow once my JIRA account is approved). ### Why are the changes needed? Since 4.2.0, regular Python UDFs are Arrow-optimized by default (SPARK-54555). The python3 images ship without PyArrow or pandas, so a plain `@udf` fails on the unmodified image with `ModuleNotFoundError: No module named 'pyarrow'`, and `@pandas_udf` fails with `No module named 'pandas'`. The floors are PySpark 4.2's own requirements. `PIP_BREAK_SYSTEM_PACKAGES=1` keeps the line working on the Ubuntu 26.04 base proposed in #133, where a plain system pip install is refused (PEP 668); pip 22.0 on jammy ignores it. `<3` because PySpark currently warns that pandas 3.x isn't fully supported, and pip picks 3.x on Python 3.14. ### Does this PR introduce _any_ user-facing change? Yes. Python images built from this template can run default Python UDFs and pandas UDFs without a custom image. The image grows by about 320 MB uncompressed (1.32 GB to 1.64 GB for the 4.2.0 java21 python3 image in a local build). ### How was this patch tested? Rendered the template for 4.2.0, built `scala2.13-java21-python3-ubuntu` locally, started a Spark Connect server from the stock and patched images, and ran three UDFs from `pyspark-client==4.2.0` on Python 3.10: | UDF | stock image | patched image | |---|---|---| | `@udf` (default) | `ModuleNotFoundError: No module named 'pyarrow'` | OK | | `@udf(useArrow=False)` | OK | OK | | `@pandas_udf` | `ModuleNotFoundError: No module named 'pandas'` | OK | Also ran the install line on `eclipse-temurin:21-jammy` (Python 3.10) and `eclipse-temurin:21-resolute` (Python 3.14); both got pyarrow 25.0.1 and pandas 2.3.3. Without `PIP_BREAK_SYSTEM_PACKAGES`, resolute refuses the install. This contribution is my original work, and I license it to the project under the project's open source license. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
