This is an automated email from the ASF dual-hosted git repository.
uros-b pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new d6ce1efc1fec [SPARK-58305][PYTHON] Clarify jinja2 usage
d6ce1efc1fec is described below
commit d6ce1efc1fec3425c808e5b25176cb39eae4f854
Author: Tian Gao <[email protected]>
AuthorDate: Fri Jul 24 05:57:13 2026 +0000
[SPARK-58305][PYTHON] Clarify jinja2 usage
### What changes were proposed in this pull request?
Moved `jinja2` to `pandas_on_spark_extra` in dependency groups. Removed if
from all the lint/doc docker images. Added documentation about it as the other
optional requirements for pandas on spark.
### Why are the changes needed?
`jinja2` was not well documented and it was misused in our CIs. It is a
[requirement](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_latex.html)
to use `to_latex`, not doc related. That's also the only usage we have in
pyspark code.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
CI needs to pass - if docs/lint CI fails because `jinja2` is removed, I
need to investigate again. Otherwise it proves that this package is not used
for docs/lint.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #57477 from gaogaotiantian/clarify-jinja2.
Authored-by: Tian Gao <[email protected]>
Signed-off-by: Uros Bojanic <[email protected]>
---
dev/create-release/spark-rm/Dockerfile | 2 +-
dev/spark-test-image/docs/Dockerfile | 2 +-
dev/spark-test-image/lint/Dockerfile | 1 -
pyproject.toml | 2 +-
python/docs/source/getting_started/install.rst | 1 +
5 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/dev/create-release/spark-rm/Dockerfile
b/dev/create-release/spark-rm/Dockerfile
index f34f008aa9a6..30cc1d7bdd8d 100644
--- a/dev/create-release/spark-rm/Dockerfile
+++ b/dev/create-release/spark-rm/Dockerfile
@@ -57,7 +57,7 @@ RUN python3.11 -m pip install --ignore-installed
'blinker>=1.6.2' && \
# Sphinx and documentation packages
# See 'ipython_genutils' in SPARK-38517, 'docutils' in SPARK-39421
RUN python3.11 -m pip install 'sphinx==8.2.3' mkdocs
'pydata_sphinx_theme>=0.13' \
- sphinx-copybutton nbsphinx numpydoc jinja2 markupsafe 'pyzmq<24.0.0' \
+ sphinx-copybutton nbsphinx numpydoc markupsafe 'pyzmq<24.0.0' \
ipython ipython_genutils sphinx_plotly_directive 'numpy>=1.23.2' pyarrow
pandas \
'plotly>=4.8' 'docutils' 'mypy==1.19.1' 'pytest==7.1.3' \
'pytest-mypy-plugins==1.9.3' 'ruff==0.14.8' 'pandas-stubs==1.2.0.53' \
diff --git a/dev/spark-test-image/docs/Dockerfile
b/dev/spark-test-image/docs/Dockerfile
index fdc364dbc690..22022dfb77c4 100644
--- a/dev/spark-test-image/docs/Dockerfile
+++ b/dev/spark-test-image/docs/Dockerfile
@@ -91,7 +91,7 @@ ENV PATH="$VIRTUAL_ENV/bin:$PATH"
# See 'ipython_genutils' in SPARK-38517
# See 'docutils<0.18.0' in SPARK-39421
-RUN python3.12 -m pip install 'sphinx==8.2.3' mkdocs
'pydata_sphinx_theme>=0.13' sphinx-copybutton nbsphinx numpydoc jinja2
markupsafe \
+RUN python3.12 -m pip install 'sphinx==8.2.3' mkdocs
'pydata_sphinx_theme>=0.13' sphinx-copybutton nbsphinx numpydoc markupsafe \
ipython ipython_genutils sphinx_plotly_directive 'numpy>=1.23.2'
'pyarrow>=23.0.0' 'pandas==2.3.3' 'plotly>=4.8' 'docutils' \
'mypy==1.19.1' 'pytest==7.1.3' 'pytest-mypy-plugins==1.9.3' 'ruff==0.14.8' \
'pandas-stubs==1.2.0.53' 'grpcio==1.76.0' 'grpcio-status==1.76.0'
'protobuf==6.33.5' 'grpc-stubs==1.24.11'
'googleapis-common-protos-stubs==2.2.0' \
diff --git a/dev/spark-test-image/lint/Dockerfile
b/dev/spark-test-image/lint/Dockerfile
index f3968b0498d3..ead0e6488c0f 100644
--- a/dev/spark-test-image/lint/Dockerfile
+++ b/dev/spark-test-image/lint/Dockerfile
@@ -90,7 +90,6 @@ RUN python3.12 -m pip install \
'zstandard==0.25.0' \
'ipython' \
'ipython_genutils' \
- 'jinja2' \
'matplotlib' \
'mypy==1.19.1' \
'numpy==2.4.1' \
diff --git a/pyproject.toml b/pyproject.toml
index 48bd4cc83baa..01fdcfc63de1 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -107,6 +107,7 @@ pandas_on_spark_extra = [
"plotly<6.0.0",
"matplotlib",
"mlflow>=2.3.1",
+ "jinja2",
]
ml = [
@@ -179,7 +180,6 @@ _docs = [
"ipython",
"nbsphinx",
"numpydoc",
- "jinja2",
"mkdocs",
]
diff --git a/python/docs/source/getting_started/install.rst
b/python/docs/source/getting_started/install.rst
index d137a19b30c9..558ad51ac98f 100644
--- a/python/docs/source/getting_started/install.rst
+++ b/python/docs/source/getting_started/install.rst
@@ -280,6 +280,7 @@ Additional libraries that enhance functionality but are not
included in the inst
- **mlflow**: Required for ``pyspark.pandas.mlflow``.
- **plotly**: Provide plotting for visualization. It is recommended using
**plotly** over **matplotlib**.
- **matplotlib**: Provide plotting for visualization. The default is
**plotly**.
+- **jinja2**: Required for ``pyspark.pandas.to_latex``.
MLlib DataFrame-based API
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
