This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 6e64e92 [SPARK-38032][INFRA] Upgrade Arrow version < 7.0.0 for Python
UDF tests in SQL and documentation generation
6e64e92 is described below
commit 6e64e9252a821651a8984babfaccccc79a9ea433
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Wed Jan 26 15:55:12 2022 +0900
[SPARK-38032][INFRA] Upgrade Arrow version < 7.0.0 for Python UDF tests in
SQL and documentation generation
### What changes were proposed in this pull request?
This PR proposes to use Arrow < 7.0.0 (6.0.1 latest) for
[IntegratedUDFTestUtils](https://github.com/apache/spark/blob/master/sql/core/src/test/scala/org/apache/spark/sql/IntegratedUDFTestUtils.scala),
e.g.,
https://github.com/apache/spark/tree/master/sql/core/src/test/resources/sql-tests/inputs/udf
for pandas UDFs.
Note that this PR does not change the PyArrow and pandas used for PySpark
test base because they are installed in the base image
(https://github.com/apache/spark/blob/master/.github/workflows/build_and_test.yml#L290),
and they are already using almost latest version (PyArrow 6.0.0, and pandas
1.3.3) so I think it's fine.
### Why are the changes needed?
It's better to test latest versions as they are likely more used by end
users.
### Does this PR introduce _any_ user-facing change?
No, dev-only.
### How was this patch tested?
Existing test cases should cover.
Closes #35331 from HyukjinKwon/arrow-version-sql-test.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.github/workflows/build_and_test.yml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index 32f46d3..4529cd9 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -252,7 +252,7 @@ jobs:
- name: Install Python packages (Python 3.8)
if: (contains(matrix.modules, 'sql') && !contains(matrix.modules,
'sql-'))
run: |
- python3.8 -m pip install 'numpy>=1.20.0' 'pyarrow<5.0.0' pandas scipy
xmlrunner
+ python3.8 -m pip install 'numpy>=1.20.0' 'pyarrow<7.0.0' pandas scipy
xmlrunner
python3.8 -m pip list
# Run the tests.
- name: Run tests
@@ -530,7 +530,7 @@ jobs:
# Jinja2 3.0.0+ causes error when building with Sphinx.
# See also https://issues.apache.org/jira/browse/SPARK-35375.
python3.9 -m pip install 'sphinx<3.1.0' mkdocs pydata_sphinx_theme
ipython nbsphinx numpydoc 'jinja2<3.0.0'
- python3.9 -m pip install sphinx_plotly_directive 'numpy>=1.20.0'
'pyarrow<5.0.0' pandas 'plotly>=4.8'
+ python3.9 -m pip install sphinx_plotly_directive 'numpy>=1.20.0'
'pyarrow<7.0.0' pandas 'plotly>=4.8'
apt-get update -y
apt-get install -y ruby ruby-dev
Rscript -e "install.packages(c('devtools', 'testthat', 'knitr',
'rmarkdown', 'roxygen2'), repos='https://cloud.r-project.org/')"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
