This is an automated email from the ASF dual-hosted git repository.
raulcd pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow.git
The following commit(s) were added to refs/heads/main by this push:
new 410ff4b2de GH-49524: [CI][Integration] Bump HDFS versions tested to
use latest v2 and newest v3 (#50814)
410ff4b2de is described below
commit 410ff4b2deec6b67c422d2dda3a3424d773379da
Author: Raúl Cumplido <[email protected]>
AuthorDate: Tue Sep 15 14:26:29 2026 +0200
GH-49524: [CI][Integration] Bump HDFS versions tested to use latest v2 and
newest v3 (#50814)
### Rationale for this change
When the issue was originally opened, the
[test-conda-python-3.10-hdfs-2.9.2](https://github.com/ursacomputing/crossbow/actions/runs/22944679479/job/66593987920)
failed sporadically on a PR when upgrading some dependencies due to time out
rebuilding the docker image.
A big chunk of time was spent on the job just downloading HDFS 2.9.2.
HDFS 2.9.2 was released on 2018. We test 2.9.2 and 3.2.1, released on 2019.
### What changes are included in this PR?
Bump to newer versions, latests are 3.5.0 (2026) and from the v2 series,
2.10.2 (2022).
Bump default JDK to 17. The only other place where we currently use the JDK
version on env was for Spark where we already are manually using `JDK=17` on
tasks.yml so no real change for any other job.
### Are these changes tested?
Yes via the archery jobs
### Are there any user-facing changes?
No
* GitHub Issue: #49524
Authored-by: Raúl Cumplido <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>
---
.env | 4 ++--
ci/docker/conda-python-hdfs.dockerfile | 4 ++--
ci/docker/conda-python-spark.dockerfile | 2 +-
dev/tasks/tasks.yml | 2 +-
4 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/.env b/.env
index 660f589bf4..f0d538ccd9 100644
--- a/.env
+++ b/.env
@@ -62,8 +62,8 @@ CMAKE=3.26.0
CUDA=11.7.1
DASK=latest
GCC=
-HDFS=3.2.1
-JDK=11
+HDFS=3.5.0
+JDK=17
# LLVM 12 and GCC 11 reports -Wmismatched-new-delete.
LLVM=18
MAVEN=3.9.9
diff --git a/ci/docker/conda-python-hdfs.dockerfile
b/ci/docker/conda-python-hdfs.dockerfile
index 85311c8fa1..d1c981953b 100644
--- a/ci/docker/conda-python-hdfs.dockerfile
+++ b/ci/docker/conda-python-hdfs.dockerfile
@@ -21,7 +21,7 @@ ARG arch_short
ARG python=3.11
FROM --platform=linux/${arch} ${repo}:${arch_short}-conda-python-${python}
-ARG jdk=11
+ARG jdk=17
ARG maven=3.9.9
RUN mamba install -q -y \
maven=${maven} \
@@ -30,7 +30,7 @@ RUN mamba install -q -y \
mamba clean --all --yes
# installing libhdfs (JNI)
-ARG hdfs=3.2.1
+ARG hdfs=3.5.0
ENV HADOOP_HOME=/opt/hadoop-${hdfs} \
HADOOP_OPTS=-Djava.library.path=/opt/hadoop-${hdfs}/lib/native \
PATH=$PATH:/opt/hadoop-${hdfs}/bin:/opt/hadoop-${hdfs}/sbin
diff --git a/ci/docker/conda-python-spark.dockerfile
b/ci/docker/conda-python-spark.dockerfile
index d7e74f08ee..8267c69047 100644
--- a/ci/docker/conda-python-spark.dockerfile
+++ b/ci/docker/conda-python-spark.dockerfile
@@ -21,7 +21,7 @@ ARG arch_short
ARG python=3.11
FROM --platform=linux/${arch} ${repo}:${arch_short}-conda-python-${python}
-ARG jdk=11
+ARG jdk=17
ARG maven=3.9.9
ARG numpy=latest
diff --git a/dev/tasks/tasks.yml b/dev/tasks/tasks.yml
index 65e22508bc..9ef1a9bd15 100644
--- a/dev/tasks/tasks.yml
+++ b/dev/tasks/tasks.yml
@@ -765,7 +765,7 @@ tasks:
image: conda-python-dask
{% endfor %}
-{% for hdfs_version in ["2.9.2", "3.2.1"] %}
+{% for hdfs_version in ["2.10.2", "3.5.0"] %}
test-conda-python-3.11-hdfs-{{ hdfs_version }}:
ci: github
template: docker-tests/github.linux.yml
