This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new bd30652d01e [chore](thirdparty) upgrade HADOOP_LIBS_3_4 to
hadoop-3.4.2.3-for-doris and build macOS CI on JDK 17 (#66783)
bd30652d01e is described below
commit bd30652d01ec626bda554afb16a9768479f887af
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Sat Aug 15 08:28:04 2026 +0800
[chore](thirdparty) upgrade HADOOP_LIBS_3_4 to hadoop-3.4.2.3-for-doris and
build macOS CI on JDK 17 (#66783)
## Proposed changes
Two commits, both small.
### 1. `HADOOP_LIBS_3_4` → `hadoop-3.4.2.3-for-doris`
Picks up
[apache/doris-thirdparty#407](https://github.com/apache/doris-thirdparty/pull/407),
which makes hadoop's native build work on macOS. Until now
`build_hadoop_libs_3_4` was Linux-only (`build-thirdparty.sh` adds it to
the package list under `uname -s == Linux`), because the build could not
complete on a Mac at all. Two things stopped it, both platform
differences rather than defects:
- **`find_package(ZLIB REQUIRED)` could not succeed.**
`hadoop_set_find_shared_library_version("1")` pins the search to
`libz.1.dylib`, which modern macOS keeps only in the dyld shared cache —
the SDK carries a `libz.1.tbd` stub instead — so configure aborted with
`Could NOT find ZLIB (missing: ZLIB_LIBRARY)`. zlib is never linked into
libhadoop; `ZlibCompressor.c` `dlopen()`s it by basename at run time,
and `dlopen("libz.1.dylib")` does resolve against the shared cache. The
lookup now takes the plain search for headers and names the runtime
library outright.
- **`libhdfspp.dylib` failed to link**, leaving 22 `gss_*` symbols
undefined through a static libsasl2. Linux' `ld` tolerates undefined
symbols in a `.so`; macOS' `ld` does not, and the failure took the whole
`make` down with it — `libhdfs.a` included. SASL is used by libhdfspp
alone, so it is turned off there.
Both fixes sit inside `if(APPLE)` / `if(NOT APPLE)` branches in the
fork, so the Linux build is byte for byte unchanged and produces the
same artifacts.
### 2. macOS thirdparty CI jobs: `openjdk@11` → `openjdk@17`
Doris builds on JDK 17, and hadoop 3.4.2 — the only thirdparty package
whose build needs a JDK — compiles cleanly against it. Declaring 11 no
longer reflects anything we build or test against. The Linux job stays
on openjdk-8, which it also pins through `JAVA_HOME`.
## Verification
The new tag was built on macOS 26.5 / arm64 (llvm@20, JDK 17), driven
through `thirdparty/build-thirdparty.sh hadoop_libs_3_4` with
dependencies resolved from a Doris thirdparty prefix — no extra Homebrew
packages. Tested with both cmake 3.25.3 (the version this workflow pins)
and cmake 4.4.
`build-thirdparty.sh hadoop_libs_3_4` exits 0 and installs:
```
installed/lib/hadoop_hdfs_3_4/native/ libhdfs.a 1.4M arm64
libhadoop.dylib 160K arm64
libhdfs.dylib 135K arm64
libhdfspp.dylib 8.9M arm64
installed/include/hadoop_hdfs_3_4/ hdfs.h
installed/lib/hadoop_hdfs_3_4/ common/ (5 jars) hdfs/ (10 jars)
```
`libhdfs.a` exports everything BE's `USE_HADOOP_HDFS` /
`USE_DORIS_HADOOP_HDFS` paths call, including the fork's
`hdfsSetLogger`, plus `hdfsGetLastExceptionRootCause`,
`hdfsBuilderSetKerb5Conf`, `hdfsBuilderSetKeyTabFile`,
`hdfsUnbufferFile`, `hdfsFileGetReadStatistics`, `hdfsHSync` and
`hdfsHFlush`.
The tarball MD5 was checked across three separate downloads of the
GitHub tag archive and is stable.
## Scope
This PR only bumps the source and the declared CI JDK. It deliberately
does **not** add `hadoop_libs_3_4` to the macOS branch of
`build-thirdparty.sh`'s package list, nor touch `be/CMakeLists.txt`,
which still selects libhdfs3 on macOS. Making the BE actually link
hadoop's libhdfs there is a separate change.
---
.github/workflows/build-thirdparty.yml | 4 ++--
thirdparty/CHANGELOG.md | 4 ++++
thirdparty/vars.sh | 8 ++++----
3 files changed, 10 insertions(+), 6 deletions(-)
diff --git a/.github/workflows/build-thirdparty.yml
b/.github/workflows/build-thirdparty.yml
index d5a6fe2b2bd..1fe25b781e8 100644
--- a/.github/workflows/build-thirdparty.yml
+++ b/.github/workflows/build-thirdparty.yml
@@ -227,7 +227,7 @@ jobs:
'gettext'
'wget'
'pcre'
- 'openjdk@11'
+ 'openjdk@17'
'maven'
'node'
'llvm@20'
@@ -292,7 +292,7 @@ jobs:
'gettext'
'wget'
'pcre'
- 'openjdk@11'
+ 'openjdk@17'
'maven'
'node'
'llvm@20'
diff --git a/thirdparty/CHANGELOG.md b/thirdparty/CHANGELOG.md
index eda6d280f5e..1cb2e96a53b 100644
--- a/thirdparty/CHANGELOG.md
+++ b/thirdparty/CHANGELOG.md
@@ -2,6 +2,10 @@
This file contains version of the third-party dependency libraries in the
build-env image. The docker build-env image is apache/doris, and the tag is
`build-env-${version}`
+## 20260814
+
+- Modified: hadoop-libs 3.4.2.2 -> 3.4.2.3
+
## 20260206
- Modified: jindofs 6.8.2 -> 6.10.4
diff --git a/thirdparty/vars.sh b/thirdparty/vars.sh
index da4913b78e7..211954c27a8 100644
--- a/thirdparty/vars.sh
+++ b/thirdparty/vars.sh
@@ -504,10 +504,10 @@
HADOOP_LIBS_SOURCE="doris-thirdparty-hadoop-3.3.6.6-for-doris"
HADOOP_LIBS_MD5SUM="13b66d5f2abffd1740e692b65df5962e"
# libhdfs 3.4
-HADOOP_LIBS_3_4_DOWNLOAD="https://github.com/apache/doris-thirdparty/archive/refs/tags/hadoop-3.4.2.2-for-doris.tar.gz";
-HADOOP_LIBS_3_4_NAME="hadoop-3.4.2.2-for-doris.tar.gz"
-HADOOP_LIBS_3_4_SOURCE="doris-thirdparty-hadoop-3.4.2.2-for-doris"
-HADOOP_LIBS_3_4_MD5SUM="b649508cdff5c8603abd659b754a2796"
+HADOOP_LIBS_3_4_DOWNLOAD="https://github.com/apache/doris-thirdparty/archive/refs/tags/hadoop-3.4.2.3-for-doris.tar.gz";
+HADOOP_LIBS_3_4_NAME="hadoop-3.4.2.3-for-doris.tar.gz"
+HADOOP_LIBS_3_4_SOURCE="doris-thirdparty-hadoop-3.4.2.3-for-doris"
+HADOOP_LIBS_3_4_MD5SUM="a751a42f3a927c4a272a99d4e3788648"
# AvxToNeon
AVX2NEON_DOWNLOAD="https://github.com/kunpengcompute/AvxToNeon/archive/refs/tags/v1.0.0.tar.gz";
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
