morningman opened a new pull request, #66842: URL: https://github.com/apache/doris/pull/66842
### What problem does this PR solve? Issue Number: close #xxx Related PR: apache/doris-thirdparty#410 Problem Summary: Three related third-party build changes, all measured against [apache/doris-thirdparty run 31988123966](https://github.com/apache/doris-thirdparty/actions/runs/31988123966). Supersedes #66837 and #66839, which are now closed. --- ## 1. Only build hadoop-libs 3.4.2.4 The tree carried two libhdfs builds: `build_hadoop_libs` built the **3.3.6.6** fork into `installed/{include,lib}/hadoop_hdfs/`, `build_hadoop_libs_3_4` built the **3.4.2.4** fork into `.../hadoop_hdfs_3_4/`, and on Linux both ran - 11m38s of the x86_64 build and 10m00s of the arm64 one, for a fork only the cloud module still consumed. The two prefixes were also mixed up on the consumer side: | consumer | header | library | |---|---|---| | `be/src/io/fs/hdfs.h` | `hadoop_hdfs_3_4/hdfs.h` (3.4.2.4) | — | | `be/src/io/hdfs_builder.cpp` | `hadoop_hdfs/hdfs.h` (**3.3.6.6**) | — | | `be/CMakeLists.txt` | — | `hadoop_hdfs_3_4/native/libhdfs.a` (3.4.2.4) | | `cloud/src/recycler/hdfs_accessor.{h,cpp}` | `hadoop_hdfs/hdfs.h` (**3.3.6.6**) | — | | `cloud/CMakeLists.txt` | — | `hadoop_hdfs/native/libhdfs.a` (**3.3.6.6**) | Both headers share the `LIBHDFS_HDFS_H` include guard, so `hdfs_builder.cpp` compiled against the 3.3.6.6 header (it came first) while BE linked the 3.4.2.4 archive. 3.4.2.4 stays where it is, under `hadoop_hdfs_3_4/`, so a build-env image from this change can still compile older Doris branches. The 3.3.6.6 build and its `hadoop_hdfs/` prefix are gone, and `cloud/`, `build.sh`'s `LAST_THIRDPARTY_LIB` sentinel, `run-be-ut.sh`, `run-cloud-ut.sh` and the thirdparty lifecycle test move to `hadoop_hdfs_3_4/`. The unit-test runners still stage the jars at `lib/hadoop_hdfs/`, so the classpath loops below them are unchanged. ## 2. Build azure everywhere, and stop building what Doris does not link **It was only ever built on x86_64 Linux.** `env.sh` forced `DISABLE_BUILD_AZURE=ON` on aarch64 and macOS, and `build_azure` skipped `Darwin` a second time, so BE and the cloud meta-service lost `+AZURE_BLOB` and `+AZURE_STORAGE_VAULT` there. Nothing in the recipe was genuinely x86_64-Linux-only, just two details: `-ldl` fails on Apple (there is no libdl; those symbols are in libSystem), and vcpkg ships no prebuilt cmake/ninja/curl for aarch64 Linux so it needs `VCPKG_FORCE_SYSTEM_BINARIES=1`. The triplet is now derived from `uname` rather than left to vcpkg's host detection. **It was the most expensive package in the tree** - 26m06s of the 2h43m x86_64 Linux build - almost none of it for Doris: | phase | time | |---|---| | `vcpkg install` | 22m11s | | - protobuf 5.29.3 | 12m29s | | - opentelemetry-cpp | 4m35s | | - abseil | 1m58s | | - openssl 1.1.1n | 1m38s | | - curl / libxml2 / zlib / uAMQP / utf8-range | ~1m30s | | azure SDK configure + build | ~3m55s | protobuf, abseil and utf8-range were pulled in **only** because `vcpkg.json` declared `opentelemetry-cpp` unconditionally - roughly 19 of the 26 minutes. Doris does not use azure's OpenTelemetry tracing. On top of that vcpkg builds every port twice (`Building x64-linux-dbg` then `-rel`) and Doris links only the release halves, and the SDK builds appconfiguration, attestation, eventhubs, keyvault, tables, template, uAMQP, storage-files-datalake, storage-files-shares and storage-queues, while `be/cmake/thirdparty.cmake` links exactly four targets: `azure-core`, `azure-identity`, `azure-storage-blobs`, `azure-storage-common`. So the patch drops `opentelemetry-cpp` and the uAMQP C libraries from `vcpkg.json`, flips `DISABLE_AMQP` / `DISABLE_AZURE_CORE_OPENTELEMETRY` to `ON` and trims the sub-projects to the four Doris links; `build_azure` writes an overlay triplet that sets `VCPKG_BUILD_TYPE release`. What is left for vcpkg to build is curl, libxml2, openssl 1.1.1n and zlib, release only. ## 3. ccache for the third-party build `ENABLE_THIRDPARTY_CCACHE=ON` (off by default) exports `CMAKE_C_COMPILER_LAUNCHER` / `CMAKE_CXX_COMPILER_LAUNCHER`, which CMake initialises from the environment, so no cmake invocation in the script changes. It deliberately does not prefix `CC`/`CXX`: CMake splits `"ccache clang"` into the compiler plus a `CMAKE_<LANG>_FLAGS` entry, and that leaks into whatever a package exports. Autotools packages are left alone. `.github/workflows/build-thirdparty.yml` turns it on and carries the cache with the `ccache-action` this repository already vendors. Today a change to one package recompiles all ~80, at `-j 2` on two of the three jobs. `max-size` is **1G** per job so the three together stay inside what is left of the repository's 10 GB Actions cache budget next to the 5G `BE-UT-macOS` cache. Note that a cache saved by a pull request run is only visible to that same pull request, so this pays off across pushes to one branch; sharing it across pull requests would need this workflow to run on master, the way `be-ut-mac.yml` uses a schedule for exactly that reason. Happy to drop this hunk if the cache budget is too tight. ## Also: the macOS jobs stalled for 10 minutes each Carried over from apache/doris-thirdparty#410. thrift's configure finds the dotnet the runner images ship, so `make` runs `dotnet build -c Release` for `lib/netstd`. The build finishes in seconds and then the compiler server it leaves behind holds the stdout it inherited for its full keep-alive while nothing happens: | runner | "Build succeeded" -> "Making install in compiler/cpp" | |---|---| | macos-14 | **600.03s** | | macos-15-intel | **598.79s** | | ubuntu-22.04 | 0.01s | `UseSharedCompilation=false` and `MSBUILDDISABLENODEREUSE=1` stop the server from being started. Both macOS jobs in this workflow pay this today. The other two fixes in apache/doris-thirdparty#410 do not apply here: there is no Docker job to decouple, and no macOS x86_64 job to build less often. ### Release note Azure Blob Storage support (`+AZURE_BLOB`, `+AZURE_STORAGE_VAULT`) is now built on aarch64 and macOS, not just x86_64 Linux. ### Check List (For Author) - Test - [x] Manual test (add detailed scripts or steps below) This PR touches `thirdparty/**` and `env.sh`, so its own `Build Third Party Libraries` run builds the whole tree on ubuntu-22.04, macos-15 and macos-14 - which is exactly where the hadoop change, the azure slimming and, for the first time, azure on macOS arm64 get exercised. **aarch64 Linux is the one path this workflow does not cover**; that is only built in apache/doris-thirdparty. The azure patch was verified to apply cleanly to a pristine `azure-sdk-for-cpp-azure-core_1.16.0` tarball, and the resulting `vcpkg.json` parses and declares only `curl`, `libxml2`, `openssl`, `wil`. `actionlint` is clean on the workflow. - Behavior changed: - [x] Yes. - `installed/{include,lib}/hadoop_hdfs/` no longer exists in the prebuilt third-party archive; only `hadoop_hdfs_3_4/` remains. Anything outside this repository reading the `hadoop_hdfs/` prefix needs updating. - Azure is now built and linked on aarch64 and macOS. **The third-party prebuilt has to be rebuilt before this lands**: BE/cloud on those platforms will pass `-DBUILD_AZURE=ON` and fail to link against an older prebuilt archive that has no azure libraries. `DISABLE_BUILD_AZURE=ON` still opts out. - `installed/` no longer contains the azure sub-libraries Doris does not link. - Does this need documentation? - [x] No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
