morningman opened a new pull request, #66839:
URL: https://github.com/apache/doris/pull/66839
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Two problems with `azure-sdk-for-cpp`.
**1. It was only ever built on x86_64 Linux.**
`env.sh` forced `DISABLE_BUILD_AZURE=ON` on aarch64 and macOS, and
`build_azure`
skipped `Darwin` a second time. As a result BE and the cloud meta-service
lost
`+AZURE_BLOB` and `+AZURE_STORAGE_VAULT` on ARM and macOS. Nothing in the
recipe
was genuinely x86_64-Linux-specific, only two details:
- `-ldl`: Apple has no libdl at all (dlopen and friends live in libSystem),
so
`-ldl` fails the link there. It is now passed on non-Apple platforms only,
both
in `build_azure` and in the openssl vcpkg wrapper the existing patch
touches.
- vcpkg ships no prebuilt cmake/ninja/curl for aarch64 Linux and needs
`VCPKG_FORCE_SYSTEM_BINARIES=1` there.
The vcpkg triplet is now derived from `uname` (`x64-linux`, `arm64-linux`,
`x64-osx`, `arm64-osx`) instead of being left to vcpkg's host detection.
**2. It was the most expensive package in the tree, almost none of it for
Doris.**
In `apache/doris-thirdparty` run 31988123966, azure took **26m06s** of the
2h43m
x86_64 Linux third-party build. Where it went:
| phase | time |
|---|---|
| `vcpkg install` | 22m11s |
| - protobuf 5.29.3 | 12m29s |
| - opentelemetry-cpp | 4m35s |
| - abseil | 1m58s |
| - openssl 1.1.1n | 1m38s |
| - curl / libxml2 / zlib / uAMQP / utf8-range | ~1m30s |
| azure SDK configure + build | ~3m55s |
**protobuf, abseil and utf8-range are there only because `vcpkg.json`
declares
`opentelemetry-cpp` unconditionally** - roughly 19 of the 26 minutes. Doris
does
not use azure's OpenTelemetry tracing.
On top of that, vcpkg builds every port twice (`Building x64-linux-dbg` then
`Building x64-linux-rel`) and Doris links only the release halves, and the
SDK
builds appconfiguration, attestation, eventhubs, keyvault, tables, template,
uAMQP, storage-files-datalake, storage-files-shares and storage-queues while
`be/cmake/thirdparty.cmake` links exactly four targets: `azure-core`,
`azure-identity`, `azure-storage-blobs`, `azure-storage-common`.
So this PR:
- extends `thirdparty/patches/azure-sdk-for-cpp-azure-core_1.16.0.patch` to
drop
`opentelemetry-cpp` and the uAMQP C libraries from `vcpkg.json`, flip
`DISABLE_AMQP` / `DISABLE_AZURE_CORE_OPENTELEMETRY` to `ON`, and trim the
sub-project list to core + identity + storage-{common,blobs};
- writes an overlay triplet in `build_azure` that sets `VCPKG_BUILD_TYPE
release`,
which is the only supported way to ask vcpkg for a release-only build.
What is left for vcpkg to build is curl, libxml2, openssl 1.1.1n and zlib,
release only.
**Also in this PR: an opt-in ccache for the third-party build.**
`ENABLE_THIRDPARTY_CCACHE=ON` exports `CMAKE_C_COMPILER_LAUNCHER` /
`CMAKE_CXX_COMPILER_LAUNCHER`, which CMake initialises from the environment,
so no
cmake invocation in the script changes. It deliberately does not prefix
`CC`/`CXX`
with ccache: CMake would split `"ccache clang"` into the compiler plus a
`CMAKE_<LANG>_FLAGS` entry, and that leaks into whatever a package exports.
Autotools packages are left alone. It is off by default because prefixing the
compiler changes how every package configures itself;
`apache/doris-thirdparty`
CI turns it on, where a warm ccache turns a rebuild triggered by one changed
package into minutes instead of hours.
### Release note
Azure Blob Storage support (`+AZURE_BLOB`, `+AZURE_STORAGE_VAULT`) is now
built on
aarch64 and macOS, not just x86_64 Linux.
### Check List (For Author)
- Test
- [x] Manual test (add detailed scripts or steps below)
The patch was verified to apply cleanly to a pristine
`azure-sdk-for-cpp-azure-core_1.16.0` tarball, and the resulting
`vcpkg.json`
parses and declares only `curl`, `libxml2`, `openssl`, `wil`. The build
itself is exercised by the `apache/doris-thirdparty` pipeline, which
is the
only place all four platforms are built.
- Behavior changed:
- [x] Yes.
- Azure is now built and linked on aarch64 and macOS. **The third-party
prebuilt has to be rebuilt before this lands**: BE/cloud on those
platforms
will pass `-DBUILD_AZURE=ON` and fail to link against an older
prebuilt
archive that has no azure libraries. `DISABLE_BUILD_AZURE=ON` still
opts out.
- `installed/` no longer contains the azure sub-libraries Doris does
not link
(appconfiguration, attestation, eventhubs, keyvault, tables,
template,
storage-files-datalake, storage-files-shares, storage-queues), nor
the
opentelemetry/protobuf/abseil artifacts vcpkg used to install under
the
azure build tree.
- Does this need documentation?
- [x] No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
