This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new 11286672f78 branch-4.1: [fix](be) Fix macOS build on branch-4.1
(#67011)
11286672f78 is described below
commit 11286672f7807f17d1316b1b737465658926d784
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Sat Aug 22 00:11:09 2026 +0800
branch-4.1: [fix](be) Fix macOS build on branch-4.1 (#67011)
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
`branch-4.1` does not compile on macOS / arm64 with clang 20. Most of
these are
platform assumptions that master already fixed but this branch did not
pick up;
three of them are latent portability bugs rather than macOS-only
cosmetics.
| File | Problem | Fix |
|---|---|---|
| `common/phdr_cache.h` | `_previous` is only used on Linux, so it trips
`-Werror,-Wunused-private-field` elsewhere | guard it, same as master |
| `exec/connector/jni_connector.cpp` | `int64_t` is `long long` on macOS
while `jlong` is `long`, so `FunctionCall::call()` rejects `int64_t*`
via its `requires(std::is_same_v<RETURN_TYPE, ReturnType>)` constraint |
use `jlong`, matching the other call sites in the same file |
| `exec/operator/scan_operator.h` | `atomic_shared_ptr<T> x = nullptr`
needs two user-defined conversions, which is ill-formed | declare
without initializer, same as master |
| `exec/operator/scan_operator.cpp` | the class exposes `store()`, not
`operator=` | assign through `store()`, same as master |
| `exprs/aggregate/aggregate_function_java_udaf.h` | same `jlong`
mismatch as above | `cast_set<jlong>`, exactly as master does |
| `io/cache/block_file_cache_factory.cpp` | `statfs::f_frsize` is a
Linux field and does not exist on macOS | select `f_bsize` under
`__APPLE__`, same as master |
| `util/md5.cpp` | the MD5 constants and the padding helper are only
referenced from the AVX2 path, so on arm64 they trip
`-Wunused-const-variable` / `-Wunused-function` | move them inside the
existing `#ifdef __AVX2__` block, matching master's layout |
| `CMakeLists.txt` | `ld` on macOS does not accept `--whole-archive` |
use `-Wl,-force_load` under `APPLE`, same as master |
| `CMakeLists.txt` | the Rust `sysinfo` crate inside `liblance_c.a`
calls into IOKit, which is not linked | link `IOKit` |
Everything above except the IOKit link is a straight port of what master
already
does; master simply has not been picked back to this branch.
Three of them are worth calling out as real portability issues rather
than
macOS-only noise:
- `statfs::f_frsize` does not exist outside Linux at all;
- `atomic_shared_ptr<T> x = nullptr` is ill-formed C++ regardless of
platform (it
only survives on Linux because that build selects the
`std::atomic<shared_ptr>`
alias instead of the hand-rolled libc++ fallback);
- `-Wl,--whole-archive` is GNU-ld specific.
With these, `DISABLE_BUILD_UI=ON bash build.sh --fe --be` completes on
macOS arm64
and produces a working BE.
### Release note
None
### Check List (For Author)
- Test
- [ ] Regression test
- [ ] Unit Test
- [x] Manual test (add detailed scripts or steps below)
Built on macOS 26.5 / arm64 with clang 20:
```
DISABLE_BUILD_UI=ON bash build.sh --fe --be
```
completes with `Successfully build Doris`, and the resulting BE starts
and
serves queries in a local cluster. **Not verified on Linux/x86** - the
`md5.cpp` change moves code inside the existing `#ifdef __AVX2__` block,
so CI
coverage on x86 is the check that matters there.
- [ ] No need to test or manual test. Explain why:
- Behavior changed:
- [x] No.
- [ ] Yes.
- Does this need documentation?
- [x] No.
- [ ] Yes.
### Check List (For Reviewer who merge this PR)
- [x] Confirm the release note
- [x] Confirm test cases
- [x] Confirm document
- [x] Add branch pick label
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
---
be/CMakeLists.txt | 17 +++++++++++++----
be/src/common/phdr_cache.h | 2 ++
be/src/exec/connector/jni_connector.cpp | 4 ++--
be/src/exec/operator/scan_operator.cpp | 12 ++++++------
be/src/exec/operator/scan_operator.h | 2 +-
be/src/exprs/aggregate/aggregate_function_java_udaf.h | 6 +++---
be/src/io/cache/block_file_cache_factory.cpp | 8 ++++++++
be/src/util/md5.cpp | 13 +++++++------
8 files changed, 42 insertions(+), 22 deletions(-)
diff --git a/be/CMakeLists.txt b/be/CMakeLists.txt
index fb32ae2782c..fefb8808016 100644
--- a/be/CMakeLists.txt
+++ b/be/CMakeLists.txt
@@ -692,10 +692,17 @@ set(DORIS_DEPENDENCIES ${DORIS_DEPENDENCIES}
clucene-contribs-lib)
if (ENABLE_PAIMON_CPP)
if (PAIMON_FACTORY_REGISTRY_LIBS)
- set(DORIS_DEPENDENCIES ${DORIS_DEPENDENCIES}
- -Wl,--whole-archive
- ${PAIMON_FACTORY_REGISTRY_LIBS}
- -Wl,--no-whole-archive)
+ if (APPLE)
+ foreach(lib ${PAIMON_FACTORY_REGISTRY_LIBS})
+ set(DORIS_DEPENDENCIES ${DORIS_DEPENDENCIES}
+ -Wl,-force_load,$<TARGET_FILE:${lib}>)
+ endforeach()
+ else()
+ set(DORIS_DEPENDENCIES ${DORIS_DEPENDENCIES}
+ -Wl,--whole-archive
+ ${PAIMON_FACTORY_REGISTRY_LIBS}
+ -Wl,--no-whole-archive)
+ endif()
endif()
# paimon-cpp internal dependencies (renamed with _paimon suffix)
@@ -779,6 +786,8 @@ else()
"-framework Foundation"
"-framework SystemConfiguration"
"-framework Security"
+ # liblance_c.a pulls in the Rust `sysinfo` crate, whose macOS backend
calls IOKit.
+ "-framework IOKit"
)
if (USE_JEMALLOC OR (NOT CMAKE_BUILD_TYPE STREQUAL "DEBUG" AND NOT
CMAKE_BUILD_TYPE STREQUAL "RELEASE"))
set(DORIS_LINK_LIBS
diff --git a/be/src/common/phdr_cache.h b/be/src/common/phdr_cache.h
index da18b698f0d..abf08a0500f 100644
--- a/be/src/common/phdr_cache.h
+++ b/be/src/common/phdr_cache.h
@@ -75,5 +75,7 @@ public:
ScopedPHDRCacheRead& operator=(const ScopedPHDRCacheRead&) = delete;
private:
+#if defined(__linux__) && !defined(THREAD_SANITIZER) && !defined(USE_MUSL)
bool _previous = false;
+#endif
};
diff --git a/be/src/exec/connector/jni_connector.cpp
b/be/src/exec/connector/jni_connector.cpp
index 8717341eb92..091e8889e5f 100644
--- a/be/src/exec/connector/jni_connector.cpp
+++ b/be/src/exec/connector/jni_connector.cpp
@@ -824,11 +824,11 @@ void JniConnector::_collect_profile_before_close() {
COUNTER_UPDATE(_open_scanner_time, _jni_scanner_open_watcher);
COUNTER_UPDATE(_fill_block_time, _fill_block_watcher);
- int64_t append_data_time = 0;
+ jlong append_data_time = 0;
auto append_time_status =
_jni_scanner_obj.call_long_method(env,
_jni_scanner_get_append_data_time)
.call(&append_data_time);
- int64_t create_vector_table_time = 0;
+ jlong create_vector_table_time = 0;
auto create_table_time_status =
_jni_scanner_obj.call_long_method(env,
_jni_scanner_get_create_vector_table_time)
.call(&create_vector_table_time);
diff --git a/be/src/exec/operator/scan_operator.cpp
b/be/src/exec/operator/scan_operator.cpp
index f945fa0a488..7eb6c8c2489 100644
--- a/be/src/exec/operator/scan_operator.cpp
+++ b/be/src/exec/operator/scan_operator.cpp
@@ -986,14 +986,14 @@ template <typename Derived>
Status ScanLocalState<Derived>::_start_scanners(
const std::list<std::shared_ptr<ScannerDelegate>>& scanners) {
auto& p = _parent->cast<typename Derived::Parent>();
- _scanner_ctx = ScannerContext::create_shared(state(), this,
p._output_tuple_desc,
- p.output_row_descriptor(),
scanners, p.limit(),
- _scan_dependency
+ _scanner_ctx.store(ScannerContext::create_shared(state(), this,
p._output_tuple_desc,
+
p.output_row_descriptor(), scanners, p.limit(),
+ _scan_dependency
#ifdef BE_TEST
- ,
-
max_scanners_concurrency(state())
+ ,
+
max_scanners_concurrency(state())
#endif
- );
+ ));
return Status::OK();
}
diff --git a/be/src/exec/operator/scan_operator.h
b/be/src/exec/operator/scan_operator.h
index 0086784e573..929a67d0755 100644
--- a/be/src/exec/operator/scan_operator.h
+++ b/be/src/exec/operator/scan_operator.h
@@ -320,7 +320,7 @@ protected:
VExprContextSPtrs _stale_expr_ctxs;
VExprContextSPtrs _common_expr_ctxs_push_down;
- atomic_shared_ptr<ScannerContext> _scanner_ctx = nullptr;
+ atomic_shared_ptr<ScannerContext> _scanner_ctx;
// colname -> cast dst type
std::map<std::string, DataTypePtr> _cast_types_for_variants;
diff --git a/be/src/exprs/aggregate/aggregate_function_java_udaf.h
b/be/src/exprs/aggregate/aggregate_function_java_udaf.h
index fbecd4cda0e..f92782a1e2b 100644
--- a/be/src/exprs/aggregate/aggregate_function_java_udaf.h
+++ b/be/src/exprs/aggregate/aggregate_function_java_udaf.h
@@ -125,7 +125,7 @@ public:
.with_arg((jboolean)is_single_place)
.with_arg(cast_set<jint>(row_num_start))
.with_arg(cast_set<jint>(row_num_end))
- .with_arg(places_address)
+ .with_arg(cast_set<jlong>(places_address))
.with_arg(cast_set<jint>(place_offset))
.with_arg(input_map)
.call();
@@ -170,7 +170,7 @@ public:
JNIEnv* env = nullptr;
RETURN_NOT_OK_STATUS_WITH_WARN(Jni::Env::Get(&env), "Java-Udaf reset
function");
return executor_obj.call_nonvirtual_void_method(env, executor_cl,
executor_reset_id)
- .with_arg(place)
+ .with_arg(cast_set<jlong>(place))
.call();
}
@@ -201,7 +201,7 @@ public:
long output_address;
RETURN_IF_ERROR(executor_obj.call_long_method(env,
executor_get_value_id)
- .with_arg(place)
+ .with_arg(cast_set<jlong>(place))
.with_arg(output_map)
.call(&output_address));
diff --git a/be/src/io/cache/block_file_cache_factory.cpp
b/be/src/io/cache/block_file_cache_factory.cpp
index ef0d1238c6f..c0bee420ae0 100644
--- a/be/src/io/cache/block_file_cache_factory.cpp
+++ b/be/src/io/cache/block_file_cache_factory.cpp
@@ -87,7 +87,11 @@ Status build_file_cache(const std::string& cache_base_path,
FileCacheSettings fi
LOG_ERROR("").tag("file cache path", cache_base_path).tag("error",
strerror(errno));
return Status::IOError("{} statfs error {}", cache_base_path,
strerror(errno));
}
+#if defined(__APPLE__)
+ const auto block_size = stat.f_bsize;
+#else
const auto block_size = stat.f_frsize ? stat.f_frsize : stat.f_bsize;
+#endif
size_t disk_capacity =
static_cast<size_t>(static_cast<size_t>(stat.f_blocks) *
static_cast<size_t>(block_size));
if (file_cache_settings.capacity == 0 || disk_capacity <
file_cache_settings.capacity) {
@@ -409,7 +413,11 @@ std::string validate_capacity(const std::string& path,
int64_t new_capacity,
valid_capacity = 0; // caller will handle the error
return ret;
}
+#if defined(__APPLE__)
+ const auto block_size = stat.f_bsize;
+#else
const auto block_size = stat.f_frsize ? stat.f_frsize : stat.f_bsize;
+#endif
size_t disk_capacity =
static_cast<size_t>(static_cast<size_t>(stat.f_blocks) *
static_cast<size_t>(block_size));
if (new_capacity == 0 || disk_capacity < new_capacity) {
diff --git a/be/src/util/md5.cpp b/be/src/util/md5.cpp
index b54e4eed8de..9c9b5d7f68c 100644
--- a/be/src/util/md5.cpp
+++ b/be/src/util/md5.cpp
@@ -31,10 +31,6 @@ namespace doris {
namespace {
-constexpr uint32_t MD5_A0 = 0x67452301;
-constexpr uint32_t MD5_B0 = 0xefcdab89;
-constexpr uint32_t MD5_C0 = 0x98badcfe;
-constexpr uint32_t MD5_D0 = 0x10325476;
constexpr unsigned char MD5_DUMMY_INPUT = 0;
void md5_to_hex(const unsigned char* digest, char* out) {
@@ -45,6 +41,13 @@ void md5_to_hex(const unsigned char* digest, char* out) {
}
}
+#ifdef __AVX2__
+
+constexpr uint32_t MD5_A0 = 0x67452301;
+constexpr uint32_t MD5_B0 = 0xefcdab89;
+constexpr uint32_t MD5_C0 = 0x98badcfe;
+constexpr uint32_t MD5_D0 = 0x10325476;
+
size_t md5_num_blocks(size_t len) {
return (len + 9 + 63) / 64;
}
@@ -63,8 +66,6 @@ size_t md5_pad_final_blocks(const unsigned char* data, size_t
len, unsigned char
return final_count;
}
-#ifdef __AVX2__
-
struct AVX2MD5Ops {
using Vec = __m256i;
static constexpr size_t LANES = 8;
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
