This is an automated email from the ASF dual-hosted git repository.
Gabriel39 pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new f0e00809c81 branch-4.1:[chore](thirdparty) Upgrade Apache Arrow to
24.0.0(#66221) (#66480)
f0e00809c81 is described below
commit f0e00809c812ae6d10fa70a9188a953424063626
Author: daidai <[email protected]>
AuthorDate: Thu Aug 6 21:50:34 2026 +0800
branch-4.1:[chore](thirdparty) Upgrade Apache Arrow to 24.0.0(#66221)
(#66480)
### What problem does this PR solve?
bp #66221
Problem Summary: Doris used Apache Arrow 17.0.0, which blocked consuming
newer Arrow fixes and APIs. Upgrade the C++ third-party dependency to
24.0.0, align xsimd with Arrow 24, port the Paimon row-group, INT96, and
Parquet LZO compatibility patches, add Arrow Compute to the static
dependency graph, and migrate removed Arrow and Parquet APIs while
preserving Decimal128 semantics.
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [X] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [x] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
---
.github/workflows/be-ut-mac.yml | 50 ++-
.github/workflows/build-thirdparty.yml | 38 +-
be/CMakeLists.txt | 2 +-
be/cmake/thirdparty.cmake | 1 +
be/src/format/table/iceberg/arrow_schema_util.cpp | 2 +-
.../data_type_serde_decimal_test.cpp | 4 +-
be/test/format/parquet/parquet_expr_test.cpp | 2 +-
be/test/format_v2/parquet/parquet_reader_test.cpp | 58 ++-
.../format_v2/table/remote_doris_reader_test.cpp | 2 +-
build.sh | 75 +++-
docker/compilation/Dockerfile.gcc10 | 3 +-
docker/compilation/Dockerfile.gcc7 | 119 -----
docker/compilation/arm/Dockerfile | 12 +
.../iceberg/write/test_iceberg_write_stats2.out | 2 +-
.../iceberg/iceberg_schema_change_ddl.groovy | 2 +-
run-be-ut.sh | 25 ++
thirdparty/arrow-paimon-vars.sh | 408 +++++++++++++++++
thirdparty/build-thirdparty.sh | 16 +-
thirdparty/download-thirdparty.sh | 64 ++-
thirdparty/paimon-cpp-cache.cmake | 3 +-
...rrow-24.0.0-force-write-int96-timestamps.patch} | 42 +-
...0.0-lzo.patch => apache-arrow-24.0.0-lzo.patch} | 72 ++-
...imon.patch => apache-arrow-24.0.0-paimon.patch} | 116 ++---
.../paimon-cpp-arrow-24-compatibility.patch | 64 +++
.../patches/paimon-cpp-arrow-24-compute.patch | 44 ++
thirdparty/test/arrow-paimon-lifecycle-test.sh | 492 +++++++++++++++++++++
thirdparty/vars.sh | 31 +-
27 files changed, 1442 insertions(+), 307 deletions(-)
diff --git a/.github/workflows/be-ut-mac.yml b/.github/workflows/be-ut-mac.yml
index 4f3db038dba..c64dc52e0de 100644
--- a/.github/workflows/be-ut-mac.yml
+++ b/.github/workflows/be-ut-mac.yml
@@ -47,9 +47,23 @@ jobs:
- 'be/**'
- 'gensrc/proto/**'
- 'gensrc/thrift/**'
+ arrow_paimon_changes:
+ - '.github/workflows/be-ut-mac.yml'
+ - 'build.sh'
+ - 'env.sh'
+ - 'thirdparty/arrow-paimon-vars.sh'
+ - 'thirdparty/vars.sh'
+ - 'thirdparty/download-thirdparty.sh'
+ - 'thirdparty/build-thirdparty.sh'
+ - 'thirdparty/paimon-cpp-cache.cmake'
+ - 'thirdparty/patches/apache-arrow-*.patch'
+ - 'thirdparty/patches/paimon-cpp-*.patch'
- name: Ccache ${{ github.ref }}
- if: ${{ github.event_name == 'schedule' ||
steps.filter.outputs.be_changes == 'true' }}
+ if: |
+ github.event_name == 'schedule' ||
+ steps.filter.outputs.be_changes == 'true' ||
+ steps.filter.outputs.arrow_paimon_changes == 'true'
uses: ./.github/actions/ccache-action
with:
key: BE-UT-macOS
@@ -57,7 +71,10 @@ jobs:
restore-keys: BE-UT-macOS-
- name: Run UT ${{ github.ref }}
- if: ${{ github.event_name == 'schedule' ||
steps.filter.outputs.be_changes == 'true' }}
+ if: |
+ github.event_name == 'schedule' ||
+ steps.filter.outputs.be_changes == 'true' ||
+ steps.filter.outputs.arrow_paimon_changes == 'true'
run: |
cellars=(
'm4'
@@ -84,6 +101,15 @@ jobs:
)
brew install "${cellars[@]}" || true
+ # Paimon's bundled dependencies still require pre-CMake-4 policy
+ # compatibility. Match the supported version used by the full
+ # thirdparty jobs instead of relying on Homebrew's latest CMake.
+ brew unlink cmake || true
+ wget
https://github.com/Kitware/CMake/releases/download/v3.25.3/cmake-3.25.3-macos-universal.tar.gz
+ tar -xzf cmake-3.25.3-macos-universal.tar.gz
+ sudo cp -r cmake-3.25.3-macos-universal/CMake.app/Contents/*
/usr/local/
+ export CUSTOM_CMAKE=/usr/local/bin/cmake
+ "${CUSTOM_CMAKE}" --version
pushd thirdparty
branch="${{ github.base_ref }}"
if [[ -z "${branch}" ]] || [[ "${branch}" == 'master' ]]; then
@@ -94,6 +120,26 @@ jobs:
-o doris-thirdparty-prebuilt-darwin-x86_64.tar.xz
fi
tar -xvf doris-thirdparty-prebuilt-darwin-x86_64.tar.xz
+
+ # Rebuild the Arrow/Paimon stack when its inputs change or the shared
+ # prebuilt predates the selected Arrow version/component closure.
+ # The artifact check also covers scheduled and later BE-only builds,
+ # where Paths Filter is skipped or reports no Arrow/Paimon changes.
+ # shellcheck source=thirdparty/arrow-paimon-vars.sh
+ . ./arrow-paimon-vars.sh
+ arrow_paimon_prebuilt_is_valid=false
+ if arrow_paimon_prebuilt_valid installed; then
+ arrow_paimon_prebuilt_is_valid=true
+ fi
+ if [[ "${{ steps.filter.outputs.arrow_paimon_changes }}" == "true" ||
+ "${arrow_paimon_prebuilt_is_valid}" != "true" ]]; then
+ curl -L
https://github.com/apache/doris-thirdparty/releases/download/automation/doris-thirdparty-source.tgz
\
+ -o doris-thirdparty-source.tgz
+ tar -zxvf doris-thirdparty-source.tgz
+ ./download-thirdparty.sh arrow paimon_cpp xsimd brotli
+ export MACOSX_DEPLOYMENT_TARGET=12.0
+ ./build-thirdparty.sh -j "$(nproc)" arrow paimon_cpp
+ fi
popd
export JAVA_HOME="${JAVA_HOME_17_X64%\/}"
diff --git a/.github/workflows/build-thirdparty.yml
b/.github/workflows/build-thirdparty.yml
index cc707cf516b..7144cfe8425 100644
--- a/.github/workflows/build-thirdparty.yml
+++ b/.github/workflows/build-thirdparty.yml
@@ -39,6 +39,7 @@ jobs:
github.event.issue.user.login == 'github-actions[bot]')
outputs:
thirdparty_changes: ${{ steps.filter.outputs.thirdparty_changes }}
+ arrow_paimon_lifecycle_changes: ${{
steps.filter.outputs.arrow_paimon_lifecycle_changes }}
steps:
- name: Checkout ${{ github.ref }}
uses: actions/checkout@v4
@@ -54,6 +55,30 @@ jobs:
thirdparty_changes:
- 'thirdparty/**'
- 'env.sh'
+ arrow_paimon_lifecycle_changes:
+ - 'build.sh'
+ - 'env.sh'
+ - 'thirdparty/arrow-paimon-vars.sh'
+ - 'thirdparty/vars.sh'
+ - 'thirdparty/build-thirdparty.sh'
+ - 'thirdparty/download-thirdparty.sh'
+ - 'thirdparty/paimon-cpp-cache.cmake'
+ - 'thirdparty/patches/apache-arrow-*.patch'
+ - 'thirdparty/patches/paimon-cpp-*.patch'
+ - 'thirdparty/test/arrow-paimon-lifecycle-test.sh'
+ - '.github/workflows/build-thirdparty.yml'
+
+ arrow_paimon_lifecycle_test:
+ name: Arrow/Paimon Lifecycle Test
+ needs: changes
+ if: ${{ needs.changes.outputs.arrow_paimon_lifecycle_changes == 'true' }}
+ runs-on: ubuntu-22.04
+ steps:
+ - name: Checkout ${{ github.ref }}
+ uses: actions/checkout@v4
+
+ - name: Test focused thirdparty lifecycle
+ run: thirdparty/test/arrow-paimon-lifecycle-test.sh
build_linux:
name: Build Third Party Libraries (Linux)
@@ -195,9 +220,9 @@ jobs:
# Install specific version of cmake
brew unlink cmake || true
- wget
https://github.com/Kitware/CMake/releases/download/v3.22.1/cmake-3.22.1-macos-universal.tar.gz
- tar -xzf cmake-3.22.1-macos-universal.tar.gz
- sudo cp -r cmake-3.22.1-macos-universal/CMake.app/Contents/*
/usr/local/
+ wget
https://github.com/Kitware/CMake/releases/download/v3.25.3/cmake-3.25.3-macos-universal.tar.gz
+ tar -xzf cmake-3.25.3-macos-universal.tar.gz
+ sudo cp -r cmake-3.25.3-macos-universal/CMake.app/Contents/*
/usr/local/
cmake --version
- name: Build
@@ -260,9 +285,9 @@ jobs:
# Install specific version of cmake
brew unlink cmake || true
- wget
https://github.com/Kitware/CMake/releases/download/v3.22.1/cmake-3.22.1-macos-universal.tar.gz
- tar -xzf cmake-3.22.1-macos-universal.tar.gz
- sudo cp -r cmake-3.22.1-macos-universal/CMake.app/Contents/*
/usr/local/
+ wget
https://github.com/Kitware/CMake/releases/download/v3.25.3/cmake-3.25.3-macos-universal.tar.gz
+ tar -xzf cmake-3.25.3-macos-universal.tar.gz
+ sudo cp -r cmake-3.25.3-macos-universal/CMake.app/Contents/*
/usr/local/
cmake --version
- name: Build
@@ -278,4 +303,3 @@ jobs:
cd thirdparty
#./build-thirdparty.sh -j "$(nproc)"
./build-thirdparty.sh -j 2
-
diff --git a/be/CMakeLists.txt b/be/CMakeLists.txt
index f3c6aa3379a..05bdc796351 100644
--- a/be/CMakeLists.txt
+++ b/be/CMakeLists.txt
@@ -653,7 +653,7 @@ endif()
set(PAIMON_FACTORY_REGISTRY_LIBS)
if (ENABLE_PAIMON_CPP)
# Plan B: Doris Arrow is now built with COMPUTE/DATASET/ACERO/FILESYSTEM,
- # so arrow, arrow_dataset, arrow_acero are all in COMMON_THIRDPARTY via
+ # so arrow, arrow_compute, arrow_dataset, arrow_acero are all in
COMMON_THIRDPARTY via
# thirdparty.cmake. paimon-cpp reuses the same Arrow (no paimon_deps).
# No dual-stack selection needed — single Arrow for everything.
diff --git a/be/cmake/thirdparty.cmake b/be/cmake/thirdparty.cmake
index 91e3e40dd39..eed1823515f 100644
--- a/be/cmake/thirdparty.cmake
+++ b/be/cmake/thirdparty.cmake
@@ -105,6 +105,7 @@ add_thirdparty(brotlidec LIB64)
add_thirdparty(brotlienc LIB64)
add_thirdparty(zstd LIB64)
add_thirdparty(arrow LIB64)
+add_thirdparty(arrow_compute LIB64)
add_thirdparty(arrow_flight LIB64)
add_thirdparty(arrow_flight_sql LIB64)
add_thirdparty(arrow_dataset LIB64)
diff --git a/be/src/format/table/iceberg/arrow_schema_util.cpp
b/be/src/format/table/iceberg/arrow_schema_util.cpp
index d53faf0062b..77989a64812 100644
--- a/be/src/format/table/iceberg/arrow_schema_util.cpp
+++ b/be/src/format/table/iceberg/arrow_schema_util.cpp
@@ -98,7 +98,7 @@ Status ArrowSchemaUtil::convert_to(const
iceberg::NestedField& field,
case iceberg::TypeID::DECIMAL: {
auto* dt = dynamic_cast<DecimalType*>(field.field_type());
- arrow_type = arrow::decimal(dt->get_precision(), dt->get_scale());
+ arrow_type = arrow::decimal128(dt->get_precision(), dt->get_scale());
break;
}
diff --git a/be/test/core/data_type_serde/data_type_serde_decimal_test.cpp
b/be/test/core/data_type_serde/data_type_serde_decimal_test.cpp
index 4394f012b7c..fab8453a816 100644
--- a/be/test/core/data_type_serde/data_type_serde_decimal_test.cpp
+++ b/be/test/core/data_type_serde/data_type_serde_decimal_test.cpp
@@ -292,7 +292,7 @@ TEST_F(DataTypeDecimalSerDeTest, serdes) {
// Run with UBSan enabled to catch misalignment errors.
TEST_F(DataTypeDecimalSerDeTest, ArrowMemNotAligned) {
// 1.Prepare the data.
- arrow::Decimal128Builder builder(arrow::decimal(38, 30));
+ arrow::Decimal128Builder builder(arrow::decimal128(38, 30));
std::vector<std::string> decimal_strings = {"12345.67", "89.10",
"1112.13", "1415.16",
"1718.19"};
@@ -318,7 +318,7 @@ TEST_F(DataTypeDecimalSerDeTest, ArrowMemNotAligned) {
// 4. Create Arrow array with unaligned memory
auto unaligned_buffer = arrow::Buffer::Wrap(unaligned_data, num_elements *
element_size);
- auto arr = std::make_shared<arrow::DecimalArray>(arrow::decimal(38, 30),
num_elements,
+ auto arr = std::make_shared<arrow::DecimalArray>(arrow::decimal128(38,
30), num_elements,
unaligned_buffer);
const auto* raw_values_ptr = arr->raw_values();
diff --git a/be/test/format/parquet/parquet_expr_test.cpp
b/be/test/format/parquet/parquet_expr_test.cpp
index 0e1774fa634..d8cfb8765b6 100644
--- a/be/test/format/parquet/parquet_expr_test.cpp
+++ b/be/test/format/parquet/parquet_expr_test.cpp
@@ -339,7 +339,7 @@ public:
outfile = std::move(result_file).ValueUnsafe();
::parquet::WriterProperties::Builder builder;
- builder.version(::parquet::ParquetVersion::PARQUET_2_0);
+ builder.version(::parquet::ParquetVersion::PARQUET_2_6);
builder.data_page_version(::parquet::ParquetDataPageVersion::V2);
builder.enable_write_page_index();
builder.compression(::parquet::Compression::SNAPPY);
diff --git a/be/test/format_v2/parquet/parquet_reader_test.cpp
b/be/test/format_v2/parquet/parquet_reader_test.cpp
index abc67839d2d..fca95c06177 100644
--- a/be/test/format_v2/parquet/parquet_reader_test.cpp
+++ b/be/test/format_v2/parquet/parquet_reader_test.cpp
@@ -1004,16 +1004,25 @@ void write_sparse_filter_nested_parquet_file(const
std::string& file_path) {
::parquet::WriterProperties::Builder builder;
builder.version(::parquet::ParquetVersion::PARQUET_2_6);
- // V2 and page-index writers preserve record boundaries, so use V1 here to
produce the
- // continuation pages that the reader must still handle correctly.
+ // Arrow 24 enables page indexes by default. V2 and page-index writers
preserve record
+ // boundaries, so use V1 without a page index to produce the continuation
pages that the
+ // reader must still handle correctly.
builder.data_page_version(::parquet::ParquetDataPageVersion::V1);
+ builder.disable_write_page_index();
builder.compression(::parquet::Compression::UNCOMPRESSED);
builder.disable_dictionary();
builder.write_batch_size(8);
- builder.data_pagesize(64);
+ // Arrow 24 checks repeated-column page limits only at record or
WriteBatch boundaries.
+ // A one-byte target deterministically flushes at every eligible boundary,
including the
+ // deliberately split wide record below.
+ builder.data_pagesize(1);
auto writer = ::parquet::ParquetFileWriter::Open(out, schema,
builder.build());
auto* row_group = writer->AppendRowGroup();
+ // Split each wide record across WriteBatch calls. The first half flushes
at the one-byte page
+ // limit, so the second call deterministically creates a page whose first
repetition level is 1.
+ constexpr int64_t SPANNING_BATCH_VALUES = SPANNING_NESTED_VALUES / 2;
+
auto* id_writer =
static_cast<::parquet::Int32Writer*>(row_group->NextColumn());
const int32_t ids[] = {1, 2, 3, 4, 5, 6};
EXPECT_EQ(id_writer->WriteBatch(6, nullptr, nullptr, ids), 6);
@@ -1042,16 +1051,32 @@ void write_sparse_filter_nested_parquet_file(const
std::string& file_path) {
map_keys.push_back(6000);
auto* map_key_writer =
static_cast<::parquet::Int32Writer*>(row_group->NextColumn());
-
EXPECT_EQ(map_key_writer->WriteBatch(static_cast<int64_t>(map_repetition_levels.size()),
- map_key_definition_levels.data(),
+ constexpr int64_t MAP_PREFIX_LEVELS = 4;
+ constexpr int64_t MAP_KEY_PREFIX_VALUES = 2;
+ constexpr int64_t MAP_SPLIT_LEVELS = MAP_PREFIX_LEVELS +
SPANNING_BATCH_VALUES;
+ constexpr int64_t MAP_KEY_SPLIT_VALUES = MAP_KEY_PREFIX_VALUES +
SPANNING_BATCH_VALUES;
+ EXPECT_EQ(map_key_writer->WriteBatch(MAP_SPLIT_LEVELS,
map_key_definition_levels.data(),
map_repetition_levels.data(),
map_keys.data()),
- static_cast<int64_t>(map_keys.size()));
+ MAP_KEY_SPLIT_VALUES);
+ EXPECT_EQ(map_key_writer->WriteBatch(
+ static_cast<int64_t>(map_repetition_levels.size()) -
MAP_SPLIT_LEVELS,
+ map_key_definition_levels.data() + MAP_SPLIT_LEVELS,
+ map_repetition_levels.data() + MAP_SPLIT_LEVELS,
+ map_keys.data() + MAP_KEY_SPLIT_VALUES),
+ static_cast<int64_t>(map_keys.size()) - MAP_KEY_SPLIT_VALUES);
map_key_writer->Close();
auto* map_value_writer =
static_cast<::parquet::ByteArrayWriter*>(row_group->NextColumn());
-
EXPECT_EQ(map_value_writer->WriteBatch(static_cast<int64_t>(map_repetition_levels.size()),
- map_value_definition_levels.data(),
+ constexpr int64_t MAP_VALUE_PREFIX_VALUES = 1;
+ constexpr int64_t MAP_VALUE_SPLIT_VALUES = MAP_VALUE_PREFIX_VALUES +
SPANNING_BATCH_VALUES;
+ EXPECT_EQ(map_value_writer->WriteBatch(MAP_SPLIT_LEVELS,
map_value_definition_levels.data(),
map_repetition_levels.data(),
map_values.data()),
- static_cast<int64_t>(map_values.size()));
+ MAP_VALUE_SPLIT_VALUES);
+ EXPECT_EQ(map_value_writer->WriteBatch(
+ static_cast<int64_t>(map_repetition_levels.size()) -
MAP_SPLIT_LEVELS,
+ map_value_definition_levels.data() + MAP_SPLIT_LEVELS,
+ map_repetition_levels.data() + MAP_SPLIT_LEVELS,
+ map_values.data() + MAP_VALUE_SPLIT_VALUES),
+ static_cast<int64_t>(map_values.size()) -
MAP_VALUE_SPLIT_VALUES);
map_value_writer->Close();
std::vector<int16_t> element_repetition_levels {0, 0, 0, 1, 0};
@@ -1069,10 +1094,19 @@ void write_sparse_filter_nested_parquet_file(const
std::string& file_path) {
element_definition_levels.push_back(3);
auto* element_writer =
static_cast<::parquet::Int32Writer*>(row_group->NextColumn());
-
EXPECT_EQ(element_writer->WriteBatch(static_cast<int64_t>(element_repetition_levels.size()),
- element_definition_levels.data(),
+ constexpr int64_t ELEMENT_PREFIX_LEVELS = 5;
+ constexpr int64_t ELEMENT_PREFIX_VALUES = 2;
+ constexpr int64_t ELEMENT_SPLIT_LEVELS = ELEMENT_PREFIX_LEVELS +
SPANNING_BATCH_VALUES;
+ constexpr int64_t ELEMENT_SPLIT_VALUES = ELEMENT_PREFIX_VALUES +
SPANNING_BATCH_VALUES;
+ EXPECT_EQ(element_writer->WriteBatch(ELEMENT_SPLIT_LEVELS,
element_definition_levels.data(),
element_repetition_levels.data(),
element_values.data()),
- static_cast<int64_t>(element_values.size()));
+ ELEMENT_SPLIT_VALUES);
+ EXPECT_EQ(element_writer->WriteBatch(
+ static_cast<int64_t>(element_repetition_levels.size()) -
ELEMENT_SPLIT_LEVELS,
+ element_definition_levels.data() + ELEMENT_SPLIT_LEVELS,
+ element_repetition_levels.data() + ELEMENT_SPLIT_LEVELS,
+ element_values.data() + ELEMENT_SPLIT_VALUES),
+ static_cast<int64_t>(element_values.size()) -
ELEMENT_SPLIT_VALUES);
element_writer->Close();
auto* marker_writer =
static_cast<::parquet::Int32Writer*>(row_group->NextColumn());
const int16_t marker_definition_levels[] = {1, 0, 1, 1, 1, 1};
diff --git a/be/test/format_v2/table/remote_doris_reader_test.cpp
b/be/test/format_v2/table/remote_doris_reader_test.cpp
index a8affbde311..ad8149c9d05 100644
--- a/be/test/format_v2/table/remote_doris_reader_test.cpp
+++ b/be/test/format_v2/table/remote_doris_reader_test.cpp
@@ -213,7 +213,7 @@ TFileRangeDesc remote_doris_range(const
BlockingFlightServer& server) {
auto range = remote_doris_range();
auto& params = range.table_format_params.remote_doris_params;
params.__set_location_uri("grpc://localhost:" +
std::to_string(server.port()));
- arrow::flight::Ticket ticket {.ticket = "ticket"};
+ arrow::flight::Ticket ticket {"ticket"};
params.__set_ticket(ticket.SerializeToString().ValueOrDie());
return range;
}
diff --git a/build.sh b/build.sh
index 70673a15e49..2200aa1c71d 100755
--- a/build.sh
+++ b/build.sh
@@ -432,22 +432,74 @@ fi
if [[ "${HELP}" -eq 1 ]]; then
usage
fi
+
+if [[ "${CLEAN}" -eq 1 && "${BUILD_BE}" -eq 0 && "${BUILD_FE}" -eq 0 &&
${BUILD_CLOUD} -eq 0 ]]; then
+ clean_gensrc
+ clean_be
+ clean_fe
+ exit 0
+fi
+
# build thirdparty libraries if necessary. check last thirdparty lib
installation
if [[ "${TARGET_SYSTEM}" == 'Darwin' ]]; then
LAST_THIRDPARTY_LIB='libbrotlienc.a'
else
LAST_THIRDPARTY_LIB='hadoop_hdfs/native/libhdfs.a'
fi
-if [[ ! -f "${DORIS_THIRDPARTY}/installed/lib/${LAST_THIRDPARTY_LIB}" ]]; then
- echo "Thirdparty libraries need to be build ..."
- # need remove all installed pkgs because some lib like lz4 will throw
error if its lib alreay exists
- rm -rf "${DORIS_THIRDPARTY}/installed"
- if [[ "${CLEAN}" -eq 0 ]]; then
- "${DORIS_THIRDPARTY}/build-thirdparty.sh" -j "${PARALLEL}"
- else
- "${DORIS_THIRDPARTY}/build-thirdparty.sh" -j "${PARALLEL}" --clean
+# The final-library sentinel only proves that some third-party build
completed. It cannot
+# distinguish an older prebuilt whose Arrow/Paimon closure predates the
selected sources.
+# shellcheck source=thirdparty/arrow-paimon-vars.sh
+. "${DORIS_HOME}/thirdparty/arrow-paimon-vars.sh"
+NEED_ARROW_PAIMON_THIRDPARTY=false
+if [[ "${BUILD_BE}" -eq 1 || "${BUILD_CLOUD}" -eq 1 ||
+ "${BUILD_META_TOOL}" == "ON" || "${BUILD_FILE_CACHE_MICROBENCH_TOOL}" ==
"ON" ||
+ "${BUILD_INDEX_TOOL}" == "ON" ]]; then
+ NEED_ARROW_PAIMON_THIRDPARTY=true
+fi
+
+rebuild_thirdparty_libraries() {
+ local remove_installed="$1"
+ shift
+ local build_script="${DORIS_THIRDPARTY}/build-thirdparty.sh"
+ local build_args=(-j "${PARALLEL}")
+ local selected_thirdparty_root
+ local checkout_thirdparty_root
+
+ if [[ ! -f "${build_script}" ]]; then
+ echo "Cannot rebuild thirdparty libraries: ${build_script} is
missing." >&2
+ echo "DORIS_THIRDPARTY=${DORIS_THIRDPARTY} is an install-only or
incomplete prefix. Use a matching compilation image/prebuilt, or unset
DORIS_THIRDPARTY to rebuild with this checkout's thirdparty tree." >&2
+ exit 1
+ fi
+ selected_thirdparty_root="$(cd "${DORIS_THIRDPARTY}" && pwd -P)"
+ checkout_thirdparty_root="$(cd "${DORIS_HOME}/thirdparty" && pwd -P)"
+ if [[ "${selected_thirdparty_root}" != "${checkout_thirdparty_root}" ]];
then
+ echo "Cannot rebuild thirdparty libraries with an external source
tree: ${selected_thirdparty_root}." >&2
+ echo "Unset DORIS_THIRDPARTY to rebuild with this checkout's
thirdparty tree, then use the resulting version-matched installation." >&2
+ exit 1
fi
+ build_script="${checkout_thirdparty_root}/build-thirdparty.sh"
+ if [[ "${remove_installed}" == "true" ]]; then
+ # Some libraries, such as lz4, fail when an earlier installation
remains.
+ rm -rf "${DORIS_THIRDPARTY}/installed"
+ fi
+ if [[ "${CLEAN}" -eq 1 ]]; then
+ build_args+=(--clean)
+ fi
+ bash "${build_script}" "${build_args[@]}" "$@"
+ if ! arrow_paimon_prebuilt_valid "${DORIS_THIRDPARTY}/installed"; then
+ echo "Rebuilt Arrow/Paimon artifacts do not match this checkout's
selected inputs." >&2
+ exit 1
+ fi
+}
+
+if [[ ! -f "${DORIS_THIRDPARTY}/installed/lib/${LAST_THIRDPARTY_LIB}" ]]; then
+ echo "Thirdparty libraries need to be build ..."
+ rebuild_thirdparty_libraries true
+elif [[ "${NEED_ARROW_PAIMON_THIRDPARTY}" == "true" ]] &&
+ ! arrow_paimon_prebuilt_valid "${DORIS_THIRDPARTY}/installed"; then
+ echo "Arrow/Paimon thirdparty libraries need to be rebuilt ..."
+ rebuild_thirdparty_libraries false "${ARROW_PAIMON_BUILD_PACKAGES[@]}"
fi
update_submodule() {
@@ -486,13 +538,6 @@ update_submodule() {
fi
}
-if [[ "${CLEAN}" -eq 1 && "${BUILD_BE}" -eq 0 && "${BUILD_FE}" -eq 0 &&
${BUILD_CLOUD} -eq 0 ]]; then
- clean_gensrc
- clean_be
- clean_fe
- exit 0
-fi
-
if [[ -z "${GLIBC_COMPATIBILITY}" ]]; then
if [[ "${TARGET_SYSTEM}" != 'Darwin' ]]; then
GLIBC_COMPATIBILITY='ON'
diff --git a/docker/compilation/Dockerfile.gcc10
b/docker/compilation/Dockerfile.gcc10
index e0026ce5f59..f066fd3af1b 100644
--- a/docker/compilation/Dockerfile.gcc10
+++ b/docker/compilation/Dockerfile.gcc10
@@ -26,7 +26,7 @@ RUN yum makecache && yum -y update && yum -y groupinstall
'Development Tools' &&
devtoolset-10-libasan-devel
# build cmake
-ARG CMAKE_VERSION=3.22.1
+ARG CMAKE_VERSION=3.25.3
ARG
CMAKE_BASE_URL=https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}
RUN wget ${CMAKE_BASE_URL}/cmake-${CMAKE_VERSION}-Linux-x86_64.sh -q -O
/tmp/cmake-install.sh && \
chmod u+x /tmp/cmake-install.sh && \
@@ -121,4 +121,3 @@ ENV JAVA_HOME="/usr/lib/jvm/java-11" \
WORKDIR /root
CMD ["/bin/bash"]
-
diff --git a/docker/compilation/Dockerfile.gcc7
b/docker/compilation/Dockerfile.gcc7
deleted file mode 100644
index 7ca1fb321b4..00000000000
--- a/docker/compilation/Dockerfile.gcc7
+++ /dev/null
@@ -1,119 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-FROM centos:centos7.5.1804 AS builder
-
-LABEL maintainer=tangxiaoqing214445
-
-ENV DEFAULT_DIR /var/local
-
-ARG GCC_VERSION=7.3.0
-ARG GCC_URL=https://mirrors.ustc.edu.cn/gnu/gcc/gcc-${GCC_VERSION}
-
-# install dependencies and build gcc
-RUN yum install -y bzip2 wget git gcc-c++ libstdc++-static byacc flex automake
libtool binutils-devel bison ncurses-devel make mlocate unzip patch which
vim-common redhat-lsb-core zip libcurl-devel libasan-static libtsan-static \
- && updatedb \
- && yum -y clean all \
- && rm -rf /var/cache/yum \
- && mkdir -p /var/local/gcc \
- && curl -fsSL -o /tmp/node-v14.16.0-linux-x64.tar.xz
https://nodejs.org/dist/v14.16.0/node-v14.16.0-linux-x64.tar.xz \
- && tar xf /tmp/node-v14.16.0-linux-x64.tar.xz -C ${DEFAULT_DIR}/ \
- && rm /tmp/node-v14.16.0-linux-x64.tar.xz \
- && curl -fsSL -o /tmp/gcc.tar.gz ${GCC_URL}/gcc-${GCC_VERSION}.tar.gz \
- && tar -xzf /tmp/gcc.tar.gz -C /var/local/gcc --strip-components=1 \
- && cd /var/local/gcc \
- && sed -i
's/ftp:\/\/gcc.gnu.org\/pub\/gcc\/infrastructure\//http:\/\/mirror.linux-ia64.org\/gnu\/gcc\/infrastructure\//g'
contrib/download_prerequisites \
- && ./contrib/download_prerequisites \
- && ./configure --disable-multilib --enable-languages=c,c++ --prefix=/usr \
- && make -j$[$(nproc)/4+1] && make install \
- && rm -rf /var/local/gcc \
- && rm -f /tmp/gcc.tar.gz
-
-ENV PATH ${DEFAULT_DIR}/node-v14.16.0-linux-x64/bin:$PATH
-
-# build cmake
-ARG CMAKE_VERSION=3.12.3
-ARG
CMAKE_DOWNLOAD_URL=https://cmake.org/files/v3.12/cmake-${CMAKE_VERSION}.tar.gz
-RUN mkdir -p /tmp/cmake && curl -fsSL -o /tmp/cmake.tar.gz
${CMAKE_DOWNLOAD_URL} \
- && tar -zxf /tmp/cmake.tar.gz -C /tmp/cmake --strip-components=1 \
- && cd /tmp/cmake \
- && ./bootstrap --system-curl \
- && gmake -j$[$(nproc)/4+1] \
- && gmake install \
- && rm -rf /tmp/cmake.tar.gz \
- && rm -rf /tmp/cmake
-
-# install jdk
-COPY ./jdk.rpm ./
-RUN wget --no-cookies --no-check-certificate --header "Cookie:
oraclelicense=accept-securebackup-cookie"
https://javadl.oracle.com/webapps/download/GetFile/1.8.0_281-b09/89d678f2be164786b292527658ca1605/linux-i586/jdk-8u281-linux-x64.rpm
-O jdk.rpm \
- && touch ${DEFAULT_DIR}/install_jdk.sh \
- && echo '#!/bin/bash' >> ${DEFAULT_DIR}/install_jdk.sh \
- && echo 'rpm -Uvh jdk.rpm > /dev/null 2>&1' >>
${DEFAULT_DIR}/install_jdk.sh \
- && echo 'mv /usr/java/jdk* /usr/java/jdk' >>
${DEFAULT_DIR}/install_jdk.sh \
- && echo 'export JAVA_HOME=/usr/java/jdk' >> ${DEFAULT_DIR}/install_jdk.sh
\
- && echo 'echo "export JAVA_HOME=/usr/java/jdk" >> /etc/environment' >>
${DEFAULT_DIR}/install_jdk.sh \
- && echo 'export JRE_HOME=/usr/java/jdk/jre' >>
${DEFAULT_DIR}/install_jdk.sh \
- && echo 'echo "export JRE_HOME=/usr/java/jdk/jre" >> /etc/environment' >>
${DEFAULT_DIR}/install_jdk.sh \
- && echo 'ls /usr/java/jdk > /dev/null 2>&1' >>
${DEFAULT_DIR}/install_jdk.sh \
- && echo 'echo "export JAVA_HOME=/usr/java/jdk" >> /root/.bashrc' >>
${DEFAULT_DIR}/install_jdk.sh \
- && chmod 777 ${DEFAULT_DIR}/install_jdk.sh \
- && /bin/bash ${DEFAULT_DIR}/install_jdk.sh \
- && rm -rf *.rpm \
- && rm ${DEFAULT_DIR}/install_jdk.sh
-
-ENV JAVA_HOME /usr/java/jdk
-
-# install maven 3.6.3
-ARG
SHA=c35a1803a6e70a126e80b2b3ae33eed961f83ed74d18fcd16909b2d44d7dada3203f1ffe726c17ef8dcca2dcaa9fca676987befeadc9b9f759967a8cb77181c0
-ARG
MAVEN_URL=http://mirror.bit.edu.cn/apache/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz
-
-RUN mkdir -p /usr/share/maven /usr/share/maven/ref \
- && curl -fsSL -o /tmp/apache-maven.tar.gz ${MAVEN_URL} \
- && echo "${SHA} /tmp/apache-maven.tar.gz" | sha512sum -c - \
- && tar -xzf /tmp/apache-maven.tar.gz -C /usr/share/maven
--strip-components=1 \
- && rm -f /tmp/apache-maven.tar.gz \
- && ln -s /usr/share/maven/bin/mvn /usr/bin/mvn
-
-ENV MAVEN_HOME /usr/share/maven
-
-# build environment
-WORKDIR ${DEFAULT_DIR}
-
-# there is a repo which is included all of thirdparty
-ENV REPOSITORY_URL=https://doris-thirdparty-repo.bj.bcebos.com/thirdparty
-
-# clone lastest source code, download and build third party
-COPY doris ${DEFAULT_DIR}/doris
-RUN cd ${DEFAULT_DIR}/doris && /bin/bash thirdparty/build-thirdparty.sh \
- && rm -rf ${DEFAULT_DIR}/doris/thirdparty/src \
- && rm -rf ${DEFAULT_DIR}/doris-thirdparty.tar.gz \
- && rm -rf ${DEFAULT_DIR}/doris-thirdparty \
- && mkdir -p ${DEFAULT_DIR}/thirdparty \
- && mv ${DEFAULT_DIR}/doris/thirdparty/installed ${DEFAULT_DIR}/thirdparty/
\
- && rm -rf ${DEFAULT_DIR}/doris
-
-FROM scratch
-COPY --from=builder / /
-
-ENV DEFAULT_DIR /var/local
-ENV PATH ${DEFAULT_DIR}/node-v14.8.0-linux-x64/bin:$PATH
-ENV JAVA_HOME /usr/java/jdk
-ENV MAVEN_HOME /usr/share/maven
-ENV DORIS_THIRDPARTY /var/local/thirdparty
-WORKDIR /root
-CMD ["/bin/bash"]
-
diff --git a/docker/compilation/arm/Dockerfile
b/docker/compilation/arm/Dockerfile
index aa0ce5eea4d..656ee43d39a 100644
--- a/docker/compilation/arm/Dockerfile
+++ b/docker/compilation/arm/Dockerfile
@@ -47,10 +47,21 @@ RUN wget
https://github.com/amosbird/ldb_toolchain_gen/releases/download/v0.9.1/
&& sh /tmp/ldb_toolchain_gen.aarch64.sh /var/local/ldb-toolchain/ \
&& rm /tmp/ldb_toolchain_gen.aarch64.sh
+# Arrow 24 requires CMake 3.25 or newer. The bundled ldb toolchain only
+# provides CMake 3.22, so install and select a compatible native build.
+ARG CMAKE_VERSION=3.25.3
+ARG
CMAKE_BASE_URL=https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}
+RUN wget ${CMAKE_BASE_URL}/cmake-${CMAKE_VERSION}-Linux-aarch64.sh \
+ -q -O /tmp/cmake-install.sh \
+ && chmod u+x /tmp/cmake-install.sh \
+ && /tmp/cmake-install.sh --skip-license --prefix=/usr --exclude-subdir \
+ && rm /tmp/cmake-install.sh
+
# there is a repo which is included all of thirdparty
ENV REPOSITORY_URL="https://doris-thirdparty-repo.bj.bcebos.com/thirdparty/"; \
DEFAULT_DIR="/var/local" \
JAVA_HOME="/usr/lib/jvm/java-11" \
+ CUSTOM_CMAKE="/usr/bin/cmake" \
PATH="/var/local/ldb-toolchain/bin/:$PATH"
# disable auto enable ccache
@@ -76,6 +87,7 @@ ENV JAVA_HOME="/usr/lib/jvm/java-11" \
MAVEN_HOME="/usr/share/maven" \
REPOSITORY_URL="" \
DEFAULT_DIR="/var/local" \
+ CUSTOM_CMAKE="/usr/bin/cmake" \
PATH="/var/local/ldb-toolchain/bin/:/var/local/thirdparty/installed/bin/:$PATH"
\
DORIS_THIRDPARTY="/var/local/thirdparty"
WORKDIR /root
diff --git
a/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_stats2.out
b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_stats2.out
index 48765f6c4dd..0094b0f105f 100644
---
a/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_stats2.out
+++
b/regression-test/data/external_table_p0/iceberg/write/test_iceberg_write_stats2.out
@@ -7,7 +7,7 @@ true 11 111 1.1 1.1 1111 1234.5678
1234.567890 123456789012345678.123456789012 a
0 PARQUET 2 {1:2, 2:2, 3:2, 4:2, 5:2, 6:2, 7:2, 8:2, 9:2, 10:2,
11:2, 12:2} {1:0, 2:0, 3:0, 4:0, 5:0, 6:0, 7:0, 8:0, 9:0, 10:0, 11:0, 12:0}
{1:0x00, 2:0x0B000000, 3:0x6F00000000000000, 4:0xCDCC8C3F,
5:0x9A9999999999F13F, 6:0x00000457, 7:0x00BC614E, 8:0x00000000499602D2,
9:0x000000018EE90FF6C373E0393713FA14, 10:0x616161, 11:0x9E4B0000,
12:0x005CE70F33F10500} {1:0x01, 2:0x16000000, 3:0xDE00000000000000,
4:0xCDCC0C40, 5:0x9A99999999990140, 6:0x000008AE, 7:0x05397FB1,
8:0x000000020A75E124, 9 [...]
-- !sql_2 --
-{"bigint_col":{"column_size":118, "value_count":2, "null_value_count":0,
"nan_value_count":null, "lower_bound":111, "upper_bound":222},
"boolean_col":{"column_size":49, "value_count":2, "null_value_count":0,
"nan_value_count":null, "lower_bound":0, "upper_bound":1},
"date_col":{"column_size":94, "value_count":2, "null_value_count":0,
"nan_value_count":null, "lower_bound":"2023-01-01",
"upper_bound":"2023-06-15"}, "datetime_col1":{"column_size":118,
"value_count":2, "null_value_count":0, [...]
+{"bigint_col":{"column_size":74, "value_count":2, "null_value_count":0,
"nan_value_count":null, "lower_bound":111, "upper_bound":222},
"boolean_col":{"column_size":33, "value_count":2, "null_value_count":0,
"nan_value_count":null, "lower_bound":0, "upper_bound":1},
"date_col":{"column_size":66, "value_count":2, "null_value_count":0,
"nan_value_count":null, "lower_bound":"2023-01-01",
"upper_bound":"2023-06-15"}, "datetime_col1":{"column_size":74,
"value_count":2, "null_value_count":0, "n [...]
-- !sql_3 --
false 22 222 2.2 2.2 2222 8765.4321 8765.432100
987654321098765432.987654321099 bbb 2023-06-15 2023-06-15T23:45:01
diff --git
a/regression-test/suites/external_table_p0/iceberg/iceberg_schema_change_ddl.groovy
b/regression-test/suites/external_table_p0/iceberg/iceberg_schema_change_ddl.groovy
index c197e3f372c..d40c1eb29c9 100644
---
a/regression-test/suites/external_table_p0/iceberg/iceberg_schema_change_ddl.groovy
+++
b/regression-test/suites/external_table_p0/iceberg/iceberg_schema_change_ddl.groovy
@@ -204,7 +204,7 @@ suite("iceberg_schema_change_ddl",
"p0,external,doris,external_docker,external_d
// Add test columns for type conversion tests
sql """ ALTER TABLE ${table_name} ADD COLUMN test_float FLOAT """
sql """ ALTER TABLE ${table_name} ADD COLUMN test_decimal DECIMAL(5,2) """
- sql """ INSERT INTO ${table_name} (id, test_float, test_decimal) VALUES
(7, 3.14, 123.45) """
+ sql """ INSERT INTO ${table_name} (id, grade, test_float, test_decimal)
VALUES (7, 0, 3.14, 123.45) """
// Positive case: float -> double
sql """ ALTER TABLE ${table_name} MODIFY COLUMN test_float DOUBLE """
diff --git a/run-be-ut.sh b/run-be-ut.sh
index 65d1ff68558..97e5759a9a4 100755
--- a/run-be-ut.sh
+++ b/run-be-ut.sh
@@ -48,6 +48,29 @@ export TP_INCLUDE_DIR="${DORIS_THIRDPARTY}/installed/include"
export TP_INSTALLED_DIR="${DORIS_THIRDPARTY}/installed"
export TP_LIB_DIR="${DORIS_THIRDPARTY}/installed/lib"
. "${DORIS_HOME}/env.sh"
+# shellcheck source=thirdparty/arrow-paimon-vars.sh
+. "${DORIS_HOME}/thirdparty/arrow-paimon-vars.sh"
+
+prepare_build_image_arrow_paimon_prebuilt() {
+ local selected_thirdparty_root
+ local checkout_thirdparty_root
+
+ selected_thirdparty_root="$(cd "${DORIS_THIRDPARTY}" && pwd -P)"
+ checkout_thirdparty_root="$(cd "${DORIS_HOME}/thirdparty" && pwd -P)"
+ if [[ "${selected_thirdparty_root}" == "${checkout_thirdparty_root}" ]];
then
+ return 0
+ fi
+
+ # The official Linux x86_64 build image carries an install-only thirdparty
+ # tree. Refresh it from the shared automation asset when the image predates
+ # the Arrow/Paimon closure selected by this checkout.
+ if [[ "${selected_thirdparty_root}" != "/var/local/thirdparty" ||
+ "$(uname -s)" != "Linux" || "$(uname -m)" != "x86_64" ]]; then
+ return 0
+ fi
+ ensure_arrow_paimon_prebuilt_from_url "${selected_thirdparty_root}" \
+ "${ARROW_PAIMON_SHARED_PREBUILT_LINUX_X86_64_URL}"
+}
trim_whitespace() {
local value="$1"
@@ -228,6 +251,8 @@ echo "Get params:
"
echo "Build Backend UT"
+prepare_build_image_arrow_paimon_prebuilt
+
update_submodule() {
local submodule_path=$1
local submodule_name=$2
diff --git a/thirdparty/arrow-paimon-vars.sh b/thirdparty/arrow-paimon-vars.sh
new file mode 100644
index 00000000000..b5c6d6955c6
--- /dev/null
+++ b/thirdparty/arrow-paimon-vars.sh
@@ -0,0 +1,408 @@
+#!/bin/bash
+# shellcheck disable=2034
+
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Keep the Arrow/Paimon source closure in a dedicated file so targeted CI can
+# distinguish this stack from unrelated thirdparty changes.
+
+# arrow
+ARROW_VERSION="24.0.0"
+ARROW_DOWNLOAD="https://github.com/apache/arrow/archive/refs/tags/apache-arrow-${ARROW_VERSION}.tar.gz";
+ARROW_NAME="apache-arrow-${ARROW_VERSION}.tar.gz"
+ARROW_SOURCE="arrow-apache-arrow-${ARROW_VERSION}"
+ARROW_MD5SUM="66c53bd00baa79034bd2ca167beea436"
+
+# Arrow bundled dependencies
+BROTLI_DOWNLOAD="https://github.com/google/brotli/archive/v1.0.9.tar.gz";
+BROTLI_NAME="brotli-1.0.9.tar.gz"
+BROTLI_SOURCE="brotli-1.0.9"
+BROTLI_MD5SUM="c2274f0c7af8470ad514637c35bcee7d"
+
+XSIMD_DOWNLOAD="https://github.com/xtensor-stack/xsimd/archive/refs/tags/14.0.0.tar.gz";
+XSIMD_NAME="14.0.0.tar.gz"
+XSIMD_SOURCE=xsimd-14.0.0
+XSIMD_MD5SUM="75c0d34cf7011924ba19978076c76dc1"
+
+# paimon-cpp
+PAIMON_CPP_DOWNLOAD="https://github.com/apache/doris-thirdparty/archive/refs/tags/paimon-cpp-0a4f4e2.tar.gz";
+PAIMON_CPP_NAME="paimon-cpp-0a4f4e2.tar.gz"
+PAIMON_CPP_SOURCE="doris-thirdparty-paimon-cpp-0a4f4e2"
+PAIMON_CPP_MD5SUM="b8599a0421dbf1ec05e2f1a481d64e87"
+
+ARROW_PAIMON_SHARED_PREBUILT_LINUX_X86_64_URL="${ARROW_PAIMON_SHARED_PREBUILT_LINUX_X86_64_URL:-https://github.com/apache/doris-thirdparty/releases/download/automation/doris-thirdparty-prebuilt-linux-x86_64.tar.xz}";
+
+# Bump the corresponding schema version whenever output-affecting build
options or
+# helper behavior in build_arrow() or build_paimon_cpp() changes. The
fingerprints
+# below intentionally describe only this component stack so master and release
+# branches can reuse the same shared prebuilt when their semantic inputs match.
+ARROW_BUILD_SCHEMA_VERSION="1"
+PAIMON_BUILD_SCHEMA_VERSION="1"
+
+# The current shared automation prebuilt was published from master with the
former
+# whole-script fingerprint. Keep these exact markers during the schema
transition;
+# version and complete artifact validation are still mandatory before reuse.
+ARROW_LEGACY_COMPATIBLE_SEMANTIC_FINGERPRINT="ab79ab0bbfbf93f9860050fb751b20fee9e40d96"
+PAIMON_LEGACY_COMPATIBLE_SEMANTIC_FINGERPRINT="cb82e41ba46f534e611cdd52e66b53c227d49bf8"
+ARROW_LEGACY_BUILD_FINGERPRINTS=(
+ 9d03645dd1cded5184a8126f5c7f4a6eb9b92b53
+)
+PAIMON_LEGACY_BUILD_FINGERPRINTS=(
+ dbb6ca6e243cb3aa783b7a8011f97afda9e7ea28
+)
+
+# Arrow consumes xsimd and Brotli as bundled source archives, but neither is a
+# build target in the focused Arrow/Paimon recovery path.
+ARROW_PAIMON_BUILD_PACKAGES=(arrow paimon_cpp)
+ARROW_BUNDLED_SOURCE_PACKAGES=(xsimd brotli)
+ARROW_PAIMON_DOWNLOAD_PACKAGES=()
+
+prepare_arrow_paimon_download_packages() {
+ ARROW_PAIMON_DOWNLOAD_PACKAGES=("$@")
+
+ local package
+ local source_package
+ local arrow_requested=false
+ local source_requested
+ for package in "$@"; do
+ if [[ "${package}" == "arrow" ]]; then
+ arrow_requested=true
+ break
+ fi
+ done
+ if [[ "${arrow_requested}" != "true" ]]; then
+ return
+ fi
+
+ for source_package in "${ARROW_BUNDLED_SOURCE_PACKAGES[@]}"; do
+ source_requested=false
+ for package in "${ARROW_PAIMON_DOWNLOAD_PACKAGES[@]}"; do
+ if [[ "${package}" == "${source_package}" ]]; then
+ source_requested=true
+ break
+ fi
+ done
+ if [[ "${source_requested}" != "true" ]]; then
+ ARROW_PAIMON_DOWNLOAD_PACKAGES+=("${source_package}")
+ fi
+ done
+}
+
+# Print stable path-and-content records for fingerprint inputs. Including the
path
+# makes patch selection and ordering part of the contract, not only file
contents.
+arrow_paimon_fingerprint_files() {
+ local file
+ local blob
+ while IFS= read -r file; do
+ blob="$(git hash-object "${file}")" || return 1
+ printf 'file=%s\n' "${file}"
+ printf 'blob=%s\n' "${blob}"
+ done < <(printf '%s\n' "$@" | LC_ALL=C sort)
+}
+
+# Identify only the source, patch, and explicit build-schema inputs selected
for
+# Arrow. Arrow and Paimon publish separate installed markers so a package-only
+# build cannot certify a component that it did not rebuild.
+arrow_build_fingerprint() {
+ local vars_dir
+ vars_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ (
+ set -o pipefail
+ cd "${vars_dir}" || return 1
+ {
+ printf 'schema=%s\n' "${ARROW_BUILD_SCHEMA_VERSION}"
+ printf 'ARROW_VERSION=%s\n' "${ARROW_VERSION}"
+ printf 'ARROW_NAME=%s\n' "${ARROW_NAME}"
+ printf 'ARROW_SOURCE=%s\n' "${ARROW_SOURCE}"
+ printf 'ARROW_MD5SUM=%s\n' "${ARROW_MD5SUM}"
+ printf 'BROTLI_NAME=%s\n' "${BROTLI_NAME}"
+ printf 'BROTLI_SOURCE=%s\n' "${BROTLI_SOURCE}"
+ printf 'BROTLI_MD5SUM=%s\n' "${BROTLI_MD5SUM}"
+ printf 'XSIMD_NAME=%s\n' "${XSIMD_NAME}"
+ printf 'XSIMD_SOURCE=%s\n' "${XSIMD_SOURCE}"
+ printf 'XSIMD_MD5SUM=%s\n' "${XSIMD_MD5SUM}"
+ arrow_paimon_fingerprint_files \
+ patches/apache-arrow-"${ARROW_VERSION}"-*.patch
+ } | git hash-object --stdin
+ )
+}
+
+paimon_build_fingerprint() {
+ local vars_dir
+ vars_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ (
+ set -o pipefail
+ cd "${vars_dir}" || return 1
+ {
+ arrow_build_fingerprint
+ printf 'schema=%s\n' "${PAIMON_BUILD_SCHEMA_VERSION}"
+ printf 'PAIMON_CPP_NAME=%s\n' "${PAIMON_CPP_NAME}"
+ printf 'PAIMON_CPP_SOURCE=%s\n' "${PAIMON_CPP_SOURCE}"
+ printf 'PAIMON_CPP_MD5SUM=%s\n' "${PAIMON_CPP_MD5SUM}"
+ arrow_paimon_fingerprint_files \
+ paimon-cpp-cache.cmake \
+ patches/paimon-cpp-*.patch
+ } | git hash-object --stdin
+ )
+}
+
+# Source patch markers use a combined value because either component's inputs
+# may change the external-Arrow contract applied to both source trees.
+arrow_paimon_build_fingerprint() {
+ {
+ arrow_build_fingerprint
+ paimon_build_fingerprint
+ } | git hash-object --stdin
+}
+
+arrow_paimon_fingerprint_matches() {
+ local installed_fingerprint="$1"
+ local expected_fingerprint="$2"
+ local legacy_compatible_expected_fingerprint="$3"
+ shift 3
+ local compatible_fingerprint
+
+ if [[ "${installed_fingerprint}" == "${expected_fingerprint}" ]]; then
+ return 0
+ fi
+ if [[ "${expected_fingerprint}" !=
"${legacy_compatible_expected_fingerprint}" ]]; then
+ return 1
+ fi
+ for compatible_fingerprint in "$@"; do
+ if [[ "${installed_fingerprint}" == "${compatible_fingerprint}" ]];
then
+ return 0
+ fi
+ done
+ return 1
+}
+
+ARROW_REQUIRED_LIBRARIES=(
+ libbrotlicommon.a
+ libbrotlidec.a
+ libbrotlienc.a
+ libarrow.a
+ libarrow_compute.a
+ libarrow_flight.a
+ libarrow_flight_sql.a
+ libarrow_dataset.a
+ libarrow_acero.a
+ libarrow_bundled_dependencies.a
+ libparquet.a
+)
+
+PAIMON_REQUIRED_LIBRARIES=(
+ libpaimon.a
+ libpaimon_parquet_file_format.a
+ libpaimon_orc_file_format.a
+ libpaimon_blob_file_format.a
+ libpaimon_local_file_system.a
+ libpaimon_file_index.a
+ libpaimon_global_index.a
+ libroaring_bitmap_paimon.a
+ libxxhash_paimon.a
+ libfmt_paimon.a
+ libtbb_paimon.a
+)
+
+ARROW_PAIMON_REQUIRED_LIBRARIES=(
+ "${ARROW_REQUIRED_LIBRARIES[@]}"
+ "${PAIMON_REQUIRED_LIBRARIES[@]}"
+)
+
+arrow_artifacts_valid() {
+ local install_dir="$1"
+ local installed_arrow_version
+ local library
+
+ if [[ ! -f "${install_dir}/include/arrow/util/config.h" ]]; then
+ echo "Missing installed Arrow version header" >&2
+ return 1
+ fi
+ installed_arrow_version="$(
+ awk '$1 == "#define" && $2 == "ARROW_VERSION_STRING" {
+ gsub(/"/, "", $3); print $3; exit
+ }' "${install_dir}/include/arrow/util/config.h"
+ )"
+ if [[ "${installed_arrow_version}" != "${ARROW_VERSION}" ]]; then
+ echo "Installed Arrow version ${installed_arrow_version} does not
match ${ARROW_VERSION}" >&2
+ return 1
+ fi
+
+ for library in "${ARROW_REQUIRED_LIBRARIES[@]}"; do
+ if [[ ! -f "${install_dir}/lib64/${library}" ]]; then
+ echo "Missing Arrow library: ${library}" >&2
+ return 1
+ fi
+ done
+ return 0
+}
+
+paimon_artifacts_valid() {
+ local install_dir="$1"
+ local library
+
+ for library in "${PAIMON_REQUIRED_LIBRARIES[@]}"; do
+ if [[ ! -f "${install_dir}/lib64/${library}" ]]; then
+ echo "Missing Paimon library: ${library}" >&2
+ return 1
+ fi
+ done
+ return 0
+}
+
+arrow_prebuilt_valid() {
+ local install_dir="$1"
+ local arrow_fingerprint_mark="${install_dir}/arrow-build-fingerprint.txt"
+ local expected_fingerprint
+ local installed_fingerprint
+
+ if [[ ! -f "${arrow_fingerprint_mark}" ]]; then
+ echo "Missing Arrow build fingerprint: ${arrow_fingerprint_mark}" >&2
+ return 1
+ fi
+ expected_fingerprint="$(arrow_build_fingerprint)"
+ installed_fingerprint="$(<"${arrow_fingerprint_mark}")"
+ if ! arrow_paimon_fingerprint_matches "${installed_fingerprint}" \
+ "${expected_fingerprint}"
"${ARROW_LEGACY_COMPATIBLE_SEMANTIC_FINGERPRINT}" \
+ "${ARROW_LEGACY_BUILD_FINGERPRINTS[@]}"; then
+ echo "Arrow build fingerprint does not match selected inputs" >&2
+ return 1
+ fi
+ arrow_artifacts_valid "${install_dir}"
+}
+
+paimon_prebuilt_valid() {
+ local install_dir="$1"
+ local paimon_fingerprint_mark="${install_dir}/paimon-build-fingerprint.txt"
+ local expected_fingerprint
+ local installed_fingerprint
+
+ if [[ ! -f "${paimon_fingerprint_mark}" ]]; then
+ echo "Missing Paimon build fingerprint: ${paimon_fingerprint_mark}" >&2
+ return 1
+ fi
+ expected_fingerprint="$(paimon_build_fingerprint)"
+ installed_fingerprint="$(<"${paimon_fingerprint_mark}")"
+ if ! arrow_paimon_fingerprint_matches "${installed_fingerprint}" \
+ "${expected_fingerprint}"
"${PAIMON_LEGACY_COMPATIBLE_SEMANTIC_FINGERPRINT}" \
+ "${PAIMON_LEGACY_BUILD_FINGERPRINTS[@]}"; then
+ echo "Paimon build fingerprint does not match selected inputs" >&2
+ return 1
+ fi
+ paimon_artifacts_valid "${install_dir}"
+}
+
+arrow_paimon_prebuilt_valid() {
+ local install_dir="$1"
+ arrow_prebuilt_valid "${install_dir}" && paimon_prebuilt_valid
"${install_dir}"
+}
+
+# Validate an extracted candidate before replacing the selected thirdparty
+# installation. This keeps an older build image usable when a download is
+# incomplete or contains an incompatible Arrow/Paimon closure.
+install_arrow_paimon_prebuilt_archive() {
+ local archive="$1"
+ local thirdparty_root="$2"
+ local staging_dir
+ local had_previous_installed=false
+
+ staging_dir="$(mktemp -d
"${thirdparty_root}/.arrow-paimon-install.XXXXXX")"
+ mkdir -p "${staging_dir}/candidate"
+ if ! tar -xf "${archive}" -C "${staging_dir}/candidate"; then
+ rm -rf "${staging_dir}"
+ return 1
+ fi
+ if ! arrow_paimon_prebuilt_valid "${staging_dir}/candidate/installed"; then
+ rm -rf "${staging_dir}"
+ return 1
+ fi
+
+ if [[ -e "${thirdparty_root}/installed" || -L
"${thirdparty_root}/installed" ]]; then
+ if ! mv "${thirdparty_root}/installed"
"${staging_dir}/previous-installed"; then
+ rm -rf "${staging_dir}"
+ return 1
+ fi
+ had_previous_installed=true
+ fi
+ if ! mv "${staging_dir}/candidate/installed"
"${thirdparty_root}/installed"; then
+ if [[ "${had_previous_installed}" == "true" ]]; then
+ mv "${staging_dir}/previous-installed"
"${thirdparty_root}/installed"
+ fi
+ rm -rf "${staging_dir}"
+ return 1
+ fi
+ rm -rf "${staging_dir}"
+}
+
+ensure_arrow_paimon_prebuilt_from_url() {
+ local thirdparty_root="$1"
+ local prebuilt_url="$2"
+ local download_dir
+ local archive
+
+ if arrow_paimon_prebuilt_valid "${thirdparty_root}/installed"; then
+ return 0
+ fi
+
+ echo "Refreshing Arrow/Paimon thirdparty libraries from ${prebuilt_url}"
+ download_dir="$(mktemp -d
"${thirdparty_root}/.arrow-paimon-download.XXXXXX")"
+ archive="${download_dir}/doris-thirdparty-prebuilt.tar.xz"
+ if ! curl --fail --location --retry 3 --show-error \
+ --output "${archive}" "${prebuilt_url}"; then
+ rm -rf "${download_dir}"
+ return 1
+ fi
+ if ! install_arrow_paimon_prebuilt_archive "${archive}"
"${thirdparty_root}"; then
+ rm -rf "${download_dir}"
+ return 1
+ fi
+ rm -rf "${download_dir}"
+ arrow_paimon_prebuilt_valid "${thirdparty_root}/installed"
+}
+
+invalidate_arrow_prebuilt_marker() {
+ local install_dir="$1"
+ rm -f "${install_dir}/arrow-build-fingerprint.txt" \
+ "${install_dir}/arrow-paimon-build-fingerprint.txt"
+}
+
+publish_arrow_prebuilt_marker() {
+ local install_dir="$1"
+ arrow_artifacts_valid "${install_dir}"
+ arrow_build_fingerprint >"${install_dir}/arrow-build-fingerprint.txt"
+}
+
+invalidate_paimon_prebuilt_marker() {
+ local install_dir="$1"
+ rm -f "${install_dir}/paimon-build-fingerprint.txt" \
+ "${install_dir}/arrow-paimon-build-fingerprint.txt"
+}
+
+publish_paimon_prebuilt_marker() {
+ local install_dir="$1"
+ paimon_artifacts_valid "${install_dir}"
+ paimon_build_fingerprint >"${install_dir}/paimon-build-fingerprint.txt"
+}
+
+require_arrow_prebuilt_for_paimon() {
+ local install_dir="$1"
+ if ! arrow_prebuilt_valid "${install_dir}"; then
+ echo "Paimon requires Arrow to be built from the currently selected
inputs first" >&2
+ return 1
+ fi
+}
diff --git a/thirdparty/build-thirdparty.sh b/thirdparty/build-thirdparty.sh
index 7e460ade7d8..35437afcdf1 100755
--- a/thirdparty/build-thirdparty.sh
+++ b/thirdparty/build-thirdparty.sh
@@ -155,7 +155,8 @@ if [[ "${CLEAN}" -eq 1 ]] && [[ -d "${TP_SOURCE_DIR}" ]];
then
fi
# Download thirdparties.
-eval "${TP_DIR}/download-thirdparty.sh ${packages[*]}"
+prepare_arrow_paimon_download_packages "${packages[@]}"
+bash "${TP_DIR}/download-thirdparty.sh" "${ARROW_PAIMON_DOWNLOAD_PACKAGES[@]}"
export LD_LIBRARY_PATH="${TP_DIR}/installed/lib:${LD_LIBRARY_PATH}"
@@ -1064,6 +1065,7 @@ build_grpc() {
# arrow
build_arrow() {
check_if_source_exist "${ARROW_SOURCE}"
+ invalidate_arrow_prebuilt_marker "${TP_INSTALL_DIR}"
cd "${TP_SOURCE_DIR}/${ARROW_SOURCE}/cpp"
mkdir -p release
@@ -1088,9 +1090,7 @@ build_arrow() {
ldflags="-L${TP_LIB_DIR}"
fi
- CPPFLAGS="-I${TP_INCLUDE_DIR}" \
- CXXFLAGS="-I${TP_INCLUDE_DIR}" \
- LDFLAGS="${ldflags}" \
+ LDFLAGS="${ldflags}" \
"${CMAKE_CMD}" -DCMAKE_POLICY_VERSION_MINIMUM=3.5 \
-DCMAKE_CXX_STANDARD="${TP_CXX_STANDARD}" \
-G "${GENERATOR}" -DARROW_PARQUET=ON -DARROW_IPC=ON
-DARROW_BUILD_SHARED=OFF \
@@ -1125,6 +1125,7 @@ build_arrow() {
-Dxsimd_SOURCE=BUNDLED \
-DBrotli_SOURCE=BUNDLED \
-DARROW_LZ4_USE_SHARED=OFF \
+ -DLZ4_ROOT="${TP_INSTALL_DIR};${TP_INSTALL_DIR}/include/lz4" \
-DLZ4_LIB="${TP_INSTALL_DIR}/lib/liblz4.a"
-DLZ4_INCLUDE_DIR="${TP_INSTALL_DIR}/include/lz4" \
-DLz4_SOURCE=SYSTEM \
-DARROW_ZSTD_USE_SHARED=OFF \
@@ -1147,9 +1148,12 @@ build_arrow() {
cp -rf ./brotli_ep/src/brotli_ep-install/lib/libbrotlidec-static.a
"${TP_INSTALL_DIR}/lib64/libbrotlidec.a"
cp -rf ./brotli_ep/src/brotli_ep-install/lib/libbrotlicommon-static.a
"${TP_INSTALL_DIR}/lib64/libbrotlicommon.a"
strip_lib libarrow.a
+ strip_lib libarrow_compute.a
strip_lib libparquet.a
strip_lib libarrow_dataset.a
strip_lib libarrow_acero.a
+
+ publish_arrow_prebuilt_marker "${TP_INSTALL_DIR}"
}
# abseil
@@ -2042,6 +2046,8 @@ build_pugixml() {
# paimon-cpp
build_paimon_cpp() {
check_if_source_exist "${PAIMON_CPP_SOURCE}"
+ require_arrow_prebuilt_for_paimon "${TP_INSTALL_DIR}"
+ invalidate_paimon_prebuilt_marker "${TP_INSTALL_DIR}"
cd "${TP_SOURCE_DIR}/${PAIMON_CPP_SOURCE}"
rm -rf "${BUILD_DIR}"
@@ -2088,6 +2094,7 @@ build_paimon_cpp() {
mkdir -p "${paimon_deps_dir}"
for paimon_arrow_dep in \
libarrow.a \
+ libarrow_compute.a \
libarrow_filesystem.a \
libarrow_dataset.a \
libarrow_acero.a \
@@ -2121,6 +2128,7 @@ build_paimon_cpp() {
fi
echo "Paimon-cpp internal dependencies installed successfully"
+ publish_paimon_prebuilt_marker "${TP_INSTALL_DIR}"
}
# lance-c
diff --git a/thirdparty/download-thirdparty.sh
b/thirdparty/download-thirdparty.sh
index c278b15c6f7..87eae93eaf4 100755
--- a/thirdparty/download-thirdparty.sh
+++ b/thirdparty/download-thirdparty.sh
@@ -334,6 +334,20 @@ echo "===== Patching thirdparty archives..."
# This is to avoid duplicated patch.
###################################################################################
PATCHED_MARK="patched_mark"
+ARROW_PAIMON_PATCH_FINGERPRINT_MARK="patched_mark_arrow_paimon_fingerprint"
+ARROW_PAIMON_BUILD_FINGERPRINT=""
+if [[ " ${TP_ARCHIVES[*]} " =~ " ARROW " ||
+ " ${TP_ARCHIVES[*]} " =~ " PAIMON_CPP " ]]; then
+ ARROW_PAIMON_BUILD_FINGERPRINT="$(arrow_paimon_build_fingerprint)"
+fi
+
+reset_arrow_paimon_source() {
+ local archive_name="$1"
+ local source_name="$2"
+ echo "Resetting ${source_name} because its patch state is incomplete or
stale"
+ rm -rf "${TP_SOURCE_DIR:?}/${source_name}"
+ "${TAR_CMD}" xzf "${TP_SOURCE_DIR}/${archive_name}" -C "${TP_SOURCE_DIR}/"
+}
# glog patch
if [[ " ${TP_ARCHIVES[*]} " =~ " GLOG " ]]; then
@@ -438,22 +452,28 @@ if [[ " ${TP_ARCHIVES[*]} " =~ " ARROW " ]]; then
fi
cd -
fi
- if [[ "${ARROW_SOURCE}" == "arrow-apache-arrow-17.0.0" ]]; then
- cd "${TP_SOURCE_DIR}/${ARROW_SOURCE}"
- if [[ ! -f "${PATCHED_MARK}" ]]; then
+ if [[ "${ARROW_SOURCE}" == "arrow-apache-arrow-24.0.0" ]]; then
+
arrow_fingerprint_mark="${TP_SOURCE_DIR}/${ARROW_SOURCE}/${ARROW_PAIMON_PATCH_FINGERPRINT_MARK}"
+ if ! [[ -f "${TP_SOURCE_DIR}/${ARROW_SOURCE}/${PATCHED_MARK}" &&
+ -f "${arrow_fingerprint_mark}" ]] ||
+ [[ "$(<"${arrow_fingerprint_mark}")" !=
"${ARROW_PAIMON_BUILD_FINGERPRINT}" ]]; then
+ reset_arrow_paimon_source "${ARROW_NAME}" "${ARROW_SOURCE}"
+ cd "${TP_SOURCE_DIR}/${ARROW_SOURCE}"
# Paimon-cpp parquet patches: row-group-aware batch reader,
max_row_group_size,
# GetBufferedSize(), int96 NANO guard, and Thrift_VERSION empty
fix.
- patch -p1 <"${TP_PATCH_DIR}/apache-arrow-17.0.0-paimon.patch"
+ patch -p1 <"${TP_PATCH_DIR}/apache-arrow-24.0.0-paimon.patch"
- # apache-arrow-17.0.0-force-write-int96-timestamps.patch :
- # Introducing the parameter that forces writing int96 timestampes
for compatibility with Paimon cpp.
- patch -p1
<"${TP_PATCH_DIR}/apache-arrow-17.0.0-force-write-int96-timestamps.patch"
+ # Introducing the parameter that forces writing INT96 timestamps
for
+ # compatibility with the Doris Parquet writer.
+ patch -p1
<"${TP_PATCH_DIR}/apache-arrow-24.0.0-force-write-int96-timestamps.patch"
# Add Parquet LZO page decompression support used by file scanner
v2.
- patch -p1 <"${TP_PATCH_DIR}/apache-arrow-17.0.0-lzo.patch"
+ patch -p1 <"${TP_PATCH_DIR}/apache-arrow-24.0.0-lzo.patch"
touch "${PATCHED_MARK}"
+ printf '%s\n' "${ARROW_PAIMON_BUILD_FINGERPRINT}" \
+ >"${ARROW_PAIMON_PATCH_FINGERPRINT_MARK}"
+ cd -
fi
- cd -
fi
echo "Finished patching ${ARROW_SOURCE}"
fi
@@ -731,16 +751,26 @@ fi
# patch paimon-cpp
if [[ " ${TP_ARCHIVES[*]} " =~ " PAIMON_CPP " ]]; then
- cd "${TP_SOURCE_DIR}/${PAIMON_CPP_SOURCE}"
- if [[ ! -f "${PATCHED_MARK}" ]]; then
- if patch -p1 -N --batch --dry-run
<"${TP_PATCH_DIR}/paimon-cpp-buildutils-static-deps.patch" >/dev/null 2>&1; then
- patch -p1 -N --batch
<"${TP_PATCH_DIR}/paimon-cpp-buildutils-static-deps.patch"
- else
- echo "Skip paimon-cpp patch: already applied or not applicable for
current source"
- fi
+ PAIMON_CPP_ARROW_24_PATCHED_MARK="patched_mark_arrow_24"
+ PAIMON_CPP_ARROW_24_COMPUTE_PATCHED_MARK="patched_mark_arrow_24_compute"
+
paimon_fingerprint_mark="${TP_SOURCE_DIR}/${PAIMON_CPP_SOURCE}/${ARROW_PAIMON_PATCH_FINGERPRINT_MARK}"
+ if ! [[ -f "${TP_SOURCE_DIR}/${PAIMON_CPP_SOURCE}/${PATCHED_MARK}" &&
+ -f
"${TP_SOURCE_DIR}/${PAIMON_CPP_SOURCE}/${PAIMON_CPP_ARROW_24_PATCHED_MARK}" &&
+ -f
"${TP_SOURCE_DIR}/${PAIMON_CPP_SOURCE}/${PAIMON_CPP_ARROW_24_COMPUTE_PATCHED_MARK}"
&&
+ -f "${paimon_fingerprint_mark}" ]] ||
+ [[ "$(<"${paimon_fingerprint_mark}")" !=
"${ARROW_PAIMON_BUILD_FINGERPRINT}" ]]; then
+ reset_arrow_paimon_source "${PAIMON_CPP_NAME}" "${PAIMON_CPP_SOURCE}"
+ cd "${TP_SOURCE_DIR}/${PAIMON_CPP_SOURCE}"
+ patch -p1 <"${TP_PATCH_DIR}/paimon-cpp-buildutils-static-deps.patch"
+ patch -p1 <"${TP_PATCH_DIR}/paimon-cpp-arrow-24-compatibility.patch"
+ patch -p1 <"${TP_PATCH_DIR}/paimon-cpp-arrow-24-compute.patch"
touch "${PATCHED_MARK}"
+ touch "${PAIMON_CPP_ARROW_24_PATCHED_MARK}"
+ touch "${PAIMON_CPP_ARROW_24_COMPUTE_PATCHED_MARK}"
+ printf '%s\n' "${ARROW_PAIMON_BUILD_FINGERPRINT}" \
+ >"${ARROW_PAIMON_PATCH_FINGERPRINT_MARK}"
+ cd -
fi
- cd -
echo "Finished patching ${PAIMON_CPP_SOURCE}"
fi
diff --git a/thirdparty/paimon-cpp-cache.cmake
b/thirdparty/paimon-cpp-cache.cmake
index dbebd94a0cc..6ab6049c59b 100644
--- a/thirdparty/paimon-cpp-cache.cmake
+++ b/thirdparty/paimon-cpp-cache.cmake
@@ -65,7 +65,7 @@ set(LZ4_INCLUDE_DIR "${DORIS_INCLUDE_DIR}" CACHE PATH "LZ4
include directory")
# ============================================================================
# Arrow - Reuse from Doris (Doris Arrow now includes
COMPUTE/DATASET/ACERO/FILESYSTEM)
-# Doris's Arrow 17.0.0 is built with the full module set that paimon-cpp
+# Doris's Arrow 24.0.0 is built with the full module set that paimon-cpp
# needs, so we skip paimon-cpp's internal externalproject_add(arrow_ep ...).
# ============================================================================
set(PAIMON_USE_EXTERNAL_ARROW ON CACHE BOOL "Use pre-built Arrow from Doris
instead of building from source")
@@ -74,6 +74,7 @@ set(DORIS_LIB64_DIR "${DORIS_THIRDPARTY_DIR}/lib64" CACHE
PATH "Doris lib64 dire
set(PAIMON_EXTERNAL_ARROW_INCLUDE_DIR "${DORIS_INCLUDE_DIR}" CACHE PATH "Arrow
include directory")
set(PAIMON_EXTERNAL_ARROW_LIB "${DORIS_LIB64_DIR}/libarrow.a" CACHE FILEPATH
"Arrow core library")
+set(PAIMON_EXTERNAL_ARROW_COMPUTE_LIB "${DORIS_LIB64_DIR}/libarrow_compute.a"
CACHE FILEPATH "Arrow Compute library")
set(PAIMON_EXTERNAL_ARROW_DATASET_LIB "${DORIS_LIB64_DIR}/libarrow_dataset.a"
CACHE FILEPATH "Arrow Dataset library")
set(PAIMON_EXTERNAL_ARROW_ACERO_LIB "${DORIS_LIB64_DIR}/libarrow_acero.a"
CACHE FILEPATH "Arrow Acero library")
set(PAIMON_EXTERNAL_PARQUET_LIB "${DORIS_LIB64_DIR}/libparquet.a" CACHE
FILEPATH "Parquet library")
diff --git
a/thirdparty/patches/apache-arrow-17.0.0-force-write-int96-timestamps.patch
b/thirdparty/patches/apache-arrow-24.0.0-force-write-int96-timestamps.patch
similarity index 74%
rename from
thirdparty/patches/apache-arrow-17.0.0-force-write-int96-timestamps.patch
rename to
thirdparty/patches/apache-arrow-24.0.0-force-write-int96-timestamps.patch
index 5a754247566..1847a50694f 100644
--- a/thirdparty/patches/apache-arrow-17.0.0-force-write-int96-timestamps.patch
+++ b/thirdparty/patches/apache-arrow-24.0.0-force-write-int96-timestamps.patch
@@ -1,20 +1,20 @@
-diff -ruN
arrow-apache-arrow-17.0.0-after-paimon/cpp/src/parquet/arrow/schema.cc
arrow-apache-arrow-17.0.0/cpp/src/parquet/arrow/schema.cc
---- arrow-apache-arrow-17.0.0-after-paimon/cpp/src/parquet/arrow/schema.cc
2026-03-27 01:23:23.651831424 +0800
-+++ arrow-apache-arrow-17.0.0/cpp/src/parquet/arrow/schema.cc 2026-03-27
01:28:36.855281965 +0800
-@@ -178,7 +178,8 @@
+--- a/cpp/src/parquet/arrow/schema.cc
++++ b/cpp/src/parquet/arrow/schema.cc
+@@ -215,8 +215,9 @@
// The user is explicitly asking for Impala int96 encoding, there is no
// logical type.
-- if (arrow_properties.support_deprecated_int96_timestamps() && target_unit
== ::arrow::TimeUnit::NANO) {
+- if (arrow_properties.support_deprecated_int96_timestamps() &&
+- target_unit == ::arrow::TimeUnit::NANO) {
+ if (arrow_properties.force_write_int96_timestamps() ||
-+ (arrow_properties.support_deprecated_int96_timestamps() && target_unit
== ::arrow::TimeUnit::NANO)) {
++ (arrow_properties.support_deprecated_int96_timestamps() &&
++ target_unit == ::arrow::TimeUnit::NANO)) {
*physical_type = ParquetType::INT96;
return Status::OK();
}
-diff -ruN arrow-apache-arrow-17.0.0-after-paimon/cpp/src/parquet/properties.h
arrow-apache-arrow-17.0.0/cpp/src/parquet/properties.h
---- arrow-apache-arrow-17.0.0-after-paimon/cpp/src/parquet/properties.h
2026-03-27 01:23:23.643831362 +0800
-+++ arrow-apache-arrow-17.0.0/cpp/src/parquet/properties.h 2026-03-27
01:27:47.717897537 +0800
-@@ -980,6 +980,7 @@
+--- a/cpp/src/parquet/properties.h
++++ b/cpp/src/parquet/properties.h
+@@ -1309,6 +1309,7 @@
public:
Builder()
: write_timestamps_as_int96_(false),
@@ -22,7 +22,7 @@ diff -ruN
arrow-apache-arrow-17.0.0-after-paimon/cpp/src/parquet/properties.h ar
coerce_timestamps_enabled_(false),
coerce_timestamps_unit_(::arrow::TimeUnit::SECOND),
truncated_timestamps_allowed_(false),
-@@ -1005,6 +1006,21 @@
+@@ -1334,6 +1335,21 @@
return this;
}
@@ -44,7 +44,7 @@ diff -ruN
arrow-apache-arrow-17.0.0-after-paimon/cpp/src/parquet/properties.h ar
/// \brief Coerce all timestamps to the specified time unit.
/// \param unit time unit to truncate to.
/// For Parquet versions 1.0 and 2.4, nanoseconds are casted to
microseconds.
-@@ -1085,7 +1101,8 @@
+@@ -1423,7 +1439,8 @@
/// Create the final properties.
std::shared_ptr<ArrowWriterProperties> build() {
return std::shared_ptr<ArrowWriterProperties>(new ArrowWriterProperties(
@@ -52,9 +52,9 @@ diff -ruN
arrow-apache-arrow-17.0.0-after-paimon/cpp/src/parquet/properties.h ar
+ write_timestamps_as_int96_, force_write_int96_timestamps_,
+ coerce_timestamps_enabled_, coerce_timestamps_unit_,
truncated_timestamps_allowed_, store_schema_,
compliant_nested_types_,
- engine_version_, use_threads_, executor_));
+ engine_version_, use_threads_, executor_,
write_time_adjusted_to_utc_));
}
-@@ -1093,6 +1110,8 @@
+@@ -1431,6 +1448,8 @@
private:
bool write_timestamps_as_int96_;
@@ -63,7 +63,7 @@ diff -ruN
arrow-apache-arrow-17.0.0-after-paimon/cpp/src/parquet/properties.h ar
bool coerce_timestamps_enabled_;
::arrow::TimeUnit::type coerce_timestamps_unit_;
bool truncated_timestamps_allowed_;
-@@ -1107,6 +1126,8 @@
+@@ -1447,6 +1466,8 @@
bool support_deprecated_int96_timestamps() const { return
write_timestamps_as_int96_; }
@@ -72,7 +72,7 @@ diff -ruN
arrow-apache-arrow-17.0.0-after-paimon/cpp/src/parquet/properties.h ar
bool coerce_timestamps_enabled() const { return coerce_timestamps_enabled_;
}
::arrow::TimeUnit::type coerce_timestamps_unit() const {
return coerce_timestamps_unit_;
-@@ -1138,6 +1159,7 @@
+@@ -1483,6 +1504,7 @@
private:
explicit ArrowWriterProperties(bool write_nanos_as_int96,
@@ -80,16 +80,16 @@ diff -ruN
arrow-apache-arrow-17.0.0-after-paimon/cpp/src/parquet/properties.h ar
bool coerce_timestamps_enabled,
::arrow::TimeUnit::type
coerce_timestamps_unit,
bool truncated_timestamps_allowed, bool
store_schema,
-@@ -1145,6 +1167,7 @@
- EngineVersion engine_version, bool
use_threads,
- ::arrow::internal::Executor* executor)
+@@ -1491,6 +1513,7 @@
+ ::arrow::internal::Executor* executor,
+ bool write_time_adjusted_to_utc)
: write_timestamps_as_int96_(write_nanos_as_int96),
+ force_write_int96_timestamps_(force_write_int96_timestamps),
coerce_timestamps_enabled_(coerce_timestamps_enabled),
coerce_timestamps_unit_(coerce_timestamps_unit),
truncated_timestamps_allowed_(truncated_timestamps_allowed),
-@@ -1155,6 +1178,7 @@
- executor_(executor) {}
+@@ -1502,6 +1525,7 @@
+ write_time_adjusted_to_utc_(write_time_adjusted_to_utc) {}
const bool write_timestamps_as_int96_;
+ const bool force_write_int96_timestamps_;
diff --git a/thirdparty/patches/apache-arrow-17.0.0-lzo.patch
b/thirdparty/patches/apache-arrow-24.0.0-lzo.patch
similarity index 55%
rename from thirdparty/patches/apache-arrow-17.0.0-lzo.patch
rename to thirdparty/patches/apache-arrow-24.0.0-lzo.patch
index a983818413a..d7076509756 100644
--- a/thirdparty/patches/apache-arrow-17.0.0-lzo.patch
+++ b/thirdparty/patches/apache-arrow-24.0.0-lzo.patch
@@ -1,11 +1,26 @@
--- a/cpp/src/parquet/column_reader.cc
+++ b/cpp/src/parquet/column_reader.cc
-@@ -30,0 +31,2 @@
-+
+@@ -29,6 +29,8 @@
+ #include <utility>
+ #include <vector>
+
+#include <lzo/lzo1x.h>
-@@ -268,0 +269 @@
++
+ #include "arrow/array.h"
+ #include "arrow/array/array_binary.h"
+ #include "arrow/array/builder_binary.h"
+@@ -228,6 +230,7 @@
+ const CryptoContext* crypto_ctx, bool
always_compressed)
+ : properties_(properties),
+ stream_(std::move(stream)),
+ compression_codec_(codec),
-@@ -279 +282,7 @@
+ decompression_buffer_(AllocateBuffer(properties_.memory_pool(), 0)),
+ page_ordinal_(0),
+ seen_num_values_(0),
+@@ -237,7 +240,13 @@
+ InitDecryption();
+ }
+ max_page_header_size_ = kDefaultMaxPageHeaderSize;
- decompressor_ = GetCodec(codec);
+ if (compression_codec_ == Compression::LZO) {
+ if (lzo_init() != LZO_E_OK) {
@@ -14,12 +29,30 @@
+ } else {
+ decompressor_ = GetCodec(codec);
+ }
-@@ -315,0 +325 @@
+ always_compressed_ = always_compressed;
+ }
+
+@@ -271,6 +280,7 @@
+ format::PageHeader current_page_header_;
+
+ // Compression codec to use.
+ Compression::type compression_codec_;
-@@ -585 +595 @@
+ std::unique_ptr<::arrow::util::Codec> decompressor_;
+ std::shared_ptr<ResizableBuffer> decompression_buffer_;
+
+@@ -548,7 +558,7 @@
+ std::shared_ptr<Buffer> SerializedPageReader::DecompressIfNeeded(
+ std::shared_ptr<Buffer> page_buffer, int compressed_len, int
uncompressed_len,
+ int levels_byte_len) {
- if (decompressor_ == nullptr) {
+ if (decompressor_ == nullptr && compression_codec_ != Compression::LZO) {
-@@ -601,0 +612,61 @@
+ return page_buffer;
+ }
+ if (compressed_len < levels_byte_len || uncompressed_len < levels_byte_len)
{
+@@ -565,6 +575,74 @@
+ memcpy(decompressed, page_buffer->data(), levels_byte_len);
+ }
+
+ if (compression_codec_ == Compression::LZO) {
+ const uint8_t* input = page_buffer->data() + levels_byte_len;
+ const uint8_t* const input_end = page_buffer->data() + compressed_len;
@@ -34,7 +67,8 @@
+
+ while (input < input_end) {
+ if (input_end - input < 4) {
-+ throw ParquetException("LZO page decompression failed: truncated
large block length");
++ throw ParquetException(
++ "LZO page decompression failed: truncated large block length");
+ }
+
+ uint32_t large_block_uncompressed_len = load_big_endian_u32(input);
@@ -45,13 +79,15 @@
+
+ while (large_block_uncompressed_len > 0) {
+ if (input_end - input < 4) {
-+ throw ParquetException("LZO page decompression failed: truncated
small block length");
++ throw ParquetException(
++ "LZO page decompression failed: truncated small block length");
+ }
+
+ uint32_t small_block_compressed_len = load_big_endian_u32(input);
+ input += 4;
+ if (static_cast<size_t>(input_end - input) <
small_block_compressed_len) {
-+ throw ParquetException("LZO page decompression failed: truncated
small block data");
++ throw ParquetException(
++ "LZO page decompression failed: truncated small block data");
+ }
+
+ auto small_block_uncompressed_len =
@@ -64,7 +100,8 @@
+ std::to_string(result));
+ }
+ if (small_block_uncompressed_len > large_block_uncompressed_len) {
-+ throw ParquetException("LZO page decompression failed: invalid
small block size");
++ throw ParquetException(
++ "LZO page decompression failed: invalid small block size");
+ }
+
+ input += small_block_compressed_len;
@@ -73,12 +110,17 @@
+ }
+ }
+ if (output != output_end) {
-+ throw ParquetException("Page didn't decompress to expected size,
expected: " +
-+ std::to_string(uncompressed_len -
levels_byte_len) + ", but got:" +
-+ std::to_string(output -
(decompression_buffer_->mutable_data() +
-+ levels_byte_len)));
++ throw ParquetException(
++ "Page didn't decompress to expected size, expected: " +
++ std::to_string(uncompressed_len - levels_byte_len) +
++ ", but got:" +
++ std::to_string(
++ output - (decompression_buffer_->mutable_data() +
levels_byte_len)));
+ }
+
+ return decompression_buffer_;
+ }
+
+ // GH-31992: DataPageV2 may store only levels and no values when all
+ // values are null. In this case, Parquet java is known to produce a
+ // 0-len compressed area (which is invalid compressed input).
diff --git a/thirdparty/patches/apache-arrow-17.0.0-paimon.patch
b/thirdparty/patches/apache-arrow-24.0.0-paimon.patch
similarity index 68%
rename from thirdparty/patches/apache-arrow-17.0.0-paimon.patch
rename to thirdparty/patches/apache-arrow-24.0.0-paimon.patch
index 4e53117b79b..583f85b4400 100644
--- a/thirdparty/patches/apache-arrow-17.0.0-paimon.patch
+++ b/thirdparty/patches/apache-arrow-24.0.0-paimon.patch
@@ -1,23 +1,19 @@
-diff --git a/cpp/src/parquet/arrow/schema.cc b/cpp/src/parquet/arrow/schema.cc
-index ec3890a41f..943f69bb6c 100644
--- a/cpp/src/parquet/arrow/schema.cc
+++ b/cpp/src/parquet/arrow/schema.cc
-@@ -178,7 +178,7 @@ static Status GetTimestampMetadata(const
::arrow::TimestampType& type,
+@@ -215,7 +215,8 @@
// The user is explicitly asking for Impala int96 encoding, there is no
// logical type.
- if (arrow_properties.support_deprecated_int96_timestamps()) {
-+ if (arrow_properties.support_deprecated_int96_timestamps() && target_unit
== ::arrow::TimeUnit::NANO) {
++ if (arrow_properties.support_deprecated_int96_timestamps() &&
++ target_unit == ::arrow::TimeUnit::NANO) {
*physical_type = ParquetType::INT96;
return Status::OK();
}
-
-diff --git a/cpp/src/parquet/arrow/reader.cc b/cpp/src/parquet/arrow/reader.cc
-index 285e2a5973..aa6f92f077 100644
--- a/cpp/src/parquet/arrow/reader.cc
+++ b/cpp/src/parquet/arrow/reader.cc
-@@ -1013,25 +1013,32 @@ Status FileReaderImpl::GetRecordBatchReader(const
std::vector<int>& row_groups,
- return Status::OK();
+@@ -1041,25 +1041,32 @@
+ ::arrow::MakeVectorIterator(std::move(batches)),
std::move(batch_schema));
}
- int64_t num_rows = 0;
@@ -28,7 +24,7 @@ index 285e2a5973..aa6f92f077 100644
}
using ::arrow::RecordBatchIterator;
-+ int row_group_idx = 0;
++ size_t row_group_idx = 0;
// NB: This lambda will be invoked outside the scope of this call to
// `GetRecordBatchReader()`, so it must capture `readers` and
`batch_schema` by value.
@@ -40,26 +36,23 @@ index 285e2a5973..aa6f92f077 100644
this]() mutable -> ::arrow::Result<RecordBatchIterator> {
::arrow::ChunkedArrayVector columns(readers.size());
-- // don't reserve more rows than necessary
+ // don't reserve more rows than necessary
- int64_t batch_size = std::min(properties().batch_size(), num_rows);
- num_rows -= batch_size;
+ int64_t batch_size = 0;
+ if (!num_rows.empty()) {
-+ // don't reserve more rows than necessary
+ batch_size = std::min(properties().batch_size(),
num_rows[row_group_idx]);
+ num_rows[row_group_idx] -= batch_size;
-+ if (num_rows[row_group_idx] == 0 && (num_rows.size() - 1) !=
row_group_idx) {
-+ row_group_idx++;
++ if (num_rows[row_group_idx] == 0 && row_group_idx + 1 <
num_rows.size()) {
++ ++row_group_idx;
+ }
+ }
RETURN_NOT_OK(::arrow::internal::OptionalParallelFor(
reader_properties_.use_threads(),
static_cast<int>(readers.size()),
-diff --git a/cpp/src/parquet/arrow/writer.cc b/cpp/src/parquet/arrow/writer.cc
-index 4fd7ef1b47..87326a54f1 100644
--- a/cpp/src/parquet/arrow/writer.cc
+++ b/cpp/src/parquet/arrow/writer.cc
-@@ -314,6 +314,14 @@ class FileWriterImpl : public FileWriter {
+@@ -341,6 +341,14 @@
return Status::OK();
}
@@ -68,13 +61,13 @@ index 4fd7ef1b47..87326a54f1 100644
+ return 0;
+ }
+ return row_group_writer_->total_compressed_bytes() +
-+ row_group_writer_->total_compressed_bytes_written();
++ row_group_writer_->total_compressed_bytes_written();
+ }
+
Status Close() override {
if (!closed_) {
// Make idempotent
-@@ -418,10 +426,13 @@ class FileWriterImpl : public FileWriter {
+@@ -444,10 +452,14 @@
// Max number of rows allowed in a row group.
const int64_t max_row_group_length =
this->properties().max_row_group_length();
@@ -84,16 +77,15 @@ index 4fd7ef1b47..87326a54f1 100644
if (row_group_writer_ == nullptr || !row_group_writer_->buffered() ||
- row_group_writer_->num_rows() >= max_row_group_length) {
+ row_group_writer_->num_rows() >= max_row_group_length ||
-+ (row_group_writer_->total_compressed_bytes_written() +
-+ row_group_writer_->total_compressed_bytes() >= max_row_group_size)) {
++ row_group_writer_->total_compressed_bytes_written() +
++ row_group_writer_->total_compressed_bytes() >=
++ max_row_group_size) {
RETURN_NOT_OK(NewBufferedRowGroup());
}
-diff --git a/cpp/src/parquet/arrow/writer.h b/cpp/src/parquet/arrow/writer.h
-index 4a1a033a7b..0f13d05e44 100644
--- a/cpp/src/parquet/arrow/writer.h
+++ b/cpp/src/parquet/arrow/writer.h
-@@ -138,6 +138,9 @@ class PARQUET_EXPORT FileWriter {
+@@ -124,6 +124,9 @@
/// option in this case.
virtual ::arrow::Status WriteRecordBatch(const ::arrow::RecordBatch& batch)
= 0;
@@ -103,11 +95,9 @@ index 4a1a033a7b..0f13d05e44 100644
/// \brief Write the footer and close the file.
virtual ::arrow::Status Close() = 0;
virtual ~FileWriter();
-diff --git a/cpp/src/parquet/properties.h b/cpp/src/parquet/properties.h
-index 4d3acb491e..3906ff3c59 100644
--- a/cpp/src/parquet/properties.h
+++ b/cpp/src/parquet/properties.h
-@@ -139,6 +139,7 @@ static constexpr bool DEFAULT_IS_DICTIONARY_ENABLED = true;
+@@ -160,6 +160,7 @@
static constexpr int64_t DEFAULT_DICTIONARY_PAGE_SIZE_LIMIT =
kDefaultDataPageSize;
static constexpr int64_t DEFAULT_WRITE_BATCH_SIZE = 1024;
static constexpr int64_t DEFAULT_MAX_ROW_GROUP_LENGTH = 1024 * 1024;
@@ -115,28 +105,28 @@ index 4d3acb491e..3906ff3c59 100644
static constexpr bool DEFAULT_ARE_STATISTICS_ENABLED = true;
static constexpr int64_t DEFAULT_MAX_STATISTICS_SIZE = 4096;
static constexpr Encoding::type DEFAULT_ENCODING = Encoding::UNKNOWN;
-@@ -232,6 +233,7 @@ class PARQUET_EXPORT WriterProperties {
+@@ -343,6 +344,7 @@
dictionary_pagesize_limit_(DEFAULT_DICTIONARY_PAGE_SIZE_LIMIT),
write_batch_size_(DEFAULT_WRITE_BATCH_SIZE),
max_row_group_length_(DEFAULT_MAX_ROW_GROUP_LENGTH),
+ max_row_group_size_(DEFAULT_MAX_ROW_GROUP_SIZE),
pagesize_(kDefaultDataPageSize),
+ max_rows_per_page_(kDefaultMaxRowsPerPage),
version_(ParquetVersion::PARQUET_2_6),
- data_page_version_(ParquetDataPageVersion::V1),
-@@ -244,6 +246,7 @@ class PARQUET_EXPORT WriterProperties {
+@@ -359,6 +361,7 @@
dictionary_pagesize_limit_(properties.dictionary_pagesize_limit()),
write_batch_size_(properties.write_batch_size()),
max_row_group_length_(properties.max_row_group_length()),
+ max_row_group_size_(properties.max_row_group_size()),
pagesize_(properties.data_pagesize()),
+ max_rows_per_page_(properties.max_rows_per_page()),
version_(properties.version()),
- data_page_version_(properties.data_page_version()),
-@@ -321,6 +324,13 @@ class PARQUET_EXPORT WriterProperties {
+@@ -468,6 +471,13 @@
return this;
}
+ /// Specify the max bytes size to put in a single row group.
-+ /// Default 128 M.
++ /// Default 128MB.
+ Builder* max_row_group_size(int64_t max_row_group_size) {
+ max_row_group_size_ = max_row_group_size;
+ return this;
@@ -145,24 +135,33 @@ index 4d3acb491e..3906ff3c59 100644
/// Specify the data page size.
/// Default 1MB.
Builder* data_pagesize(int64_t pg_size) {
-@@ -664,7 +674,7 @@ class PARQUET_EXPORT WriterProperties {
+@@ -867,11 +877,12 @@
return std::shared_ptr<WriterProperties>(new WriterProperties(
pool_, dictionary_pagesize_limit_, write_batch_size_,
max_row_group_length_,
-- pagesize_, version_, created_by_, page_checksum_enabled_,
-+ max_row_group_size_, pagesize_, version_, created_by_,
page_checksum_enabled_,
- std::move(file_encryption_properties_), default_column_properties_,
- column_properties, data_page_version_, store_decimal_as_integer_,
- std::move(sorting_columns_)));
-@@ -675,6 +685,7 @@ class PARQUET_EXPORT WriterProperties {
+- pagesize_, max_rows_per_page_, version_, created_by_,
page_checksum_enabled_,
+- size_statistics_level_, std::move(file_encryption_properties_),
+- default_column_properties_, column_properties, data_page_version_,
+- store_decimal_as_integer_, std::move(sorting_columns_),
+- content_defined_chunking_enabled_,
content_defined_chunking_options_));
++ max_row_group_size_, pagesize_, max_rows_per_page_, version_,
created_by_,
++ page_checksum_enabled_, size_statistics_level_,
++ std::move(file_encryption_properties_), default_column_properties_,
++ column_properties, data_page_version_, store_decimal_as_integer_,
++ std::move(sorting_columns_), content_defined_chunking_enabled_,
++ content_defined_chunking_options_));
+ }
+
+ private:
+@@ -881,6 +892,7 @@
int64_t dictionary_pagesize_limit_;
int64_t write_batch_size_;
int64_t max_row_group_length_;
+ int64_t max_row_group_size_;
int64_t pagesize_;
+ int64_t max_rows_per_page_;
ParquetVersion::type version_;
- ParquetDataPageVersion data_page_version_;
-@@ -705,6 +716,8 @@ class PARQUET_EXPORT WriterProperties {
+@@ -917,6 +929,8 @@
inline int64_t max_row_group_length() const { return max_row_group_length_;
}
@@ -170,40 +169,43 @@ index 4d3acb491e..3906ff3c59 100644
+
inline int64_t data_pagesize() const { return pagesize_; }
- inline ParquetDataPageVersion data_page_version() const {
-@@ -810,7 +823,7 @@ class PARQUET_EXPORT WriterProperties {
+ inline int64_t max_rows_per_page() const { return max_rows_per_page_; }
+@@ -1046,9 +1060,10 @@
private:
explicit WriterProperties(
MemoryPool* pool, int64_t dictionary_pagesize_limit, int64_t
write_batch_size,
-- int64_t max_row_group_length, int64_t pagesize, ParquetVersion::type
version,
-+ int64_t max_row_group_length, int64_t max_row_group_size, int64_t
pagesize, ParquetVersion::type version,
- const std::string& created_by, bool page_write_checksum_enabled,
+- int64_t max_row_group_length, int64_t pagesize, int64_t
max_rows_per_page,
+- ParquetVersion::type version, const std::string& created_by,
+- bool page_write_checksum_enabled, SizeStatisticsLevel
size_statistics_level,
++ int64_t max_row_group_length, int64_t max_row_group_size, int64_t
pagesize,
++ int64_t max_rows_per_page, ParquetVersion::type version,
++ const std::string& created_by, bool page_write_checksum_enabled,
++ SizeStatisticsLevel size_statistics_level,
std::shared_ptr<FileEncryptionProperties> file_encryption_properties,
const ColumnProperties& default_column_properties,
-@@ -821,6 +834,7 @@ class PARQUET_EXPORT WriterProperties {
+ const std::unordered_map<std::string, ColumnProperties>&
column_properties,
+@@ -1059,6 +1074,7 @@
dictionary_pagesize_limit_(dictionary_pagesize_limit),
write_batch_size_(write_batch_size),
max_row_group_length_(max_row_group_length),
+ max_row_group_size_(max_row_group_size),
pagesize_(pagesize),
+ max_rows_per_page_(max_rows_per_page),
parquet_data_page_version_(data_page_version),
- parquet_version_(version),
-@@ -836,6 +850,7 @@ class PARQUET_EXPORT WriterProperties {
+@@ -1078,6 +1094,7 @@
int64_t dictionary_pagesize_limit_;
int64_t write_batch_size_;
int64_t max_row_group_length_;
+ int64_t max_row_group_size_;
int64_t pagesize_;
+ int64_t max_rows_per_page_;
ParquetDataPageVersion parquet_data_page_version_;
- ParquetVersion::type parquet_version_;
-diff --git a/cpp/cmake_modules/ThirdpartyToolchain.cmake
b/cpp/cmake_modules/ThirdpartyToolchain.cmake
-index 9df922afa2..5c8b3d4d07 100644
--- a/cpp/cmake_modules/ThirdpartyToolchain.cmake
+++ b/cpp/cmake_modules/ThirdpartyToolchain.cmake
-@@ -1789,7 +1789,20 @@ if(ARROW_WITH_THRIFT)
- REQUIRED_VERSION
- 0.11.0)
-
+@@ -1884,7 +1884,20 @@
+ REQUIRED_VERSION
+ 0.11.0)
+
- string(REPLACE "." ";" Thrift_VERSION_LIST ${Thrift_VERSION})
+ if(NOT Thrift_VERSION)
+ if(DEFINED thrift_PC_VERSION AND thrift_PC_VERSION)
diff --git a/thirdparty/patches/paimon-cpp-arrow-24-compatibility.patch
b/thirdparty/patches/paimon-cpp-arrow-24-compatibility.patch
new file mode 100644
index 00000000000..d0e723a34d9
--- /dev/null
+++ b/thirdparty/patches/paimon-cpp-arrow-24-compatibility.patch
@@ -0,0 +1,64 @@
+diff --git a/include/paimon/status.h b/include/paimon/status.h
+--- a/include/paimon/status.h
++++ b/include/paimon/status.h
+@@ -458,9 +458,4 @@
+ } \
+ } while (false)
+
+-// This is an internal-use macro and should not be used in public headers.
+-#ifndef RETURN_NOT_OK
+-#define RETURN_NOT_OK(s) PAIMON_RETURN_NOT_OK(s)
+-#endif
+-
+ } // namespace paimon
+diff --git a/src/paimon/format/parquet/parquet_input_stream_impl.cpp
b/src/paimon/format/parquet/parquet_input_stream_impl.cpp
+--- a/src/paimon/format/parquet/parquet_input_stream_impl.cpp
++++ b/src/paimon/format/parquet/parquet_input_stream_impl.cpp
+@@ -59,7 +59,7 @@ arrow::Result<std::shared_ptr<arrow::Buffer>>
ParquetInputStreamImpl::Read(int64
+ arrow::AllocateResizableBuffer(nbytes,
pool_.get()));
+ ARROW_ASSIGN_OR_RAISE(int64_t read_bytes, Read(nbytes,
buffer->mutable_data()));
+ if (read_bytes < nbytes) {
+- RETURN_NOT_OK(buffer->Resize(read_bytes));
++ ARROW_RETURN_NOT_OK(buffer->Resize(read_bytes));
+ }
+ return std::shared_ptr<arrow::Buffer>(std::move(buffer));
+ }
+@@ -78,7 +78,7 @@ arrow::Result<std::shared_ptr<arrow::Buffer>>
ParquetInputStreamImpl::ReadAt(int
+ arrow::AllocateResizableBuffer(nbytes,
pool_.get()));
+ ARROW_ASSIGN_OR_RAISE(int64_t read_bytes, ReadAt(position, nbytes,
buffer->mutable_data()));
+ if (read_bytes < nbytes) {
+- RETURN_NOT_OK(buffer->Resize(read_bytes));
++ ARROW_RETURN_NOT_OK(buffer->Resize(read_bytes));
+ }
+ return std::shared_ptr<arrow::Buffer>(std::move(buffer));
+ }
+diff --git a/src/paimon/format/parquet/file_reader_wrapper.cpp
b/src/paimon/format/parquet/file_reader_wrapper.cpp
+--- a/src/paimon/format/parquet/file_reader_wrapper.cpp
++++ b/src/paimon/format/parquet/file_reader_wrapper.cpp
+@@ -81,8 +81,9 @@ Status FileReaderWrapper::SeekToRow(uint64_t row_number) {
+ PAIMON_ASSIGN_OR_RAISE(int32_t row_group_id,
GetRowGroupId(target_row_groups_[j]));
+ target_row_group_indices.push_back(row_group_id);
+ }
+-
PAIMON_RETURN_NOT_OK_FROM_ARROW(file_reader_->GetRecordBatchReader(
+- target_row_group_indices, target_column_indices_,
&batch_reader_));
++ PAIMON_ASSIGN_OR_RAISE_FROM_ARROW(
++ batch_reader_,
file_reader_->GetRecordBatchReader(target_row_group_indices,
++
target_column_indices_));
+ return Status::OK();
+ }
+ }
+@@ -152,9 +153,11 @@ Status FileReaderWrapper::PrepareForReading(
+ std::vector<std::pair<uint64_t, uint64_t>> target_row_groups;
+ PAIMON_ASSIGN_OR_RAISE(target_row_groups,
GetRowGroupRanges(target_row_group_indices));
+ std::unique_ptr<arrow::RecordBatchReader> batch_reader;
+- PAIMON_RETURN_NOT_OK_FROM_ARROW(file_reader_->GetRecordBatchReader(
+- std::vector<int32_t>(target_row_group_indices.begin(),
target_row_group_indices.end()),
+- column_indices, &batch_reader));
++ PAIMON_ASSIGN_OR_RAISE_FROM_ARROW(
++ batch_reader,
++ file_reader_->GetRecordBatchReader(
++ std::vector<int32_t>(target_row_group_indices.begin(),
target_row_group_indices.end()),
++ column_indices));
+ target_row_groups_ = target_row_groups;
+ target_column_indices_ = column_indices;
+ batch_reader_ = std::move(batch_reader);
diff --git a/thirdparty/patches/paimon-cpp-arrow-24-compute.patch
b/thirdparty/patches/paimon-cpp-arrow-24-compute.patch
new file mode 100644
index 00000000000..1b581b8f93e
--- /dev/null
+++ b/thirdparty/patches/paimon-cpp-arrow-24-compute.patch
@@ -0,0 +1,44 @@
+diff --git a/cmake_modules/ThirdpartyToolchain.cmake
b/cmake_modules/ThirdpartyToolchain.cmake
+--- a/cmake_modules/ThirdpartyToolchain.cmake
++++ b/cmake_modules/ThirdpartyToolchain.cmake
+@@ -38,6 +38,8 @@
+ set(PAIMON_EXTERNAL_ARROW_INCLUDE_DIR "" CACHE PATH
+ "Include directory for external Arrow/Parquet headers")
+ set(PAIMON_EXTERNAL_ARROW_LIB "" CACHE FILEPATH "Path to external libarrow.a")
++set(PAIMON_EXTERNAL_ARROW_COMPUTE_LIB "" CACHE FILEPATH
++ "Path to external libarrow_compute.a")
+ set(PAIMON_EXTERNAL_ARROW_DATASET_LIB "" CACHE FILEPATH "Path to external
libarrow_dataset.a")
+ set(PAIMON_EXTERNAL_ARROW_ACERO_LIB "" CACHE FILEPATH "Path to external
libarrow_acero.a")
+ set(PAIMON_EXTERNAL_PARQUET_LIB "" CACHE FILEPATH "Path to external
libparquet.a")
+@@ -1008,6 +1010,7 @@
+
+ foreach(_paimon_external_lib
+ IN ITEMS PAIMON_EXTERNAL_ARROW_LIB
++ PAIMON_EXTERNAL_ARROW_COMPUTE_LIB
+ PAIMON_EXTERNAL_ARROW_DATASET_LIB
+ PAIMON_EXTERNAL_ARROW_ACERO_LIB
+ PAIMON_EXTERNAL_PARQUET_LIB
+@@ -1028,6 +1031,13 @@
+ INTERFACE_INCLUDE_DIRECTORIES
+ "${ARROW_INCLUDE_DIR}")
+
++ add_library(arrow_compute STATIC IMPORTED)
++ set_target_properties(arrow_compute
++ PROPERTIES IMPORTED_LOCATION
++
"${PAIMON_EXTERNAL_ARROW_COMPUTE_LIB}"
++ INTERFACE_INCLUDE_DIRECTORIES
++ "${ARROW_INCLUDE_DIR}")
++
+ add_library(arrow_dataset STATIC IMPORTED)
+ set_target_properties(arrow_dataset
+ PROPERTIES IMPORTED_LOCATION
+@@ -1055,7 +1065,9 @@
+ INTERFACE_INCLUDE_DIRECTORIES
+ "${ARROW_INCLUDE_DIR}")
+
+- target_link_libraries(arrow_acero INTERFACE arrow)
++ target_link_libraries(arrow_compute INTERFACE arrow)
++
++ target_link_libraries(arrow_acero INTERFACE arrow_compute)
+
+ target_link_libraries(arrow_dataset INTERFACE arrow_acero)
diff --git a/thirdparty/test/arrow-paimon-lifecycle-test.sh
b/thirdparty/test/arrow-paimon-lifecycle-test.sh
new file mode 100755
index 00000000000..9873218a40a
--- /dev/null
+++ b/thirdparty/test/arrow-paimon-lifecycle-test.sh
@@ -0,0 +1,492 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+set -eo pipefail
+
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." &>/dev/null && pwd)"
+
+fail() {
+ echo "FAIL: $*" >&2
+ exit 1
+}
+
+tmpdir="$(mktemp -d)"
+trap 'rm -rf "${tmpdir}"' EXIT
+
+create_fingerprint_fixture() {
+ local destination="$1"
+
+ mkdir -p "${destination}/patches"
+ cp "${ROOT}/arrow-paimon-vars.sh" "${destination}/arrow-paimon-vars.sh"
+ cp "${ROOT}/paimon-cpp-cache.cmake" "${destination}/paimon-cpp-cache.cmake"
+ cp "${ROOT}"/patches/apache-arrow-24.0.0-*.patch "${destination}/patches/"
+ cp "${ROOT}"/patches/paimon-cpp-*.patch "${destination}/patches/"
+}
+
+fingerprint_from_fixture() {
+ local fixture="$1"
+ local component="$2"
+ local result_variable="$3"
+ local fingerprint
+
+ fingerprint="$(
+ set -e
+ # shellcheck source=/dev/null
+ . "${fixture}/arrow-paimon-vars.sh"
+ "${component}_build_fingerprint"
+ )"
+ printf -v "${result_variable}" '%s' "${fingerprint}"
+}
+
+exercise_semantic_fingerprints() {
+ local first_fixture="${tmpdir}/fingerprint-first"
+ local second_fixture="${tmpdir}/fingerprint-second"
+ local first_arrow
+ local first_paimon
+ local second_arrow
+ local second_paimon
+ local changed_arrow
+ local changed_paimon
+
+ create_fingerprint_fixture "${first_fixture}"
+ create_fingerprint_fixture "${second_fixture}"
+
+ printf '%s\n' first-env >"${first_fixture}/env.sh"
+ printf '%s\n' second-env >"${second_fixture}/env.sh"
+ printf '%s\n' first-vars >"${first_fixture}/vars.sh"
+ printf '%s\n' second-vars >"${second_fixture}/vars.sh"
+ printf '%s\n' first-download >"${first_fixture}/download-thirdparty.sh"
+ printf '%s\n' second-download >"${second_fixture}/download-thirdparty.sh"
+ printf '%s\n' first-build >"${first_fixture}/build-thirdparty.sh"
+ printf '%s\n' second-build >"${second_fixture}/build-thirdparty.sh"
+ printf '%s\n' '# release-branch-only comment'
>>"${second_fixture}/arrow-paimon-vars.sh"
+
+ fingerprint_from_fixture "${first_fixture}" arrow first_arrow
+ fingerprint_from_fixture "${first_fixture}" paimon first_paimon
+ [[ "${first_arrow}" == "${ARROW_LEGACY_COMPATIBLE_SEMANTIC_FINGERPRINT}"
]] ||
+ fail "the Arrow legacy marker migration target is stale"
+ [[ "${first_paimon}" == "${PAIMON_LEGACY_COMPATIBLE_SEMANTIC_FINGERPRINT}"
]] ||
+ fail "the Paimon legacy marker migration target is stale"
+ fingerprint_from_fixture "${second_fixture}" arrow second_arrow
+ fingerprint_from_fixture "${second_fixture}" paimon second_paimon
+ [[ "${first_arrow}" == "${second_arrow}" ]] ||
+ fail "unrelated branch scripts changed the Arrow fingerprint"
+ [[ "${first_paimon}" == "${second_paimon}" ]] ||
+ fail "unrelated branch scripts changed the Paimon fingerprint"
+
+ printf '%s\n' semantic-change \
+ >>"${second_fixture}/patches/apache-arrow-24.0.0-lzo.patch"
+ fingerprint_from_fixture "${second_fixture}" arrow changed_arrow
+ fingerprint_from_fixture "${second_fixture}" paimon changed_paimon
+ [[ "${changed_arrow}" != "${first_arrow}" ]] ||
+ fail "an Arrow patch change did not change the Arrow fingerprint"
+ [[ "${changed_paimon}" != "${first_paimon}" ]] ||
+ fail "an Arrow patch change did not change the Paimon fingerprint"
+
+ cp "${first_fixture}/patches/apache-arrow-24.0.0-lzo.patch" \
+ "${second_fixture}/patches/apache-arrow-24.0.0-lzo.patch"
+ printf '%s\n' semantic-change >>"${second_fixture}/paimon-cpp-cache.cmake"
+ fingerprint_from_fixture "${second_fixture}" arrow second_arrow
+ fingerprint_from_fixture "${second_fixture}" paimon changed_paimon
+ [[ "${second_arrow}" == "${first_arrow}" ]] ||
+ fail "a Paimon-only cache change changed the Arrow fingerprint"
+ [[ "${changed_paimon}" != "${first_paimon}" ]] ||
+ fail "a Paimon cache change did not change the Paimon fingerprint"
+
+ changed_arrow="$(
+ set -e
+ # shellcheck source=/dev/null
+ . "${first_fixture}/arrow-paimon-vars.sh"
+ ARROW_BUILD_SCHEMA_VERSION="${ARROW_BUILD_SCHEMA_VERSION}-changed"
+ arrow_build_fingerprint
+ )"
+ [[ "${changed_arrow}" != "${first_arrow}" ]] ||
+ fail "an Arrow build-schema change did not change its fingerprint"
+}
+
+# shellcheck source=../arrow-paimon-vars.sh
+. "${ROOT}/arrow-paimon-vars.sh"
+exercise_semantic_fingerprints
+
+harness="${tmpdir}/harness"
+mkdir -p "${harness}/src" "${harness}/patches" "${harness}/installed"
+cp "${ROOT}/download-thirdparty.sh" "${harness}/download-thirdparty.sh"
+
+create_patch() {
+ local patch_file="$1"
+ local source_file="$2"
+ {
+ printf '%s\n' "--- a/${source_file}"
+ printf '%s\n' "+++ b/${source_file}"
+ printf '%s\n' '@@ -1 +1 @@'
+ printf '%s\n' '-original'
+ printf '%s\n' '+patched'
+ } >"${patch_file}"
+}
+
+create_archive() {
+ local source_name="$1"
+ local archive_name="$2"
+ local prefix="$3"
+ local index
+
+ mkdir -p "${harness}/src/${source_name}"
+ for index in 1 2 3; do
+ printf '%s\n' original
>"${harness}/src/${source_name}/${prefix}-${index}.txt"
+ done
+ tar -czf "${harness}/src/${archive_name}" -C "${harness}/src"
"${source_name}"
+ rm -rf "${harness}/src/${source_name}"
+}
+
+arrow_source="arrow-apache-arrow-24.0.0"
+arrow_archive="apache-arrow-24.0.0.tar.gz"
+paimon_source="doris-thirdparty-paimon-cpp-0a4f4e2"
+paimon_archive="paimon-cpp-0a4f4e2.tar.gz"
+
+create_archive "${arrow_source}" "${arrow_archive}" arrow
+create_archive "${paimon_source}" "${paimon_archive}" paimon
+
+arrow_patches=(
+ apache-arrow-24.0.0-paimon.patch
+ apache-arrow-24.0.0-force-write-int96-timestamps.patch
+ apache-arrow-24.0.0-lzo.patch
+)
+paimon_patches=(
+ paimon-cpp-buildutils-static-deps.patch
+ paimon-cpp-arrow-24-compatibility.patch
+ paimon-cpp-arrow-24-compute.patch
+)
+
+for index in 0 1 2; do
+ create_patch "${harness}/patches/${arrow_patches[${index}]}"
"arrow-$((index + 1)).txt"
+ create_patch "${harness}/patches/${paimon_patches[${index}]}"
"paimon-$((index + 1)).txt"
+done
+
+arrow_md5="$(md5sum "${harness}/src/${arrow_archive}" | awk '{print $1}')"
+paimon_md5="$(md5sum "${harness}/src/${paimon_archive}" | awk '{print $1}')"
+{
+ printf 'TP_SOURCE_DIR="%s"\n' "${harness}/src"
+ printf 'TP_INSTALL_DIR="%s"\n' "${harness}/installed"
+ printf 'TP_PATCH_DIR="%s"\n' "${harness}/patches"
+ printf '%s\n' 'TP_ARCHIVES=(ARROW PAIMON_CPP)'
+ printf 'ARROW_NAME="%s"\n' "${arrow_archive}"
+ printf 'ARROW_SOURCE="%s"\n' "${arrow_source}"
+ printf 'ARROW_MD5SUM="%s"\n' "${arrow_md5}"
+ printf '%s\n' 'ARROW_DOWNLOAD="unused"'
+ printf 'PAIMON_CPP_NAME="%s"\n' "${paimon_archive}"
+ printf 'PAIMON_CPP_SOURCE="%s"\n' "${paimon_source}"
+ printf 'PAIMON_CPP_MD5SUM="%s"\n' "${paimon_md5}"
+ printf '%s\n' 'PAIMON_CPP_DOWNLOAD="unused"'
+ printf '%s\n' 'arrow_paimon_build_fingerprint() { printf "%s\n"
test-fingerprint; }'
+} >"${harness}/vars.sh"
+
+exercise_interrupted_patch_set() {
+ local package="$1"
+ local source_name="$2"
+ local archive_name="$3"
+ local prefix="$4"
+ shift 4
+ local patches=("$@")
+ local applied
+ local index
+
+ for applied in 1 2 3; do
+ rm -rf "${harness}/src/${source_name}"
+ tar -xzf "${harness}/src/${archive_name}" -C "${harness}/src"
+ for ((index = 0; index < applied; ++index)); do
+ (
+ cd "${harness}/src/${source_name}"
+ patch -s -p1 <"${harness}/patches/${patches[${index}]}"
+ )
+ done
+
+ TP_DIR="${harness}" DORIS_HOME="${tmpdir}" \
+ bash "${harness}/download-thirdparty.sh" "${package}" >/dev/null
+
+ for index in 1 2 3; do
+ [[ "$(<"${harness}/src/${source_name}/${prefix}-${index}.txt")" ==
"patched" ]] ||
+ fail "${package} did not recover after interruption boundary
${applied}"
+ done
+ [[
"$(<"${harness}/src/${source_name}/patched_mark_arrow_paimon_fingerprint")" ==
"test-fingerprint" ]] ||
+ fail "${package} fingerprint marker is missing"
+
+ touch "${harness}/src/${source_name}/idempotence-sentinel"
+ TP_DIR="${harness}" DORIS_HOME="${tmpdir}" \
+ bash "${harness}/download-thirdparty.sh" "${package}" >/dev/null
+ [[ -f "${harness}/src/${source_name}/idempotence-sentinel" ]] ||
+ fail "${package} reset a completely patched source tree"
+ done
+}
+
+exercise_interrupted_patch_set ARROW "${arrow_source}" "${arrow_archive}"
arrow \
+ "${arrow_patches[@]}"
+exercise_interrupted_patch_set PAIMON_CPP "${paimon_source}"
"${paimon_archive}" paimon \
+ "${paimon_patches[@]}"
+
+exercise_generic_recovery_dispatch() {
+ local generic="${tmpdir}/generic-recovery"
+ local thirdparty_dir="${generic}/thirdparty"
+ local external_thirdparty_dir="${generic}/external-thirdparty"
+ local args_file="${generic}/build-args.txt"
+ local output_file="${generic}/build-output.txt"
+ local external_builder_called="${generic}/external-builder-called"
+ local fake_mvn="${generic}/fake-mvn"
+ local non_native_status=79
+ local non_native_target
+ local status
+ local flag
+ local parallel
+ local clean
+ local package1
+ local package2
+ local extra
+
+ mkdir -p "${thirdparty_dir}/installed/lib/hadoop_hdfs/native" \
+ "${external_thirdparty_dir}/installed/lib/hadoop_hdfs/native" \
+ "${generic}/gensrc" "${generic}/fe" "${generic}/be/build_Release" \
+ "${generic}/be/output"
+ cp "${ROOT}/../build.sh" "${generic}/build.sh"
+ cp "${ROOT}/arrow-paimon-vars.sh" "${thirdparty_dir}/arrow-paimon-vars.sh"
+ touch "${thirdparty_dir}/installed/lib/hadoop_hdfs/native/libhdfs.a"
+ touch
"${external_thirdparty_dir}/installed/lib/hadoop_hdfs/native/libhdfs.a"
+ printf '%s\n' 'clean: ; @:' >"${generic}/gensrc/Makefile"
+ # shellcheck disable=SC2016
+ printf '%s\n' '#!/usr/bin/env bash' '[[ "$1" == "clean" ]]' >"${fake_mvn}"
+ chmod +x "${fake_mvn}"
+ {
+ printf '%s\n' '#!/usr/bin/env bash'
+ printf 'exit %q\n' "${non_native_status}"
+ } >"${generic}/generated-source.sh"
+ chmod +x "${generic}/generated-source.sh"
+ {
+ printf '%s\n' 'DORIS_BUILD_PROFILE=0'
+ printf '%s\n' 'TARGET_SYSTEM=Linux'
+ printf 'MVN_CMD=%q\n' "${fake_mvn}"
+ } >"${generic}/env.sh"
+ {
+ printf '%s\n' '#!/usr/bin/env bash'
+ # RECOVERY_ARGS_FILE is expanded when the generated builder runs.
+ # shellcheck disable=SC2016
+ printf '%s\n' 'printf "%s\n" "$*" >"${RECOVERY_ARGS_FILE:?}"'
+ # shellcheck disable=SC2016
+ printf '%s\n' 'exit "${RECOVERY_EXIT_STATUS:-73}"'
+ } >"${thirdparty_dir}/build-thirdparty.sh"
+
+ DORIS_THIRDPARTY="${thirdparty_dir}" RECOVERY_ARGS_FILE="${args_file}" \
+ bash "${generic}/build.sh" --clean >"${output_file}" 2>&1 ||
+ fail "bare clean failed before reaching its clean-only exit"
+ [[ ! -e "${args_file}" ]] || fail "bare clean invoked the thirdparty
builder"
+ [[ ! -d "${generic}/be/build_Release" && ! -d "${generic}/be/output" ]] ||
+ fail "bare clean did not remove BE build outputs"
+
+ for non_native_target in --fe --hive-udf; do
+ if DORIS_THIRDPARTY="${thirdparty_dir}"
RECOVERY_ARGS_FILE="${args_file}" \
+ bash "${generic}/build.sh" "${non_native_target}"
>"${output_file}" 2>&1; then
+ fail "${non_native_target} did not reach the generated-source
sentinel"
+ else
+ status=$?
+ fi
+ [[ "${status}" -eq "${non_native_status}" ]] ||
+ fail "${non_native_target} failed before the generated-source
sentinel"
+ [[ ! -e "${args_file}" ]] ||
+ fail "${non_native_target} invoked the native Arrow/Paimon builder"
+ done
+
+ if DORIS_THIRDPARTY="${thirdparty_dir}" RECOVERY_ARGS_FILE="${args_file}" \
+ bash "${generic}/build.sh" --fe --clean >"${output_file}" 2>&1; then
+ fail "--fe --clean did not reach the generated-source sentinel"
+ else
+ status=$?
+ fi
+ [[ "${status}" -eq "${non_native_status}" ]] ||
+ fail "--fe --clean failed before the generated-source sentinel"
+ [[ ! -e "${args_file}" ]] ||
+ fail "--fe --clean invoked the native Arrow/Paimon builder"
+
+ if DORIS_THIRDPARTY="${thirdparty_dir}" RECOVERY_ARGS_FILE="${args_file}" \
+ bash "${generic}/build.sh" --be >"${output_file}" 2>&1; then
+ fail "generic stale-prebuilt recovery did not invoke the focused
builder"
+ else
+ status=$?
+ fi
+ [[ "${status}" -eq 73 ]] || fail "generic recovery failed before invoking
its builder"
+ read -r flag parallel package1 package2 extra <"${args_file}"
+ [[ "${flag}" == "-j" && "${parallel}" =~ ^[0-9]+$ &&
+ "${package1}" == "arrow" && "${package2}" == "paimon_cpp" && -z
"${extra}" ]] ||
+ fail "generic recovery dispatched the wrong build package set"
+
+ if DORIS_THIRDPARTY="${thirdparty_dir}" RECOVERY_ARGS_FILE="${args_file}" \
+ bash "${generic}/build.sh" --be --clean >"${output_file}" 2>&1; then
+ fail "generic clean recovery did not invoke the focused builder"
+ else
+ status=$?
+ fi
+ [[ "${status}" -eq 73 ]] || fail "generic clean recovery failed before
invoking its builder"
+ read -r flag parallel clean package1 package2 extra <"${args_file}"
+ [[ "${flag}" == "-j" && "${parallel}" =~ ^[0-9]+$ && "${clean}" ==
"--clean" &&
+ "${package1}" == "arrow" && "${package2}" == "paimon_cpp" && -z
"${extra}" ]] ||
+ fail "generic clean recovery dispatched the wrong build package set"
+
+ if DORIS_THIRDPARTY="${thirdparty_dir}" RECOVERY_ARGS_FILE="${args_file}" \
+ RECOVERY_EXIT_STATUS=0 bash "${generic}/build.sh" --be
>"${output_file}" 2>&1; then
+ fail "generic recovery accepted artifacts that failed current-checkout
validation"
+ fi
+ grep -Fq "Rebuilt Arrow/Paimon artifacts do not match this checkout's
selected inputs" \
+ "${output_file}" || fail "generic recovery did not validate artifacts
after its builder returned"
+
+ if DORIS_THIRDPARTY="${external_thirdparty_dir}" \
+ bash "${generic}/build.sh" --be >"${output_file}" 2>&1; then
+ fail "generic recovery accepted an invalid install-only thirdparty
prefix"
+ fi
+ grep -Fq "is an install-only or incomplete prefix" "${output_file}" ||
+ fail "generic recovery did not explain how to refresh an install-only
prebuilt"
+
+ {
+ printf '%s\n' '#!/usr/bin/env bash'
+ # EXTERNAL_BUILDER_CALLED is expanded when the generated builder runs.
+ # shellcheck disable=SC2016
+ printf '%s\n' 'touch "${EXTERNAL_BUILDER_CALLED:?}"'
+ } >"${external_thirdparty_dir}/build-thirdparty.sh"
+ if DORIS_THIRDPARTY="${external_thirdparty_dir}" \
+ EXTERNAL_BUILDER_CALLED="${external_builder_called}" \
+ bash "${generic}/build.sh" --be >"${output_file}" 2>&1; then
+ fail "generic recovery accepted an external thirdparty source tree"
+ fi
+ grep -Fq "Cannot rebuild thirdparty libraries with an external source
tree" "${output_file}" ||
+ fail "generic recovery did not reject an external thirdparty source
tree"
+ [[ ! -e "${external_builder_called}" ]] ||
+ fail "generic recovery invoked an external thirdparty builder"
+}
+
+exercise_generic_recovery_dispatch
+
+# A Paimon-only build may publish only its own fingerprint. It must not make a
+# stale Arrow installation pass the shared prebuilt validation.
+prebuilt="${tmpdir}/prebuilt"
+mkdir -p "${prebuilt}/include/arrow/util" "${prebuilt}/lib64"
+printf '#define ARROW_VERSION_STRING "%s"\n' "${ARROW_VERSION}" \
+ >"${prebuilt}/include/arrow/util/config.h"
+for library in "${ARROW_PAIMON_REQUIRED_LIBRARIES[@]}"; do
+ touch "${prebuilt}/lib64/${library}"
+done
+
+prepare_arrow_paimon_download_packages "${ARROW_PAIMON_BUILD_PACKAGES[@]}"
+[[ "${ARROW_PAIMON_BUILD_PACKAGES[*]}" == "arrow paimon_cpp" ]] ||
+ fail "focused recovery dispatches a bundled source package as a build
target"
+[[ "${ARROW_PAIMON_DOWNLOAD_PACKAGES[*]}" == "arrow paimon_cpp xsimd brotli"
]] ||
+ fail "focused recovery does not download the complete Arrow source closure"
+
+# A legacy prebuilt may have the old combined marker but no component markers.
+# Generic build.sh consumers must reject it before importing Arrow Compute.
+arrow_paimon_build_fingerprint
>"${prebuilt}/arrow-paimon-build-fingerprint.txt"
+if arrow_paimon_prebuilt_valid "${prebuilt}" >/dev/null 2>&1; then
+ fail "legacy combined marker certified an unversioned component closure"
+fi
+
+printf '%s\n' "${ARROW_LEGACY_BUILD_FINGERPRINTS[0]}" \
+ >"${prebuilt}/arrow-build-fingerprint.txt"
+printf '%s\n' "${PAIMON_LEGACY_BUILD_FINGERPRINTS[0]}" \
+ >"${prebuilt}/paimon-build-fingerprint.txt"
+arrow_paimon_prebuilt_valid "${prebuilt}" ||
+ fail "the complete shared prebuilt was rejected during fingerprint
migration"
+if (
+ ARROW_BUILD_SCHEMA_VERSION="${ARROW_BUILD_SCHEMA_VERSION}-changed"
+ arrow_prebuilt_valid "${prebuilt}"
+) >/dev/null 2>&1; then
+ fail "the legacy Arrow marker survived a semantic fingerprint change"
+fi
+if (
+ PAIMON_BUILD_SCHEMA_VERSION="${PAIMON_BUILD_SCHEMA_VERSION}-changed"
+ paimon_prebuilt_valid "${prebuilt}"
+) >/dev/null 2>&1; then
+ fail "the legacy Paimon marker survived a semantic fingerprint change"
+fi
+
+prebuilt_archive_root="${tmpdir}/prebuilt-archive-root"
+prebuilt_archive="${tmpdir}/prebuilt.tar.xz"
+mkdir -p "${prebuilt_archive_root}/installed"
+cp -a "${prebuilt}/." "${prebuilt_archive_root}/installed/"
+tar -C "${prebuilt_archive_root}" -cJf "${prebuilt_archive}" installed
+
+external_thirdparty="${tmpdir}/external-thirdparty"
+mkdir -p "${external_thirdparty}/installed"
+touch "${external_thirdparty}/installed/old-image-artifact"
+ensure_arrow_paimon_prebuilt_from_url "${external_thirdparty}" \
+ "file://${prebuilt_archive}" >/dev/null 2>&1 ||
+ fail "BE UT could not refresh an outdated build-image prebuilt"
+arrow_paimon_prebuilt_valid "${external_thirdparty}/installed" ||
+ fail "BE UT installed an invalid build-image prebuilt"
+[[ ! -e "${external_thirdparty}/installed/old-image-artifact" ]] ||
+ fail "BE UT kept files from the outdated build-image prebuilt"
+ensure_arrow_paimon_prebuilt_from_url "${external_thirdparty}" \
+ "file://${tmpdir}/missing-prebuilt.tar.xz" >/dev/null 2>&1 ||
+ fail "BE UT tried to download over a valid build-image prebuilt"
+
+invalid_archive_root="${tmpdir}/invalid-prebuilt-archive-root"
+invalid_archive="${tmpdir}/invalid-prebuilt.tar.xz"
+mkdir -p "${invalid_archive_root}/installed"
+cp -a "${prebuilt}/." "${invalid_archive_root}/installed/"
+rm "${invalid_archive_root}/installed/lib64/libarrow_compute.a"
+tar -C "${invalid_archive_root}" -cJf "${invalid_archive}" installed
+
+invalid_external_thirdparty="${tmpdir}/invalid-external-thirdparty"
+mkdir -p "${invalid_external_thirdparty}/installed"
+touch
"${invalid_external_thirdparty}/installed/preserved-after-invalid-download"
+if ensure_arrow_paimon_prebuilt_from_url "${invalid_external_thirdparty}" \
+ "file://${invalid_archive}" >/dev/null 2>&1; then
+ fail "BE UT accepted a downloaded prebuilt without Arrow Compute"
+fi
+[[ -e
"${invalid_external_thirdparty}/installed/preserved-after-invalid-download" ]]
||
+ fail "BE UT replaced the build-image prebuilt before validating its
download"
+
+publish_arrow_prebuilt_marker "${prebuilt}"
+publish_paimon_prebuilt_marker "${prebuilt}"
+rm "${prebuilt}/lib64/libarrow_compute.a"
+if arrow_paimon_prebuilt_valid "${prebuilt}" >/dev/null 2>&1; then
+ fail "prebuilt validation accepted a missing Arrow Compute archive"
+fi
+touch "${prebuilt}/lib64/libarrow_compute.a"
+
+printf '%s\n' stale-arrow >"${prebuilt}/arrow-build-fingerprint.txt"
+if arrow_paimon_prebuilt_valid "${prebuilt}" >/dev/null 2>&1; then
+ fail "Paimon-only marker update certified a stale Arrow build"
+fi
+if require_arrow_prebuilt_for_paimon "${prebuilt}" >/dev/null 2>&1; then
+ fail "Paimon build accepted a stale installed Arrow"
+fi
+
+publish_arrow_prebuilt_marker "${prebuilt}"
+require_arrow_prebuilt_for_paimon "${prebuilt}" ||
+ fail "Paimon build rejected the selected installed Arrow"
+arrow_paimon_prebuilt_valid "${prebuilt}" || fail "matching component markers
were rejected"
+
+invalidate_paimon_prebuilt_marker "${prebuilt}"
+if arrow_paimon_prebuilt_valid "${prebuilt}" >/dev/null 2>&1; then
+ fail "an interrupted Paimon rebuild left its old marker valid"
+fi
+publish_paimon_prebuilt_marker "${prebuilt}"
+
+invalidate_arrow_prebuilt_marker "${prebuilt}"
+if require_arrow_prebuilt_for_paimon "${prebuilt}" >/dev/null 2>&1; then
+ fail "an interrupted Arrow rebuild left its old marker valid"
+fi
+publish_arrow_prebuilt_marker "${prebuilt}"
+arrow_paimon_prebuilt_valid "${prebuilt}" ||
+ fail "republished component markers were rejected"
+
+echo "PASS"
diff --git a/thirdparty/vars.sh b/thirdparty/vars.sh
index c941548c05c..47c6f04b61f 100644
--- a/thirdparty/vars.sh
+++ b/thirdparty/vars.sh
@@ -50,6 +50,10 @@ export TP_JAR_DIR="${TP_INSTALL_DIR}/lib/jar"
# export REPOSITORY_URL=
DORIS_THIRDPARTY_REPOSITORY_URL="${DORIS_THIRDPARTY_REPOSITORY_URL:-https://doris-thirdparty-repo.bj.bcebos.com/thirdparty}";
+DORIS_THIRDPARTY_VARS_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=arrow-paimon-vars.sh
+. "${DORIS_THIRDPARTY_VARS_DIR}/arrow-paimon-vars.sh"
+
#####################################################
# Download url, filename and unpaced filename
# of all thirdparties
@@ -235,12 +239,6 @@ ZSTD_NAME=zstd-1.5.7.tar.gz
ZSTD_SOURCE=zstd-1.5.7
ZSTD_MD5SUM="780fc1896922b1bc52a4e90980cdda48"
-# brotli
-BROTLI_DOWNLOAD="https://github.com/google/brotli/archive/v1.0.9.tar.gz";
-BROTLI_NAME="brotli-1.0.9.tar.gz"
-BROTLI_SOURCE="brotli-1.0.9"
-BROTLI_MD5SUM="c2274f0c7af8470ad514637c35bcee7d"
-
# flatbuffers
FLATBUFFERS_DOWNLOAD="https://github.com/google/flatbuffers/archive/v23.5.26.tar.gz";
FLATBUFFERS_NAME=flatbuffers-23.5.26.tar.gz
@@ -260,14 +258,6 @@ GRPC_NAME="grpc-v1.54.3.tar.gz"
GRPC_SOURCE=grpc-1.54.3
GRPC_MD5SUM="af00a2edeae0f02bb25917cc3473b7de"
-# arrow
-# Arrow 19.0.1 will MacOS compile error and decimal type error when convert to
Parquet.
-# https://github.com/apache/doris/pull/51217
-ARROW_DOWNLOAD="https://github.com/apache/arrow/archive/refs/tags/apache-arrow-17.0.0.tar.gz";
-ARROW_NAME="apache-arrow-17.0.0.tar.gz"
-ARROW_SOURCE="arrow-apache-arrow-17.0.0"
-ARROW_MD5SUM="ba18bf83e2164abd34b9ac4cb164f0f0"
-
# Abseil
ABSEIL_DOWNLOAD="https://github.com/abseil/abseil-cpp/releases/download/20250512.1/abseil-cpp-20250512.1.tar.gz";
ABSEIL_NAME="abseil-cpp-20250512.1.tar.gz"
@@ -428,13 +418,6 @@ BENCHMARK_NAME=benchmark-v1.8.0.tar.gz
BENCHMARK_SOURCE=benchmark-1.8.0
BENCHMARK_MD5SUM="8ddf8571d3f6198d37852bcbd964f817"
-# xsimd
-# for arrow-17.0.0, if arrow upgrade, this version may also need to be changed
-XSIMD_DOWNLOAD="https://github.com/xtensor-stack/xsimd/archive/refs/tags/13.0.0.tar.gz";
-XSIMD_NAME="13.0.0.tar.gz"
-XSIMD_SOURCE=xsimd-13.0.0
-XSIMD_MD5SUM="c661deb91836e82d3070f81032014fe6"
-
# simdjson
SIMDJSON_DOWNLOAD="https://github.com/simdjson/simdjson/archive/refs/tags/v3.11.6.tar.gz";
SIMDJSON_NAME=simdjson-3.11.6.tar.gz
@@ -568,12 +551,6 @@ PUGIXML_NAME=pugixml-1.15.tar.gz
PUGIXML_SOURCE=pugixml-1.15
PUGIXML_MD5SUM="3b894c29455eb33a40b165c6e2de5895"
-# paimon-cpp
-PAIMON_CPP_DOWNLOAD="https://github.com/apache/doris-thirdparty/archive/refs/tags/paimon-cpp-0a4f4e2.tar.gz";
-PAIMON_CPP_NAME="paimon-cpp-0a4f4e2.tar.gz"
-PAIMON_CPP_SOURCE="doris-thirdparty-paimon-cpp-0a4f4e2"
-PAIMON_CPP_MD5SUM="b8599a0421dbf1ec05e2f1a481d64e87"
-
# lance-c
LANCE_C_DOWNLOAD="https://github.com/lance-format/lance-c/archive/refs/tags/v0.1.2.tar.gz";
LANCE_C_NAME="lance-c-v0.1.2.tar.gz"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
