This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 43a31acc4b4 [fix](be) Validate Arrow input buffers before column
conversion (#64796)
43a31acc4b4 is described below
commit 43a31acc4b45647b84d5298cfd0e2ec1aca1540a
Author: Mryange <[email protected]>
AuthorDate: Tue Jul 14 17:40:48 2026 +0800
[fix](be) Validate Arrow input buffers before column conversion (#64796)
### What problem does this PR solve?
Doris converts Arrow arrays into Doris columns through
`DataTypeSerDe::read_column_from_arrow`. If the Arrow producer sends
malformed array metadata, such as truncated validity bitmaps, truncated
offsets buffers, non-monotonic string/list offsets, or offsets pointing
past the child/value buffer, the existing conversion code may read
invalid Arrow memory and crash BE.
Root cause: the Arrow-to-Doris serde path trusted Arrow array metadata
before accessing Arrow buffers. Several hot paths call `IsNull()`,
`Value()`, raw value offsets, list offsets, or child arrays directly, so
malformed Arrow buffers can trigger out-of-bounds reads before Doris
reports a clean error.
This PR adds lightweight, type-specific Arrow input validation before
those buffer accesses. The checks are modeled as local preflight checks
rather than full `ValidateFull()`: validity bitmap size, fixed-width
data buffer size, boolean bitmap size, binary/string offsets buffer
size, per-value data range, and list/map offsets monotonicity plus child
length bounds. A BE config `enable_arrow_input_validation` is added and
defaults to `true`.
The change also fixes an existing `FixedSizeBinaryArray` sliced-read
null check: the loop uses a relative index after `GetValue(start)`, but
`IsNull()` expects the original Arrow row index, so it must check `start
+ offset_i`.
| Type | Rows | Check disabled | Check enabled | Overhead |
| --- | ---: | ---: | ---: | ---: |
| String | 4096 | 39,352 ns | 42,032 ns | +6.8% |
| String | 65536 | 604,782 ns | 624,064 ns | +3.2% |
| Int64 | 4096 | 1,480 ns | 1,523 ns | +2.9% |
| Int64 | 65536 | 16,160 ns | 15,883 ns | -1.7% |
| Boolean | 4096 | 4,784 ns | 4,888 ns | +2.2% |
| Boolean | 65536 | 79,017 ns | 80,453 ns | +1.8% |
| ArrayString | 4096 | 139,747 ns | 147,793 ns | +5.8% |
| ArrayString | 65536 | 2,384,377 ns | 2,477,601 ns | +3.9% |
| MapStringInt | 4096 | 84,375 ns | 96,022 ns | +13.8% |
| MapStringInt | 65536 | 2,742,538 ns | 2,903,358 ns | +5.9% |
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
---
be/benchmark/benchmark_arrow_validation.hpp | 382 +++++++++++++++++++++
be/benchmark/benchmark_main.cpp | 1 +
be/src/common/config.cpp | 3 +
be/src/common/config.h | 3 +
be/src/core/data_type_serde/arrow_validation.h | 320 +++++++++++++++++
.../core/data_type_serde/data_type_array_serde.cpp | 5 +
.../data_type_date_or_datetime_serde.cpp | 5 +
.../data_type_serde/data_type_datetimev2_serde.cpp | 5 +
.../data_type_serde/data_type_datev2_serde.cpp | 5 +
.../data_type_serde/data_type_decimal_serde.cpp | 5 +
.../core/data_type_serde/data_type_ipv4_serde.cpp | 5 +
.../core/data_type_serde/data_type_ipv6_serde.cpp | 5 +
.../core/data_type_serde/data_type_jsonb_serde.cpp | 5 +
.../core/data_type_serde/data_type_map_serde.cpp | 5 +
.../data_type_serde/data_type_nullable_serde.cpp | 6 +
.../data_type_serde/data_type_number_serde.cpp | 26 +-
.../data_type_serde/data_type_string_serde.cpp | 39 ++-
.../data_type_serde/data_type_struct_serde.cpp | 5 +
.../data_type_serde_arrow_validation_test.cpp | 275 +++++++++++++++
19 files changed, 1095 insertions(+), 10 deletions(-)
diff --git a/be/benchmark/benchmark_arrow_validation.hpp
b/be/benchmark/benchmark_arrow_validation.hpp
new file mode 100644
index 00000000000..f5a2da45fb4
--- /dev/null
+++ b/be/benchmark/benchmark_arrow_validation.hpp
@@ -0,0 +1,382 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <arrow/array/array_binary.h>
+#include <arrow/array/array_nested.h>
+#include <arrow/array/array_primitive.h>
+#include <arrow/array/builder_binary.h>
+#include <arrow/array/builder_nested.h>
+#include <arrow/array/builder_primitive.h>
+#include <arrow/buffer.h>
+#include <arrow/type.h>
+#include <benchmark/benchmark.h>
+#include <cctz/time_zone.h>
+
+#include <chrono>
+#include <cstdint>
+#include <memory>
+#include <stdexcept>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "common/config.h"
+#include "core/assert_cast.h"
+#include "core/column/column_array.h"
+#include "core/column/column_map.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_string.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_map.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_string.h"
+#include "core/data_type/primitive_type.h"
+#include "util/unaligned.h"
+
+namespace doris {
+namespace {
+
+enum class ArrowValidationBenchMode { OLD, CHECK_DISABLED, CHECK_ENABLED };
+
+void throw_if_not_ok(const arrow::Status& status) {
+ if (!status.ok()) {
+ throw std::runtime_error(status.ToString());
+ }
+}
+
+void throw_if_not_ok(const Status& status) {
+ if (!status.ok()) {
+ throw std::runtime_error(status.to_string());
+ }
+}
+
+void keep_column_size(size_t size) {
+ benchmark::DoNotOptimize(size);
+}
+
+template <typename Func>
+double measure_seconds(Func&& func) {
+ const auto start = std::chrono::steady_clock::now();
+ std::forward<Func>(func)();
+ const auto finish = std::chrono::steady_clock::now();
+ return std::chrono::duration<double>(finish - start).count();
+}
+
+std::shared_ptr<arrow::StringArray> make_string_array(int64_t rows) {
+ arrow::StringBuilder builder;
+ throw_if_not_ok(builder.Reserve(rows));
+ for (int64_t i = 0; i < rows; ++i) {
+ const std::string value = "arrow_validation_" + std::to_string(i %
1000);
+ throw_if_not_ok(builder.Append(value));
+ }
+ std::shared_ptr<arrow::Array> array;
+ throw_if_not_ok(builder.Finish(&array));
+ return std::static_pointer_cast<arrow::StringArray>(array);
+}
+
+std::shared_ptr<arrow::Int64Array> make_int64_array(int64_t rows) {
+ arrow::Int64Builder builder;
+ throw_if_not_ok(builder.Reserve(rows));
+ for (int64_t i = 0; i < rows; ++i) {
+ throw_if_not_ok(builder.Append(i));
+ }
+ std::shared_ptr<arrow::Array> array;
+ throw_if_not_ok(builder.Finish(&array));
+ return std::static_pointer_cast<arrow::Int64Array>(array);
+}
+
+std::shared_ptr<arrow::BooleanArray> make_boolean_array(int64_t rows) {
+ arrow::BooleanBuilder builder;
+ throw_if_not_ok(builder.Reserve(rows));
+ for (int64_t i = 0; i < rows; ++i) {
+ throw_if_not_ok(builder.Append((i & 1) == 0));
+ }
+ std::shared_ptr<arrow::Array> array;
+ throw_if_not_ok(builder.Finish(&array));
+ return std::static_pointer_cast<arrow::BooleanArray>(array);
+}
+
+std::shared_ptr<arrow::ListArray> make_string_list_array(int64_t rows) {
+ auto value_builder = std::make_shared<arrow::StringBuilder>();
+ arrow::ListBuilder builder(arrow::default_memory_pool(), value_builder);
+ throw_if_not_ok(builder.Reserve(rows));
+ throw_if_not_ok(value_builder->Reserve(rows * 3));
+ for (int64_t i = 0; i < rows; ++i) {
+ throw_if_not_ok(builder.Append());
+ for (int64_t j = 0; j < 3; ++j) {
+ const std::string value = "item_" + std::to_string((i + j) % 1000);
+ throw_if_not_ok(value_builder->Append(value));
+ }
+ }
+ std::shared_ptr<arrow::Array> array;
+ throw_if_not_ok(builder.Finish(&array));
+ return std::static_pointer_cast<arrow::ListArray>(array);
+}
+
+std::shared_ptr<arrow::MapArray> make_string_int_map_array(int64_t rows) {
+ auto keys = make_string_array(rows * 2);
+ auto items = make_int64_array(rows * 2);
+ std::vector<int32_t> offsets(rows + 1);
+ for (int64_t i = 0; i <= rows; ++i) {
+ offsets[i] = static_cast<int32_t>(i * 2);
+ }
+ auto offsets_buffer = arrow::Buffer::FromVector(std::move(offsets));
+ auto map_type = arrow::map(arrow::utf8(), arrow::int64());
+ return std::make_shared<arrow::MapArray>(map_type, rows, offsets_buffer,
keys, items);
+}
+
+void read_string_old(ColumnString& column, const arrow::BinaryArray& array,
int64_t start,
+ int64_t end) {
+ std::shared_ptr<arrow::Buffer> buffer = array.value_data();
+ const uint8_t* offsets_data = array.value_offsets()->data();
+ const size_t offset_size = sizeof(int32_t);
+
+ for (auto offset_i = start; offset_i < end; ++offset_i) {
+ if (!array.IsNull(offset_i)) {
+ auto start_offset = unaligned_load<int32_t>(offsets_data +
offset_i * offset_size);
+ auto end_offset = unaligned_load<int32_t>(offsets_data + (offset_i
+ 1) * offset_size);
+ int32_t length = end_offset - start_offset;
+ const auto* raw_data = buffer->data() + start_offset;
+ column.insert_data(reinterpret_cast<const char*>(raw_data),
length);
+ } else {
+ column.insert_default();
+ }
+ }
+}
+
+void read_int64_old(ColumnInt64& column, const arrow::Array& array, int64_t
start, int64_t end) {
+ auto& col_data = column.get_data();
+ std::shared_ptr<arrow::Buffer> buffer = array.data()->buffers[1];
+ const auto* raw_data = reinterpret_cast<const int64_t*>(buffer->data()) +
start;
+ col_data.insert(raw_data, raw_data + end - start);
+}
+
+void read_boolean_old(ColumnUInt8& column, const arrow::BooleanArray& array,
int64_t start,
+ int64_t end) {
+ auto& col_data = column.get_data();
+ for (int64_t bool_i = start; bool_i != end; ++bool_i) {
+ col_data.emplace_back(array.Value(bool_i));
+ }
+}
+
+void read_array_string_old(ColumnArray& column, const arrow::ListArray& array,
int64_t start,
+ int64_t end) {
+ auto& offsets_data = column.get_offsets();
+ auto arrow_offsets_array = array.offsets();
+ auto* arrow_offsets =
dynamic_cast<arrow::Int32Array*>(arrow_offsets_array.get());
+ auto prev_size = offsets_data.back();
+ const auto* base_offsets_ptr = reinterpret_cast<const
uint8_t*>(arrow_offsets->raw_values());
+ const size_t offset_element_size = sizeof(int32_t);
+ const uint8_t* start_offset_ptr = base_offsets_ptr + start *
offset_element_size;
+ const uint8_t* end_offset_ptr = base_offsets_ptr + end *
offset_element_size;
+ auto arrow_nested_start_offset = unaligned_load<int32_t>(start_offset_ptr);
+ auto arrow_nested_end_offset = unaligned_load<int32_t>(end_offset_ptr);
+
+ for (auto i = start + 1; i < end + 1; ++i) {
+ const uint8_t* current_offset_ptr = base_offsets_ptr + i *
offset_element_size;
+ auto current_offset = unaligned_load<int32_t>(current_offset_ptr);
+ offsets_data.emplace_back(prev_size + current_offset -
arrow_nested_start_offset);
+ }
+ auto& nested_nullable = assert_cast<ColumnNullable&>(column.get_data());
+
read_string_old(assert_cast<ColumnString&>(nested_nullable.get_nested_column()),
+ static_cast<const arrow::BinaryArray&>(*array.values()),
+ arrow_nested_start_offset, arrow_nested_end_offset);
+ auto& null_map = nested_nullable.get_null_map_data();
+ null_map.resize_fill(null_map.size() + arrow_nested_end_offset -
arrow_nested_start_offset, 0);
+}
+
+void read_map_string_int_old(ColumnMap& column, const arrow::MapArray& array,
int64_t start,
+ int64_t end) {
+ auto& offsets_data = column.get_offsets();
+ auto arrow_offsets_array = array.offsets();
+ auto* arrow_offsets =
dynamic_cast<arrow::Int32Array*>(arrow_offsets_array.get());
+ auto prev_size = offsets_data.back();
+ const auto* base_offsets_ptr = reinterpret_cast<const
uint8_t*>(arrow_offsets->raw_values());
+ const size_t offset_element_size = sizeof(int32_t);
+ const uint8_t* start_offset_ptr = base_offsets_ptr + start *
offset_element_size;
+ const uint8_t* end_offset_ptr = base_offsets_ptr + end *
offset_element_size;
+ auto arrow_nested_start_offset = unaligned_load<int32_t>(start_offset_ptr);
+ auto arrow_nested_end_offset = unaligned_load<int32_t>(end_offset_ptr);
+ for (int64_t i = start + 1; i < end + 1; ++i) {
+ const uint8_t* current_offset_ptr = base_offsets_ptr + i *
offset_element_size;
+ auto current_offset = unaligned_load<int32_t>(current_offset_ptr);
+ offsets_data.emplace_back(prev_size + current_offset -
arrow_nested_start_offset);
+ }
+ read_string_old(assert_cast<ColumnString&>(column.get_keys()),
+ static_cast<const arrow::BinaryArray&>(*array.keys()),
+ arrow_nested_start_offset, arrow_nested_end_offset);
+ read_int64_old(assert_cast<ColumnInt64&>(column.get_values()),
*array.items(),
+ arrow_nested_start_offset, arrow_nested_end_offset);
+}
+
+template <ArrowValidationBenchMode mode, typename DataType, typename
ArrowArray>
+void run_serde_benchmark(benchmark::State& state, const
std::shared_ptr<DataType>& type,
+ const std::shared_ptr<ArrowArray>& array) {
+ const bool old_config = config::enable_arrow_input_validation;
+ if constexpr (mode == ArrowValidationBenchMode::CHECK_DISABLED) {
+ config::enable_arrow_input_validation = false;
+ } else if constexpr (mode == ArrowValidationBenchMode::CHECK_ENABLED) {
+ config::enable_arrow_input_validation = true;
+ }
+
+ for (auto _ : state) {
+ const double seconds = measure_seconds([&] {
+ auto column = type->create_column();
+ throw_if_not_ok(type->get_serde()->read_column_from_arrow(
+ *column, array.get(), 0, array->length(),
cctz::utc_time_zone()));
+ keep_column_size(column->size());
+ });
+ state.SetIterationTime(seconds);
+ }
+ config::enable_arrow_input_validation = old_config;
+}
+
+template <ArrowValidationBenchMode mode>
+void BM_ArrowValidation_String(benchmark::State& state) {
+ auto array = make_string_array(state.range(0));
+ auto type = std::make_shared<DataTypeString>();
+
+ if constexpr (mode == ArrowValidationBenchMode::OLD) {
+ for (auto _ : state) {
+ const double seconds = measure_seconds([&] {
+ auto column = ColumnString::create();
+ read_string_old(*column, *array, 0, array->length());
+ keep_column_size(column->size());
+ });
+ state.SetIterationTime(seconds);
+ }
+ } else {
+ run_serde_benchmark<mode>(state, type, array);
+ }
+}
+
+template <ArrowValidationBenchMode mode>
+void BM_ArrowValidation_Int64(benchmark::State& state) {
+ auto array = make_int64_array(state.range(0));
+ auto type = std::make_shared<DataTypeInt64>();
+
+ if constexpr (mode == ArrowValidationBenchMode::OLD) {
+ for (auto _ : state) {
+ const double seconds = measure_seconds([&] {
+ auto column = ColumnInt64::create();
+ read_int64_old(*column, *array, 0, array->length());
+ keep_column_size(column->size());
+ });
+ state.SetIterationTime(seconds);
+ }
+ } else {
+ run_serde_benchmark<mode>(state, type, array);
+ }
+}
+
+template <ArrowValidationBenchMode mode>
+void BM_ArrowValidation_Boolean(benchmark::State& state) {
+ auto array = make_boolean_array(state.range(0));
+ auto type = std::make_shared<DataTypeBool>();
+
+ if constexpr (mode == ArrowValidationBenchMode::OLD) {
+ for (auto _ : state) {
+ const double seconds = measure_seconds([&] {
+ auto column = ColumnUInt8::create();
+ read_boolean_old(*column, *array, 0, array->length());
+ keep_column_size(column->size());
+ });
+ state.SetIterationTime(seconds);
+ }
+ } else {
+ run_serde_benchmark<mode>(state, type, array);
+ }
+}
+
+template <ArrowValidationBenchMode mode>
+void BM_ArrowValidation_ArrayString(benchmark::State& state) {
+ auto array = make_string_list_array(state.range(0));
+ auto type =
std::make_shared<DataTypeArray>(std::make_shared<DataTypeString>());
+
+ if constexpr (mode == ArrowValidationBenchMode::OLD) {
+ for (auto _ : state) {
+ const double seconds = measure_seconds([&] {
+ auto column = type->create_column();
+ read_array_string_old(assert_cast<ColumnArray&>(*column),
*array, 0,
+ array->length());
+ keep_column_size(column->size());
+ });
+ state.SetIterationTime(seconds);
+ }
+ } else {
+ run_serde_benchmark<mode>(state, type, array);
+ }
+}
+
+template <ArrowValidationBenchMode mode>
+void BM_ArrowValidation_MapStringInt(benchmark::State& state) {
+ auto array = make_string_int_map_array(state.range(0));
+ auto type =
std::make_shared<DataTypeMap>(std::make_shared<DataTypeString>(),
+
std::make_shared<DataTypeInt64>());
+
+ if constexpr (mode == ArrowValidationBenchMode::OLD) {
+ for (auto _ : state) {
+ const double seconds = measure_seconds([&] {
+ auto column = type->create_column();
+ read_map_string_int_old(assert_cast<ColumnMap&>(*column),
*array, 0,
+ array->length());
+ keep_column_size(column->size());
+ });
+ state.SetIterationTime(seconds);
+ }
+ } else {
+ run_serde_benchmark<mode>(state, type, array);
+ }
+}
+
+#define REGISTER_ARROW_VALIDATION_BENCH(FUNC, MODE, NAME) \
+ BENCHMARK_TEMPLATE(FUNC, ArrowValidationBenchMode::MODE) \
+ ->Name("BM_ArrowValidation_" #NAME "_" #MODE "/rows:4096") \
+ ->Arg(4096) \
+ ->UseManualTime(); \
+ BENCHMARK_TEMPLATE(FUNC, ArrowValidationBenchMode::MODE) \
+ ->Name("BM_ArrowValidation_" #NAME "_" #MODE "/rows:65536") \
+ ->Arg(65536) \
+ ->UseManualTime()
+
+REGISTER_ARROW_VALIDATION_BENCH(BM_ArrowValidation_String, OLD, String);
+REGISTER_ARROW_VALIDATION_BENCH(BM_ArrowValidation_String, CHECK_DISABLED,
String);
+REGISTER_ARROW_VALIDATION_BENCH(BM_ArrowValidation_String, CHECK_ENABLED,
String);
+
+REGISTER_ARROW_VALIDATION_BENCH(BM_ArrowValidation_Int64, OLD, Int64);
+REGISTER_ARROW_VALIDATION_BENCH(BM_ArrowValidation_Int64, CHECK_DISABLED,
Int64);
+REGISTER_ARROW_VALIDATION_BENCH(BM_ArrowValidation_Int64, CHECK_ENABLED,
Int64);
+
+REGISTER_ARROW_VALIDATION_BENCH(BM_ArrowValidation_Boolean, OLD, Boolean);
+REGISTER_ARROW_VALIDATION_BENCH(BM_ArrowValidation_Boolean, CHECK_DISABLED,
Boolean);
+REGISTER_ARROW_VALIDATION_BENCH(BM_ArrowValidation_Boolean, CHECK_ENABLED,
Boolean);
+
+REGISTER_ARROW_VALIDATION_BENCH(BM_ArrowValidation_ArrayString, OLD,
ArrayString);
+REGISTER_ARROW_VALIDATION_BENCH(BM_ArrowValidation_ArrayString,
CHECK_DISABLED, ArrayString);
+REGISTER_ARROW_VALIDATION_BENCH(BM_ArrowValidation_ArrayString, CHECK_ENABLED,
ArrayString);
+
+REGISTER_ARROW_VALIDATION_BENCH(BM_ArrowValidation_MapStringInt, OLD,
MapStringInt);
+REGISTER_ARROW_VALIDATION_BENCH(BM_ArrowValidation_MapStringInt,
CHECK_DISABLED, MapStringInt);
+REGISTER_ARROW_VALIDATION_BENCH(BM_ArrowValidation_MapStringInt,
CHECK_ENABLED, MapStringInt);
+
+#undef REGISTER_ARROW_VALIDATION_BENCH
+
+} // namespace
+} // namespace doris
diff --git a/be/benchmark/benchmark_main.cpp b/be/benchmark/benchmark_main.cpp
index 53ac0c1df00..efb2930f836 100644
--- a/be/benchmark/benchmark_main.cpp
+++ b/be/benchmark/benchmark_main.cpp
@@ -17,6 +17,7 @@
#include <benchmark/benchmark.h>
+#include "benchmark_arrow_validation.hpp"
#include "benchmark_bit_pack.hpp"
#include "benchmark_bits.hpp"
#include "benchmark_block_bloom_filter.hpp"
diff --git a/be/src/common/config.cpp b/be/src/common/config.cpp
index b193797a863..e8ca2a9de7e 100644
--- a/be/src/common/config.cpp
+++ b/be/src/common/config.cpp
@@ -72,6 +72,9 @@ DEFINE_Int32(brpc_port, "8060");
DEFINE_Int32(arrow_flight_sql_port, "8050");
+// Validate Arrow input buffers in opted-in Arrow readers before converting
them to Doris columns.
+DEFINE_Bool(enable_arrow_input_validation, "true");
+
DEFINE_Int32(cdc_client_port, "9096");
DEFINE_String(cdc_client_java_opts, "");
diff --git a/be/src/common/config.h b/be/src/common/config.h
index b79e10e8f69..cd50e4fb08f 100644
--- a/be/src/common/config.h
+++ b/be/src/common/config.h
@@ -119,6 +119,9 @@ DECLARE_Int32(brpc_port);
// Default -1, do not start arrow flight sql server.
DECLARE_Int32(arrow_flight_sql_port);
+// Validate Arrow input buffers in opted-in Arrow readers before converting
them to Doris columns.
+DECLARE_Bool(enable_arrow_input_validation);
+
// port for cdc client scan oltp cdc data
DECLARE_Int32(cdc_client_port);
diff --git a/be/src/core/data_type_serde/arrow_validation.h
b/be/src/core/data_type_serde/arrow_validation.h
new file mode 100644
index 00000000000..9d8b09b0cf1
--- /dev/null
+++ b/be/src/core/data_type_serde/arrow_validation.h
@@ -0,0 +1,320 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <arrow/array/array_base.h>
+#include <arrow/array/array_binary.h>
+#include <arrow/array/array_nested.h>
+#include <arrow/array/array_primitive.h>
+
+#include <cstddef>
+#include <cstdint>
+#include <limits>
+#include <memory>
+#include <string>
+#include <string_view>
+#include <utility>
+
+#include "common/compiler_util.h"
+#include "common/exception.h"
+#include "util/unaligned.h"
+
+namespace doris {
+namespace arrow_validation_detail {
+
+inline std::string arrow_type_name(const arrow::Array& array) {
+ return array.type() ? array.type()->name() : "unknown";
+}
+
+inline void throw_invalid_arrow(std::string_view arrow_type, std::string_view
message) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT, "Invalid Arrow {}: {}",
arrow_type, message);
+}
+
+inline void throw_invalid_arrow(const arrow::Array& array, std::string_view
message) {
+ throw_invalid_arrow(arrow_type_name(array), message);
+}
+
+template <typename... Args>
+inline void throw_invalid_arrow(std::string_view arrow_type, std::string_view
format,
+ Args&&... args) {
+ throw Exception(ErrorCode::INVALID_ARGUMENT, "Invalid Arrow {}: {}",
arrow_type,
+ fmt::format(std::string(format),
std::forward<Args>(args)...));
+}
+
+template <typename... Args>
+inline void throw_invalid_arrow(const arrow::Array& array, std::string_view
format,
+ Args&&... args) {
+ throw_invalid_arrow(arrow_type_name(array), format,
std::forward<Args>(args)...);
+}
+
+inline void check_arrow_length_and_offset(const arrow::Array& array) {
+ if (UNLIKELY(array.length() < 0 || array.offset() < 0)) {
+ throw_invalid_arrow(array, "negative length or offset: length={},
offset={}",
+ array.length(), array.offset());
+ }
+}
+
+inline void check_arrow_no_offset(const arrow::Array& array) {
+ check_arrow_length_and_offset(array);
+ if (UNLIKELY(array.offset() != 0)) {
+ throw_invalid_arrow(array, "non-zero array offset is not supported:
offset={}",
+ array.offset());
+ }
+}
+
+inline void check_add_overflow(size_t left, size_t right, const arrow::Array&
array,
+ std::string_view item) {
+ if (UNLIKELY(left > std::numeric_limits<size_t>::max() - right)) {
+ throw_invalid_arrow(array, "{} size overflow", item);
+ }
+}
+
+inline std::shared_ptr<arrow::Int32Array> get_int32_offsets_array(const
arrow::Array& array) {
+ auto offsets_array = static_cast<const arrow::ListArray&>(array).offsets();
+ auto offsets = std::dynamic_pointer_cast<arrow::Int32Array>(offsets_array);
+ if (UNLIKELY(!offsets)) {
+ throw_invalid_arrow(array, "offsets array is not Int32Array");
+ }
+ return offsets;
+}
+
+} // namespace arrow_validation_detail
+
+inline void check_arrow_no_offset(const arrow::Array& array) {
+ arrow_validation_detail::check_arrow_no_offset(array);
+}
+
+// Validate the caller's requested read range before any Arrow buffer access.
+// This rejects negative or out-of-array start/end values up front.
+inline void check_arrow_array_range(const arrow::Array& array, int64_t start,
int64_t end) {
+ arrow_validation_detail::check_arrow_no_offset(array);
+ if (UNLIKELY(start < 0 || end < start || end > array.length())) {
+ arrow_validation_detail::throw_invalid_arrow(
+ array, "read range is invalid: start={}, end={}, length={}",
start, end,
+ array.length());
+ }
+}
+
+// Validate buffers[0] when a validity bitmap exists. This must run before
+// IsNull()/IsValid()/null_count(), because those APIs may scan the bitmap.
+inline void check_arrow_validity_bitmap(const arrow::Array& array) {
+ arrow_validation_detail::check_arrow_length_and_offset(array);
+ const auto& buffers = array.data()->buffers;
+ if (buffers.empty() || !buffers[0]) {
+ if (UNLIKELY(array.data()->null_count > 0)) {
+ arrow_validation_detail::throw_invalid_arrow(
+ array, "validity bitmap is missing but null_count={}",
+ array.data()->null_count.load());
+ }
+ return;
+ }
+
+ const size_t offset = static_cast<size_t>(array.offset());
+ const size_t length = static_cast<size_t>(array.length());
+ arrow_validation_detail::check_add_overflow(offset, length, array,
"validity bitmap");
+ const size_t count = offset + length;
+ const size_t required = count / 8 + (count % 8 != 0 ? 1 : 0);
+ const size_t available = static_cast<size_t>(buffers[0]->size());
+ if (UNLIKELY(available < required)) {
+ arrow_validation_detail::throw_invalid_arrow(
+ array, "validity bitmap too small: {} bytes available, {}
required", available,
+ required);
+ }
+}
+
+// Validate buffers[1] for fixed-width arrays before raw_values(), Value(), or
+// direct buffer memcpy. elem_size is the physical byte width of one value.
+inline void check_arrow_fixed_width_buffer(const arrow::Array& array, size_t
elem_size) {
+ check_arrow_validity_bitmap(array);
+ const auto& buffers = array.data()->buffers;
+ if (UNLIKELY(buffers.size() <= 1 || !buffers[1])) {
+ if (array.length() == 0) {
+ return;
+ }
+ arrow_validation_detail::throw_invalid_arrow(array, "data buffer is
missing");
+ }
+
+ const size_t offset = static_cast<size_t>(array.offset());
+ const size_t length = static_cast<size_t>(array.length());
+ arrow_validation_detail::check_add_overflow(offset, length, array, "data
buffer");
+ const size_t count = offset + length;
+ if (UNLIKELY(elem_size != 0 && count > std::numeric_limits<size_t>::max()
/ elem_size)) {
+ arrow_validation_detail::throw_invalid_arrow(array, "data buffer size
overflow");
+ }
+ const size_t required = elem_size * count;
+ const size_t available = static_cast<size_t>(buffers[1]->size());
+ if (UNLIKELY(available < required)) {
+ arrow_validation_detail::throw_invalid_arrow(
+ array, "data buffer too small: {} bytes available, {}
required", available,
+ required);
+ }
+}
+
+// Validate the bit-packed boolean data bitmap before BooleanArray::Value().
+inline void check_arrow_boolean_buffer(const arrow::Array& array) {
+ check_arrow_validity_bitmap(array);
+ const auto& buffers = array.data()->buffers;
+ if (UNLIKELY(buffers.size() <= 1 || !buffers[1])) {
+ if (array.length() == 0) {
+ return;
+ }
+ arrow_validation_detail::throw_invalid_arrow(array, "data bitmap is
missing");
+ }
+
+ const size_t offset = static_cast<size_t>(array.offset());
+ const size_t length = static_cast<size_t>(array.length());
+ arrow_validation_detail::check_add_overflow(offset, length, array, "data
bitmap");
+ const size_t count = offset + length;
+ const size_t required = count / 8 + (count % 8 != 0 ? 1 : 0);
+ const size_t available = static_cast<size_t>(buffers[1]->size());
+ if (UNLIKELY(available < required)) {
+ arrow_validation_detail::throw_invalid_arrow(
+ array, "data bitmap too small: {} bytes available, {}
required", available,
+ required);
+ }
+}
+
+// Validate one variable-width value range after reading offsets and before
+// forming value_data() + offset.
+inline void check_arrow_value_range(const arrow::Array& array, int64_t offset,
int64_t length,
+ size_t buffer_size) {
+ if (UNLIKELY(offset < 0 || length < 0)) {
+ arrow_validation_detail::throw_invalid_arrow(
+ array, "value range has negative offset or length: offset={},
length={}", offset,
+ length);
+ }
+ const size_t safe_offset = static_cast<size_t>(offset);
+ const size_t safe_length = static_cast<size_t>(length);
+ if (UNLIKELY(safe_offset > buffer_size || safe_length > buffer_size -
safe_offset)) {
+ arrow_validation_detail::throw_invalid_arrow(
+ array, "value range exceeds data buffer: offset={}, length={},
buffer_size={}",
+ offset, length, buffer_size);
+ }
+}
+
+namespace arrow_validation_detail {
+
+// Offsets buffers may come from external Arrow producers through Buffer::Wrap
or FFI and are not
+// guaranteed to be aligned to int32_t. Do not use Int32Array::Value() here
because it performs a
+// typed raw_values()[i] load and can trigger UBSan on misaligned buffers.
Keep this validation path
+// consistent with the array/map readers below, which load offsets through
unaligned_load().
+inline int32_t read_int32_offset(const arrow::Int32Array& offsets, int64_t
index) {
+ const auto* data = reinterpret_cast<const uint8_t*>(offsets.raw_values());
+ return unaligned_load<int32_t>(data + index * sizeof(int32_t));
+}
+
+inline int64_t check_arrow_offsets_range(const arrow::Int32Array& offsets,
int64_t start,
+ int64_t end) {
+ check_arrow_array_range(offsets, 0, offsets.length());
+ check_arrow_fixed_width_buffer(offsets, sizeof(int32_t));
+ if (UNLIKELY(start < 0 || end < start || end >= offsets.length())) {
+ arrow_validation_detail::throw_invalid_arrow(
+ offsets, "offsets read range is invalid: start={}, end={},
offsets_length={}",
+ start, end, offsets.length());
+ }
+
+ int64_t previous_offset = read_int32_offset(offsets, start);
+ if (UNLIKELY(previous_offset < 0)) {
+ arrow_validation_detail::throw_invalid_arrow(
+ offsets, "offsets contain negative value: offset[{}]={}",
start, previous_offset);
+ }
+ for (int64_t i = start + 1; i <= end; ++i) {
+ const int64_t current_offset = read_int32_offset(offsets, i);
+ if (UNLIKELY(current_offset < previous_offset)) {
+ arrow_validation_detail::throw_invalid_arrow(
+ offsets,
+ "offsets are not monotonically non-decreasing:
offset[{}]={} < offset[{}]={}",
+ i, current_offset, i - 1, previous_offset);
+ }
+ previous_offset = current_offset;
+ }
+ return previous_offset;
+}
+
+} // namespace arrow_validation_detail
+
+// Validate List offsets before reading offsets or recursing into values.
+inline void check_arrow_list_offsets(const arrow::ListArray& array, int64_t
start, int64_t end) {
+ check_arrow_array_range(array, start, end);
+ const auto offsets =
arrow_validation_detail::get_int32_offsets_array(array);
+ const int64_t last_offset =
+ arrow_validation_detail::check_arrow_offsets_range(*offsets,
start, end);
+ const int64_t values_length = array.values() ? array.values()->length() :
0;
+ if (UNLIKELY(last_offset > values_length)) {
+ arrow_validation_detail::throw_invalid_arrow(
+ array, "offsets exceed values length: last_offset={},
values_length={}",
+ last_offset, values_length);
+ }
+}
+
+// Validate Map offsets before reading offsets or recursing into keys/items.
+inline void check_arrow_map_offsets(const arrow::MapArray& array, int64_t
start, int64_t end) {
+ check_arrow_array_range(array, start, end);
+ const auto offsets =
arrow_validation_detail::get_int32_offsets_array(array);
+ const int64_t last_offset =
+ arrow_validation_detail::check_arrow_offsets_range(*offsets,
start, end);
+ const int64_t keys_length = array.keys() ? array.keys()->length() : 0;
+ if (UNLIKELY(last_offset > keys_length)) {
+ arrow_validation_detail::throw_invalid_arrow(
+ array, "offsets exceed keys length: last_offset={},
keys_length={}", last_offset,
+ keys_length);
+ }
+ const int64_t items_length = array.items() ? array.items()->length() : 0;
+ if (UNLIKELY(last_offset > items_length)) {
+ arrow_validation_detail::throw_invalid_arrow(
+ array, "offsets exceed items length: last_offset={},
items_length={}", last_offset,
+ items_length);
+ }
+}
+
+// Validate String/Binary offsets buffer before value_offset(), value_length(),
+// raw_value_offsets(), or manual offset reads. Variable-width Arrow arrays
need
+// offset + length + 1 offset entries.
+template <typename ArrowBinaryArray>
+inline void check_arrow_binary_offsets_buffer(const ArrowBinaryArray& array) {
+ check_arrow_validity_bitmap(array);
+ const auto& buffers = array.data()->buffers;
+ if (UNLIKELY(buffers.size() <= 1 || !buffers[1])) {
+ arrow_validation_detail::throw_invalid_arrow(array, "offsets buffer is
missing");
+ }
+
+ const size_t offset = static_cast<size_t>(array.offset());
+ const size_t length = static_cast<size_t>(array.length());
+ if (UNLIKELY(offset > std::numeric_limits<size_t>::max() - length)) {
+ arrow_validation_detail::throw_invalid_arrow(array, "offsets entry
count overflow");
+ }
+ const size_t count = offset + length;
+ if (UNLIKELY(count == std::numeric_limits<size_t>::max())) {
+ arrow_validation_detail::throw_invalid_arrow(array, "offsets entry
count overflow");
+ }
+ const size_t count_plus_one = count + 1;
+ if (UNLIKELY(count_plus_one > std::numeric_limits<size_t>::max() /
+ sizeof(typename
ArrowBinaryArray::offset_type))) {
+ arrow_validation_detail::throw_invalid_arrow(array, "offsets buffer
size overflow");
+ }
+
+ const size_t required = count_plus_one * sizeof(typename
ArrowBinaryArray::offset_type);
+ const size_t available = static_cast<size_t>(buffers[1]->size());
+ if (UNLIKELY(available < required)) {
+ arrow_validation_detail::throw_invalid_arrow(
+ array, "offsets buffer too small: {} bytes available, {}
required", available,
+ required);
+ }
+}
+
+} // namespace doris
diff --git a/be/src/core/data_type_serde/data_type_array_serde.cpp
b/be/src/core/data_type_serde/data_type_array_serde.cpp
index c54f3e1b46a..dd3a8e8ca73 100644
--- a/be/src/core/data_type_serde/data_type_array_serde.cpp
+++ b/be/src/core/data_type_serde/data_type_array_serde.cpp
@@ -19,6 +19,7 @@
#include <arrow/array/builder_nested.h>
+#include "common/config.h"
#include "common/status.h"
#include "core/assert_cast.h"
#include "core/column/column.h"
@@ -27,6 +28,7 @@
#include "core/data_type/data_type.h"
#include "core/data_type/data_type_array.h"
#include "core/data_type/get_least_supertype.h"
+#include "core/data_type_serde/arrow_validation.h"
#include "core/data_type_serde/complex_type_deserialize_util.h"
#include "core/string_ref.h"
#include "exprs/function/function_helpers.h"
@@ -323,6 +325,9 @@ Status DataTypeArraySerDe::read_column_from_arrow(IColumn&
column, const arrow::
const auto* concrete_array = dynamic_cast<const
arrow::ListArray*>(arrow_array);
auto arrow_offsets_array = concrete_array->offsets();
auto* arrow_offsets =
dynamic_cast<arrow::Int32Array*>(arrow_offsets_array.get());
+ if (config::enable_arrow_input_validation) {
+ check_arrow_list_offsets(*concrete_array, start, end);
+ }
auto prev_size = offsets_data.back();
const auto* base_offsets_ptr = reinterpret_cast<const
uint8_t*>(arrow_offsets->raw_values());
const size_t offset_element_size = sizeof(int32_t);
diff --git a/be/src/core/data_type_serde/data_type_date_or_datetime_serde.cpp
b/be/src/core/data_type_serde/data_type_date_or_datetime_serde.cpp
index e948529fac3..1ff362c6d5e 100644
--- a/be/src/core/data_type_serde/data_type_date_or_datetime_serde.cpp
+++ b/be/src/core/data_type_serde/data_type_date_or_datetime_serde.cpp
@@ -20,10 +20,12 @@
#include <arrow/builder.h>
#include <cctz/time_zone.h>
+#include "common/config.h"
#include "common/status.h"
#include "core/column/column_const.h"
#include "core/data_type/data_type_decimal.h"
#include "core/data_type/data_type_number.h"
+#include "core/data_type_serde/arrow_validation.h"
#include "core/value/vdatetime_value.h"
#include "exprs/function/cast/cast_base.h"
#include "exprs/function/cast/cast_to_date_or_datetime_impl.hpp"
@@ -191,6 +193,9 @@ Status
DataTypeDateSerDe<T>::_read_column_from_arrow(IColumn& column,
const arrow::Array*
arrow_array, int64_t start,
int64_t end,
const cctz::time_zone&
ctz) const {
+ if (config::enable_arrow_input_validation) {
+ check_arrow_no_offset(*arrow_array);
+ }
auto& col_data = static_cast<ColumnVector<T>&>(column).get_data();
int64_t divisor = 1;
int64_t multiplier = 1;
diff --git a/be/src/core/data_type_serde/data_type_datetimev2_serde.cpp
b/be/src/core/data_type_serde/data_type_datetimev2_serde.cpp
index 0eb5e4d44a3..44402bb5b53 100644
--- a/be/src/core/data_type_serde/data_type_datetimev2_serde.cpp
+++ b/be/src/core/data_type_serde/data_type_datetimev2_serde.cpp
@@ -23,11 +23,13 @@
#include <chrono> // IWYU pragma: keep
#include <cstdint>
+#include "common/config.h"
#include "common/status.h"
#include "core/column/column_const.h"
#include "core/data_type/data_type_decimal.h"
#include "core/data_type/data_type_number.h"
#include "core/data_type/primitive_type.h"
+#include "core/data_type_serde/arrow_validation.h"
#include "core/data_type_serde/decoded_column_view.h"
#include "core/types.h"
#include "core/value/vdatetime_value.h"
@@ -487,6 +489,9 @@ Status
DataTypeDateTimeV2SerDe::read_column_from_arrow(IColumn& column,
const arrow::Array*
arrow_array,
int64_t start, int64_t
end,
const cctz::time_zone&
ctz) const {
+ if (config::enable_arrow_input_validation) {
+ check_arrow_no_offset(*arrow_array);
+ }
auto& col_data = static_cast<ColumnDateTimeV2&>(column).get_data();
int64_t divisor = 1;
if (arrow_array->type()->id() == arrow::Type::TIMESTAMP) {
diff --git a/be/src/core/data_type_serde/data_type_datev2_serde.cpp
b/be/src/core/data_type_serde/data_type_datev2_serde.cpp
index 33e484ef946..c46ce4d1086 100644
--- a/be/src/core/data_type_serde/data_type_datev2_serde.cpp
+++ b/be/src/core/data_type_serde/data_type_datev2_serde.cpp
@@ -23,10 +23,12 @@
#include <cstdint>
+#include "common/config.h"
#include "core/column/column_const.h"
#include "core/data_type/data_type_decimal.h"
#include "core/data_type/data_type_number.h"
#include "core/data_type/define_primitive_type.h"
+#include "core/data_type_serde/arrow_validation.h"
#include "core/data_type_serde/decoded_column_view.h"
#include "core/types.h"
#include "core/value/vdatetime_value.h"
@@ -110,6 +112,9 @@ Status DataTypeDateV2SerDe::write_column_to_arrow(const
IColumn& column, const N
Status DataTypeDateV2SerDe::read_column_from_arrow(IColumn& column, const
arrow::Array* arrow_array,
int64_t start, int64_t end,
const cctz::time_zone& ctz)
const {
+ if (config::enable_arrow_input_validation) {
+ check_arrow_no_offset(*arrow_array);
+ }
auto& col_data = static_cast<ColumnDateV2&>(column).get_data();
const auto* concrete_array = dynamic_cast<const
arrow::Date32Array*>(arrow_array);
const auto* base_ptr = reinterpret_cast<const
uint8_t*>(concrete_array->raw_values());
diff --git a/be/src/core/data_type_serde/data_type_decimal_serde.cpp
b/be/src/core/data_type_serde/data_type_decimal_serde.cpp
index 726263b705a..35208450c39 100644
--- a/be/src/core/data_type_serde/data_type_decimal_serde.cpp
+++ b/be/src/core/data_type_serde/data_type_decimal_serde.cpp
@@ -27,11 +27,13 @@
#include "arrow/type.h"
#include "common/cast_set.h"
+#include "common/config.h"
#include "common/consts.h"
#include "core/column/column.h"
#include "core/column/column_decimal.h"
#include "core/data_type/data_type_decimal.h"
#include "core/data_type/define_primitive_type.h"
+#include "core/data_type_serde/arrow_validation.h"
#include "core/data_type_serde/decoded_column_view.h"
#include "core/types.h"
#include "exec/common/arithmetic_overflow.h"
@@ -456,6 +458,9 @@ Status
DataTypeDecimalSerDe<T>::read_column_from_arrow(IColumn& column,
const arrow::Array*
arrow_array,
int64_t start, int64_t
end,
const cctz::time_zone&
ctz) const {
+ if (config::enable_arrow_input_validation) {
+ check_arrow_no_offset(*arrow_array);
+ }
auto& column_data = static_cast<ColumnDecimal<T>&>(column).get_data();
// Decimal<Int128> for decimalv2
// Decimal<Int128I> for deicmalv3
diff --git a/be/src/core/data_type_serde/data_type_ipv4_serde.cpp
b/be/src/core/data_type_serde/data_type_ipv4_serde.cpp
index df6174ec319..ef6da635e13 100644
--- a/be/src/core/data_type_serde/data_type_ipv4_serde.cpp
+++ b/be/src/core/data_type_serde/data_type_ipv4_serde.cpp
@@ -19,7 +19,9 @@
#include <arrow/builder.h>
+#include "common/config.h"
#include "core/column/column_const.h"
+#include "core/data_type_serde/arrow_validation.h"
#include "core/types.h"
#include "exprs/function/cast/cast_to_ip.h"
#include "exprs/function/cast/cast_to_string.h"
@@ -113,6 +115,9 @@ Status DataTypeIPv4SerDe::write_column_to_arrow(const
IColumn& column, const Nul
Status DataTypeIPv4SerDe::read_column_from_arrow(IColumn& column, const
arrow::Array* arrow_array,
int64_t start, int64_t end,
const cctz::time_zone& ctz)
const {
+ if (config::enable_arrow_input_validation) {
+ check_arrow_no_offset(*arrow_array);
+ }
auto& col_data = assert_cast<ColumnIPv4&>(column).get_data();
int64_t row_count = end - start;
/// buffers[0] is a null bitmap and buffers[1] are actual values
diff --git a/be/src/core/data_type_serde/data_type_ipv6_serde.cpp
b/be/src/core/data_type_serde/data_type_ipv6_serde.cpp
index 9c9ba1a2972..3239a4da498 100644
--- a/be/src/core/data_type_serde/data_type_ipv6_serde.cpp
+++ b/be/src/core/data_type_serde/data_type_ipv6_serde.cpp
@@ -22,7 +22,9 @@
#include <cstddef>
#include <string>
+#include "common/config.h"
#include "core/column/column_const.h"
+#include "core/data_type_serde/arrow_validation.h"
#include "core/types.h"
#include "exprs/function/cast/cast_to_ip.h"
#include "exprs/function/cast/cast_to_string.h"
@@ -142,6 +144,9 @@ Status DataTypeIPv6SerDe::write_column_to_arrow(const
IColumn& column, const Nul
Status DataTypeIPv6SerDe::read_column_from_arrow(IColumn& column, const
arrow::Array* arrow_array,
int64_t start, int64_t end,
const cctz::time_zone& ctz)
const {
+ if (config::enable_arrow_input_validation) {
+ check_arrow_no_offset(*arrow_array);
+ }
auto& col_data = assert_cast<ColumnIPv6&>(column).get_data();
const auto* concrete_array = assert_cast<const
arrow::StringArray*>(arrow_array);
std::shared_ptr<arrow::Buffer> buffer = concrete_array->value_data();
diff --git a/be/src/core/data_type_serde/data_type_jsonb_serde.cpp
b/be/src/core/data_type_serde/data_type_jsonb_serde.cpp
index d7111a412b2..e53e4d23a74 100644
--- a/be/src/core/data_type_serde/data_type_jsonb_serde.cpp
+++ b/be/src/core/data_type_serde/data_type_jsonb_serde.cpp
@@ -22,8 +22,10 @@
#include <memory>
#include "arrow/array/builder_binary.h"
+#include "common/config.h"
#include "common/exception.h"
#include "common/status.h"
+#include "core/data_type_serde/arrow_validation.h"
#include "core/value/jsonb_value.h"
#include "exprs/json_functions.h"
#include "util/jsonb_parser_simd.h"
@@ -116,6 +118,9 @@ Status DataTypeJsonbSerDe::write_column_to_arrow(const
IColumn& column, const Nu
Status DataTypeJsonbSerDe::read_column_from_arrow(IColumn& column, const
arrow::Array* arrow_array,
int64_t start, int64_t end,
const cctz::time_zone& ctz)
const {
+ if (config::enable_arrow_input_validation) {
+ check_arrow_no_offset(*arrow_array);
+ }
if (arrow_array->type_id() == arrow::Type::STRING ||
arrow_array->type_id() == arrow::Type::BINARY) {
const auto* concrete_array = dynamic_cast<const
arrow::BinaryArray*>(arrow_array);
diff --git a/be/src/core/data_type_serde/data_type_map_serde.cpp
b/be/src/core/data_type_serde/data_type_map_serde.cpp
index 9721e6eed99..c906e6aa5ad 100644
--- a/be/src/core/data_type_serde/data_type_map_serde.cpp
+++ b/be/src/core/data_type_serde/data_type_map_serde.cpp
@@ -18,11 +18,13 @@
#include "core/data_type_serde/data_type_map_serde.h"
#include "arrow/array/builder_nested.h"
+#include "common/config.h"
#include "common/exception.h"
#include "common/status.h"
#include "core/column/column.h"
#include "core/column/column_const.h"
#include "core/column/column_map.h"
+#include "core/data_type_serde/arrow_validation.h"
#include "core/data_type_serde/complex_type_deserialize_util.h"
#include "core/string_ref.h"
#include "util/jsonb_document.h"
@@ -388,6 +390,9 @@ Status DataTypeMapSerDe::read_column_from_arrow(IColumn&
column, const arrow::Ar
const auto* concrete_map = dynamic_cast<const
arrow::MapArray*>(arrow_array);
auto arrow_offsets_array = concrete_map->offsets();
auto* arrow_offsets =
dynamic_cast<arrow::Int32Array*>(arrow_offsets_array.get());
+ if (config::enable_arrow_input_validation) {
+ check_arrow_map_offsets(*concrete_map, start, end);
+ }
auto prev_size = offsets_data.back();
const auto* base_offsets_ptr = reinterpret_cast<const
uint8_t*>(arrow_offsets->raw_values());
diff --git a/be/src/core/data_type_serde/data_type_nullable_serde.cpp
b/be/src/core/data_type_serde/data_type_nullable_serde.cpp
index 7c6ce46e1cd..d41c7a39208 100644
--- a/be/src/core/data_type_serde/data_type_nullable_serde.cpp
+++ b/be/src/core/data_type_serde/data_type_nullable_serde.cpp
@@ -24,11 +24,13 @@
#include <boost/iterator/iterator_facade.hpp>
#include <vector>
+#include "common/config.h"
#include "core/assert_cast.h"
#include "core/column/column.h"
#include "core/column/column_const.h"
#include "core/column/column_nullable.h"
#include "core/column/column_vector.h"
+#include "core/data_type_serde/arrow_validation.h"
#include "core/data_type_serde/data_type_serde.h"
#include "core/data_type_serde/data_type_string_serde.h"
#include "core/data_type_serde/decoded_column_view.h"
@@ -342,6 +344,10 @@ Status
DataTypeNullableSerDe::read_column_from_arrow(IColumn& column,
const arrow::Array*
arrow_array, int64_t start,
int64_t end,
const cctz::time_zone&
ctz) const {
+ if (config::enable_arrow_input_validation) {
+ check_arrow_array_range(*arrow_array, start, end);
+ check_arrow_validity_bitmap(*arrow_array);
+ }
auto& col = reinterpret_cast<ColumnNullable&>(column);
NullMap& map_data = col.get_null_map_data();
for (auto i = start; i < end; ++i) {
diff --git a/be/src/core/data_type_serde/data_type_number_serde.cpp
b/be/src/core/data_type_serde/data_type_number_serde.cpp
index d788bbb2981..3900f25a6d6 100644
--- a/be/src/core/data_type_serde/data_type_number_serde.cpp
+++ b/be/src/core/data_type_serde/data_type_number_serde.cpp
@@ -23,11 +23,13 @@
#include <limits>
#include <type_traits>
+#include "common/config.h"
#include "common/exception.h"
#include "common/status.h"
#include "core/column/column_nullable.h"
#include "core/data_type/define_primitive_type.h"
#include "core/data_type/primitive_type.h"
+#include "core/data_type_serde/arrow_validation.h"
#include "core/data_type_serde/data_type_serde.h"
#include "core/data_type_serde/decoded_column_view.h"
#include "core/packed_int128.h"
@@ -404,13 +406,19 @@ Status
DataTypeNumberSerDe<T>::read_column_from_arrow(IColumn& column,
const arrow::Array*
arrow_array,
int64_t start, int64_t
end,
const cctz::time_zone&
ctz) const {
+ if (config::enable_arrow_input_validation) {
+ check_arrow_array_range(*arrow_array, start, end);
+ }
auto row_count = end - start;
auto& col_data = static_cast<ColumnType&>(column).get_data();
// now uint8 for bool
if constexpr (T == TYPE_BOOLEAN) {
const auto* concrete_array = dynamic_cast<const
arrow::BooleanArray*>(arrow_array);
- for (size_t bool_i = 0; bool_i !=
static_cast<size_t>(concrete_array->length()); ++bool_i) {
+ if (config::enable_arrow_input_validation) {
+ check_arrow_boolean_buffer(*concrete_array);
+ }
+ for (int64_t bool_i = start; bool_i != end; ++bool_i) {
col_data.emplace_back(concrete_array->Value(bool_i));
}
return Status::OK();
@@ -419,7 +427,11 @@ Status
DataTypeNumberSerDe<T>::read_column_from_arrow(IColumn& column,
// only for largeint(int128) type
if (arrow_array->type_id() == arrow::Type::STRING) {
const auto* concrete_array = dynamic_cast<const
arrow::StringArray*>(arrow_array);
+ if (config::enable_arrow_input_validation) {
+ check_arrow_binary_offsets_buffer(*concrete_array);
+ }
std::shared_ptr<arrow::Buffer> buffer = concrete_array->value_data();
+ const size_t buffer_size = buffer ?
static_cast<size_t>(buffer->size()) : 0;
CastParameters params;
const auto* offsets_data = concrete_array->value_offsets()->data();
@@ -430,13 +442,17 @@ Status
DataTypeNumberSerDe<T>::read_column_from_arrow(IColumn& column,
auto end_offset =
unaligned_load<int32_t>(offsets_data + (offset_i + 1)
* offset_size);
- const auto* raw_data = buffer->data() + start_offset;
const auto raw_data_len = end_offset - start_offset;
-
+ if (config::enable_arrow_input_validation) {
+ check_arrow_value_range(*concrete_array, start_offset,
raw_data_len,
+ buffer_size);
+ }
if (raw_data_len == 0) {
col_data.emplace_back(
typename PrimitiveTypeTraits<T>::CppType()); //
Int128() is NULL
} else {
+ const auto* raw_data =
+ reinterpret_cast<const char*>(buffer->data() +
start_offset);
if constexpr (T == TYPE_DATETIMEV2 || T ==
TYPE_TIMESTAMPTZ) {
StringRef str_ref(raw_data, raw_data_len);
UInt64 val = 0;
@@ -487,6 +503,10 @@ Status
DataTypeNumberSerDe<T>::read_column_from_arrow(IColumn& column,
}
/// buffers[0] is a null bitmap and buffers[1] are actual values
+ if (config::enable_arrow_input_validation) {
+ check_arrow_fixed_width_buffer(*arrow_array,
+ sizeof(typename
PrimitiveTypeTraits<T>::CppType));
+ }
std::shared_ptr<arrow::Buffer> buffer = arrow_array->data()->buffers[1];
// Handle empty array case: buffer can be null when row_count is 0.
diff --git a/be/src/core/data_type_serde/data_type_string_serde.cpp
b/be/src/core/data_type_serde/data_type_string_serde.cpp
index 4c7c9d02475..bd1c6e5b4f1 100644
--- a/be/src/core/data_type_serde/data_type_string_serde.cpp
+++ b/be/src/core/data_type_serde/data_type_string_serde.cpp
@@ -20,8 +20,10 @@
#include <array>
#include <cstring>
+#include "common/config.h"
#include "core/column/column_string.h"
#include "core/data_type/define_primitive_type.h"
+#include "core/data_type_serde/arrow_validation.h"
#include "core/data_type_serde/decoded_column_view.h"
#include "util/jsonb_document_cast.h"
#include "util/jsonb_utils.h"
@@ -406,7 +408,12 @@ Status
DataTypeStringSerDeBase<ColumnType>::read_column_from_arrow(
if (arrow_array->type_id() == arrow::Type::STRING ||
arrow_array->type_id() == arrow::Type::BINARY) {
const auto* concrete_array = dynamic_cast<const
arrow::BinaryArray*>(arrow_array);
+ if (config::enable_arrow_input_validation) {
+ check_arrow_array_range(*concrete_array, start, end);
+ check_arrow_binary_offsets_buffer(*concrete_array);
+ }
std::shared_ptr<arrow::Buffer> buffer = concrete_array->value_data();
+ const size_t buffer_size = buffer ?
static_cast<size_t>(buffer->size()) : 0;
const uint8_t* offsets_data = concrete_array->value_offsets()->data();
const size_t offset_size = sizeof(int32_t);
@@ -417,16 +424,23 @@ Status
DataTypeStringSerDeBase<ColumnType>::read_column_from_arrow(
unaligned_load<int32_t>(offsets_data + (offset_i + 1)
* offset_size);
int32_t length = end_offset - start_offset;
- const auto* raw_data = buffer->data() + start_offset;
-
- assert_cast<ColumnType&>(column).insert_data(
- reinterpret_cast<const char*>(raw_data), length);
+ if (config::enable_arrow_input_validation) {
+ check_arrow_value_range(*concrete_array, start_offset,
length, buffer_size);
+ }
+ // insert_data() does not read the input pointer when length
is zero.
+ const auto* raw_data = reinterpret_cast<const
char*>(buffer->data() + start_offset);
+ assert_cast<ColumnType&>(column).insert_data(raw_data, length);
} else {
assert_cast<ColumnType&>(column).insert_default();
}
}
} else if (arrow_array->type_id() == arrow::Type::FIXED_SIZE_BINARY) {
const auto* concrete_array = dynamic_cast<const
arrow::FixedSizeBinaryArray*>(arrow_array);
+ if (config::enable_arrow_input_validation) {
+ check_arrow_array_range(*concrete_array, start, end);
+ check_arrow_fixed_width_buffer(*concrete_array,
+
static_cast<size_t>(concrete_array->byte_width()));
+ }
uint32_t width = concrete_array->byte_width();
for (auto offset_i = start; offset_i < end; ++offset_i) {
@@ -440,13 +454,24 @@ Status
DataTypeStringSerDeBase<ColumnType>::read_column_from_arrow(
} else if (arrow_array->type_id() == arrow::Type::LARGE_STRING ||
arrow_array->type_id() == arrow::Type::LARGE_BINARY) {
const auto* concrete_array = dynamic_cast<const
arrow::LargeBinaryArray*>(arrow_array);
+ if (config::enable_arrow_input_validation) {
+ check_arrow_array_range(*concrete_array, start, end);
+ check_arrow_binary_offsets_buffer(*concrete_array);
+ }
std::shared_ptr<arrow::Buffer> buffer = concrete_array->value_data();
+ const size_t buffer_size = buffer ?
static_cast<size_t>(buffer->size()) : 0;
for (auto offset_i = start; offset_i < end; ++offset_i) {
if (!concrete_array->IsNull(offset_i)) {
- const auto* raw_data = buffer->data() +
concrete_array->value_offset(offset_i);
- assert_cast<ColumnType&>(column).insert_data(
- (char*)raw_data,
concrete_array->value_length(offset_i));
+ const auto value_offset =
concrete_array->value_offset(offset_i);
+ const auto value_length =
concrete_array->value_length(offset_i);
+ if (config::enable_arrow_input_validation) {
+ check_arrow_value_range(*concrete_array, value_offset,
value_length,
+ buffer_size);
+ }
+ // insert_data() does not read the input pointer when length
is zero.
+ const auto* raw_data = reinterpret_cast<const
char*>(buffer->data() + value_offset);
+ assert_cast<ColumnType&>(column).insert_data(raw_data,
value_length);
} else {
assert_cast<ColumnType&>(column).insert_default();
}
diff --git a/be/src/core/data_type_serde/data_type_struct_serde.cpp
b/be/src/core/data_type_serde/data_type_struct_serde.cpp
index 0530e1fce66..15960382c87 100644
--- a/be/src/core/data_type_serde/data_type_struct_serde.cpp
+++ b/be/src/core/data_type_serde/data_type_struct_serde.cpp
@@ -20,10 +20,12 @@
#include <algorithm>
#include "arrow/array/builder_nested.h"
+#include "common/config.h"
#include "common/status.h"
#include "core/column/column.h"
#include "core/column/column_const.h"
#include "core/column/column_struct.h"
+#include "core/data_type_serde/arrow_validation.h"
#include "core/data_type_serde/complex_type_deserialize_util.h"
#include "core/data_type_serde/data_type_serde.h"
#include "core/string_ref.h"
@@ -417,6 +419,9 @@ Status DataTypeStructSerDe::write_column_to_arrow(const
IColumn& column, const N
Status DataTypeStructSerDe::read_column_from_arrow(IColumn& column, const
arrow::Array* arrow_array,
int64_t start, int64_t end,
const cctz::time_zone& ctz)
const {
+ if (config::enable_arrow_input_validation) {
+ check_arrow_no_offset(*arrow_array);
+ }
auto& struct_column = static_cast<ColumnStruct&>(column);
const auto* concrete_struct = dynamic_cast<const
arrow::StructArray*>(arrow_array);
DCHECK_EQ(struct_column.tuple_size(), concrete_struct->num_fields());
diff --git
a/be/test/core/data_type_serde/data_type_serde_arrow_validation_test.cpp
b/be/test/core/data_type_serde/data_type_serde_arrow_validation_test.cpp
new file mode 100644
index 00000000000..fdce7af8f89
--- /dev/null
+++ b/be/test/core/data_type_serde/data_type_serde_arrow_validation_test.cpp
@@ -0,0 +1,275 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/api.h>
+#include <cctz/time_zone.h>
+#include <gtest/gtest-message.h>
+#include <gtest/gtest-test-part.h>
+#include <gtest/gtest.h>
+
+#include <cstdint>
+#include <memory>
+#include <string>
+#include <string_view>
+#include <utility>
+#include <vector>
+
+#include "common/config.h"
+#include "common/exception.h"
+#include "core/column/column_array.h"
+#include "core/column/column_map.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_string.h"
+#include "core/data_type/primitive_type.h"
+#include "core/data_type_serde/data_type_array_serde.h"
+#include "core/data_type_serde/data_type_map_serde.h"
+#include "core/data_type_serde/data_type_nullable_serde.h"
+#include "core/data_type_serde/data_type_number_serde.h"
+#include "core/data_type_serde/data_type_string_serde.h"
+
+namespace doris {
+namespace {
+
+class ScopedArrowInputValidation {
+public:
+ explicit ScopedArrowInputValidation(bool enabled)
+ : _old_value(config::enable_arrow_input_validation) {
+ config::enable_arrow_input_validation = enabled;
+ }
+
+ ~ScopedArrowInputValidation() { config::enable_arrow_input_validation =
_old_value; }
+
+private:
+ bool _old_value;
+};
+
+template <typename Func>
+void expect_invalid_arrow(Func&& func, std::string_view message) {
+ bool thrown = false;
+ try {
+ std::forward<Func>(func)();
+ } catch (const Exception& e) {
+ thrown = true;
+ EXPECT_EQ(e.code(), ErrorCode::INVALID_ARGUMENT) << e.to_string();
+ }
+ EXPECT_TRUE(thrown) << message;
+}
+
+std::shared_ptr<arrow::Buffer> wrap_offsets(const std::vector<int32_t>&
offsets) {
+ return arrow::Buffer::Wrap(offsets);
+}
+
+struct StringArrayHolder {
+ StringArrayHolder(std::vector<int32_t> offsets_, std::string_view values_)
+ : offsets(std::move(offsets_)), values(values_) {
+ auto value_buffer = arrow::Buffer::Wrap(values.data(), values.size());
+ array = std::make_shared<arrow::StringArray>(offsets.size() - 1,
wrap_offsets(offsets),
+ value_buffer);
+ }
+
+ std::vector<int32_t> offsets;
+ std::string values;
+ std::shared_ptr<arrow::StringArray> array;
+};
+
+} // namespace
+
+TEST(DataTypeSerDeArrowValidationTest, RejectsShortStringOffsetsBuffer) {
+ ScopedArrowInputValidation validation(true);
+
+ std::vector<int32_t> offsets = {0};
+ std::string_view values = "abc";
+ auto value_buffer = arrow::Buffer::Wrap(values.data(), values.size());
+ auto array = std::make_shared<arrow::StringArray>(1,
wrap_offsets(offsets), value_buffer);
+ auto column = ColumnString::create();
+ DataTypeStringSerDe serde(TYPE_STRING);
+
+ expect_invalid_arrow(
+ [&] {
+ static_cast<void>(serde.read_column_from_arrow(*column,
array.get(), 0, 1,
+
cctz::utc_time_zone()));
+ },
+ "short string offsets buffer should be rejected");
+}
+
+TEST(DataTypeSerDeArrowValidationTest, RejectsStringValueRangeBeyondBuffer) {
+ ScopedArrowInputValidation validation(true);
+
+ StringArrayHolder array({0, 8}, "abc");
+ auto column = ColumnString::create();
+ DataTypeStringSerDe serde(TYPE_STRING);
+
+ expect_invalid_arrow(
+ [&] {
+ static_cast<void>(serde.read_column_from_arrow(*column,
array.array.get(), 0, 1,
+
cctz::utc_time_zone()));
+ },
+ "string value range beyond data buffer should be rejected");
+}
+
+TEST(DataTypeSerDeArrowValidationTest, RejectsNonMonotonicStringOffsets) {
+ ScopedArrowInputValidation validation(true);
+
+ StringArrayHolder array({3, 1}, "abcd");
+ auto column = ColumnString::create();
+ DataTypeStringSerDe serde(TYPE_STRING);
+
+ expect_invalid_arrow(
+ [&] {
+ static_cast<void>(serde.read_column_from_arrow(*column,
array.array.get(), 0, 1,
+
cctz::utc_time_zone()));
+ },
+ "non-monotonic string offsets should be rejected");
+}
+
+TEST(DataTypeSerDeArrowValidationTest, RejectsShortFixedWidthDataBuffer) {
+ ScopedArrowInputValidation validation(true);
+
+ std::vector<int64_t> values = {1};
+ auto data_buffer = arrow::Buffer::Wrap(values);
+ auto array = std::make_shared<arrow::Int64Array>(2, data_buffer);
+ auto column = ColumnInt64::create();
+ DataTypeNumberSerDe<TYPE_BIGINT> serde;
+
+ expect_invalid_arrow(
+ [&] {
+ static_cast<void>(serde.read_column_from_arrow(*column,
array.get(), 0, 2,
+
cctz::utc_time_zone()));
+ },
+ "short int64 data buffer should be rejected");
+}
+
+TEST(DataTypeSerDeArrowValidationTest, RejectsSlicedArrowArray) {
+ ScopedArrowInputValidation validation(true);
+
+ std::vector<int64_t> values = {1, 2, 3};
+ auto original = std::make_shared<arrow::Int64Array>(3,
arrow::Buffer::Wrap(values));
+ auto sliced = original->Slice(1, 2);
+ auto column = ColumnInt64::create();
+ DataTypeNumberSerDe<TYPE_BIGINT> serde;
+
+ expect_invalid_arrow(
+ [&] {
+ static_cast<void>(serde.read_column_from_arrow(*column,
sliced.get(), 0, 2,
+
cctz::utc_time_zone()));
+ },
+ "sliced Arrow array should be rejected");
+}
+
+TEST(DataTypeSerDeArrowValidationTest, RejectsShortBooleanDataBitmap) {
+ ScopedArrowInputValidation validation(true);
+
+ std::vector<uint8_t> bits = {0xFF};
+ auto data_buffer = arrow::Buffer::Wrap(bits);
+ auto array = std::make_shared<arrow::BooleanArray>(9, data_buffer);
+ auto column = ColumnUInt8::create();
+ DataTypeNumberSerDe<TYPE_BOOLEAN> serde;
+
+ expect_invalid_arrow(
+ [&] {
+ static_cast<void>(serde.read_column_from_arrow(*column,
array.get(), 0, 9,
+
cctz::utc_time_zone()));
+ },
+ "short boolean data bitmap should be rejected");
+}
+
+TEST(DataTypeSerDeArrowValidationTest, RejectsShortValidityBitmap) {
+ ScopedArrowInputValidation validation(true);
+
+ std::vector<uint8_t> validity = {0xFF};
+ std::vector<int64_t> values(9, 1);
+ auto validity_buffer = arrow::Buffer::Wrap(validity);
+ auto data_buffer = arrow::Buffer::Wrap(values);
+ auto array = std::make_shared<arrow::Int64Array>(9, data_buffer,
validity_buffer);
+ auto column = ColumnNullable::create(ColumnInt64::create(),
ColumnUInt8::create());
+ auto nested_serde = std::make_shared<DataTypeNumberSerDe<TYPE_BIGINT>>();
+ DataTypeNullableSerDe serde(nested_serde);
+
+ expect_invalid_arrow(
+ [&] {
+ static_cast<void>(serde.read_column_from_arrow(*column,
array.get(), 0, 9,
+
cctz::utc_time_zone()));
+ },
+ "short validity bitmap should be rejected before IsNull");
+}
+
+TEST(DataTypeSerDeArrowValidationTest,
RejectsMissingValidityBitmapWithNullCount) {
+ ScopedArrowInputValidation validation(true);
+
+ std::vector<int64_t> values = {1, 2};
+ auto data_buffer = arrow::Buffer::Wrap(values);
+ auto array = std::make_shared<arrow::Int64Array>(2, data_buffer,
+
std::shared_ptr<arrow::Buffer>(), 1);
+ auto column = ColumnNullable::create(ColumnInt64::create(),
ColumnUInt8::create());
+ auto nested_serde = std::make_shared<DataTypeNumberSerDe<TYPE_BIGINT>>();
+ DataTypeNullableSerDe serde(nested_serde);
+
+ expect_invalid_arrow(
+ [&] {
+ static_cast<void>(serde.read_column_from_arrow(*column,
array.get(), 0, 2,
+
cctz::utc_time_zone()));
+ },
+ "missing validity bitmap with positive null_count should be
rejected");
+}
+
+TEST(DataTypeSerDeArrowValidationTest, RejectsListOffsetsBeyondValuesLength) {
+ ScopedArrowInputValidation validation(true);
+
+ StringArrayHolder values({0, 1}, "a");
+ std::vector<int32_t> offsets = {0, 2};
+ auto offsets_buffer = wrap_offsets(offsets);
+ auto array =
std::make_shared<arrow::ListArray>(arrow::list(arrow::utf8()), 1,
offsets_buffer,
+ values.array);
+ auto column = ColumnArray::create(
+ ColumnNullable::create(ColumnString::create(),
ColumnUInt8::create()),
+ ColumnOffset64::create());
+ auto nested_serde = std::make_shared<DataTypeNullableSerDe>(
+ std::make_shared<DataTypeStringSerDe>(TYPE_STRING));
+ DataTypeArraySerDe serde(nested_serde);
+
+ expect_invalid_arrow(
+ [&] {
+ static_cast<void>(serde.read_column_from_arrow(*column,
array.get(), 0, 1,
+
cctz::utc_time_zone()));
+ },
+ "list offsets beyond values length should be rejected");
+}
+
+TEST(DataTypeSerDeArrowValidationTest, RejectsMapOffsetsBeyondKeysLength) {
+ ScopedArrowInputValidation validation(true);
+
+ StringArrayHolder keys({0, 1}, "k");
+ std::vector<int64_t> item_values = {1};
+ auto items = std::make_shared<arrow::Int64Array>(1,
arrow::Buffer::Wrap(item_values));
+ std::vector<int32_t> offsets = {0, 2};
+ auto offsets_buffer = wrap_offsets(offsets);
+ auto array = std::make_shared<arrow::MapArray>(arrow::map(arrow::utf8(),
arrow::int64()), 1,
+ offsets_buffer, keys.array,
items);
+ auto column = ColumnMap::create(ColumnString::create(),
ColumnInt64::create(),
+ ColumnOffset64::create());
+ DataTypeMapSerDe serde(std::make_shared<DataTypeStringSerDe>(TYPE_STRING),
+
std::make_shared<DataTypeNumberSerDe<TYPE_BIGINT>>());
+
+ expect_invalid_arrow(
+ [&] {
+ static_cast<void>(serde.read_column_from_arrow(*column,
array.get(), 0, 1,
+
cctz::utc_time_zone()));
+ },
+ "map offsets beyond keys length should be rejected");
+}
+
+} // namespace doris
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
