This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
     new 489eecba70c [refactor](exec) Extract COW-safe tablet column validation 
(#70562)
489eecba70c is described below

commit 489eecba70cba635e077666bbb79a0dc41edd976
Author: Mryange <[email protected]>
AuthorDate: Wed Nov 18 15:12:49 2026 +0800

    [refactor](exec) Extract COW-safe tablet column validation (#70562)
    
    ### Background
    
    Tablet write validation can truncate strings, round legacy decimals, and
    deduplicate map keys. Expression outputs can share column trees, so
    validation must preserve copy-on-write ownership before changing them.
    
    ### What problem does this PR solve?
    
    Issue Number: None
    Related PR: #70421 (superseded)
    PR Type: Refactor
    
    Problem Summary:
    
    Tablet validation is embedded in the block converter as a large
    recursive function that mixes type dispatch, nullable handling, row
    filtering, and mutation. Extract it into a standalone, testable column
    validator with per-type helpers and a shared validation context. The new
    entry reads an immutable input tree and returns a validated column.
    Scalar checks reuse input storage; complex types build new shells around
    validated children, avoiding shared-column corruption and full-buffer
    copies for read-only validation.
    
    ### Details
    
    - **Change map:** Roughly 500 lines of embedded validation are replaced
    by a standalone implementation and interface; the converter keeps block
    orchestration. Add ownership/validation UTs and a 55-line regression
    with generated output. No new column mutation accessor is needed.
    - **Moves vs. logic changes:** Move scalar checks and decimal bound
    caches into per-type helpers/context. Centralize physical nullable
    handling and original-row mapping. Replace const-cast mutation with
    immutable recursive traversal and fresh container shells. String
    truncation returns a replacement; DecimalV2 clones only if rounding is
    needed. Each new map shell deduplicates once, detaching child data only
    when filtering duplicates.
    - **Behavior change:** Validation no longer modifies shared source
    columns or retains stale views after nested string replacement.
    Inherited NULLs skip nested payload checks and rejected elements filter
    their original outer row.
    - **Compatibility:** No wire or storage format changes. DecimalV2 keeps
    historical execution precision/scale (27, 9); enforcing declared schema
    parameters and resolving map-key collisions after string truncation
    remain outside this change. This refactoring targets master without
    release pick requests.
    - **Performance impact:** Read-only scalar columns and unchanged leaf
    buffers remain shared, including offsets and null maps. Complex types
    allocate new shells even when their children are unchanged; this trades
    small container allocations for simple ownership without recursive
    detachment. UTs verify storage reuse and source isolation. No throughput
    benchmark was run.
    - **Rollback:** Revert this commit; no metadata or data migration is
    needed.
    
    ### Testing
    
    - `BUILD_TYPE_UT=ASAN sh run-be-ut.sh --run
    --filter='TabletColumnValidatorTest.*' -j48`: 24 tests passed on the
    development machine. Covers shared read-only leaves, nested shell
    construction, replacement/source isolation, map deduplication, NULL
    propagation and decimal compatibility.
    - Repository clang-format 16 scripts, build hygiene, `git diff --check`,
    and prescribed clang-tidy on the changed source TUs and validator UT:
    passed.
    - No standalone BE build or end-to-end regression was rerun for this
    revision. The ASAN UT script compiled the affected production code and
    test binary; the existing shared-map regression and generated output are
    retained.
    
    ### Release note
    
    Fix incorrect results or failures when inserting shared complex columns
    that require tablet validation.
    
    ### Check List (For Author)
    
    - Behavior changed: Yes (see Details)
    - Does this need documentation: No
    - Pre-submission status: Partially verified: focused ASAN UTs and static
    checks passed; standalone BE build and end-to-end regression not rerun
    after this change.
    
    ### AI Disclosure
    
    - Tool: Codex (GPT-6).
    - Generated: Code, tests, and this description.
    - Human review: The author reviewed the design and an earlier source
    version; the final changes remain available for review.
---
 be/src/exec/sink/tablet_column_validator.cpp       | 475 +++++++++++++++++++
 be/src/exec/sink/tablet_column_validator.h         |  61 +++
 be/src/exec/sink/vtablet_block_convertor.cpp       | 497 +-------------------
 be/src/exec/sink/vtablet_block_convertor.h         |  41 +-
 be/test/exec/sink/tablet_column_validator_test.cpp | 511 +++++++++++++++++++++
 .../map/test_map_insert_shared_column.out          |   4 +
 .../map/test_map_insert_shared_column.groovy       |  55 +++
 7 files changed, 1127 insertions(+), 517 deletions(-)

diff --git a/be/src/exec/sink/tablet_column_validator.cpp 
b/be/src/exec/sink/tablet_column_validator.cpp
new file mode 100644
index 00000000000..0d0b7c049cf
--- /dev/null
+++ b/be/src/exec/sink/tablet_column_validator.cpp
@@ -0,0 +1,475 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#include "exec/sink/tablet_column_validator.h"
+
+#include <algorithm>
+#include <string>
+#include <type_traits>
+#include <utility>
+
+#include "common/config.h"
+#include "common/exception.h"
+#include "core/assert_cast.h"
+#include "core/binary_cast.hpp"
+#include "core/block/block.h"
+#include "core/column/column_array.h"
+#include "core/column/column_decimal.h"
+#include "core/column/column_map.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_string.h"
+#include "core/column/column_struct.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_decimal.h"
+#include "core/data_type/data_type_factory.hpp"
+#include "core/data_type/data_type_map.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_string.h"
+#include "core/data_type/data_type_struct.h"
+#include "core/data_type/define_primitive_type.h"
+#include "core/field.h"
+#include "core/wide_integer_to_string.h"
+#include "exprs/function/function_helpers.h"
+#include "exprs/function/simple_function_factory.h"
+#include "runtime/runtime_state.h"
+#include "storage/olap_common.h"
+#include "util/defer_op.h"
+
+namespace doris {
+
+Status TabletValidateContext::reject(size_t row, const fmt::memory_buffer& 
message) const {
+    filter_map[row] = true;
+    return state->append_error_msg_to_file(
+            []() -> std::string { return ""; },
+            [&]() { return fmt::to_string(error_prefix) + 
fmt::to_string(message); });
+}
+
+namespace {
+
+template <bool is_min>
+DecimalV2Value get_decimalv2_min_or_max(TabletValidateContext& context, const 
DataTypePtr& type) {
+    auto& cache = is_min ? context.min_decimalv2_val : 
context.max_decimalv2_val;
+    const auto precision = type->get_precision();
+    const auto scale = type->get_scale();
+    const auto key = std::pair {precision, scale};
+    auto iter = cache.find(key);
+    if (iter != cache.end()) {
+        return iter->second;
+    }
+    DecimalV2Value value;
+    if constexpr (is_min) {
+        value.to_min_decimal(precision, scale);
+    } else {
+        value.to_max_decimal(precision, scale);
+    }
+    cache.emplace(key, value);
+    return value;
+}
+
+template <typename DecimalType, bool IsMin>
+DecimalType get_decimalv3_min_or_max(TabletValidateContext& context, const 
DataTypePtr& type) {
+    std::map<int, typename DecimalType::NativeType>* pmap;
+    if constexpr (std::is_same_v<DecimalType, Decimal32>) {
+        pmap = IsMin ? &context.min_decimal32_val : &context.max_decimal32_val;
+    } else if constexpr (std::is_same_v<DecimalType, Decimal64>) {
+        pmap = IsMin ? &context.min_decimal64_val : &context.max_decimal64_val;
+    } else if constexpr (std::is_same_v<DecimalType, Decimal128V3>) {
+        pmap = IsMin ? &context.min_decimal128_val : 
&context.max_decimal128_val;
+    } else {
+        pmap = IsMin ? &context.min_decimal256_val : 
&context.max_decimal256_val;
+    }
+
+    // found
+    auto iter = pmap->find(type->get_precision());
+    if (iter != pmap->end()) {
+        return DecimalType(iter->second);
+    }
+
+    DecimalType value;
+    if constexpr (IsMin) {
+        value = min_decimal_value<DecimalType::PType>(type->get_precision());
+    } else {
+        value = max_decimal_value<DecimalType::PType>(type->get_precision());
+    }
+    pmap->emplace(type->get_precision(), value.value);
+    return value;
+}
+
+// Borrowed views: local positions map to rows in the original output block.
+// null_map includes NULLs inherited from enclosing nullable containers.
+struct ValidationRows {
+    const IColumn::Permutation* rows = nullptr;
+    const UInt8* null_map = nullptr;
+
+    size_t original_row(size_t i) const { return rows ? (*rows)[i] : i; }
+
+    bool should_validate(size_t i, const TabletValidateContext& context) const 
{
+        return !context.filter_map[original_row(i)] && (!null_map || 
!null_map[i]);
+    }
+};
+
+Result<ColumnPtr> validate_column(const ColumnPtr& column, const DataTypePtr& 
type,
+                                  TabletValidateContext& context, const 
ValidationRows& rows);
+
+Result<ColumnPtr> validate_string(ColumnPtr column, const DataTypePtr& type,
+                                  TabletValidateContext& context, const 
ValidationRows& rows) {
+    int limit = config::string_type_length_soft_limit_bytes;
+    int len = -1;
+    const auto* type_str = check_and_get_data_type<DataTypeString>(type.get());
+    if (type_str && type_str->len() >= 0) {
+        len = type_str->len();
+        limit = std::min(limit, len);
+    }
+
+    const auto& strings = assert_cast<const ColumnString&>(*column);
+    const auto& offsets = strings.get_offsets();
+    size_t invalid_count = 0;
+    for (size_t i = 0; i < column->size(); ++i) {
+        invalid_count += rows.should_validate(i, context) && offsets[i] - 
offsets[i - 1] > limit;
+    }
+    if (!invalid_count) {
+        return column;
+    }
+
+    // Non-strict loads truncate in characters, but the tablet limit is in 
bytes.
+    // For example, two Chinese characters can still exceed a VARCHAR(2) byte 
limit.
+    if (type_str && !context.state->enable_insert_strict()) {
+        auto int_type =
+                
DataTypeFactory::instance().create_data_type(FieldType::OLAP_FIELD_TYPE_INT, 0, 
0);
+        ColumnsWithTypeAndName arguments {{nullptr, type, "string column"},
+                                          {nullptr, int_type, "pos"},
+                                          {nullptr, int_type, "len"}};
+        auto function = SimpleFunctionFactory::instance().get_function(
+                "substring", arguments, type, {}, 
context.state->be_exec_version());
+        if (!function) {
+            return ResultError(Status::InternalError("get function substring 
failed"));
+        }
+        ColumnPtr result;
+        {
+            const auto size = column->size();
+            auto pos = int_type->create_column_const(size, 
to_field<TYPE_INT>(1));
+            auto length = int_type->create_column_const(size, 
to_field<TYPE_INT>(limit));
+            Block block({{column->get_ptr(), type, "string column"},
+                         {pos, int_type, "pos"},
+                         {length, int_type, "len"},
+                         {nullptr, type, "result"}});
+            RETURN_IF_ERROR_RESULT(function->execute(nullptr, block, {0, 1, 
2}, 3, size));
+            result = std::move(block.get_by_position(3).column);
+        }
+        column = std::move(result);
+    }
+
+    const auto& checked = assert_cast<const ColumnString&>(*column);
+    for (size_t i = 0; i < column->size(); ++i) {
+        if (!rows.should_validate(i, context)) {
+            continue;
+        }
+        auto value = checked.get_data_at(i);
+        if (std::cmp_greater(value.size, static_cast<size_t>(limit))) {
+            fmt::memory_buffer message;
+            if (std::cmp_greater(value.size, static_cast<size_t>(len))) {
+                fmt::format_to(message, "the length of input is too long than 
schema. ");
+                fmt::format_to(message, "first 32 bytes of input str: [{}] ", 
value.to_prefix(32));
+                fmt::format_to(message, "schema length: {}; actual length: {}; 
", len, value.size);
+            } else {
+                fmt::format_to(message, "the length of input string is too 
long than vec schema. ");
+                fmt::format_to(message, "first 32 bytes of input str: [{}] ", 
value.to_prefix(32));
+                fmt::format_to(message, "schema length: {}; limit length: {}; 
actual length: {}; ",
+                               len, limit, value.size);
+            }
+            RETURN_IF_ERROR_RESULT(context.reject(rows.original_row(i), 
message));
+        }
+    }
+    return column;
+}
+
+Status validate_jsonb(const ColumnString& column, TabletValidateContext& 
context,
+                      const ValidationRows& rows) {
+    for (size_t i = 0; i < column.size(); ++i) {
+        if (rows.should_validate(i, context) && column.get_data_at(i).size == 
0) {
+            fmt::memory_buffer message;
+            fmt::format_to(message, "jsonb with size 0 is invalid");
+            RETURN_IF_ERROR(context.reject(rows.original_row(i), message));
+        }
+    }
+    return Status::OK();
+}
+
+Result<ColumnPtr> validate_decimal_v2(const ColumnPtr& input, const 
DataTypePtr& type,
+                                      TabletValidateContext& context, const 
ValidationRows& rows) {
+    // DecimalV2 is deprecated. Preserve its historical validation using 
execution
+    // precision/scale (27, 9), rather than the original schema parameters. For
+    // example, DECIMAL(9, 2) still validates 1.235 at scale 9 here; enforcing 
the
+    // declared scale and range is outside this COW refactoring.
+    const auto& column = assert_cast<const ColumnDecimal128V2&>(*input);
+    MutableColumnPtr rewritten;
+    const auto precision = type->get_precision();
+    const auto scale = type->get_scale();
+    const auto max = get_decimalv2_min_or_max<false>(context, type);
+    const auto min = get_decimalv2_min_or_max<true>(context, type);
+    for (size_t i = 0; i < column.size(); ++i) {
+        if (!rows.should_validate(i, context)) {
+            continue;
+        }
+        auto value = binary_cast<Int128, DecimalV2Value>(column.get_data()[i]);
+        fmt::memory_buffer message;
+        if (value.greater_than_scale(scale)) {
+            auto code = value.round(&value, scale, HALF_UP);
+            if (!rewritten) {
+                rewritten = column.clone();
+            }
+            assert_cast<ColumnDecimal128V2&>(*rewritten).get_data()[i] = value;
+            if (code != E_DEC_OK) {
+                fmt::format_to(message, "round one decimal failed.value={}; ", 
value.to_string());
+            }
+        }
+        if (value > max || value < min) {
+            fmt::format_to(message,
+                           "decimal value is not valid for definition, 
value={}, "
+                           "precision={}, scale={}, min={}, max={}; ",
+                           value.to_string(), precision, scale, 
min.to_string(), max.to_string());
+        }
+        if (message.size()) {
+            RETURN_IF_ERROR_RESULT(context.reject(rows.original_row(i), 
message));
+        }
+    }
+    return rewritten ? ColumnPtr(std::move(rewritten)) : input;
+}
+
+// DecimalV3 needs only a range check; use one template instead of four macro 
bodies.
+template <typename DecimalType>
+Status validate_decimal_v3(const ColumnDecimal<DecimalType::PType>& column, 
const DataTypePtr& type,
+                           TabletValidateContext& context, const 
ValidationRows& rows) {
+    const auto max = get_decimalv3_min_or_max<DecimalType, false>(context, 
type);
+    const auto min = get_decimalv3_min_or_max<DecimalType, true>(context, 
type);
+    size_t invalid_count = 0;
+    for (const auto value : column.get_data()) {
+        invalid_count += value > max || value < min;
+    }
+    if (!invalid_count) {
+        return Status::OK();
+    }
+    for (size_t i = 0; i < column.size(); ++i) {
+        if (!rows.should_validate(i, context)) {
+            continue;
+        }
+        const auto value = column.get_data()[i];
+        if (value > max || value < min) {
+            fmt::memory_buffer message;
+            fmt::format_to(message,
+                           "decimal value is not valid for definition, 
value={}, "
+                           "precision={}, scale={}, min={}, max={}; ",
+                           value.value, type->get_precision(), 
type->get_scale(), min.value,
+                           max.value);
+            RETURN_IF_ERROR(context.reject(rows.original_row(i), message));
+        }
+    }
+    return Status::OK();
+}
+
+// For [NULL, [x, y]], both x and y map to outer row 1; elements underneath a
+// NULL parent are skipped even if their physical payload happens to be 
invalid.
+struct NestedRows {
+    IColumn::Permutation rows;
+    NullMap null_map;
+
+    NestedRows(const ColumnArray::Offsets64& offsets, const ValidationRows& 
parent)
+            : rows(offsets.back()) {
+        if (parent.null_map) {
+            null_map.resize(rows.size());
+        }
+        for (size_t i = 0; i < offsets.size(); ++i) {
+            for (size_t j = offsets[i - 1]; j < offsets[i]; ++j) {
+                rows[j] = parent.original_row(i);
+                if (parent.null_map) {
+                    null_map[j] = parent.null_map[i];
+                }
+            }
+        }
+    }
+
+    ValidationRows view() const {
+        return {.rows = &rows, .null_map = null_map.empty() ? nullptr : 
null_map.data()};
+    }
+};
+
+Result<ColumnPtr> validate_array(const ColumnArray& column, const 
DataTypeArray& type,
+                                 TabletValidateContext& context, const 
ValidationRows& rows) {
+    NestedRows nested(column.get_offsets(), rows);
+    const auto prefix_size = context.error_prefix.size();
+    Defer restore_prefix {[&] { context.error_prefix.resize(prefix_size); }};
+    fmt::format_to(context.error_prefix, "ARRAY type failed: ");
+    auto data =
+            validate_column(column.get_data_ptr(), type.get_nested_type(), 
context, nested.view());
+    if (!data) {
+        return unexpected(std::move(data.error()));
+    }
+    return ColumnPtr(ColumnArray::create(data.value(), 
column.get_offsets_ptr()));
+}
+
+Result<ColumnPtr> validate_map(const ColumnMap& input, const DataTypeMap& type,
+                               TabletValidateContext& context, const 
ValidationRows& rows) {
+    // Deduplication changes only this new shell. The existing implementation
+    // detaches child data only when duplicate keys require filtering.
+    auto result = ColumnMap::create(input.get_keys_ptr(), 
input.get_values_ptr(),
+                                    input.get_offsets_ptr());
+    RETURN_IF_ERROR_RESULT(result->deduplicate_keys(false));
+    const auto& column = *result;
+    NestedRows nested(column.get_offsets(), rows);
+    const auto prefix_size = context.error_prefix.size();
+    Defer restore_prefix {[&] { context.error_prefix.resize(prefix_size); }};
+    fmt::format_to(context.error_prefix, "MAP type failed: ");
+    auto keys = validate_column(column.get_keys_ptr(), type.get_key_type(), 
context, nested.view());
+    if (!keys) {
+        return unexpected(std::move(keys.error()));
+    }
+    auto values =
+            validate_column(column.get_values_ptr(), type.get_value_type(), 
context, nested.view());
+    if (!values) {
+        return unexpected(std::move(values.error()));
+    }
+    result->get_keys_ptr() = std::move(keys.value());
+    result->get_values_ptr() = std::move(values.value());
+    return ColumnPtr(std::move(result));
+}
+
+Result<ColumnPtr> validate_struct(const ColumnStruct& column, const 
DataTypeStruct& type,
+                                  TabletValidateContext& context, const 
ValidationRows& rows) {
+    DCHECK_EQ(type.get_elements().size(), column.tuple_size());
+    const auto prefix_size = context.error_prefix.size();
+    Defer restore_prefix {[&] { context.error_prefix.resize(prefix_size); }};
+    fmt::format_to(context.error_prefix, "STRUCT type failed: ");
+    Columns children;
+    children.reserve(column.tuple_size());
+    for (size_t i = 0; i < column.tuple_size(); ++i) {
+        auto child = validate_column(column.get_column_ptr(i), 
type.get_element(i), context, rows);
+        if (!child) {
+            return unexpected(std::move(child.error()));
+        }
+        children.push_back(std::move(child.value()));
+    }
+    return ColumnPtr(ColumnStruct::create(std::move(children)));
+}
+
+Result<ColumnPtr> validate_nullable(const ColumnNullable& column, const 
DataTypePtr& type,
+                                    TabletValidateContext& context, const 
ValidationRows& rows) {
+    const auto& null_map = column.get_null_map_data();
+    // A physically nullable column can target NOT NULL (including BITMAP). 
NULLs
+    // beneath a NULL ancestor do not represent values being written to that 
target.
+    if (!type->is_nullable() || type->get_primitive_type() == TYPE_BITMAP) {
+        for (size_t i = 0; i < column.size(); ++i) {
+            if (null_map[i] && rows.should_validate(i, context)) {
+                fmt::memory_buffer message;
+                fmt::format_to(message, "null value for not null column, 
type={}",
+                               type->get_name());
+                RETURN_IF_ERROR_RESULT(context.reject(rows.original_row(i), 
message));
+            }
+        }
+    }
+    NullMap combined;
+    const UInt8* effective_null_map = null_map.data();
+    if (rows.null_map) {
+        combined.resize(column.size());
+        for (size_t i = 0; i < column.size(); ++i) {
+            combined[i] = null_map[i] | rows.null_map[i];
+        }
+        effective_null_map = combined.data();
+    }
+    auto nested = validate_column(column.get_nested_column_ptr(), 
remove_nullable(type), context,
+                                  {.rows = rows.rows, .null_map = 
effective_null_map});
+    if (!nested) {
+        return unexpected(std::move(nested.error()));
+    }
+    return ColumnPtr(ColumnNullable::create(nested.value(), 
column.get_null_map_column_ptr()));
+}
+
+Result<ColumnPtr> validate_column(const ColumnPtr& column, const DataTypePtr& 
type,
+                                  TabletValidateContext& context, const 
ValidationRows& rows) {
+    DCHECK(!rows.rows || rows.rows->size() == column->size());
+    const auto& source = *column;
+    if (const auto* nullable = check_and_get_column<ColumnNullable>(source)) {
+        return validate_nullable(*nullable, type, context, rows);
+    }
+    // A non-nullable input can target Nullable(T); physical nullability is
+    // handled above, while dispatch must always use the unwrapped target type.
+    const auto nested_type = remove_nullable(type);
+    switch (nested_type->get_primitive_type()) {
+    case TYPE_CHAR:
+    case TYPE_VARCHAR:
+    case TYPE_STRING:
+        return validate_string(column, nested_type, context, rows);
+    case TYPE_JSONB:
+        RETURN_IF_ERROR_RESULT(
+                validate_jsonb(assert_cast<const ColumnString&>(source), 
context, rows));
+        return column;
+    case TYPE_DECIMALV2:
+        return validate_decimal_v2(column, nested_type, context, rows);
+    case TYPE_DECIMAL32:
+        RETURN_IF_ERROR_RESULT(validate_decimal_v3<Decimal32>(
+                assert_cast<const ColumnDecimal32&>(source), nested_type, 
context, rows));
+        return column;
+    case TYPE_DECIMAL64:
+        RETURN_IF_ERROR_RESULT(validate_decimal_v3<Decimal64>(
+                assert_cast<const ColumnDecimal64&>(source), nested_type, 
context, rows));
+        return column;
+    case TYPE_DECIMAL128I:
+        RETURN_IF_ERROR_RESULT(validate_decimal_v3<Decimal128V3>(
+                assert_cast<const ColumnDecimal128V3&>(source), nested_type, 
context, rows));
+        return column;
+    case TYPE_DECIMAL256:
+        RETURN_IF_ERROR_RESULT(validate_decimal_v3<Decimal256>(
+                assert_cast<const ColumnDecimal256&>(source), nested_type, 
context, rows));
+        return column;
+    case TYPE_ARRAY:
+        return validate_array(assert_cast<const ColumnArray&>(source),
+                              assert_cast<const DataTypeArray&>(*nested_type), 
context, rows);
+    case TYPE_MAP:
+        return validate_map(assert_cast<const ColumnMap&>(source),
+                            assert_cast<const DataTypeMap&>(*nested_type), 
context, rows);
+    case TYPE_STRUCT:
+        return validate_struct(assert_cast<const ColumnStruct&>(source),
+                               assert_cast<const 
DataTypeStruct&>(*nested_type), context, rows);
+    case TYPE_AGG_STATE:
+        // Some aggregate states are serialized into a ColumnString.
+        if (check_and_get_column<ColumnString>(source)) {
+            return validate_string(column, nested_type, context, rows);
+        }
+        return column;
+    default:
+        return column;
+    }
+}
+
+} // namespace
+
+Result<ColumnPtr> tablet_validate_column(const ColumnPtr& column, const 
DataTypePtr& type,
+                                         TabletValidateContext& context) {
+    ColumnPtr result;
+    auto validate = [&]() -> Status {
+        RETURN_IF_CATCH_EXCEPTION({
+            auto checked = validate_column(column, type, context, {});
+            if (!checked) {
+                return std::move(checked.error());
+            }
+            result = std::move(checked.value());
+            return Status::OK();
+        });
+    };
+    RETURN_IF_ERROR_RESULT(validate());
+    return result;
+}
+
+} // namespace doris
diff --git a/be/src/exec/sink/tablet_column_validator.h 
b/be/src/exec/sink/tablet_column_validator.h
new file mode 100644
index 00000000000..093fa1ca9f8
--- /dev/null
+++ b/be/src/exec/sink/tablet_column_validator.h
@@ -0,0 +1,61 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#pragma once
+
+#include <fmt/format.h>
+
+#include <map>
+#include <vector>
+
+#include "common/status.h"
+#include "core/column/column.h"
+#include "core/data_type/data_type.h"
+#include "core/types.h"
+#include "core/value/decimalv2_value.h"
+
+namespace doris {
+
+class RuntimeState;
+
+// State shared by recursive tablet validation of one output block.
+struct TabletValidateContext {
+    explicit TabletValidateContext(std::vector<char>& filter_map) : 
filter_map(filter_map) {}
+
+    Status reject(size_t row, const fmt::memory_buffer& message) const;
+
+    RuntimeState* state = nullptr;
+    fmt::memory_buffer error_prefix;
+    std::vector<char>& filter_map;
+
+    std::map<std::pair<int, int>, DecimalV2Value> min_decimalv2_val;
+    std::map<std::pair<int, int>, DecimalV2Value> max_decimalv2_val;
+    std::map<int, int32_t> min_decimal32_val;
+    std::map<int, int32_t> max_decimal32_val;
+    std::map<int, int64_t> min_decimal64_val;
+    std::map<int, int64_t> max_decimal64_val;
+    std::map<int, int128_t> min_decimal128_val;
+    std::map<int, int128_t> max_decimal128_val;
+    std::map<int, wide::Int256> min_decimal256_val;
+    std::map<int, wide::Int256> max_decimal256_val;
+};
+
+// Validate a full (non-const) column without modifying its input tree. Scalar
+// checks reuse input storage; complex types build shells around validated 
children.
+Result<ColumnPtr> tablet_validate_column(const ColumnPtr& column, const 
DataTypePtr& type,
+                                         TabletValidateContext& context);
+
+} // namespace doris
diff --git a/be/src/exec/sink/vtablet_block_convertor.cpp 
b/be/src/exec/sink/vtablet_block_convertor.cpp
index 134aa2a2140..99e4794a5d4 100644
--- a/be/src/exec/sink/vtablet_block_convertor.cpp
+++ b/be/src/exec/sink/vtablet_block_convertor.cpp
@@ -18,51 +18,30 @@
 #include "exec/sink/vtablet_block_convertor.h"
 
 #include <fmt/format.h>
-#include <gen_cpp/FrontendService.h>
 #include <glog/logging.h>
-#include <google/protobuf/stubs/common.h>
 
-#include <algorithm>
 #include <memory>
-#include <string>
-#include <unordered_map>
 #include <utility>
+#include <vector>
 
 #include "common/compiler_util.h" // IWYU pragma: keep
 #include "common/consts.h"
 #include "common/status.h"
 #include "core/assert_cast.h"
-#include "core/binary_cast.hpp"
 #include "core/block/block.h"
 #include "core/column/column.h"
-#include "core/column/column_array.h"
 #include "core/column/column_const.h"
-#include "core/column/column_decimal.h"
-#include "core/column/column_map.h"
 #include "core/column/column_nullable.h"
-#include "core/column/column_string.h"
-#include "core/column/column_struct.h"
-#include "core/data_type/data_type.h"
-#include "core/data_type/data_type_array.h"
-#include "core/data_type/data_type_decimal.h"
-#include "core/data_type/data_type_factory.hpp"
-#include "core/data_type/data_type_map.h"
+#include "core/column/column_vector.h"
 #include "core/data_type/data_type_nullable.h"
-#include "core/data_type/data_type_struct.h"
-#include "core/data_type/define_primitive_type.h"
+#include "core/data_type/data_type_number.h"
 #include "core/data_type/primitive_type.h"
-#include "core/types.h"
-#include "core/wide_integer_to_string.h"
-#include "exprs/function/function_helpers.h"
-#include "exprs/function/simple_function_factory.h"
-#include "exprs/vexpr.h"
+#include "exec/sink/tablet_column_validator.h"
 #include "exprs/vexpr_context.h"
 #include "runtime/descriptors.h"
+#include "runtime/runtime_profile.h"
 #include "runtime/runtime_state.h"
-#include "service/brpc.h"
-#include "storage/olap_common.h"
-#include "util/brpc_client_cache.h"
-#include "util/thread.h"
+#include "util/defer_op.h"
 
 namespace doris {
 
@@ -103,7 +82,8 @@ Status 
OlapTableBlockConvertor::validate_and_convert_block(RuntimeState* state,
         SCOPED_RAW_TIMER(&_validate_data_ns);
         _filter_map.clear();
         _filter_map.resize(rows, 0);
-        auto st = _validate_data(state, block.get(), rows, filtered_rows);
+        _tablet_validate_context.state = state;
+        auto st = _validate_data(block.get(), rows, filtered_rows);
         _num_filtered_rows += filtered_rows;
         has_filtered_rows = filtered_rows > 0;
         if (!st.ok()) {
@@ -136,457 +116,11 @@ void OlapTableBlockConvertor::init_autoinc_info(int64_t 
db_id, int64_t table_id,
     }
 }
 
-template <bool is_min>
-DecimalV2Value OlapTableBlockConvertor::_get_decimalv2_min_or_max(const 
DataTypePtr& type) {
-    std::map<std::pair<int, int>, DecimalV2Value>* pmap;
-    if constexpr (is_min) {
-        pmap = &_min_decimalv2_val;
-    } else {
-        pmap = &_max_decimalv2_val;
-    }
-
-    // found
-    auto iter = pmap->find(
-            {remove_nullable(type)->get_precision(), 
remove_nullable(type)->get_scale()});
-    if (iter != pmap->end()) {
-        return iter->second;
-    }
-
-    // save min or max DecimalV2Value for next time
-    DecimalV2Value value;
-    if constexpr (is_min) {
-        value.to_min_decimal(type->get_precision(), type->get_scale());
-    } else {
-        value.to_max_decimal(type->get_precision(), type->get_scale());
-    }
-    pmap->emplace(std::pair<int, int> {type->get_precision(), 
type->get_scale()}, value);
-    return value;
-}
-
-template <typename DecimalType, bool IsMin>
-DecimalType OlapTableBlockConvertor::_get_decimalv3_min_or_max(const 
DataTypePtr& type) {
-    std::map<int, typename DecimalType::NativeType>* pmap;
-    if constexpr (std::is_same_v<DecimalType, Decimal32>) {
-        pmap = IsMin ? &_min_decimal32_val : &_max_decimal32_val;
-    } else if constexpr (std::is_same_v<DecimalType, Decimal64>) {
-        pmap = IsMin ? &_min_decimal64_val : &_max_decimal64_val;
-    } else if constexpr (std::is_same_v<DecimalType, Decimal128V3>) {
-        pmap = IsMin ? &_min_decimal128_val : &_max_decimal128_val;
-    } else {
-        pmap = IsMin ? &_min_decimal256_val : &_max_decimal256_val;
-    }
-
-    // found
-    auto iter = pmap->find(type->get_precision());
-    if (iter != pmap->end()) {
-        return DecimalType(iter->second);
-    }
-
-    DecimalType value;
-    if constexpr (IsMin) {
-        value = min_decimal_value<DecimalType::PType>(type->get_precision());
-    } else {
-        value = max_decimal_value<DecimalType::PType>(type->get_precision());
-    }
-    pmap->emplace(type->get_precision(), value.value);
-    return value;
-}
-
-Status OlapTableBlockConvertor::_internal_validate_column(RuntimeState* state, 
Block* block,
-                                                          const DataTypePtr& 
type, ColumnPtr column,
-                                                          size_t slot_index,
-                                                          fmt::memory_buffer& 
error_prefix,
-                                                          const size_t 
row_count,
-                                                          
IColumn::Permutation* rows) {
-    DCHECK((rows == nullptr) || (rows->size() == row_count));
-    fmt::memory_buffer error_msg;
-    auto set_invalid_and_append_error_msg = [&](size_t row) {
-        _filter_map[row] = true;
-        auto ret = state->append_error_msg_to_file([]() -> std::string { 
return ""; },
-                                                   [&error_prefix, 
&error_msg]() -> std::string {
-                                                       return 
fmt::to_string(error_prefix) +
-                                                              
fmt::to_string(error_msg);
-                                                   });
-        error_msg.clear();
-        return ret;
-    };
-
-    const auto* column_ptr = check_and_get_column<ColumnNullable>(*column);
-    const auto& real_column_ptr =
-            column_ptr == nullptr ? column : 
(column_ptr->get_nested_column_ptr());
-    const auto* null_map = column_ptr == nullptr ? nullptr : 
column_ptr->get_null_map_data().data();
-    auto need_to_validate = [](size_t j, size_t row, const std::vector<char>& 
filter_map,
-                               const unsigned char* null_map) {
-        return !filter_map[row] && (null_map == nullptr || null_map[j] == 0);
-    };
-
-    // may change orig_column if substring function is performed
-    auto string_column_checker = [&state, &error_msg, need_to_validate,
-                                  set_invalid_and_append_error_msg](
-                                         ColumnPtr& orig_column, const 
DataTypePtr& orig_type,
-                                         IColumn::Permutation* rows,
-                                         const std::vector<char>& filter_map) {
-        int limit = config::string_type_length_soft_limit_bytes;
-        int len = -1;
-        // when type.len is negative, std::min will return overflow value, so 
we need to check it
-        const auto* type_str =
-                
check_and_get_data_type<DataTypeString>(remove_nullable(orig_type).get());
-        if (type_str) {
-            if (type_str->len() >= 0) {
-                len = type_str->len();
-                limit = std::min(limit, type_str->len());
-            }
-        }
-
-        auto tmp_column_ptr = 
check_and_get_column<ColumnNullable>(*orig_column);
-        auto tmp_real_column_ptr =
-                tmp_column_ptr == nullptr ? orig_column : 
(tmp_column_ptr->get_nested_column_ptr());
-        const auto* column_string = assert_cast<const 
ColumnString*>(tmp_real_column_ptr.get());
-        const auto* null_map =
-                tmp_column_ptr == nullptr ? nullptr : 
tmp_column_ptr->get_null_map_data().data();
-
-        const auto* __restrict offsets = column_string->get_offsets().data();
-        int invalid_count = 0;
-        size_t row_count = orig_column->size();
-        for (int64_t j = 0; j < row_count; ++j) {
-            invalid_count += (offsets[j] - offsets[j - 1]) > limit;
-        }
-
-        if (invalid_count) {
-            // For string column, if in non-strict load mode(for both insert 
stmt and stream load),
-            // truncate the string to schema len.
-            // After truncation, still need to check if byte len of each row 
exceed the schema len,
-            // because currently the schema len is defined in bytes, and 
substring works by unit of chars.
-            // This is a workaround for now, need to improve it after better 
support of multi-byte chars.
-            if (type_str && !state->enable_insert_strict()) {
-                ColumnsWithTypeAndName argument_template;
-                auto input_type = remove_nullable(orig_type);
-                auto pos_type = DataTypeFactory::instance().create_data_type(
-                        FieldType::OLAP_FIELD_TYPE_INT, 0, 0);
-                auto len_type = DataTypeFactory::instance().create_data_type(
-                        FieldType::OLAP_FIELD_TYPE_INT, 0, 0);
-                argument_template.emplace_back(nullptr, input_type, "string 
column");
-                argument_template.emplace_back(nullptr, pos_type, "pos 
column");
-                argument_template.emplace_back(nullptr, len_type, "len 
column");
-                auto func = SimpleFunctionFactory::instance().get_function(
-                        "substring", argument_template, input_type, {}, 
state->be_exec_version());
-                if (!func) {
-                    return Status::InternalError("get function substring 
failed");
-                }
-                auto pos_column = pos_type->create_column_const(row_count, 
to_field<TYPE_INT>(1));
-                auto len_column =
-                        len_type->create_column_const(row_count, 
to_field<TYPE_INT>(limit));
-                Block tmp_block({{remove_nullable(orig_column), input_type, 
"string column"},
-                                 {pos_column, pos_type, "pos"},
-                                 {len_column, len_type, "len"},
-                                 {nullptr, input_type, "result"}});
-                RETURN_IF_ERROR(func->execute(nullptr, tmp_block, {0, 1, 2}, 
3, row_count));
-                auto result_column =
-                        
IColumn::mutate(std::move(tmp_block.get_by_position(3).column));
-                if (orig_column->is_nullable()) {
-                    orig_column = ColumnNullable::create(
-                            std::move(result_column),
-                            
IColumn::mutate(tmp_column_ptr->get_null_map_column_ptr()));
-                } else {
-                    orig_column = std::move(result_column);
-                }
-                tmp_column_ptr = 
check_and_get_column<ColumnNullable>(*orig_column);
-                tmp_real_column_ptr = tmp_column_ptr == nullptr
-                                              ? orig_column
-                                              : 
tmp_column_ptr->get_nested_column_ptr();
-                column_string = assert_cast<const 
ColumnString*>(tmp_real_column_ptr.get());
-                null_map = tmp_column_ptr == nullptr ? nullptr
-                                                     : 
tmp_column_ptr->get_null_map_data().data();
-            }
-            for (size_t j = 0; j < row_count; ++j) {
-                auto row = rows ? (*rows)[j] : j;
-                if (need_to_validate(j, row, filter_map, null_map)) {
-                    auto str_val = column_string->get_data_at(j);
-                    bool invalid = str_val.size > limit;
-                    if (invalid) {
-                        if (str_val.size > len) {
-                            fmt::format_to(error_msg, "{}",
-                                           "the length of input is too long 
than schema. ");
-                            fmt::format_to(error_msg, "first 32 bytes of input 
str: [{}] ",
-                                           str_val.to_prefix(32));
-                            fmt::format_to(error_msg, "schema length: {}; ", 
len);
-                            fmt::format_to(error_msg, "actual length: {}; ", 
str_val.size);
-                        } else if (str_val.size > limit) {
-                            fmt::format_to(
-                                    error_msg, "{}",
-                                    "the length of input string is too long 
than vec schema. ");
-                            fmt::format_to(error_msg, "first 32 bytes of input 
str: [{}] ",
-                                           str_val.to_prefix(32));
-                            fmt::format_to(error_msg, "schema length: {}; ", 
len);
-                            fmt::format_to(error_msg, "limit length: {}; ", 
limit);
-                            fmt::format_to(error_msg, "actual length: {}; ", 
str_val.size);
-                        }
-                        RETURN_IF_ERROR(set_invalid_and_append_error_msg(row));
-                    }
-                }
-            }
-        }
-        return Status::OK();
-    };
-
-    switch (type->get_primitive_type()) {
-    case TYPE_CHAR:
-    case TYPE_VARCHAR:
-    case TYPE_STRING: {
-        RETURN_IF_ERROR(string_column_checker(column, type, rows, 
_filter_map));
-        block->get_by_position(slot_index).column = std::move(column);
-        break;
-    }
-    case TYPE_JSONB: {
-        const auto* column_string = assert_cast<const 
ColumnString*>(real_column_ptr.get());
-        for (size_t j = 0; j < row_count; ++j) {
-            if (!_filter_map[j]) {
-                if (type->is_nullable() && column_ptr && 
column_ptr->is_null_at(j)) {
-                    continue;
-                }
-                auto str_val = column_string->get_data_at(j);
-                bool invalid = str_val.size == 0;
-                if (invalid) {
-                    error_msg.clear();
-                    fmt::format_to(error_msg, "{}", "jsonb with size 0 is 
invalid");
-                    RETURN_IF_ERROR(set_invalid_and_append_error_msg(j));
-                }
-            }
-        }
-        break;
-    }
-    case TYPE_DECIMALV2: {
-        // column_decimal utilizes the ColumnPtr from the block* block in 
_validate_data and can be modified.
-        auto* column_decimal = const_cast<ColumnDecimal128V2*>(
-                assert_cast<const ColumnDecimal128V2*>(real_column_ptr.get()));
-        const auto& max_decimalv2 = _get_decimalv2_min_or_max<false>(type);
-        const auto& min_decimalv2 = _get_decimalv2_min_or_max<true>(type);
-        for (size_t j = 0; j < row_count; ++j) {
-            auto row = rows ? (*rows)[j] : j;
-            if (need_to_validate(j, row, _filter_map, null_map)) {
-                auto dec_val = binary_cast<Int128, 
DecimalV2Value>(column_decimal->get_data()[j]);
-                bool invalid = false;
-
-                if (dec_val.greater_than_scale(type->get_scale())) {
-                    auto code =
-                            dec_val.round(&dec_val, 
remove_nullable(type)->get_scale(), HALF_UP);
-                    column_decimal->get_data()[j] = dec_val;
-
-                    if (code != E_DEC_OK) {
-                        fmt::format_to(error_msg, "round one decimal 
failed.value={}; ",
-                                       dec_val.to_string());
-                        invalid = true;
-                    }
-                }
-                if (dec_val > max_decimalv2 || dec_val < min_decimalv2) {
-                    fmt::format_to(error_msg, "{}", "decimal value is not 
valid for definition");
-                    fmt::format_to(error_msg, ", value={}", 
dec_val.to_string());
-                    fmt::format_to(error_msg, ", precision={}, scale={}", 
type->get_precision(),
-                                   type->get_scale());
-                    fmt::format_to(error_msg, ", min={}, max={}; ", 
min_decimalv2.to_string(),
-                                   max_decimalv2.to_string());
-                    invalid = true;
-                }
-
-                if (invalid) {
-                    RETURN_IF_ERROR(set_invalid_and_append_error_msg(row));
-                }
-            }
-        }
-        break;
-    }
-    case TYPE_DECIMAL32: {
-#define CHECK_VALIDATION_FOR_DECIMALV3(DecimalType)                            
                   \
-    auto column_decimal =                                                      
                   \
-            assert_cast<const 
ColumnDecimal<DecimalType::PType>*>(real_column_ptr.get());         \
-    const auto& max_decimal = _get_decimalv3_min_or_max<DecimalType, 
false>(type);                \
-    const auto& min_decimal = _get_decimalv3_min_or_max<DecimalType, 
true>(type);                 \
-    const auto* __restrict datas = column_decimal->get_data().data();          
                   \
-    int invalid_count = 0;                                                     
                   \
-    for (int j = 0; j < row_count; ++j) {                                      
                   \
-        const auto dec_val = datas[j];                                         
                   \
-        invalid_count += dec_val > max_decimal || dec_val < min_decimal;       
                   \
-    }                                                                          
                   \
-    if (invalid_count) {                                                       
                   \
-        for (size_t j = 0; j < row_count; ++j) {                               
                   \
-            auto row = rows ? (*rows)[j] : j;                                  
                   \
-            if (need_to_validate(j, row, _filter_map, null_map)) {             
                   \
-                auto dec_val = column_decimal->get_data()[j];                  
                   \
-                bool invalid = false;                                          
                   \
-                if (dec_val > max_decimal || dec_val < min_decimal) {          
                   \
-                    fmt::format_to(error_msg, "{}", "decimal value is not 
valid for definition"); \
-                    fmt::format_to(error_msg, ", value={}", dec_val.value);    
                   \
-                    fmt::format_to(error_msg, ", precision={}, scale={}", 
type->get_precision(),  \
-                                   type->get_scale());                         
                   \
-                    fmt::format_to(error_msg, ", min={}, max={}; ", 
min_decimal.value,            \
-                                   max_decimal.value);                         
                   \
-                    invalid = true;                                            
                   \
-                }                                                              
                   \
-                if (invalid) {                                                 
                   \
-                    RETURN_IF_ERROR(set_invalid_and_append_error_msg(row));    
                   \
-                }                                                              
                   \
-            }                                                                  
                   \
-        }                                                                      
                   \
-    }
-        CHECK_VALIDATION_FOR_DECIMALV3(Decimal32);
-        break;
-    }
-    case TYPE_DECIMAL64: {
-        CHECK_VALIDATION_FOR_DECIMALV3(Decimal64);
-        break;
-    }
-    case TYPE_DECIMAL128I: {
-        CHECK_VALIDATION_FOR_DECIMALV3(Decimal128V3);
-        break;
-    }
-    case TYPE_DECIMAL256: {
-        CHECK_VALIDATION_FOR_DECIMALV3(Decimal256);
-        break;
-    }
-#undef CHECK_VALIDATION_FOR_DECIMALV3
-    case TYPE_ARRAY: {
-        const auto* column_array = assert_cast<const 
ColumnArray*>(real_column_ptr.get());
-        const auto* type_array = assert_cast<const 
DataTypeArray*>(remove_nullable(type).get());
-        auto nested_type = type_array->get_nested_type();
-        const auto& offsets = column_array->get_offsets();
-        IColumn::Permutation permutation(offsets.back());
-        for (size_t r = 0; r < row_count; ++r) {
-            for (size_t c = offsets[r - 1]; c < offsets[r]; ++c) {
-                permutation[c] = rows ? (*rows)[r] : r;
-            }
-        }
-        fmt::format_to(error_prefix, "ARRAY type failed: ");
-        auto data_column_ptr = column_array->get_data_ptr();
-        switch (nested_type->get_primitive_type()) {
-        case TYPE_CHAR:
-        case TYPE_VARCHAR:
-        case TYPE_STRING: {
-            RETURN_IF_ERROR(
-                    string_column_checker(data_column_ptr, nested_type, 
&permutation, _filter_map));
-            const_cast<ColumnArray*>(column_array)->get_data_ptr() = 
std::move(data_column_ptr);
-            break;
-        }
-        default:
-            RETURN_IF_ERROR(_validate_column(state, block, nested_type, 
data_column_ptr, slot_index,
-                                             error_prefix, permutation.size(), 
&permutation));
-            break;
-        }
-        break;
-    }
-    case TYPE_MAP: {
-        const auto* column_map = assert_cast<const 
ColumnMap*>(real_column_ptr.get());
-        // column_map utilizes the ColumnPtr from the block* block in 
_validate_data and can be modified.
-        
RETURN_IF_ERROR((const_cast<ColumnMap*>(column_map))->deduplicate_keys(true));
-
-        const auto* type_map = assert_cast<const 
DataTypeMap*>(remove_nullable(type).get());
-        auto key_type = type_map->get_key_type();
-        auto val_type = type_map->get_value_type();
-        const auto& offsets = column_map->get_offsets();
-        IColumn::Permutation permutation(offsets.back());
-        for (size_t r = 0; r < row_count; ++r) {
-            for (size_t c = offsets[r - 1]; c < offsets[r]; ++c) {
-                permutation[c] = rows ? (*rows)[r] : r;
-            }
-        }
-
-        fmt::format_to(error_prefix, "MAP type failed: ");
-        switch (key_type->get_primitive_type()) {
-        case TYPE_CHAR:
-        case TYPE_VARCHAR:
-        case TYPE_STRING: {
-            auto key_column_ptr = column_map->get_keys_ptr();
-            RETURN_IF_ERROR(
-                    string_column_checker(key_column_ptr, key_type, 
&permutation, _filter_map));
-            const_cast<ColumnMap*>(column_map)->get_keys_ptr() = 
std::move(key_column_ptr);
-            break;
-        }
-        default:
-            RETURN_IF_ERROR(_validate_column(state, block, key_type, 
column_map->get_keys_ptr(),
-                                             slot_index, error_prefix, 
permutation.size(),
-                                             &permutation));
-            break;
-        }
-
-        switch (val_type->get_primitive_type()) {
-        case TYPE_CHAR:
-        case TYPE_VARCHAR:
-        case TYPE_STRING: {
-            auto value_column_ptr = column_map->get_values_ptr();
-            RETURN_IF_ERROR(
-                    string_column_checker(value_column_ptr, val_type, 
&permutation, _filter_map));
-            const_cast<ColumnMap*>(column_map)->get_values_ptr() = 
std::move(value_column_ptr);
-            break;
-        }
-        default:
-            RETURN_IF_ERROR(_validate_column(state, block, val_type, 
column_map->get_values_ptr(),
-                                             slot_index, error_prefix, 
permutation.size(),
-                                             &permutation));
-            break;
-        }
-        break;
-    }
-    case TYPE_STRUCT: {
-        const auto column_struct = assert_cast<const 
ColumnStruct*>(real_column_ptr.get());
-        const auto* type_struct = assert_cast<const 
DataTypeStruct*>(remove_nullable(type).get());
-        DCHECK(type_struct->get_elements().size() == 
column_struct->tuple_size());
-        fmt::format_to(error_prefix, "STRUCT type failed: ");
-        for (size_t sc = 0; sc < column_struct->tuple_size(); ++sc) {
-            auto element_type = type_struct->get_element(sc);
-            switch (element_type->get_primitive_type()) {
-            case TYPE_CHAR:
-            case TYPE_VARCHAR:
-            case TYPE_STRING: {
-                auto element_column_ptr = column_struct->get_column_ptr(sc);
-                RETURN_IF_ERROR(string_column_checker(element_column_ptr, 
element_type, nullptr,
-                                                      _filter_map));
-                const_cast<ColumnStruct*>(column_struct)->get_column_ptr(sc) =
-                        std::move(element_column_ptr);
-                break;
-            }
-            default:
-                RETURN_IF_ERROR(_validate_column(state, block, 
type_struct->get_element(sc),
-                                                 
column_struct->get_column_ptr(sc), slot_index,
-                                                 error_prefix,
-                                                 
column_struct->get_column_ptr(sc)->size()));
-                break;
-            }
-        }
-        break;
-    }
-    case TYPE_AGG_STATE: {
-        auto* column_string = 
check_and_get_column<ColumnString>(*real_column_ptr);
-        if (column_string) {
-            RETURN_IF_ERROR(string_column_checker(column, type, rows, 
_filter_map));
-        }
-        break;
-    }
-    default:
-        break;
-    }
-
-    // Dispose the column should do not contain the NULL value
-    // Only two case:
-    // 1. column is nullable but the desc is not nullable
-    // 2. desc->type is BITMAP
-    if ((!type->is_nullable() || type->get_primitive_type() == TYPE_BITMAP) && 
column_ptr) {
-        for (int j = 0; j < row_count; ++j) {
-            auto row = rows ? (*rows)[j] : j;
-            if (null_map[j] && !_filter_map[row]) {
-                fmt::format_to(error_msg, "null value for not null column, 
type={}",
-                               type->get_name());
-                RETURN_IF_ERROR(set_invalid_and_append_error_msg(row));
-            }
-        }
-    }
-
-    return Status::OK();
-}
-
-Status OlapTableBlockConvertor::_validate_data(RuntimeState* state, Block* 
block, const size_t rows,
+Status OlapTableBlockConvertor::_validate_data(Block* block, const size_t rows,
                                                int& filtered_rows) {
     filtered_rows = 0;
     Defer defer {[&] {
-        for (int i = 0; i < rows; ++i) {
+        for (size_t i = 0; i < rows; ++i) {
             filtered_rows += _filter_map[i];
         }
     }};
@@ -595,12 +129,13 @@ Status 
OlapTableBlockConvertor::_validate_data(RuntimeState* state, Block* block
         // VRowDistribution's _deal_missing_map() relies on these slots being 
full columns.
         block->get_by_position(i).column =
                 
block->get_by_position(i).column->convert_to_full_column_if_const();
-        const auto& column = block->get_by_position(i).column;
+        auto& column = block->get_by_position(i).column;
 
-        fmt::memory_buffer error_prefix;
-        fmt::format_to(error_prefix, "column_name[{}], ", desc->col_name());
-        RETURN_IF_ERROR(
-                _validate_column(state, block, desc->type(), column, i, 
error_prefix, rows));
+        DCHECK_EQ(column->size(), rows);
+        _tablet_validate_context.error_prefix.clear();
+        fmt::format_to(_tablet_validate_context.error_prefix, 
"column_name[{}], ",
+                       desc->col_name());
+        column = DORIS_TRY(tablet_validate_column(column, desc->type(), 
_tablet_validate_context));
     }
     return Status::OK();
 }
diff --git a/be/src/exec/sink/vtablet_block_convertor.h 
b/be/src/exec/sink/vtablet_block_convertor.h
index f529b7da869..afd2f1136a7 100644
--- a/be/src/exec/sink/vtablet_block_convertor.h
+++ b/be/src/exec/sink/vtablet_block_convertor.h
@@ -22,14 +22,13 @@
 // IWYU pragma: no_include <bits/chrono.h>
 
 #include <chrono> // IWYU pragma: keep
-#include <map>
 
 #include "common/status.h"
 #include "core/block/block.h"
 #include "core/column/column.h"
 #include "core/data_type/data_type_factory.hpp"
-#include "core/value/decimalv2_value.h"
 #include "exec/sink/autoinc_buffer.h"
+#include "exec/sink/tablet_column_validator.h"
 #include "exprs/vexpr_fwd.h"
 #include "runtime/descriptors.h"
 #include "util/bitmap.h"
@@ -39,7 +38,7 @@ namespace doris {
 class OlapTableBlockConvertor {
 public:
     OlapTableBlockConvertor(TupleDescriptor* output_tuple_desc)
-            : _output_tuple_desc(output_tuple_desc) {}
+            : _output_tuple_desc(output_tuple_desc), 
_tablet_validate_context(_filter_map) {}
 
     Status validate_and_convert_block(RuntimeState* state, Block* input_block,
                                       std::shared_ptr<Block>& block,
@@ -59,30 +58,10 @@ public:
     AutoIncIDAllocator& auto_inc_id_allocator() { return 
_auto_inc_id_allocator; }
 
 private:
-    template <bool is_min>
-    DecimalV2Value _get_decimalv2_min_or_max(const DataTypePtr& type);
-
-    template <typename DecimalType, bool IsMin>
-    DecimalType _get_decimalv3_min_or_max(const DataTypePtr& type);
-
-    Status _validate_column(RuntimeState* state, Block* block, const 
DataTypePtr& type,
-                            ColumnPtr column, size_t slot_index, 
fmt::memory_buffer& error_prefix,
-                            const size_t row_count, IColumn::Permutation* rows 
= nullptr) {
-        RETURN_IF_CATCH_EXCEPTION({
-            return _internal_validate_column(state, block, type, column, 
slot_index, error_prefix,
-                                             row_count, rows);
-        });
-    }
-
-    Status _internal_validate_column(RuntimeState* state, Block* block, const 
DataTypePtr& type,
-                                     ColumnPtr column, size_t slot_index,
-                                     fmt::memory_buffer& error_prefix, const 
size_t row_count,
-                                     IColumn::Permutation* rows = nullptr);
-
     // make input data valid for OLAP table
     // return number of invalid/filtered rows.
     // invalid row number is set in Bitmap
-    Status _validate_data(RuntimeState* state, Block* block, const size_t 
rows, int& filtered_rows);
+    Status _validate_data(Block* block, const size_t rows, int& filtered_rows);
 
     // some output column of output expr may have different nullable property 
with dest slot desc
     // so here need to do the convert operation
@@ -94,20 +73,10 @@ private:
 
     TupleDescriptor* _output_tuple_desc = nullptr;
 
-    std::map<std::pair<int, int>, DecimalV2Value> _max_decimalv2_val;
-    std::map<std::pair<int, int>, DecimalV2Value> _min_decimalv2_val;
-
-    std::map<int, int32_t> _max_decimal32_val;
-    std::map<int, int32_t> _min_decimal32_val;
-    std::map<int, int64_t> _max_decimal64_val;
-    std::map<int, int64_t> _min_decimal64_val;
-    std::map<int, int128_t> _max_decimal128_val;
-    std::map<int, int128_t> _min_decimal128_val;
-    std::map<int, wide::Int256> _max_decimal256_val;
-    std::map<int, wide::Int256> _min_decimal256_val;
-
     std::vector<char> _filter_map;
 
+    TabletValidateContext _tablet_validate_context;
+
     int64_t _validate_data_ns = 0;
     int64_t _num_filtered_rows = 0;
 
diff --git a/be/test/exec/sink/tablet_column_validator_test.cpp 
b/be/test/exec/sink/tablet_column_validator_test.cpp
new file mode 100644
index 00000000000..b6b954a8a5b
--- /dev/null
+++ b/be/test/exec/sink/tablet_column_validator_test.cpp
@@ -0,0 +1,511 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "exec/sink/tablet_column_validator.h"
+
+#include <gtest/gtest.h>
+
+#include <initializer_list>
+#include <memory>
+#include <string>
+#include <vector>
+
+#include "agent/be_exec_version_manager.h"
+#include "core/assert_cast.h"
+#include "core/column/column_array.h"
+#include "core/column/column_decimal.h"
+#include "core/column/column_map.h"
+#include "core/column/column_nullable.h"
+#include "core/column/column_string.h"
+#include "core/column/column_struct.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_decimal.h"
+#include "core/data_type/data_type_factory.hpp"
+#include "core/data_type/data_type_jsonb.h"
+#include "core/data_type/data_type_map.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_number.h"
+#include "core/data_type/data_type_string.h"
+#include "core/data_type/data_type_struct.h"
+#include "runtime/runtime_state.h"
+#include "storage/olap_common.h"
+
+namespace doris {
+namespace {
+
+// Capture leaf storage without keeping owning pointers. Complex shells may 
change.
+struct StorageSnapshot {
+    const IColumn* column;
+    const void* data;
+};
+
+std::vector<StorageSnapshot> snapshot(const IColumn& column) {
+    std::vector<StorageSnapshot> result;
+    const auto visit = [&](const auto& self, const IColumn& current) -> void {
+        const void* data = nullptr;
+        if (const auto* ints = check_and_get_column<ColumnInt32>(current)) {
+            data = ints->get_data().data();
+        } else if (const auto* bytes = 
check_and_get_column<ColumnUInt8>(current)) {
+            data = bytes->get_data().data();
+        } else if (const auto* offsets =
+                           
check_and_get_column<ColumnArray::ColumnOffsets>(current)) {
+            data = offsets->get_data().data();
+        } else if (const auto* strings = 
check_and_get_column<ColumnString>(current)) {
+            data = strings->get_chars().data();
+        } else if (const auto* decimals = 
check_and_get_column<ColumnDecimal128V2>(current)) {
+            data = decimals->get_data().data();
+        }
+        result.push_back({&current, data});
+        if (const auto* strings = check_and_get_column<ColumnString>(current)) 
{
+            result.push_back({&current, strings->get_offsets().data()});
+        } else if (const auto* nullable = 
check_and_get_column<ColumnNullable>(current)) {
+            self(self, nullable->get_nested_column());
+            self(self, nullable->get_null_map_column());
+        } else if (const auto* array = 
check_and_get_column<ColumnArray>(current)) {
+            self(self, array->get_data());
+            self(self, array->get_offsets_column());
+        } else if (const auto* map = check_and_get_column<ColumnMap>(current)) 
{
+            self(self, map->get_keys());
+            self(self, map->get_values());
+            self(self, *map->get_offsets_ptr());
+        } else if (const auto* tuple = 
check_and_get_column<ColumnStruct>(current)) {
+            for (size_t i = 0; i < tuple->tuple_size(); ++i) {
+                self(self, tuple->get_column(i));
+            }
+        }
+    };
+    visit(visit, column);
+    return result;
+}
+
+void expect_same_storage(const std::vector<StorageSnapshot>& before, const 
IColumn& column) {
+    const auto after = snapshot(column);
+    ASSERT_EQ(before.size(), after.size());
+    for (size_t i = 0; i < before.size(); ++i) {
+        if (before[i].data) {
+            EXPECT_EQ(before[i].column, after[i].column) << "leaf " << i;
+            EXPECT_EQ(before[i].data, after[i].data) << "leaf " << i;
+        }
+    }
+}
+
+ColumnInt32::MutablePtr ints(std::initializer_list<Int32> values) {
+    auto column = ColumnInt32::create();
+    for (auto value : values) {
+        column->insert_value(value);
+    }
+    return column;
+}
+
+ColumnArray::ColumnOffsets::MutablePtr offsets(std::initializer_list<UInt64> 
values) {
+    auto column = ColumnArray::ColumnOffsets::create();
+    for (auto value : values) {
+        column->insert_value(value);
+    }
+    return column;
+}
+
+ColumnString::MutablePtr strings(std::initializer_list<std::string> values) {
+    auto column = ColumnString::create();
+    for (const auto& value : values) {
+        column->insert_data(value.data(), value.size());
+    }
+    return column;
+}
+
+DataTypePtr decimal_v2_type() {
+    return 
DataTypeFactory::instance().create_data_type(FieldType::OLAP_FIELD_TYPE_DECIMAL,
 9, 2);
+}
+
+TQueryOptions options(bool strict) {
+    TQueryOptions options;
+    options.__set_enable_insert_strict(strict);
+    options.__set_be_exec_version(BeExecVersionManager::get_newest_version());
+    return options;
+}
+
+} // namespace
+
+class TabletColumnValidatorTest : public testing::Test {
+protected:
+    TabletColumnValidatorTest() : state(options(true), TQueryGlobals {}), 
context(filter_map) {
+        context.state = &state;
+    }
+
+    Result<ColumnPtr> validate(const ColumnPtr& column, const DataTypePtr& 
type) {
+        filter_map.assign(column->size(), 0);
+        return tablet_validate_column(column, type, context);
+    }
+
+    RuntimeState state;
+    std::vector<char> filter_map;
+    TabletValidateContext context;
+};
+
+TEST_F(TabletColumnValidatorTest, 
UniqueScalarKeepsObjectBufferAndReferenceCount) {
+    ColumnPtr column = ints({7, 8});
+    auto before = snapshot(*column);
+    column = TEST_TRY(validate(column, std::make_shared<DataTypeInt32>()));
+    expect_same_storage(before, *column);
+    EXPECT_EQ(column->use_count(), 1);
+}
+
+TEST_F(TabletColumnValidatorTest, NullableArrayStructReusesLeafStorage) {
+    ColumnPtr field = ColumnNullable::create(ints({7, 8}), 
ColumnUInt8::create(2, 0));
+    auto tuple = ColumnStruct::create(Columns {std::move(field), strings({"a", 
"b"})});
+    auto array = ColumnArray::create(std::move(tuple), offsets({0, 2}));
+    ColumnPtr column = ColumnNullable::create(std::move(array), 
ColumnUInt8::create(2, 0));
+    auto tuple_type = std::make_shared<DataTypeStruct>(DataTypes {
+            make_nullable(std::make_shared<DataTypeInt32>()), 
std::make_shared<DataTypeString>()});
+    auto type = make_nullable(std::make_shared<DataTypeArray>(tuple_type));
+    auto before = snapshot(*column);
+    column = TEST_TRY(validate(column, type));
+    expect_same_storage(before, *column);
+    EXPECT_EQ(filter_map, (std::vector<char> {0, 0}));
+}
+
+TEST_F(TabletColumnValidatorTest, MapWithoutDuplicatesReusesLeafStorage) {
+    ColumnPtr column = ColumnMap::create(ints({1, 2}), ints({10, 20}), 
offsets({2}));
+    auto type = 
std::make_shared<DataTypeMap>(std::make_shared<DataTypeInt32>(),
+                                              
std::make_shared<DataTypeInt32>());
+    auto before = snapshot(*column);
+    column = TEST_TRY(validate(column, type));
+    expect_same_storage(before, *column);
+}
+
+TEST_F(TabletColumnValidatorTest, StructReusesSharedReadOnlyFields) {
+    ColumnPtr shared = ints({7});
+    ColumnPtr column = ColumnStruct::create(Columns {shared, ints({8}), 
strings({"a"})});
+    const auto& tuple_before = assert_cast<const ColumnStruct&>(*column);
+    auto second_before = snapshot(tuple_before.get_column(1));
+    auto third_before = snapshot(tuple_before.get_column(2));
+    const auto* parent_before = column.get();
+    const auto* shared_data = assert_cast<const 
ColumnInt32&>(*shared).get_data().data();
+    auto type = std::make_shared<DataTypeStruct>(DataTypes 
{std::make_shared<DataTypeInt32>(),
+                                                            
std::make_shared<DataTypeInt32>(),
+                                                            
std::make_shared<DataTypeString>()});
+
+    column = TEST_TRY(validate(column, type));
+    const auto& tuple = assert_cast<const ColumnStruct&>(*column);
+    EXPECT_NE(column.get(), parent_before);
+    EXPECT_EQ(&tuple.get_column(0), shared.get());
+    EXPECT_EQ(assert_cast<const 
ColumnInt32&>(tuple.get_column(0)).get_data().data(), shared_data);
+    EXPECT_EQ(assert_cast<const 
ColumnInt32&>(tuple.get_column(0)).get_data()[0], 7);
+    EXPECT_EQ(shared->use_count(), 2);
+    expect_same_storage(second_before, tuple.get_column(1));
+    expect_same_storage(third_before, tuple.get_column(2));
+}
+
+TEST_F(TabletColumnValidatorTest, 
DecimalV2PreservesLegacyScaleAndUniqueStorage) {
+    auto decimals = ColumnDecimal128V2::create(0, 9);
+    
decimals->get_data().push_back(DecimalV2Value(std::string("1.235")).value());
+    ColumnPtr column = std::move(decimals);
+    auto before = snapshot(*column);
+    column = TEST_TRY(validate(column, decimal_v2_type()));
+    expect_same_storage(before, *column);
+    EXPECT_EQ(assert_cast<const 
ColumnDecimal128V2&>(*column).get_data()[0].value(),
+              DecimalV2Value(std::string("1.235")).value());
+}
+
+TEST_F(TabletColumnValidatorTest, 
DecimalV2PrecisionEqualsScalePreservesLegacyValue) {
+    auto decimals = ColumnDecimal128V2::create(0, 9);
+    
decimals->get_data().push_back(DecimalV2Value(std::string("1.235")).value());
+    ColumnPtr column = std::move(decimals);
+    auto before = snapshot(*column);
+    // DECIMAL(2,2) still uses DecimalV2 execution precision 27 and scale 9.
+    auto type =
+            
DataTypeFactory::instance().create_data_type(FieldType::OLAP_FIELD_TYPE_DECIMAL,
 2, 2);
+    column = TEST_TRY(validate(column, type));
+    expect_same_storage(before, *column);
+    EXPECT_EQ(assert_cast<const 
ColumnDecimal128V2&>(*column).get_data()[0].value(),
+              DecimalV2Value(std::string("1.235")).value());
+}
+
+TEST_F(TabletColumnValidatorTest, SharedNullableDecimalV2PreservesLegacyValue) 
{
+    auto decimals = ColumnDecimal128V2::create(0, 9);
+    
decimals->get_data().push_back(DecimalV2Value(std::string("1.235")).value());
+    ColumnPtr column = ColumnNullable::create(std::move(decimals), 
ColumnUInt8::create(1, 0));
+    ColumnPtr original = column;
+    auto before = snapshot(*original);
+    column = TEST_TRY(validate(column, make_nullable(decimal_v2_type())));
+    const auto& result = assert_cast<const ColumnNullable&>(*column);
+    EXPECT_EQ(assert_cast<const 
ColumnDecimal128V2&>(result.get_nested_column())
+                      .get_data()[0]
+                      .value(),
+              DecimalV2Value(std::string("1.235")).value());
+    const auto& old = assert_cast<const ColumnNullable&>(*original);
+    EXPECT_EQ(assert_cast<const 
ColumnDecimal128V2&>(old.get_nested_column()).get_data()[0].value(),
+              DecimalV2Value(std::string("1.235")).value());
+    expect_same_storage(before, *original);
+}
+
+TEST_F(TabletColumnValidatorTest, SharedMapColumnIsIsolated) {
+    ColumnPtr column = ColumnMap::create(ints({7, 7}), ints({10, 20}), 
offsets({2}));
+    ColumnPtr original = column;
+    auto type = 
std::make_shared<DataTypeMap>(std::make_shared<DataTypeInt32>(),
+                                              
std::make_shared<DataTypeInt32>());
+    column = TEST_TRY(validate(column, type));
+    const auto& result = assert_cast<const ColumnMap&>(*column);
+    const auto& old = assert_cast<const ColumnMap&>(*original);
+    EXPECT_EQ(result.get_keys().size(), 1);
+    EXPECT_EQ(assert_cast<const 
ColumnInt32&>(result.get_values()).get_data()[0], 20);
+    EXPECT_EQ(old.get_keys().size(), 2);
+    EXPECT_EQ(old.get_values().size(), 2);
+    EXPECT_EQ(old.get_offsets()[0], 2);
+}
+
+TEST_F(TabletColumnValidatorTest, StructInsideArrayReportsOriginalRow) {
+    auto tuple = ColumnStruct::create(Columns {strings({""})});
+    ColumnPtr column = ColumnArray::create(std::move(tuple), offsets({0, 1}));
+    auto tuple_type =
+            std::make_shared<DataTypeStruct>(DataTypes 
{std::make_shared<DataTypeJsonb>()});
+    column = TEST_TRY(validate(column, 
std::make_shared<DataTypeArray>(tuple_type)));
+    EXPECT_EQ(filter_map, (std::vector<char> {0, 1}));
+    EXPECT_EQ(context.error_prefix.size(), 0);
+}
+
+TEST_F(TabletColumnValidatorTest, NullArraySkipsInvalidElementPayload) {
+    auto array = ColumnArray::create(strings({"", ""}), offsets({1, 2}));
+    auto nulls = ColumnUInt8::create();
+    nulls->insert_value(1);
+    nulls->insert_value(0);
+    ColumnPtr column = ColumnNullable::create(std::move(array), 
std::move(nulls));
+    auto type = 
make_nullable(std::make_shared<DataTypeArray>(std::make_shared<DataTypeJsonb>()));
+    column = TEST_TRY(validate(column, type));
+    EXPECT_EQ(filter_map, (std::vector<char> {0, 1}));
+}
+
+TEST_F(TabletColumnValidatorTest, ParentAndChildNullMapsAreCombined) {
+    ColumnPtr child = ColumnNullable::create(strings({"", "", ""}), 
ColumnUInt8::create(3, 0));
+    auto& child_nulls =
+            
assert_cast<ColumnNullable&>(child->assert_mutable_ref()).get_null_map_data();
+    child_nulls[1] = 1;
+    auto tuple = ColumnStruct::create(Columns {std::move(child)});
+    auto parent_nulls = ColumnUInt8::create();
+    parent_nulls->insert_value(1);
+    parent_nulls->insert_value(0);
+    parent_nulls->insert_value(0);
+    ColumnPtr column = ColumnNullable::create(std::move(tuple), 
std::move(parent_nulls));
+    auto type = make_nullable(std::make_shared<DataTypeStruct>(
+            DataTypes {make_nullable(std::make_shared<DataTypeJsonb>())}));
+    column = TEST_TRY(validate(column, type));
+    EXPECT_EQ(filter_map, (std::vector<char> {0, 0, 1}));
+}
+
+TEST_F(TabletColumnValidatorTest, NullParentSkipsNotNullChildViolation) {
+    ColumnPtr child = ColumnNullable::create(ints({0}), ColumnUInt8::create(1, 
1));
+    auto tuple = ColumnStruct::create(Columns {std::move(child)});
+    ColumnPtr column = ColumnNullable::create(std::move(tuple), 
ColumnUInt8::create(1, 1));
+    auto type = make_nullable(
+            std::make_shared<DataTypeStruct>(DataTypes 
{std::make_shared<DataTypeInt32>()}));
+    column = TEST_TRY(validate(column, type));
+    EXPECT_EQ(filter_map, (std::vector<char> {0}));
+}
+
+TEST_F(TabletColumnValidatorTest, NonStrictTruncationReplacesOnlyNestedString) 
{
+    RuntimeState non_strict(options(false), TQueryGlobals {});
+    context.state = &non_strict;
+    ColumnPtr column = ColumnNullable::create(strings({"abcd"}), 
ColumnUInt8::create(1, 0));
+    const auto* original_column = column.get();
+    const auto& old = assert_cast<const ColumnNullable&>(*column);
+    const auto* original_nulls = old.get_null_map_data().data();
+    auto type = make_nullable(std::make_shared<DataTypeString>(2, 
TYPE_VARCHAR));
+    column = TEST_TRY(validate(column, type));
+    const auto& result = assert_cast<const ColumnNullable&>(*column);
+    EXPECT_NE(column.get(), original_column);
+    EXPECT_EQ(result.get_null_map_data().data(), original_nulls);
+    EXPECT_EQ(result.get_nested_column().get_data_at(0).to_string(), "ab");
+    EXPECT_EQ(filter_map, (std::vector<char> {0}));
+}
+
+TEST_F(TabletColumnValidatorTest, StrictStringLengthFailurePreservesStorage) {
+    ColumnPtr column = strings({"abcd"});
+    auto before = snapshot(*column);
+    column = TEST_TRY(validate(column, std::make_shared<DataTypeString>(2, 
TYPE_VARCHAR)));
+    expect_same_storage(before, *column);
+    EXPECT_EQ(filter_map, (std::vector<char> {1}));
+}
+
+TEST_F(TabletColumnValidatorTest, EmptyNestedColumnsAreValid) {
+    ColumnPtr column = ColumnArray::create(strings({}), offsets({0, 0}));
+    auto before = snapshot(*column);
+    auto type = 
std::make_shared<DataTypeArray>(std::make_shared<DataTypeJsonb>());
+    column = TEST_TRY(validate(column, type));
+    expect_same_storage(before, *column);
+    EXPECT_EQ(filter_map, (std::vector<char> {0, 0}));
+}
+
+TEST_F(TabletColumnValidatorTest, NestedNullableMapReusesLeafStorage) {
+    auto inner = ColumnMap::create(ints({3, 4}), ints({30, 40}), offsets({2}));
+    auto nullable = ColumnNullable::create(std::move(inner), 
ColumnUInt8::create(1, 0));
+    ColumnPtr column = ColumnMap::create(ints({1}), std::move(nullable), 
offsets({1}));
+    auto inner_type = 
std::make_shared<DataTypeMap>(std::make_shared<DataTypeInt32>(),
+                                                    
std::make_shared<DataTypeInt32>());
+    auto type = 
std::make_shared<DataTypeMap>(std::make_shared<DataTypeInt32>(),
+                                              make_nullable(inner_type));
+    auto before = snapshot(*column);
+    column = TEST_TRY(validate(column, type));
+    expect_same_storage(before, *column);
+}
+
+TEST_F(TabletColumnValidatorTest, 
SharedNestedNullableMapDeduplicatesWithoutChangingSource) {
+    auto inner = ColumnMap::create(ints({3, 3}), ints({30, 40}), offsets({2}));
+    auto nullable = ColumnNullable::create(std::move(inner), 
ColumnUInt8::create(1, 0));
+    ColumnPtr column = ColumnMap::create(ints({1}), std::move(nullable), 
offsets({1}));
+    ColumnPtr original = column;
+    auto inner_type = 
std::make_shared<DataTypeMap>(std::make_shared<DataTypeInt32>(),
+                                                    
std::make_shared<DataTypeInt32>());
+    auto type = 
std::make_shared<DataTypeMap>(std::make_shared<DataTypeInt32>(),
+                                              make_nullable(inner_type));
+    column = TEST_TRY(validate(column, type));
+    const auto& result = assert_cast<const ColumnMap&>(*column);
+    const auto& result_nullable = assert_cast<const 
ColumnNullable&>(result.get_values());
+    const auto& result_inner = assert_cast<const 
ColumnMap&>(result_nullable.get_nested_column());
+    EXPECT_EQ(result_inner.get_keys().size(), 1);
+    EXPECT_EQ(assert_cast<const 
ColumnInt32&>(result_inner.get_values()).get_data()[0], 40);
+    const auto& old = assert_cast<const ColumnMap&>(*original);
+    const auto& old_nullable = assert_cast<const 
ColumnNullable&>(old.get_values());
+    EXPECT_EQ(assert_cast<const 
ColumnMap&>(old_nullable.get_nested_column()).get_keys().size(), 2);
+}
+
+TEST_F(TabletColumnValidatorTest, MapDeduplicationRebuildsElementRowMapping) {
+    ColumnPtr column =
+            ColumnMap::create(ints({1, 1, 2}), strings({"", "valid", ""}), 
offsets({2, 3}));
+    auto type = 
std::make_shared<DataTypeMap>(std::make_shared<DataTypeInt32>(),
+                                              
std::make_shared<DataTypeJsonb>());
+    column = TEST_TRY(validate(column, type));
+    EXPECT_EQ(filter_map, (std::vector<char> {0, 1}));
+    const auto& result = assert_cast<const ColumnMap&>(*column);
+    EXPECT_EQ(result.get_offsets()[0], 1);
+    EXPECT_EQ(result.get_offsets()[1], 2);
+}
+
+TEST_F(TabletColumnValidatorTest, NotNullRejectsNullRow) {
+    ColumnPtr column = ColumnNullable::create(ints({0}), 
ColumnUInt8::create(1, 1));
+    column = TEST_TRY(validate(column, std::make_shared<DataTypeInt32>()));
+    EXPECT_EQ(filter_map, (std::vector<char> {1}));
+}
+
+TEST_F(TabletColumnValidatorTest, DecimalV3RangeChecksAllWidths) {
+    const auto check = [this]<PrimitiveType T>() {
+        using ColumnType = ColumnDecimal<T>;
+        using Decimal = typename PrimitiveTypeTraits<T>::CppType;
+        auto values = ColumnType::create(0, 0);
+        values->get_data().push_back(Decimal(9));
+        values->get_data().push_back(Decimal(10));
+        values->get_data().push_back(Decimal(-10));
+        ColumnPtr column = std::move(values);
+        const auto* object = column.get();
+        const auto* data = assert_cast<const 
ColumnType&>(*column).get_data().data();
+        auto result = validate(column, std::make_shared<DataTypeDecimal<T>>(1, 
0));
+        ASSERT_TRUE(result.has_value()) << result.error();
+        EXPECT_EQ(result.value().get(), object);
+        EXPECT_EQ(assert_cast<const 
ColumnType&>(*result.value()).get_data().data(), data);
+        EXPECT_EQ(filter_map, (std::vector<char> {0, 1, 1}));
+    };
+    check.template operator()<TYPE_DECIMAL32>();
+    check.template operator()<TYPE_DECIMAL64>();
+    check.template operator()<TYPE_DECIMAL128I>();
+    check.template operator()<TYPE_DECIMAL256>();
+}
+
+TEST_F(TabletColumnValidatorTest, NonNullableInputAcceptsNullableTargetType) {
+    auto tuple = ColumnStruct::create(Columns {strings({"abc"})});
+    ColumnPtr column = ColumnArray::create(std::move(tuple), offsets({1}));
+    auto tuple_type = make_nullable(std::make_shared<DataTypeStruct>(
+            DataTypes {make_nullable(std::make_shared<DataTypeString>(2, 
TYPE_VARCHAR))}));
+    auto type = make_nullable(std::make_shared<DataTypeArray>(tuple_type));
+    auto before = snapshot(*column);
+    column = TEST_TRY(validate(column, type));
+    expect_same_storage(before, *column);
+    EXPECT_EQ(filter_map, (std::vector<char> {1}));
+}
+
+TEST_F(TabletColumnValidatorTest, 
NestedReplacementBuildsShellsAndPreservesSource) {
+    RuntimeState non_strict(options(false), TQueryGlobals {});
+    context.state = &non_strict;
+    auto nullable = ColumnNullable::create(strings({"abcd"}), 
ColumnUInt8::create(1, 0));
+    auto tuple = ColumnStruct::create(Columns {std::move(nullable), 
ints({7})});
+    ColumnPtr column = ColumnArray::create(std::move(tuple), offsets({1}));
+    const auto& array_before = assert_cast<const ColumnArray&>(*column);
+    const auto& tuple_before = assert_cast<const 
ColumnStruct&>(array_before.get_data());
+    const auto& nullable_before = assert_cast<const 
ColumnNullable&>(tuple_before.get_column(0));
+    ColumnPtr original = column;
+    const auto* array_address = column.get();
+    const auto* tuple_address = &tuple_before;
+    const auto* nullable_address = &nullable_before;
+    const auto* offsets_address = array_before.get_offsets().data();
+    const auto* nulls_address = nullable_before.get_null_map_data().data();
+    auto sibling_before = snapshot(tuple_before.get_column(1));
+    auto tuple_type = std::make_shared<DataTypeStruct>(
+            DataTypes {make_nullable(std::make_shared<DataTypeString>(2, 
TYPE_VARCHAR)),
+                       std::make_shared<DataTypeInt32>()});
+
+    column = TEST_TRY(validate(column, 
std::make_shared<DataTypeArray>(tuple_type)));
+
+    const auto& array = assert_cast<const ColumnArray&>(*column);
+    const auto& checked_tuple = assert_cast<const 
ColumnStruct&>(array.get_data());
+    const auto& checked_nullable = assert_cast<const 
ColumnNullable&>(checked_tuple.get_column(0));
+    EXPECT_NE(column.get(), array_address);
+    EXPECT_NE(&checked_tuple, tuple_address);
+    EXPECT_NE(&checked_nullable, nullable_address);
+    EXPECT_EQ(array.get_offsets().data(), offsets_address);
+    EXPECT_EQ(checked_nullable.get_null_map_data().data(), nulls_address);
+    EXPECT_EQ(checked_nullable.get_nested_column().get_data_at(0).to_string(), 
"ab");
+    EXPECT_EQ(checked_nullable.get_nested_column().use_count(), 1);
+    expect_same_storage(sibling_before, checked_tuple.get_column(1));
+    EXPECT_EQ(filter_map, (std::vector<char> {0}));
+    EXPECT_EQ(nullable_before.get_nested_column().get_data_at(0).to_string(), 
"abcd");
+    column->sanity_check();
+}
+
+TEST_F(TabletColumnValidatorTest, SharedReadOnlyLeavesKeepStorage) {
+    const auto check = [this](ColumnPtr original, const DataTypePtr& type) {
+        auto before = snapshot(*original);
+        auto result = TEST_TRY(validate(original, type));
+        EXPECT_EQ(result.get(), original.get());
+        expect_same_storage(before, *result);
+        EXPECT_EQ(original->use_count(), 2);
+    };
+    check(ints({7, 8}), std::make_shared<DataTypeInt32>());
+    check(strings({"valid"}), std::make_shared<DataTypeJsonb>());
+    auto values = ColumnDecimal32::create(0, 0);
+    values->get_data().push_back(Decimal32(1));
+    ColumnPtr original = std::move(values);
+    const auto* data = assert_cast<const 
ColumnDecimal32&>(*original).get_data().data();
+    auto result =
+            TEST_TRY(validate(original, 
std::make_shared<DataTypeDecimal<TYPE_DECIMAL32>>(1, 0)));
+    EXPECT_EQ(result.get(), original.get());
+    EXPECT_EQ(assert_cast<const ColumnDecimal32&>(*result).get_data().data(), 
data);
+}
+
+TEST_F(TabletColumnValidatorTest, SharedComplexTreeReusesReadOnlyLeaves) {
+    auto tuple = ColumnStruct::create(Columns {ints({7}), strings({"abc"})});
+    ColumnPtr original = 
ColumnNullable::create(ColumnArray::create(std::move(tuple), offsets({1})),
+                                                ColumnUInt8::create(1, 0));
+    auto tuple_type = std::make_shared<DataTypeStruct>(
+            DataTypes {std::make_shared<DataTypeInt32>(), 
std::make_shared<DataTypeString>()});
+    auto before = snapshot(*original);
+    auto result = TEST_TRY(
+            validate(original, 
make_nullable(std::make_shared<DataTypeArray>(tuple_type))));
+    EXPECT_NE(result.get(), original.get());
+    expect_same_storage(before, *result);
+    expect_same_storage(before, *original);
+}
+
+} // namespace doris
diff --git 
a/regression-test/data/datatype_p0/map/test_map_insert_shared_column.out 
b/regression-test/data/datatype_p0/map/test_map_insert_shared_column.out
new file mode 100644
index 00000000000..5303e0253aa
--- /dev/null
+++ b/regression-test/data/datatype_p0/map/test_map_insert_shared_column.out
@@ -0,0 +1,4 @@
+-- This file is automatically generated. You should know what you did if you 
want to edit this
+-- !shared_column --
+401    {1:"abc"}       {1:"abcdef"}
+402    {2:"uvw"}       {2:"uvwxyz"}
diff --git 
a/regression-test/suites/datatype_p0/map/test_map_insert_shared_column.groovy 
b/regression-test/suites/datatype_p0/map/test_map_insert_shared_column.groovy
new file mode 100644
index 00000000000..1623ecb7678
--- /dev/null
+++ 
b/regression-test/suites/datatype_p0/map/test_map_insert_shared_column.groovy
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("test_map_insert_shared_column", "p0") {
+    sql "drop table if exists test_map_insert_shared_column_src"
+    sql "drop table if exists test_map_insert_shared_column_dst"
+
+    sql """
+        create table test_map_insert_shared_column_src (
+            id int,
+            m map<int, string>
+        ) duplicate key(id)
+        distributed by hash(id) buckets 1
+        properties("replication_num" = "1")
+    """
+    sql """
+        create table test_map_insert_shared_column_dst (
+            id int,
+            m_short map<int, varchar(3)>,
+            m_full map<int, string>
+        ) duplicate key(id)
+        distributed by hash(id) buckets 1
+        properties("replication_num" = "1")
+    """
+    sql """
+        insert into test_map_insert_shared_column_src values
+            (401, {1: "abcdef"}),
+            (402, {2: "uvwxyz"})
+    """
+
+    sql "set enable_insert_strict = false"
+    sql """
+        insert into test_map_insert_shared_column_dst
+        select id, m, m from test_map_insert_shared_column_src
+    """
+    order_qt_shared_column """
+        select id, m_short, m_full
+        from test_map_insert_shared_column_dst
+        order by id
+    """
+}


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to