This is an automated email from the ASF dual-hosted git repository.
Mryange pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 5862b8737d9 [opt](be) Avoid redundant column copies in functions and
writers (#70606)
5862b8737d9 is described below
commit 5862b8737d90c5e39ed73e12af7e90cd691732d4
Author: Mryange <[email protected]>
AuthorDate: Wed Nov 18 14:25:30 2026 +0800
[opt](be) Avoid redundant column copies in functions and writers (#70606)
### Background
Doris columns use copy-on-write ownership, and composite columns can
retain immutable shared subcolumns. Filtering shared columns already
creates an output containing the selected rows.
### What problem does this PR solve?
Issue Number: None
Related PR: None
PR Type: Performance
Problem Summary:
Several function and writer paths eagerly copy data that is only read or
immediately filtered. Share the null map for null-safe equality, share
unchanged subcolumns during Iceberg ORC normalization, and filter shared
Hive input columns without first cloning the whole input.
### Details
- **Why it was slow:** Hive partition filtering copied every input row
per partition before discarding unselected rows. Null-safe equality
copied an unchanged null map, and Iceberg ORC normalization cloned
unchanged null maps and offsets and detached unmodified children.
- **Optimization:** Reuse existing immutable ColumnPtr ownership, shared
composite constructors, and the existing COW-aware block filter.
- **Results:** Source inspection confirms removal of the eager copy
operations. No performance benchmark was run; no throughput or latency
improvement is claimed.
- **Applicability and cost:** Applies to the two all-null operand
branches of null-safe equality, Hive partition filtering, and Iceberg
ORC binary normalization. Later writes can still trigger normal COW
detachment.
### Testing
- ASAN BE build: `sh build.sh --be -j48` passed on a Linux development
machine.
- `build-support/clang-format.sh`, `build-support/check-format.sh`
(clang-format 16), and `git diff --check` passed.
- `build-support/run-clang-tidy.sh` passed for all three changed files,
with no warnings on changed lines. Analysis used the build flags, exact
source entries derived from Unity compilation where needed, and the
matching toolchain resource directory.
- No new BE unit tests were added. Unit tests, regression tests, and
performance benchmarks were not run.
### Release note
Reduce intermediate column copying in null-safe equality, Hive partition
filtering, and Iceberg ORC normalization.
### Check List (For Author)
- Behavior changed: No
- Does this need documentation: No
- Pre-submission status: Partially verified: ASAN BE build, formatting,
and changed-line static analysis passed; runtime tests and performance
benchmarks were not run.
### AI Disclosure
- Tool: Codex (GPT-6)
- Generated: Code changes and PR description.
- Human review: The author discussed and approved the implementation
approach; final code review is pending.
---
be/src/exec/sink/writer/vhive_table_writer.cpp | 4 ++--
be/src/exprs/function/comparison_equal_for_null.cpp | 4 ++--
be/src/format/transformer/vorc_transformer.cpp | 12 ++++--------
3 files changed, 8 insertions(+), 12 deletions(-)
diff --git a/be/src/exec/sink/writer/vhive_table_writer.cpp
b/be/src/exec/sink/writer/vhive_table_writer.cpp
index 5d68d50caa3..0e239af8eae 100644
--- a/be/src/exec/sink/writer/vhive_table_writer.cpp
+++ b/be/src/exec/sink/writer/vhive_table_writer.cpp
@@ -231,8 +231,8 @@ Status VHiveTableWriter::_filter_block(doris::Block& block,
const IColumn::Filte
ColumnsWithTypeAndName result_columns;
for (int i = 0; i < columns_with_type_and_name.size(); ++i) {
const auto& col = columns_with_type_and_name[i];
-
result_columns.emplace_back(col.column->clone_resized(col.column->size()),
col.type,
- col.name);
+ // Keep the input owner so filtering copies only this partition's
selected rows.
+ result_columns.emplace_back(col.column, col.type, col.name);
}
*output_block = {std::move(result_columns)};
diff --git a/be/src/exprs/function/comparison_equal_for_null.cpp
b/be/src/exprs/function/comparison_equal_for_null.cpp
index 32ffc7b1d88..fa2f67cefff 100644
--- a/be/src/exprs/function/comparison_equal_for_null.cpp
+++ b/be/src/exprs/function/comparison_equal_for_null.cpp
@@ -115,7 +115,7 @@ public:
nullable_right_col = assert_cast<const
ColumnNullable*>(col_right.column.get());
// left column is all null, so result has same nullmap
with right column.
block.get_by_position(result).column =
- nullable_right_col->get_null_map_column().clone();
+ nullable_right_col->get_null_map_column_ptr();
}
}
return Status::OK();
@@ -138,7 +138,7 @@ public:
nullable_left_col = assert_cast<const
ColumnNullable*>(col_left.column.get());
// right column is all null, so result has same nullmap
with left column.
block.get_by_position(result).column =
- nullable_left_col->get_null_map_column().clone();
+ nullable_left_col->get_null_map_column_ptr();
}
}
return Status::OK();
diff --git a/be/src/format/transformer/vorc_transformer.cpp
b/be/src/format/transformer/vorc_transformer.cpp
index 28799eb8a63..abd6e5e73fd 100644
--- a/be/src/format/transformer/vorc_transformer.cpp
+++ b/be/src/format/transformer/vorc_transformer.cpp
@@ -786,9 +786,8 @@ static Status normalize_iceberg_nullable_column(const
ColumnPtr& column, const D
RETURN_IF_ERROR(normalize_iceberg_binary_column(nullable_column.get_nested_column_ptr(),
remove_nullable(type),
nested_field,
&nested_column,
combined_skipped_rows));
- *normalized_column = ColumnNullable::create(
- IColumn::mutate(std::move(nested_column)),
-
IColumn::mutate(nullable_column.get_null_map_column_ptr()->clone()));
+ *normalized_column =
+ ColumnNullable::create(nested_column,
nullable_column.get_null_map_column_ptr());
return Status::OK();
}
@@ -887,8 +886,7 @@ static Status normalize_iceberg_array_column(const
ColumnPtr& column, const Data
array_column.get_data_ptr(), array_type.get_nested_type(),
nested_field.field_type()->as_list_type()->element_field(),
&elements,
expanded_skipped_rows));
- *normalized_column =
ColumnArray::create(IColumn::mutate(std::move(elements)),
-
array_column.get_offsets_ptr()->clone());
+ *normalized_column = ColumnArray::create(elements,
array_column.get_offsets_ptr());
return Status::OK();
}
@@ -910,9 +908,7 @@ static Status normalize_iceberg_map_column(const ColumnPtr&
column, const DataTy
RETURN_IF_ERROR(normalize_iceberg_binary_column(
map_column.get_values_ptr(), map_type.get_value_type(),
iceberg_map->value_field(),
&values, expanded_skipped_rows));
- *normalized_column =
- ColumnMap::create(IColumn::mutate(std::move(keys)),
IColumn::mutate(std::move(values)),
- map_column.get_offsets_ptr()->clone());
+ *normalized_column = ColumnMap::create(keys, values,
map_column.get_offsets_ptr());
return Status::OK();
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]