This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 547dc0f0e98 [fix](storage) Release ANN resources and seal rebuilt
indexes per segment (#70591)
547dc0f0e98 is described below
commit 547dc0f0e984775790c1c1b3279171500a7492ef
Author: Mryange <[email protected]>
AuthorDate: Thu Nov 19 16:49:57 2026 +0800
[fix](storage) Release ANN resources and seal rebuilt indexes per segment
(#70591)
### Background
Historical ANN index builds buffer vectors and construct an in-memory
index for each segment. IndexBuilder also owns the per-segment column
and file writers until their index files are sealed.
### What problem does this PR solve?
Issue Number: None
Related PR: #70394 (extends the build-resource cleanup with per-segment
file sealing)
PR Type: Bugfix
Problem Summary:
Building ANN indexes on a rowset with many segments retains completed
build objects and serialized index buffers, increasing peak memory as
more segments finish. Release the ANN build resources after each segment
and seal each segment's independent index file before processing the
next segment. Preserve the existing rowset publication boundary and
propagate write/close failures.
### Details
- **Frequency:** Multi-segment historical index builds; the
serialized-buffer accumulation is visible when index RAM directories are
enabled.
- **Root cause:** Completed ANN objects/input buffers and per-segment
index directories outlive their useful build phase because cleanup and
file sealing occur at rowset scope.
- **Why existing tests missed it:** Index correctness checks do not
assert peak memory or the lifetime of completed segment build resources.
- **Fix:** Release build-only resources on success, skip and error
paths, free input-buffer capacity before serialization, and
close/release segment writers after all requested logical indexes for
that segment finish. Segments with no columns to rebuild still seal
inherited indexes.
- **Other paths and branches:** Apply segment sealing to both regular
and SNII IndexBuilder paths. The common close placement also applies to
inverted-index rebuilds; dropping indexes and rowset publication are
unchanged. No format, protocol or configuration changes.
- **Behavior change:** Completed segment resources are reclaimed
earlier; a rowset is still published only after the whole operation
succeeds.
### Testing
- Final patch: `sh build.sh --be -j48` (ASAN) completed successfully on
the development host.
- Official `build-support/clang-format.sh`,
`build-support/check-format.sh`, and
`build-support/check-build-hygiene.sh`: passed; formatting made no
source changes.
- Official `build-support/run-clang-tidy.sh --base 08231e3a635` with a
per-file compilation database derived from the ASAN unity build: passed
for all three changed files in changed-line filtering mode. This does
not claim that entire files or included headers have no pre-existing
diagnostics.
- `git diff --check`: passed.
Earlier runtime measurements of this
resource-release/per-segment-sealing implementation, before the final
no-columns control-flow adjustment:
- Same 500,000 float32 vectors, dimension 128, one rowset with 14
segments; HNSW, max_degree=16, ef_construction=100, L2, flat
quantization. RELEASE build, local disk, 32 MiB write buffer, automatic
compaction disabled, index RAM directories enabled.
- Process-wide peak increases relative to the pre-build baseline (MiB):
| Implementation | RSS peak increase | jemalloc allocated peak increase
|
| --- | ---: | ---: |
| Baseline master 08231e3a635 | 695.4 | 701.8 |
| Release ANN build resources only | 403.9 | 366.2 |
| Release resources + seal each segment | 90.7 | 73.9 |
- The combined implementation reduced the observed RSS peak increase by
about 87% and allocated peak increase by about 89.5% versus baseline.
These are single-run, process-wide observations on a shared development
host; warm state differed and the sampling is not an exact accounting of
ANN-only allocations. No throughput claim is made.
- Output remained 500,000 rows / 14 segments with identical per-segment
row counts, 220,383,392 data bytes and 328,141,266 index bytes across
the Doris versions. Six indexed self-match queries spanning segment
boundaries returned the original IDs at distance zero; EXPLAIN showed
ANN SORT INFO. This is basic readability validation, not a recall
benchmark.
- A separate targeted syscall trace with sync_file_on_close=true
observed 14 index-file fdatasync calls totaling 109.5 ms, 14
directory-sync calls totaling 2.3 ms and 14 file-close calls totaling
0.42 ms. These exclude serialization, buffer copies and resource
destruction; tracing adds overhead and this does not establish an
end-to-end performance regression bound.
- No new unit tests are included. The final no-columns branch adjustment
has not been rerun in a live cluster.
### Release note
Reduce peak memory usage when building ANN indexes on historical rowsets
with multiple segments by releasing completed segment resources earlier.
### Check List (For Author)
- Behavior changed: Yes (see Details).
- Does this need documentation: No.
- Pre-submission status: Partially verified: build/static checks and
runtime limits are listed above; runtime memory/readability measurements
precede the final no-columns branch adjustment, and broader
inverted-index rebuild regression tests were not run.
### AI Disclosure
- Tool: Codex (GPT-6).
- Generated: Code changes and this description.
- Human review: The user reviewed the proposed lifecycle approach; final
revision review is pending.
---
be/src/storage/index/ann/ann_index_writer.cpp | 20 ++-
be/src/storage/index/ann/ann_index_writer.h | 5 +-
be/src/storage/task/index_builder.cpp | 187 +++++++++++++-------------
3 files changed, 114 insertions(+), 98 deletions(-)
diff --git a/be/src/storage/index/ann/ann_index_writer.cpp
b/be/src/storage/index/ann/ann_index_writer.cpp
index 686cf15208c..a56c6a97d12 100644
--- a/be/src/storage/index/ann/ann_index_writer.cpp
+++ b/be/src/storage/index/ann/ann_index_writer.cpp
@@ -26,6 +26,7 @@
#include "common/config.h"
#include "storage/index/ann/faiss_ann_index.h"
#include "storage/index/index_fs_directory.h"
+#include "util/defer_op.h"
namespace doris::segment_v2 {
static std::string get_or_default(const std::map<std::string, std::string>&
properties,
@@ -91,6 +92,11 @@ Status AnnIndexColumnWriter::add_values(const uint8_t*
values, size_t count) {
}
void AnnIndexColumnWriter::close_on_error() {
+ _release_build_resources();
+}
+
+void AnnIndexColumnWriter::_release_build_resources() {
+ _vector_index.reset();
PODArray<float> empty_buffered_vectors;
_buffered_vectors.swap(empty_buffered_vectors);
}
@@ -138,6 +144,7 @@ int64_t AnnIndexColumnWriter::size() const {
Status AnnIndexColumnWriter::finish() {
if (_total_rows == 0) {
LOG_INFO("No data to train/add for ANN index. Skipping index
building.");
+ _release_build_resources();
return _index_file_writer->delete_index(_index_meta);
}
@@ -149,8 +156,7 @@ Status AnnIndexColumnWriter::finish() {
"Total data size {} is less than minimum {} rows required for
ANN index build. "
"Skipping index building for this segment.",
_total_rows, effective_min_rows);
- PODArray<float> empty_buffered_vectors;
- _buffered_vectors.swap(empty_buffered_vectors);
+ _release_build_resources();
return _index_file_writer->delete_index(_index_meta);
}
@@ -158,6 +164,10 @@ Status AnnIndexColumnWriter::finish() {
}
Status AnnIndexColumnWriter::_build_and_save(Int64 min_train_rows, Int64
effective_min_rows) {
+ // Release the Faiss index and any remaining input buffer after saving,
including
+ // on failure, rather than retaining them until this writer is destroyed.
+ Defer release_resources([this] { _release_build_resources(); });
+
const size_t dim = _vector_index->get_dimension();
DCHECK(_buffered_vectors.size() % dim == 0);
const Int64 train_rows = cast_set<Int64>(_buffered_vectors.size() / dim);
@@ -169,8 +179,10 @@ Status AnnIndexColumnWriter::_build_and_save(Int64
min_train_rows, Int64 effecti
RETURN_IF_ERROR(_vector_index->add(train_rows, _buffered_vectors.data()));
// PODArray::clear() keeps the allocated capacity. Swap with an empty
array so the
// full-segment build buffer is released before saving the index.
- PODArray<float> empty_buffered_vectors;
- _buffered_vectors.swap(empty_buffered_vectors);
+ {
+ PODArray<float> empty_buffered_vectors;
+ _buffered_vectors.swap(empty_buffered_vectors);
+ }
Status status = _vector_index->save(_dir.get());
if (!status.ok()) {
// A failed save keeps whatever it managed to write, and under SNII
that is
diff --git a/be/src/storage/index/ann/ann_index_writer.h
b/be/src/storage/index/ann/ann_index_writer.h
index e876a0a3956..587348d28c9 100644
--- a/be/src/storage/index/ann/ann_index_writer.h
+++ b/be/src/storage/index/ann/ann_index_writer.h
@@ -75,14 +75,13 @@ public:
private:
Status _build_and_save(Int64 min_train_rows, Int64 effective_min_rows);
+ void _release_build_resources();
#ifdef BE_TEST
friend class TestAnnIndexColumnWriter;
#endif
- // VectorIndex shoule be managed by some cache.
- // VectorIndex should be weak shared by AnnIndexWriter and
VectorIndexReader
- // This should be a weak_ptr
+ // The build index is owned by this writer and released when the segment
is finished.
std::shared_ptr<VectorIndex> _vector_index;
PODArray<float> _buffered_vectors;
int64_t _total_rows = 0;
diff --git a/be/src/storage/task/index_builder.cpp
b/be/src/storage/task/index_builder.cpp
index 84b8edce3de..74f2b1c54be 100644
--- a/be/src/storage/task/index_builder.cpp
+++ b/be/src/storage/task/index_builder.cpp
@@ -719,97 +719,101 @@ Status
IndexBuilder::handle_single_rowset(RowsetMetaSharedPtr output_rowset_meta
_index_file_writers.emplace(seg_ptr->id(),
std::move(index_file_writer));
DORIS_CHECK(inserted);
DORIS_CHECK(index_file_writer_it->second != nullptr);
- if (return_columns.empty()) {
- // no columns to read
- continue;
- }
- // create iterator for each segment
- OlapReaderStatistics stats;
- StorageReadOptions read_options(stats);
- auto schema = std::make_shared<ReadSchema>(
-
project_columns_by_ordinal(output_rowset_schema->columns(), return_columns));
- std::unique_ptr<RowwiseIterator> iter;
- auto res = seg_ptr->new_iterator(schema, read_options, &iter);
-
DBUG_EXECUTE_IF("IndexBuilder::handle_single_rowset_create_iterator_error", {
- res = Status::Error<ErrorCode::INTERNAL_ERROR>(
- "debug point:
handle_single_rowset_create_iterator_error");
- })
- if (!res.ok()) {
- LOG(WARNING) << "failed to create iterator[" << seg_ptr->id()
- << "]: " << res.to_string();
- return
Status::Error<ErrorCode::ROWSET_READER_INIT>(res.to_string());
- }
+ // Even without columns to rebuild, the inherited indexes must be
sealed below.
+ if (!return_columns.empty()) {
+ // create iterator for each segment
+ OlapReaderStatistics stats;
+ StorageReadOptions read_options(stats);
+ auto schema =
std::make_shared<ReadSchema>(project_columns_by_ordinal(
+ output_rowset_schema->columns(), return_columns));
+ std::unique_ptr<RowwiseIterator> iter;
+ auto res = seg_ptr->new_iterator(schema, read_options, &iter);
+
DBUG_EXECUTE_IF("IndexBuilder::handle_single_rowset_create_iterator_error", {
+ res = Status::Error<ErrorCode::INTERNAL_ERROR>(
+ "debug point:
handle_single_rowset_create_iterator_error");
+ })
+ if (!res.ok()) {
+ LOG(WARNING) << "failed to create iterator[" <<
seg_ptr->id()
+ << "]: " << res.to_string();
+ return
Status::Error<ErrorCode::ROWSET_READER_INIT>(res.to_string());
+ }
- auto block = Block::create_unique(schema->create_read_block());
- while (true) {
- auto status = iter->next_batch(block.get());
-
DBUG_EXECUTE_IF("IndexBuilder::handle_single_rowset_iterator_next_batch_error",
{
- status =
Status::Error<ErrorCode::SCHEMA_CHANGE_INFO_INVALID>(
- "next_batch fault injection");
- });
- if (!status.ok()) {
- if (status.is<ErrorCode::END_OF_FILE>()) {
- break;
+ auto block = Block::create_unique(schema->create_read_block());
+ while (true) {
+ auto status = iter->next_batch(block.get());
+ DBUG_EXECUTE_IF(
+
"IndexBuilder::handle_single_rowset_iterator_next_batch_error", {
+ status =
Status::Error<ErrorCode::SCHEMA_CHANGE_INFO_INVALID>(
+ "next_batch fault injection");
+ });
+ if (!status.ok()) {
+ if (status.is<ErrorCode::END_OF_FILE>()) {
+ break;
+ }
+ LOG(WARNING) << "failed to read next block when schema
change for inverted "
+ "index."
+ << ", err=" << status.to_string();
+ return status;
}
- LOG(WARNING)
- << "failed to read next block when schema change
for inverted index."
- << ", err=" << status.to_string();
- return status;
- }
- // write inverted index data, or ann index data
- status = _write_inverted_index_data(output_rowset_schema,
iter->data_id(),
- block.get());
- DBUG_EXECUTE_IF(
-
"IndexBuilder::handle_single_rowset_write_inverted_index_data_error", {
- status = Status::Error<ErrorCode::INTERNAL_ERROR>(
- "debug point: "
-
"handle_single_rowset_write_inverted_index_data_error");
- })
- if (!status.ok()) {
- return status;
+ // write inverted index data, or ann index data
+ status = _write_inverted_index_data(output_rowset_schema,
iter->data_id(),
+ block.get());
+ DBUG_EXECUTE_IF(
+
"IndexBuilder::handle_single_rowset_write_inverted_index_data_error", {
+ status =
Status::Error<ErrorCode::INTERNAL_ERROR>(
+ "debug point: "
+
"handle_single_rowset_write_inverted_index_data_error");
+ })
+ if (!status.ok()) {
+ return status;
+ }
+ block->clear_column_data();
}
- block->clear_column_data();
- }
- // finish write inverted index, flush data to compound file
- for (auto& writer_sign : inverted_index_writer_signs) {
- try {
- auto index_column_writer_it =
_index_column_writers.find(writer_sign);
- DORIS_CHECK(index_column_writer_it !=
_index_column_writers.end());
- DORIS_CHECK(index_column_writer_it->second != nullptr);
- RETURN_IF_ERROR(index_column_writer_it->second->finish());
-
DBUG_EXECUTE_IF("IndexBuilder::handle_single_rowset_index_build_finish_error", {
- _CLTHROWA(CL_ERR_IO,
- "debug point:
handle_single_rowset_index_build_finish_error");
- })
- } catch (const std::exception& e) {
- return
Status::Error<ErrorCode::INVERTED_INDEX_CLUCENE_ERROR>(
- "CLuceneError occurred: {}", e.what());
+ // finish write inverted index, flush data to compound file
+ for (auto& writer_sign : inverted_index_writer_signs) {
+ try {
+ auto index_column_writer_it =
_index_column_writers.find(writer_sign);
+ DORIS_CHECK(index_column_writer_it !=
_index_column_writers.end());
+ DORIS_CHECK(index_column_writer_it->second != nullptr);
+
RETURN_IF_ERROR(index_column_writer_it->second->finish());
+ DBUG_EXECUTE_IF(
+
"IndexBuilder::handle_single_rowset_index_build_finish_error", {
+ _CLTHROWA(CL_ERR_IO,
+ "debug point: "
+
"handle_single_rowset_index_build_finish_error");
+ })
+ } catch (const std::exception& e) {
+ return
Status::Error<ErrorCode::INVERTED_INDEX_CLUCENE_ERROR>(
+ "CLuceneError occurred: {}", e.what());
+ }
}
}
- }
- for (auto&& [seg_id, index_file_writer] : _index_file_writers) {
- auto st = index_file_writer->begin_close();
-
DBUG_EXECUTE_IF("IndexBuilder::handle_single_rowset_file_writer_close_error", {
- st = Status::Error<ErrorCode::INVERTED_INDEX_CLUCENE_ERROR>(
- "debug point:
handle_single_rowset_file_writer_close_error");
- })
- if (!st.ok()) {
- LOG(ERROR) << "close index_file_writer error:" << st;
- return st;
+ // Each segment owns an independent index file. Seal it before
building the
+ // next segment: otherwise every completed ANN file remains in its
RAM directory.
+ for (auto&& [seg_id, completed_file_writer] : _index_file_writers)
{
+ auto st = completed_file_writer->begin_close();
+
DBUG_EXECUTE_IF("IndexBuilder::handle_single_rowset_file_writer_close_error", {
+ st =
Status::Error<ErrorCode::INVERTED_INDEX_CLUCENE_ERROR>(
+ "debug point:
handle_single_rowset_file_writer_close_error");
+ })
+ if (!st.ok()) {
+ LOG(ERROR) << "close completed_file_writer error:" << st;
+ return st;
+ }
+ inverted_index_size +=
completed_file_writer->get_index_file_total_size();
}
- inverted_index_size +=
index_file_writer->get_index_file_total_size();
- }
- for (auto&& [seg_id, index_file_writer] : _index_file_writers) {
- auto st = index_file_writer->finish_close();
- if (!st.ok()) {
- LOG(ERROR) << "wait close index_file_writer error:" << st;
- return st;
+ for (auto&& [seg_id, completed_file_writer] : _index_file_writers)
{
+ auto st = completed_file_writer->finish_close();
+ if (!st.ok()) {
+ LOG(ERROR) << "wait close completed_file_writer error:" <<
st;
+ return st;
+ }
}
+ _index_column_writers.clear();
+ _index_file_writers.clear();
}
- _index_column_writers.clear();
- _index_file_writers.clear();
output_rowset_meta->set_data_disk_size(output_rowset_meta->data_disk_size());
output_rowset_meta->set_total_disk_size(output_rowset_meta->total_disk_size() +
inverted_index_size);
@@ -828,21 +832,22 @@ Status IndexBuilder::_handle_single_rowset_snii(
auto input_schema_it = _input_rowset_schemas.find(rowset_id);
DORIS_CHECK(input_schema_it != _input_rowset_schemas.end());
+ size_t inverted_index_size = 0;
for (auto& seg_ptr : segments) {
RETURN_IF_ERROR(_rewrite_single_segment_snii(output_rowset_meta->fs(),
output_rowset_meta->tablet_schema(),
*input_schema_it->second,
rowset_id, seg_ptr));
+ // Blob staging and column writers belong to this segment, not the
whole rowset.
+ for (auto&& [seg_id, index_file_writer] : _index_file_writers) {
+ RETURN_IF_ERROR(index_file_writer->begin_close());
+ inverted_index_size +=
index_file_writer->get_index_file_total_size();
+ }
+ for (auto&& [seg_id, index_file_writer] : _index_file_writers) {
+ RETURN_IF_ERROR(index_file_writer->finish_close());
+ }
+ _index_column_writers.clear();
+ _index_file_writers.clear();
}
- size_t inverted_index_size = 0;
- for (auto&& [seg_id, index_file_writer] : _index_file_writers) {
- RETURN_IF_ERROR(index_file_writer->begin_close());
- inverted_index_size += index_file_writer->get_index_file_total_size();
- }
- for (auto&& [seg_id, index_file_writer] : _index_file_writers) {
- RETURN_IF_ERROR(index_file_writer->finish_close());
- }
- _index_column_writers.clear();
- _index_file_writers.clear();
output_rowset_meta->set_data_disk_size(output_rowset_meta->data_disk_size());
output_rowset_meta->set_total_disk_size(output_rowset_meta->total_disk_size() +
inverted_index_size);
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]