This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new 5c55a352e1d [fix](multi table) fix single stream multi table memory
leak (#38255)
5c55a352e1d is described below
commit 5c55a352e1d693cf6bd8e03be9a8626e1dcbabc0
Author: hui lai <[email protected]>
AuthorDate: Thu Jul 25 09:44:32 2024 +0800
[fix](multi table) fix single stream multi table memory leak (#38255)
We meet OOM when using single stream multi table
It exist memory leak, and heap profile like:
The stream load context will not release in some exception conditions as
plan failed for high concurrency causing timeout when obtaining read
lock. It is introduced by https://github.com/apache/doris/pull/35458
The solution effect is shown in the following figure, which can run
stably with a small amount of memory
---
be/src/io/fs/multi_table_pipe.cpp | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/be/src/io/fs/multi_table_pipe.cpp
b/be/src/io/fs/multi_table_pipe.cpp
index 789903cc1ca..d7fdd8a738b 100644
--- a/be/src/io/fs/multi_table_pipe.cpp
+++ b/be/src/io/fs/multi_table_pipe.cpp
@@ -326,6 +326,19 @@ void MultiTablePipe::_handle_consumer_finished() {
_ctx->number_filtered_rows = _number_filtered_rows;
_ctx->number_unselected_rows = _number_unselected_rows;
_ctx->commit_infos = _tablet_commit_infos;
+
+ // remove ctx to avoid memory leak.
+ for (const auto& pair : _planned_tables) {
+ if (pair.second) {
+
doris::ExecEnv::GetInstance()->new_load_stream_mgr()->remove(pair.second->id);
+ }
+ }
+ for (const auto& pair : _unplanned_tables) {
+ if (pair.second) {
+
doris::ExecEnv::GetInstance()->new_load_stream_mgr()->remove(pair.second->id);
+ }
+ }
+
LOG(INFO) << "all plan for multi-table load complete. number_total_rows="
<< _ctx->number_total_rows << " number_loaded_rows=" <<
_ctx->number_loaded_rows
<< " number_filtered_rows=" << _ctx->number_filtered_rows
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
