Re: [PR] [Improve] Estimated column reader memory to control segment cache [doris]
dataroaring merged PR #34526: URL: https://github.com/apache/doris/pull/34526 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improve] Estimated column reader memory to control segment cache [doris]
xinyiZzz commented on code in PR #34526: URL: https://github.com/apache/doris/pull/34526#discussion_r1593964105 ## be/src/common/config.cpp: ## @@ -1053,6 +1053,10 @@ DEFINE_mInt32(schema_cache_sweep_time_sec, "100"); // max number of segment cache, default -1 for backward compatibility fd_number*2/5 DEFINE_mInt32(segment_cache_capacity, "-1"); +DEFINE_mInt32(estimated_num_columns_per_segment, "30"); +DEFINE_mInt32(estimated_mem_per_column_reader, "1024"); +// The value is calculate by storage_page_cache_limit * index_page_cache_percentage Review Comment: delete this line -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improve] Estimated column reader memory to control segment cache [doris]
github-actions[bot] commented on PR #34526: URL: https://github.com/apache/doris/pull/34526#issuecomment-2100425644 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improve] Estimated column reader memory to control segment cache [doris]
github-actions[bot] commented on PR #34526: URL: https://github.com/apache/doris/pull/34526#issuecomment-2100425578 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improve] Estimated column reader memory to control segment cache [doris]
github-actions[bot] commented on PR #34526: URL: https://github.com/apache/doris/pull/34526#issuecomment-2100056418 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improve] Estimated column reader memory to control segment cache [doris]
Lchangliang commented on PR #34526: URL: https://github.com/apache/doris/pull/34526#issuecomment-2100045752 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improve] Estimated column reader memory to control segment cache [doris]
Lchangliang commented on code in PR #34526:
URL: https://github.com/apache/doris/pull/34526#discussion_r1593541497
##
be/src/olap/rowset/segment_v2/segment.cpp:
##
@@ -408,6 +408,7 @@ Status Segment::_create_column_readers(const
SegmentFooterPB& footer) {
RETURN_IF_ERROR(ColumnReader::create(opts,
footer.columns(iter->second), footer.num_rows(),
_file_reader, ));
_column_readers.emplace(column.unique_id(), std::move(reader));
+_meta_mem_usage += config::estimated_mem_per_column_reader;
Review Comment:
It still has some unique_ptr/shared_ptr object and the object
`SubcolumnColumnReaders`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improve] Estimated column reader memory to control segment cache [doris]
xinyiZzz commented on code in PR #34526:
URL: https://github.com/apache/doris/pull/34526#discussion_r1593531683
##
be/src/olap/rowset/segment_v2/segment.cpp:
##
@@ -408,6 +408,7 @@ Status Segment::_create_column_readers(const
SegmentFooterPB& footer) {
RETURN_IF_ERROR(ColumnReader::create(opts,
footer.columns(iter->second), footer.num_rows(),
_file_reader, ));
_column_readers.emplace(column.unique_id(), std::move(reader));
+_meta_mem_usage += config::estimated_mem_per_column_reader;
Review Comment:
in past, `_meta_mem_usage` only tracking some indexes memory,
now, add column reader memory
will there be other memory in segment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improve] Estimated column reader memory to control segment cache [doris]
xinyiZzz commented on PR #34526: URL: https://github.com/apache/doris/pull/34526#issuecomment-2099935198  ``` auto* lru_handle = LRUCachePolicy::insert( key.encode(), , 1, value.segment->meta_mem_usage(), CachePriority::NORMAL); ``` change to ``` auto* lru_handle = LRUCachePolicy::insert( key.encode(), , value.segment->meta_mem_usage(), value.segment->meta_mem_usage(), CachePriority::NORMAL); ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improve] Estimated column reader memory to control segment cache [doris]
github-actions[bot] commented on PR #34526: URL: https://github.com/apache/doris/pull/34526#issuecomment-2099934088 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improve] Estimated column reader memory to control segment cache [doris]
Lchangliang commented on PR #34526: URL: https://github.com/apache/doris/pull/34526#issuecomment-2099924079 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [Improve] Estimated column reader memory to control segment cache [doris]
doris-robot commented on PR #34526: URL: https://github.com/apache/doris/pull/34526#issuecomment-2099923883 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR) Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website). See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
