FrankChen021 commented on code in PR #20173:
URL: https://github.com/apache/druid/pull/20173#discussion_r3881085105
##########
indexing-service/src/main/java/org/apache/druid/indexing/common/task/batch/parallel/PartialSegmentMergeTask.java:
##########
@@ -295,13 +296,17 @@ private DataSegmentsWithSchemas mergeAndPushSegments(
), "version for interval[%s]", interval),
0
);
+ try (QueryableIndex index =
toolbox.getIndexIO().loadIndex(mergedFileAndDimensionNames.lhs)) {
Review Comment:
[P2] Avoid loading the full segment to read its row count
IndexIO.loadIndex(File) defaults to lazy=false. For v9 and legacy segments
this eagerly builds the column holders, and for v10 it maps all containers and
eagerly deserializes every column before getNumRows() is called. This call only
needs the row count, so a large merged segment can add substantial heap/native
memory pressure (and potentially OOM) during publishing.
Could we add or use an IndexIO.getNumRows(File) / metadata-only row-count
helper that dispatches on the segment version? For v10, read the persisted base
projection numRows from druid.segment; for v9 and legacy, read only the __time
column header totalSize. Then call that helper here instead of constructing a
QueryableIndex. Passing lazy=true is not sufficient because it still maps the
segment buffers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
