guptashivam04 commented on code in PR #69438:
URL: https://github.com/apache/doris/pull/69438#discussion_r4366381897
##########
be/src/storage/index/inverted/reader/collection_statistics.cpp:
##########
@@ -361,6 +556,33 @@ Status CollectionStatistics::process_segment(const
RowsetSharedPtr& rowset,
ws_field_name, io_ctx);
index_query::IndexScoringStats scoring_stats;
RETURN_IF_ERROR(source->scoring_stats(&scoring_stats));
+ if (hidden_rows != nullptr) {
+ std::shared_ptr<roaring::Roaring> null_docs =
std::make_shared<roaring::Roaring>();
+ uint64_t non_null_length_sum = 0;
+ if (scoring_stats.has_norms) {
+ segment_v2::InvertedIndexQueryCacheHandle
null_bitmap_cache_handle;
+ RETURN_IF_ERROR(segment_v2::read_clucene_null_bitmap(
+ *idx_file_reader, *collect_info.index_meta, io_ctx,
+ &null_bitmap_cache_handle));
+ null_docs = null_bitmap_cache_handle.get_bitmap();
+ DORIS_CHECK(null_docs != nullptr);
+ auto sum_lengths = [&source, &null_docs](uint64_t* out) {
+ std::vector<uint32_t>
null_doc_ids(null_docs->cardinality());
+ null_docs->toUint32Array(null_doc_ids.data());
+ return
collection_statistics_detail::sum_non_null_norm_lengths(
+ *source, null_doc_ids, out);
+ };
+#ifdef BE_TEST
+ RETURN_IF_ERROR(sum_lengths(&non_null_length_sum));
+#else
+
RETURN_IF_ERROR(inverted_index_cache_handle.get<CluceneSearcherCacheValue>()
Review Comment:
Fixed in 96a0b49670: the statistics no longer ask CLucene readers for their
retained norms. They read each segment's norms into their own buffer via
`IndexReader::norms(field, bytes)` and free it after use. Only small sums are
cached with the searcher: the per-field non-NULL norm-length sum, and the
hidden rows' length sum for the last few sets of hidden rows per field (queries
at one read version reuse it and read no norms). Measured on a 10M-row tablet
with 4 scored fields and a key-pruned query: memory growth went from about +51
MB to +12–15 MB, the same as before the PR. Covered by
`HiddenNormLengthSumIsComputedOncePerRowSet` and
`NormLengthSumsReadEachSegment`.
##########
be/src/storage/index/inverted/query/exec/index_source.cpp:
##########
@@ -43,4 +46,128 @@ Status IndexSource::collect_terms(std::span<const
std::string> terms, DocIdSink&
return Status::OK();
}
+namespace {
+
+// How many postings a scan decodes in the time a seek costs.
+constexpr uint64_t kPostingsPerSeek = 32;
+
+// Decodes every posting once and tests it against a bitset of `docs`.
+Status count_listed_docs_by_scan(PostingsCursor& cursor, std::span<const
uint32_t> docs,
+ uint64_t* out) {
+ std::vector<uint64_t> bits(docs.back() / 64 + 1, 0);
Review Comment:
Fixed in 96a0b49670: `count_listed_docs` no longer allocates a bitset. The
scan path walks the hidden rows and the postings together (both ascending) and
stops once every hidden row is passed, so the cost no longer depends on the
largest row ID. A term with `doc_freq() == 0` returns before opening anything.
Covered by `CountListedDocsScansFarDocsWithoutWalkingEveryBlock` (IDs up to
9,999,999) and `CountListedDocsOfNothing`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
