github-actions[bot] commented on code in PR #69438:
URL: https://github.com/apache/doris/pull/69438#discussion_r4370815980
##########
be/src/storage/index/inverted/query/exec/index_source.cpp:
##########
@@ -43,4 +46,132 @@ Status IndexSource::collect_terms(std::span<const
std::string> terms, DocIdSink&
return Status::OK();
}
+namespace {
+
+// How many postings a scan decodes in the time a seek costs.
+constexpr uint64_t kPostingsPerSeek = 32;
+
+// Decodes every posting once and walks `docs` alongside, both ascending.
+Status count_listed_docs_by_scan(PostingsCursor& cursor, std::span<const
uint32_t> docs,
+ uint64_t* out) {
+ size_t next = 0;
+ PostingsBlock block;
+ bool eof = false;
+ while (next < docs.size()) {
+ RETURN_IF_ERROR(cursor.next_block(&block, &eof));
+ if (eof) {
+ break;
+ }
+ if (block.dense) {
+ const auto begin = std::lower_bound(docs.begin() + next,
docs.end(), block.range_begin);
+ const auto end = std::lower_bound(begin, docs.end(),
block.range_end);
+ *out += static_cast<uint64_t>(end - begin);
+ next = end - docs.begin();
+ continue;
+ }
+ for (const uint32_t doc : block.docs) {
+ while (next < docs.size() && docs[next] < doc) {
+ ++next;
+ }
+ if (next == docs.size()) {
+ break;
+ }
+ if (docs[next] == doc) {
+ ++*out;
+ ++next;
+ }
+ }
+ }
+ return Status::OK();
+}
+
+} // namespace
+
+Status count_listed_docs(PostingsCursor& cursor, std::span<const uint32_t>
docs, uint64_t* out) {
+ *out = 0;
+ if (docs.empty() || cursor.doc_freq() == 0) {
+ return Status::OK();
+ }
+ // Leaping seeks once per wanted document, scanning decodes every posting:
leap only over
+ // documents much rarer than the postings.
+ if (docs.size() * kPostingsPerSeek >= cursor.doc_freq()) {
+ return count_listed_docs_by_scan(cursor, docs, out);
+ }
+ size_t next = 0;
+ PostingsBlock block;
+ bool eof = false;
+ while (next < docs.size()) {
+ RETURN_IF_ERROR(cursor.seek_block(docs[next], &block, &eof));
+ if (eof) {
+ break;
+ }
+ const uint64_t size = block.size();
+ uint64_t ordinal = 0;
+ while (ordinal < size && next < docs.size()) {
+ const uint32_t listed = block.doc_at(ordinal);
+ const uint32_t wanted = docs[next];
+ if (listed == wanted) {
+ ++*out;
+ ++ordinal;
+ ++next;
+ } else if (listed < wanted) {
+ if (block.dense) {
+ ordinal = wanted - block.range_begin;
+ } else {
+ ordinal = std::lower_bound(block.docs.begin() + ordinal,
block.docs.end(),
+ wanted) -
+ block.docs.begin();
+ }
+ } else {
+ next = std::lower_bound(docs.begin() + next, docs.end(),
listed) - docs.begin();
+ }
+ }
+ }
+ return Status::OK();
+}
+
+Status IndexSource::scoring_doc_freq_among(std::string_view term,
std::span<const uint32_t> docs,
+ uint64_t* out) {
+ *out = 0;
+ if (docs.empty()) {
+ return Status::OK();
+ }
+ std::unique_ptr<PostingsCursor> cursor;
+ RETURN_IF_ERROR(open_term(term, /*positions=*/false, /*scoring=*/false,
&cursor));
+ if (cursor == nullptr) {
+ return Status::OK();
+ }
+ return count_listed_docs(*cursor, docs, out);
+}
+
+Status IndexSource::sum_norm_lengths_of(std::span<const uint32_t> docs,
uint64_t* out) {
+ *out = 0;
+ const std::span<const float> lengths = norm_lengths();
+ std::vector<uint32_t> norms;
Review Comment:
[P2] Bound or track the SNII selected-row norm buffer
For an analyzed SNII MoW segment with 10 million NULL rows and one matching
row, `sum_non_null_norm_lengths` passes the entire NULL list here, and
`SniiIndexSource::encoded_norms` assigns one `uint32_t` per ID. This allocates
another roughly 40 MB before scan pruning, without a MemTracker charge or
reservation. The manually charged NULL-ID vector and `DorisVector` hidden IDs
cover separate inputs; the 4096-row batching in `sum_norm_lengths` does not
bound this direct call. Large hidden non-NULL sets reach the same path, so
concurrent tablet collection can exceed query memory limits. Please batch
selected rows here or sum them directly with tracked storage. This is a
distinct norm-code output buffer from the previously reported input row-ID
arrays.
##########
be/src/storage/index/inverted/clucene/spi/clucene_index_source.cpp:
##########
@@ -243,6 +243,82 @@ std::span<const float> CluceneIndexSource::norm_lengths()
const {
return BM25Similarity::lucene_norm_lengths();
}
+// The statistics read a field's norms into a buffer of their own, freed after
use: asking a reader
+// for its norms makes it keep them, uncharged to the searcher cache, for
every segment the
+// statistics visit, scored or not. Reading per segment also avoids the copy
of all segments' norms
+// a reader of several segments would keep. A segment without the field's
norms reads as zero
+// lengths, as that reader would.
+std::vector<CluceneIndexSource::ReaderPartition>
CluceneIndexSource::_norm_partitions() const {
+ if (!_partitions.empty()) {
+ return _partitions;
+ }
+ return {{.reader = _reader.get(), .begin = 0, .end = doc_count()}};
+}
+
+void CluceneIndexSource::_read_norms(const ReaderPartition& partition,
+ DorisVector<uint8_t>* norms) const {
+ norms->assign(partition.end - partition.begin, 0);
+ partition.reader->norms(_field.c_str(), norms->data());
Review Comment:
[P2] Preserve the query I/O context on statistics norm reads
For a cold remote MoW tablet, the new statistics pass reads norms from every
captured segment, including segments a selective scan later prunes. This call
never supplies the source's `_io_ctx`: the norm stream's `CSIndexInput` has no
per-read context, and the cached base stream was reset to defaults.
Consequently `CacheContext` sees `is_disposable=false`, no query ID and no
remote-scan cache-write limiter. Even with `disable_file_cache=true` or a query
cache-write budget, these new bulk reads populate normal-priority cache blocks
and bypass that budget. Ordinary scoring had the same limitation, but
previously did not read norms of all pruned segments. Please propagate the
context per norm read without retaining query-owned pointers on the shared
cached searcher.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
