airborne12 commented on PR #69438: URL: https://github.com/apache/doris/pull/69438#issuecomment-6397108477
Checked at `a0d993147352ec34120c93871c266be1e34cc22f`. The collector handles normal overwrite/delete-bitmap filtering for V2, V3 and SNII. The description already acknowledges these limits, which still restrict the visibility guarantee: 1. **Light deletes (V2/V3/SNII).** A DELETE can use a delete predicate when `enable_mow_light_delete=true`. Queries exclude the deleted rows. However, `collect_hidden_docs()` only combines the versioned delete bitmap and delete-sign rows; it does not evaluate delete predicates. Document count, term document frequency and the norm-length sum therefore still include those deleted rows. Compaction on one replica can remove them and leave replica scores different again. [Hidden-row collection](https://github.com/apache/doris/blob/a0d993147352ec34120c93871c266be1e34cc22f/be/src/storage/index/inverted/reader/collection_statistics.cpp#L200-L279). 2. **Delete-sign visibility options (V2/V3/SNII).** With `skip_delete_sign=true` or `show_hidden_columns=true`, the planner omits the delete-sign filter. The collector still excludes every nonzero delete-sign row. Those returned rows are then absent from the scoring corpus. `MowHiddenRows` does not carry the query's delete-sign filtering choice. [Planner condition](https://github.com/apache/doris/blob/a0d993147352ec34120c93871c266be1e34cc22f/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java#L491-L508), [collector contract](https://github.com/apache/doris/blob/a0d993147352ec34120c93871c266be1e34cc22f/be/src/storage/index/inverted/reader/collection_statistics.h#L67-L82). 3. **SNII repeats norm aggregation.** For fields with norms, each MoW scoring query creates a new source and sums the segment's norm lengths. SNII lacks the summary cache that CLucene uses. This adds work proportional to the stored document count per scored field. The scan repeats even at the same read version with the same hidden rows. Consider caching these sums and profiling repeated SNII queries on a large tablet. [SNII collection](https://github.com/apache/doris/blob/a0d993147352ec34120c93871c266be1e34cc22f/be/src/storage/index/inverted/reader/collection_statistics.cpp#L484-L505). The regression matrix should cover both visibility gaps across V2, V3 and SNII. For light deletes, compare scores against a fresh table containing only visible rows, including a state with one replica compacted. For the session options, check that statistics include the delete-sign rows the query returns. The release note should qualify its visible-row and replica-consistency claims until the collector handles these cases. These observations come from code inspection; I have not run a new cluster reproduction. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
