eldenmoon opened a new pull request, #68039:
URL: https://github.com/apache/doris/pull/68039
### What problem does this PR solve?
Issue Number: None
Related PR: #53980, #60722
Problem Summary:
A table with a VARIANT column that holds many sparse paths, indexed with
analyzed `field_pattern` inverted indexes, can see its index size grow by
roughly 10x after compaction. In one reported case, after an upgrade from 3.1
to 4.1, `INDEX_LENGTH` grew from 2.25 TB to 21.3 TB while the row count grew by
0.84%.
Root cause: since #53980 (BM25 scoring),
`InvertedIndexColumnWriter::create_field` turns on CLucene norms. CLucene's
`DocumentsWriter::writeNorms` stores one byte per document for every field that
keeps norms, and fills rows without a value with `defaultNorm`. So a `.nrm`
file always has exactly as many bytes as the segment has rows, however sparse
the column is. Every extracted variant subcolumn gets its own index directory,
so a segment pays `rows × indexed paths × analyzed indexes` bytes of norms.
#60722 already removed norms for non-tokenized indexes, but analyzed indexes
still write them.
Evidence from the reported cluster, read directly from the V2 `.idx` header
and the segment footer of a base-compacted segment:
- **Norms dominate the file:** 4,055,872 rows, 15,543 indexes, and 9,891
`.nrm` files of exactly 4,055,884 bytes each (12-byte header + one byte per
row). Together they are 37.36 GiB, 97.1% of the 38.47 GiB `.idx`.
- **A sparse path pays the same:** the norms of a path whose postings were 6
bytes held 2,160,806 identical `0x01` bytes.
- **3.1-written segments have no `.nrm`:** without norms, the compacted
index was no larger than before.
Fix: analyzed indexes on variant subcolumns (indexes with a non-empty index
suffix) no longer write norms. The new mutable BE config
`inverted_index_write_norms_for_variant_subcolumn` (default `false`) restores
the old behavior. Indexes on ordinary columns are unchanged.
Scoring impact:
- **Missing norms turned BM25 scores into NaN:** CLucene keeps a field's
token count in the `.nrm` header, so `sumTotalTermFreq()` is empty without
norms, `CollectionStatistics` computes avgdl = 0, and
`BM25Similarity::compute_tf_cache` divides 0 by 0. This already happened for
any norm-less segment, for example data written before norms existed.
- **This PR fixes that:** `compute_tf_cache` now skips length normalization
when avgdl is 0, so BM25 on variant subcolumns returns finite scores that
depend on term frequency and idf only.
- **Mixed segments converge:** segments written before this change keep
their norms until compaction rewrites them.
Options considered (smallest first):
1. **Cherry-pick-only fix (#60722):** does not help analyzed indexes, which
are the ones left in this case.
2. **Norms only for analyzed indexes with positions (`support_phrase`):**
does not help either, because these indexes use `support_phrase=true`.
3. **Skip norms for variant subcolumn indexes, with a BE config to restore
them:** this PR.
4. **A new index property to opt in or out of norms:** needs FE, BE, and
docs changes, and existing indexes would have to be rebuilt.
5. **Sparse norms:** a CLucene format change across writer, reader, and
merge. Too large for this fix.
SNII writes norms under its own rule (`_should_analyzer && _has_positions`),
and its scoring path requires norms, so it is left to a follow-up. The default
index storage format is V3, which uses the CLucene writer changed here.
### Release note
Analyzed inverted indexes on variant subcolumns no longer write BM25 norms,
which removes index bloat for variant columns with many sparse indexed paths.
BM25 `score()` on variant subcolumns no longer applies document-length
normalization; set BE config
`inverted_index_write_norms_for_variant_subcolumn=true` to keep norms.
### Check List (For Author)
- Test:
- Unit Test: `BM25SimilarityTest.*` and `InvertedIndexWriterTest.*`, 34
tests passed, including the new `ZeroAvgDlScoresWithoutLengthNorm` and
`NormsFileSkippedForVariantSubcolumn`.
- Regression test: `inverted_index_p0` suites
`test_variant_subcolumn_index_norms` (new), `test_bm25_score_variant` and
`test_omit_norms` pass.
- Manual test: both runs below use 1M rows, 1000 typed sparse paths and
the 11 field_pattern indexes above (3 analyzed), V2 format, ASAN build.
Reproduction: data loaded on 3.1, compacted on 3.1, then upgraded in
place to 4.1 and compacted again.
| Stage | `.idx` | `.nrm` |
|---|---|---|
| 3.1 load (50 rowsets) | 156.8 MiB | 0 |
| 3.1 full compaction | 99.6 MiB | 0 |
| 4.1 full compaction of the same data | 934.1 MiB | 834.5 MiB (875 ×
1,000,012 bytes) |
This PR: norms on/off comparison, on one cluster.
| `inverted_index_write_norms_for_variant_subcolumn` | `.idx` after
full compaction | `.nrm` | BM25 `score()` |
|---|---|---|---|
| `true` (current master behavior) | 457.3 MiB | 357.6 MiB |
0.18668175 |
| `false` (this PR's default) | 99.6 MiB | 0 | 9.656505; returned NaN
before the `compute_tf_cache` guard |
`MATCH_PHRASE`, `MATCH_ANY` and `MATCH_ALL` return the same row counts
and `objectId` sums on both tables, and rank rows in the same order.
- Behavior changed: Yes. BM25 scores on variant subcolumn indexes no longer
use length normalization by default; the index size of such indexes drops.
- Does this need documentation: No
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
