airborne12 opened a new pull request, #69649: URL: https://github.com/apache/doris/pull/69649
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: The auto .prx codec stores each position window in the smallest of PFOR, zstd and raw. Every read of a zstd window pays a decompression that PFOR does not, and the choice ignores that cost. On 99M AgentLogs rows 74% of the position windows read by an unfiltered phrase query were zstd, and decoding them took 58% of the index execution time of the query. PrxWindowLimits gets zstd_min_saving_percent, set from the new mutable config snii_prx_zstd_min_saving_percent. A readable PFOR frame is replaced by the zstd one only if zstd is smaller by at least that percentage of the PFOR frame. The default is 30; 0 picks the smaller frame as before and writes a byte-identical index (99M rows: 61,886,854,661 bytes with and without this change). Readers handle every codec, so the setting affects only segments written afterwards, and compaction rewrites with whatever is set then. The compression is still computed, so the write cost does not change. Effect of the setting, 10M rows (index size, DS7 = unfiltered phrase on the output column, position decode time from the query profile): 10 -> +0.02% size, no change for DS7; 20 -> +0.5%, no change; 30 -> +2.1%, decode 252 ms -> 85 ms; 50 -> +6.3%, no further gain; 100 (never zstd) -> +8.5%. 30 is the smallest value that removes the zstd windows of frequent terms, which zstd beats PFOR by 20-30% on and which phrase queries read most. On 99M rows with 30: index +1.9% (63.07 GB against 61.89 GB, still below the 65.70 GB of V3), load time within 2% of a control load, hot queries in the same process 0.63x (unfiltered phrase), 0.67x (phrase in input or output), 0.75x (phrase prefix), 0.77x, 0.87x, the others within 1.02x (one query measured 1.06x once and 1.00-1.05x in three repeats); cold runs 10-18% faster for the phrase queries. The best value depends on the data, since it is how much zstd beats PFOR on the window contents; it was calibrated on one corpus. ### Release note SNII position windows are stored with zstd only where it saves at least 30% over PFOR (snii_prx_zstd_min_saving_percent): phrase queries are faster and the index is about 2% larger. Set it to 0 for the previous behavior. ### Check List (For Author) - Test: Unit Test on master, 1723 tests pass with the inline-read change stacked (SniiPrxPodTest: the saving boundary, PFOR unreadable, limit validation; SniiWriterGoldenBytes: the config reaches the writer; the default is 30). Manual test: index size and load time at 10M and 99M rows, interleaved hot A/B and repeated cold runs. - Behavior changed: Yes (newly written segments are about 2% larger; set snii_prx_zstd_min_saving_percent to 0 to restore) - Does this need documentation: No 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
