airborne12 commented on code in PR #69769:
URL: https://github.com/apache/doris/pull/69769#discussion_r4400353915
##########
be/src/storage/index/inverted/snii/format/prx_pod.cpp:
##########
@@ -555,9 +637,8 @@ Status build_prx_window_auto_from_flat(std::span<const
uint32_t> positions_flat,
testing::note_prx_raw_build();
RETURN_IF_ERROR(zstd_compress(plain.view(), zstd_level,
&compressed));
}
- const AutoPrxCodecChoice choice = select_auto_prx_codec(
- payload.size(), plain.size(), compressed.size(),
has_compressed,
- limits.max_uncomp_bytes, limits.zstd_min_saving_percent);
+ const AutoPrxCodecChoice choice = choose_auto_prx_codec(
Review Comment:
This is deliberate; c360054eee makes it explicit in the class comment and
the config comment.
The budget is a share of the windows that get a zstd frame (a plain payload
of at least 512 bytes), the only ones whose codec a share decides. Singleton
and short windows are never traded, so they are left out of both the allowance
and the warmup:
- Counting their bytes would make the allowance for the large windows grow
with the number of short posting lists an index has: in an index of many rare
terms, its large windows could grow by well over 8% of their own size. The
sizes in the description (the budget below the fixed 30 on every dataset
measured) use this definition.
- Counting them toward the warmup, as `add()` does, would end the prior
share before any tradable window is seen. The share computed from an empty
histogram is 100, so the first large windows would then be traded whatever
their saving until the cap stops it.
So the 8% is a share of the windows the policy acts on, the stricter of the
two readings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
