krishvishal commented on code in PR #3746:
URL: https://github.com/apache/iggy/pull/3746#discussion_r3662116602
##########
core/partitions/src/iggy_index_reader.rs:
##########
@@ -114,4 +114,37 @@ impl IggyIndexReader {
.await?,
))
}
+
+ /// Load every whole entry into an [`IggyIndexCache`] for offset /
timestamp
+ /// lower-bound lookups. Reads the whole file in one pass (index files are
+ /// tiny: one sparse entry per flushed chunk). A trailing partial entry
+ /// (torn write) is ignored (see [`Self::entry_count`]).
+ ///
+ /// # Errors
+ ///
+ /// Returns an error if the file metadata or bytes cannot be read.
+ pub async fn load_all(&self) -> Result<IggyIndexCache, IggyError> {
Review Comment:
`load_all()` reads and materializes an entire `.index` file on the poll hot
path with no bound
`core/partitions/src/iggy_index_reader.rs:126`
Index density is one sparse entry per flush, governed by
`messages_required_to_save` and `size_of_messages_required_to_save`
(`core/server-ng/config.toml:515,520`). At defaults a 1 GiB segment yields
about 1024 entries, a 24 KiB read, and the "index files are tiny" comment
holds. Set `messages_required_to_save = 1`, which is what in-tree
`repair_config()` uses, with 100-byte messages, and the same segment yields
7.25M entries: a single 174 MB `read_exact_at` issued from inside a poll, which
compio will punt to io-wq. That is the exact stall this PR set out to remove,
now reachable by supported config. There is no cap, no chunking, and no
fallback.
This also reintroduces up to 12 resident index caches per partition, which
is the thing `core/partitions/src/iggy_partition.rs:2731-2736` deliberately
drops at rotation, with a comment naming retained hundreds of MB as the reason.
Since `IggyIndexCache` is sorted and only ever binary-searched
(`core/partitions/src/iggy_index.rs:91,105`), the bounded alternative is
roughly 20 24-byte `read_entry_at` preads above a byte budget; the reader
already has that primitive at `:67`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
