This is an automated email from the ASF dual-hosted git repository.
spetz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/iggy.git
The following commit(s) were added to refs/heads/master by this push:
new 6a1c84c39 fix(partitions): stop reserving 16 MiB per active segment
index (#3974)
6a1c84c39 is described below
commit 6a1c84c39f06e956699a93fbd9a25215f20840e8
Author: Hubert Gruszecki <[email protected]>
AuthorDate: Thu Aug 27 11:04:09 2026 +0200
fix(partitions): stop reserving 16 MiB per active segment index (#3974)
ensure_indexes reserved a 16.78 MB Vec (SIZE_16MB /
IGGY_INDEX_SIZE = 699050 entries) on the first flush of every
active segment, for a sparse index that gains ONE entry per
flush. At the shipped cadence a full 1 GiB segment holds about
1024 entries, so the reservation was roughly 700x its ceiling.
It stayed invisible until many partitions flushed at once.
Graceful shutdown force-flushes every partition regardless of
the messages_required_to_save thresholds: with 10000 partitions,
4000 of them holding writes, peak RSS went 3043 -> 6808 MB and
VmData 4.04 -> 64.05 GB right after SIGTERM (60.01 GB / 16.78 MB
= 3576 reservations). Steady state is worse than the spike, since
10000 actively written partitions would pin 168 GB.
Reserve nothing and let the Vec grow: ~1024 entries over a
segment's life is eleven reallocations and under 48 KB of memcpy,
on a path that runs once per flush. IggyIndexCache is a bare Vec
behind len / is_empty / push and two binary searches, and its
lookups borrow &self, so growth cannot invalidate a held
reference. The shutdown delta is now 0 MB / 0.00 GB, idle RSS
unchanged.
Also correct the sparse-index density in two doc comments: at
messages_required_to_save = 1 a flush still covers a whole batch,
so the file tracks every batch, not every message.
---
core/partitions/src/iggy_index_reader.rs | 3 ++-
core/partitions/src/iggy_partition.rs | 4 +---
core/partitions/src/log.rs | 7 ++-----
core/partitions/src/poll_plan.rs | 11 ++++++-----
4 files changed, 11 insertions(+), 14 deletions(-)
diff --git a/core/partitions/src/iggy_index_reader.rs
b/core/partitions/src/iggy_index_reader.rs
index 12246cb30..bccdbe27f 100644
--- a/core/partitions/src/iggy_index_reader.rs
+++ b/core/partitions/src/iggy_index_reader.rs
@@ -122,7 +122,8 @@ impl IggyIndexReader {
/// Load every whole entry into an `IggyIndexCache` for offset / timestamp
/// lower-bound lookups in one read. Density is one sparse entry per
flushed
/// chunk, so an aggressive flush cadence (`messages_required_to_save = 1`)
- /// makes the file track every message: callers that cannot afford an
+ /// makes the file track every produced batch - every message only when
+ /// producers send one message per batch: callers that cannot afford an
/// unbounded read must gate on [`Self::entry_count`] and fall back to the
/// on-file lower-bound lookups. A trailing partial entry (torn write) is
/// ignored (see [`Self::entry_count`]).
diff --git a/core/partitions/src/iggy_partition.rs
b/core/partitions/src/iggy_partition.rs
index fb1ba35c2..17bf8aefa 100644
--- a/core/partitions/src/iggy_partition.rs
+++ b/core/partitions/src/iggy_partition.rs
@@ -3769,9 +3769,7 @@ where
self.log.index_writers_mut()[old_segment_index] = None;
// Drop the sealed segment's in-memory index cache: only the ACTIVE
// segment's cache is ever read (the `commit_messages` flush staging),
- // so a sealed cache is dead weight -- and `ensure_indexes`
preallocates
- // a 16 MiB-capacity `Vec` per segment, which under small-segment
- // workloads retains hundreds of MB across thousands of sealed
segments.
+ // so a sealed cache is dead weight.
self.log.indexes_mut()[old_segment_index] = None;
self.log
diff --git a/core/partitions/src/log.rs b/core/partitions/src/log.rs
index 6c43176f3..1dc0fcd3d 100644
--- a/core/partitions/src/log.rs
+++ b/core/partitions/src/log.rs
@@ -15,7 +15,7 @@
// specific language governing permissions and limitations
// under the License.
-use crate::iggy_index::{IGGY_INDEX_SIZE, IggyIndexCache};
+use crate::iggy_index::IggyIndexCache;
use crate::iggy_index_writer::IggyIndexWriter;
use crate::messages_writer::MessagesWriter;
use crate::poll_plan::SealedSegmentHandle;
@@ -30,8 +30,6 @@ use std::rc::Rc;
const SEGMENTS_CAPACITY: usize = 1024;
const ACCESS_MAP_CAPACITY: usize = 8;
-const SIZE_16MB: usize = 16 * 1024 * 1024;
-
/// Max sealed segments per partition that keep a resident read handle (fd +
/// sparse index). Without a cap every sealed segment a reader ever touched
pins
/// one fd for the partition's lifetime; the server-wide budget is this cap
times
@@ -394,8 +392,7 @@ where
.last_mut()
.expect("active indexes called on empty log");
if indexes.is_none() {
- let capacity = SIZE_16MB / IGGY_INDEX_SIZE;
- *indexes = Some(IggyIndexCache::with_capacity(capacity));
+ *indexes = Some(IggyIndexCache::empty());
}
}
diff --git a/core/partitions/src/poll_plan.rs b/core/partitions/src/poll_plan.rs
index 03a74e2da..4fd1ea956 100644
--- a/core/partitions/src/poll_plan.rs
+++ b/core/partitions/src/poll_plan.rs
@@ -48,11 +48,12 @@ use std::sync::atomic::Ordering;
use tracing::{error, warn};
/// Byte cap for materializing a sealed segment's sparse index into its shared
-/// read-state handle. Index density is one entry per flush: at the default
-/// cadence a 1 GiB segment yields ~24 KiB of index, but
-/// `messages_required_to_save = 1` tracks every message and the same segment
-/// yields hundreds of MB, which a single poll must never read (or pin
-/// resident) in one go. At or under the cap the whole file loads once and is
+/// read-state handle. Index density is one entry per flush, never per message:
+/// at the default cadence a 1 GiB segment yields ~24 KiB of index, but
+/// `messages_required_to_save = 1` flushes once per produced batch and the
same
+/// segment yields tens of MB (hundreds only when producers send one message
per
+/// batch), which a single poll must never read (or pin resident) in one go.
+/// At or under the cap the whole file loads once and is
/// cached; above it every poll binary-searches the file with single-entry
/// preads instead, so resident index bytes per partition stay bounded by the
/// sealed-LRU capacity times this cap.
