hubcio opened a new pull request, #3974: URL: https://github.com/apache/iggy/pull/3974
ensure_indexes reserved a 16.78 MB Vec (SIZE_16MB / IGGY_INDEX_SIZE = 699050 entries) on the first flush of every active segment, for a sparse index that gains ONE entry per flush. At the shipped cadence a full 1 GiB segment holds about 1024 entries, so the reservation was roughly 700x its ceiling. It stayed invisible until many partitions flushed at once. Graceful shutdown force-flushes every partition regardless of the messages_required_to_save thresholds: with 10000 partitions, 4000 of them holding writes, peak RSS went 3043 -> 6808 MB and VmData 4.04 -> 64.05 GB right after SIGTERM (60.01 GB / 16.78 MB = 3576 reservations). Steady state is worse than the spike, since 10000 actively written partitions would pin 168 GB. Reserve nothing and let the Vec grow: ~1024 entries over a segment's life is eleven reallocations and under 48 KB of memcpy, on a path that runs once per flush. IggyIndexCache is a bare Vec behind len / is_empty / push and two binary searches, and its lookups borrow &self, so growth cannot invalidate a held reference. The shutdown delta is now 0 MB / 0.00 GB, idle RSS unchanged. Also correct the sparse-index density in two doc comments: at messages_required_to_save = 1 a flush still covers a whole batch, so the file tracks every batch, not every message. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
