numinnex commented on code in PR #2958:
URL: https://github.com/apache/iggy/pull/2958#discussion_r2969484504
##########
core/server/src/streaming/partitions/journal.rs:
##########
@@ -19,30 +19,43 @@ use crate::streaming::segments::{IggyMessagesBatchMut,
IggyMessagesBatchSet};
use iggy_common::{IggyByteSize, IggyError};
use std::fmt::Debug;
-// TODO: Will have to revisit this Journal abstraction....
-// I don't like that it has to leak impl detail via the `Inner` struct in
order to be functional.
-
#[derive(Default, Debug)]
pub struct Inner {
- pub base_offset: u64,
+ /// Offset where the next append will land. After commit(), set to
+ /// current_offset + 1. Used in `append()`: `current_offset = next_offset +
+ /// messages_count - 1`.
+ pub next_offset: u64,
Review Comment:
Why we renamed this from `base_offset` to `next_offset` ? I think the name
base_offset was better, as it represents the beginning offset of the journal
log.
##########
core/server/src/shard/system/segments.rs:
##########
@@ -172,6 +182,16 @@ impl IggyShard {
continue;
}
+ // Same barrier as delete_expired_segments_for_partition: do not
+ // delete segments containing unconsumed messages.
+ let min_committed = Self::min_committed_offset(
+ &partition.consumer_offsets,
+ &partition.consumer_group_offsets,
+ );
+ if min_committed.is_some_and(|barrier| first.end_offset > barrier)
{
+ continue;
+ }
+
Review Comment:
Do we really want to prevent deleting expired segments that have registered
consumers via committed offsets ? I am not sure whether it is an correct
behavior + it potentially could lead in some pathological cases, where we have
a zombie consumer (who doesn't consume anything for a long time), that prevents
the reclamation mechanism from deleting expired segments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
