Hi Peter, When acks = all and min.insync.replica= replication factor, HW advancement is done synchronously with acks only because HW advancement is done after all replicas have persisted the message. Other than that there is no clear relation between ISR and HW advancement policy. right?
On Tue, 8 Oct 2019 at 21:25, Peter Bukowinski <pmb...@gmail.com> wrote: > The ack method only affects the producer. With acks=0 or 1, HW advancement > is done asynchronously with the producer, so the leader will continue to > accept writes regardless of follower status. Only if you have acks=all and > min ISR set to the replication factor will your acks be synchronous with > the HW. > > If you are producing to a topic partition that is under-replicated, your > settings determine whether new data can be written to it or not. > > -- Peter (from phone) > > > On Oct 8, 2019, at 8:31 AM, Isuru Boyagane < > isuruboyagane...@cse.mrt.ac.lk> wrote: > > > > Hi Peter, > > > > Okay. Then, whatever the producer acks configuration is, Leader replica > > waits for *all* other follower replicas to persist the message before > > advancing HW(without considering the *current ISR* set)? > > > > Thanks > > > >> On Tue, 8 Oct 2019 at 14:54, Peter Bukowinski <pmb...@gmail.com> wrote: > >> > >> The behavior of the high water mark is independent of producer ack > >> settings. It is a property of topic partitions only. (Remember that > >> multiple producers with different configurations can write to the same > >> topic.) The high water mark advances to the latest offset that exists in > >> all topic partition replicas. If your topic has a replication factor of > 3, > >> then a partition’s HW will only advance toward the LEO after both > replicas > >> are in sync with the leader. > >> > >> min.insync.replicas controls how many replicas must be in sync when your > >> producer in using acks=all. This does not affect consumers. > >> > >> HW does affect consumers, since data between the HW and LEO cannot be > >> consumed yet. > >> > >> -- > >> Peter > >> > >>> On Oct 8, 2019, at 12:35 AM, Isuru Boyagane < > >> isuruboyagane...@cse.mrt.ac.lk> wrote: > >>> > >>> Hi, > >>> > >>> 4 > >>> > >>> I read about the Kafka replication protocol. I found that Kafka > maintains > >>> LEO and HW. As I understood, > >>> > >>> LEO: Offset of latest message a replica has seen. > >>> > >>> HW: Offset of the latest message which is guaranteed that each replica > >> has > >>> persisted into their local log. > >>> > >>> We can set the following acknowledgement methods in Kafka producers > >>> configurations. > >>> > >>> > >>> 1. acks = 0 (Leader replica sends an acknowledgement to the producer > >>> once it has seen the message) > >>> 2. acks = 1 (Leader replica sends an acknowledgement to the > producer > >>> once it has persisted the message to its local log) > >>> 3. acks = all (Leader replica sends an acknowledgement to the > >>> producer once every in-sync replica has persisted the message to > >>> its local > >>> log) > >>> > >>> So my question is how the leader advances the HW depending on the > >>> acknowledgement method message producer uses. > >>> > >>> What I think is, > >>> > >>> 1. > >>> > >>> for acks = 0, Leader advances the HW when it sees a new message. > >>> 2. > >>> > >>> for acks = 1, Leader advances the HW when it has written the new > >> message > >>> to its local log. > >>> 3. > >>> > >>> for acks = all, Leader advances HW when each an every follower sent > ack > >>> that they persisted the message. > >>> > >>> Is this correct? Can anyone clarify this? > >>> > >>> Thanks > >> > > > > > > -- > > isuru > -- Isuru
