so, if i follow your suggested logic correctly, there would be some sort of :
produce(partition, msg, requestedOffset) which would fail if requestedOffset is already taken (by another previous such explicit call or by another regular call that just happened to get assigned that offset by the partition leader on the target cluster). how would you meaningfully handle this failure? suppose this happens to some cross-cluster replicator (like mirror maker). there is no use in retrying. the options would be: 1. get the next available offset - which would violate what youre trying to achieve 2. skip msgs - so replication is incomplete, any offset "already taken" on the destination is not replicated from source 3. stop replication for this partition completely - because starting from now _ALL_ offsets will be taken - 1 foreign msg ruins everything for the entire partition. none of these options look good to me. On Thu, Dec 29, 2016 at 3:22 AM, Andrey L. Neporada < anepor...@yandex-team.ru> wrote: > Hi! > > > On 27 Dec 2016, at 19:35, radai <radai.rosenbl...@gmail.com> wrote: > > > > IIUC if you replicate from a single source cluster to a single target > > cluster, the topic has the same number of partitions on both, and no one > > writes directly to the target cluster (so master --> slave) the offsets > > would be preserved. > > > > Yes, exactly. When you > 1) create topic with the same number of partitions on both master and > slave clusters > 2) write only to master > 3) replicate partition to partition from master to slave > - in this case the offsets will be preserved. > > However, you usually already have cluster that works and want to replicate > some topics to another one. > IMHO, in this scenario there should be a way to make message offsets equal > on both clusters. > > > but in the general case - how would you handle the case where multiple > > producers "claim" the same offset ? > > The same way as Kafka handles concurrent produce requests for the same > partition - produce requests for partition are serialized. > If the next produce request “overlaps” with previous one, it fails. > > > > > > > On Mon, Dec 26, 2016 at 4:52 AM, Andrey L. Neporada < > > anepor...@yandex-team.ru> wrote: > > > >> Hi all! > >> > >> Suppose you have two Kafka clusters and want to replicate topics from > >> primary cluster to secondary one. > >> It would be very convenient for readers if the message offsets for > >> replicated topics would be the same as for primary topics. > >> > >> As far as I know, currently there is no way to achieve this. > >> I wonder is it possible/reasonable to add message offset to > ProduceRequest? > >> > >> > >> — > >> Andrey Neporada > >> > >> > >> > >> > >
