Hi, By committing the offsets, I meant tracking the progress of how much data is read from the upstream system. In Kafka Connect this is being referred as committing the offsets. This is the method I was talking about https://github.com/a0x8o/kafka/blob/master/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSourceTask.java#L462-L567
My doubt is that what if the connector gets restarted or the node on which connector is running goes down just before flushing the offsets <https://github.com/a0x8o/kafka/blob/master/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSourceTask.java#L521> . Thank you, Nitin On Thu, Jan 28, 2021 at 9:54 PM Matthias J. Sax <mj...@apache.org> wrote: > I don't know all details of Connect... > > However, not sure what you mean by "committing offsets"? > > A source connector takes data from an external data source and writes it > into a Kafka topic. Thus, there should not be any offsets to be > committed. (Committing offsets only applies if you read from a topic.) > > Instead, the "progress" how much data from the upstream system is read > needs to be tracked. If done right (what I assume Connect does -- not > sure if there might be a concrete connector dependency?) there should > not be out-of-order data. > > But I hope that some Connect expert can chime in... > > > -Matthias > > On 1/28/21 12:24 AM, nitin agarwal wrote: > > Assuming the configurations are as follows: > > max.inflight.requests.per.connection=1 > > enable.idempotence=false > > > > Thanks, > > Nitin > > > > > > On Thu, Jan 28, 2021 at 1:53 PM nitin agarwal <nitingarg...@gmail.com> > > wrote: > > > >> Thanks for quick reply, I have understood this behaviour now. > >> I have another follow up question. > >> > >> Can the Source connector write out of order messages in a case where > there > >> is a failure in committing the offset and the connector is restarted at > the > >> same time? > >> > >> Thanks, > >> Nitin > >> > >> On Thu, Jan 28, 2021 at 8:06 AM Matthias J. Sax <mj...@apache.org> > wrote: > >> > >>> There should not be any data loss. > >>> > >>> However, if a request fails and is retried, it may lead to reordering > of > >>> sends. Thus, records would not be ordered based on the `send()` calls > >>> any longer. > >>> > >>> If you would enable idempotent writes, ordering is guaranteed even with > >>> multiple in-flight requests per connection though. > >>> > >>> > >>> > >>> -Matthias > >>> > >>> On 1/27/21 11:35 AM, nitin agarwal wrote: > >>>> Hi All, > >>>> > >>>> I see that max.inflight.requests.per.connection is set to 1 explicitly > >>> in > >>>> Kafka Connect but there is a way to override it. I want to understand > >>> the > >>>> impact of setting its value > 1. > >>>> As per my understanding, it will lead to data loss in some cases. Is > it > >>>> correct ? > >>>> > >>>> > >>>> Thank you, > >>>> Nitin > >>>> > >>> > >> > > >
