Hi Mark, > the sink has to stop at each checkpoint, wait for all outstanding requests to finish, and force writes to happen in order so you can clearly separate "records before this checkpoint" from "records after it." At that point you’re basically running synchronously, so the whole async throughput benefit is gone.
I second Poorvank's view of it. You can work around it by making the requests themselves idempotent, after all the sink itself doesn't do any processing. Best Regards Ahmed Hamdy On Wed, 27 Aug 2025 at 19:54, Poorvank Bhatia <puravbhat...@gmail.com> wrote: > Hi Mark, > > Afaik, AsyncSinkBase was mainly introduced as a generic helper for > high-throughput async clients. The main goal was to allow many requests in > flight at once with buffering and retries, while still being > checkpoint-aware for *at-least-once*. > > From my understanding, exactly-once might not really fit this model. IMO, > to make exactly-once work, the sink has to stop at each checkpoint, wait > for all outstanding requests to finish, and force writes to happen in order > so you can clearly separate "records before this checkpoint" from "records > after it." At that point you’re basically running synchronously, so the > whole async throughput benefit is gone. > > For systems that actually support transactions (Kafka EOS, JDBC XA, File > sinks, etc.), Flink has separate *committing sink* designs that are a > better fit. > > That’s my read of it — but I might be missing some angle. Could you share > what destination / use case you’re thinking about? If the system supports > transactional semantics, a committing sink might be worth considering; if > not, AsyncSinkBase might already be the right tool (high throughput + > at-least-once). > > On Wed, Aug 27, 2025 at 3:17 PM Mark Zitnik <mark.zit...@gmail.com> wrote: > >> Hi >> >> Reading the AsyncSinkBase source code, I encountered a section of >> limitations. >> >> One of the points is "We are not considering support for exactly-once >> semantics at this point." >> >> Can someone share the reason for this? Any plans to develop it in the >> future? >> >> Regards, >> >> Mark >> >
