Hi all,

I'd like to start the discussion on KIP-1397, which adds share group
support for Kafka Connect sink tasks.
https://cwiki.apache.org/confluence/spaces/KAFKA/pages/462817279/KIP-1397+Share+Group+Queue+Support+for+Kafka+Connect+Sink+Connectors+via+ShareSinkTask

It is meant to complement KIP-1302 (cc'ing Shekhar Rajak). KIP-1302
introduces the opt-in share runtime for Connect sinks and a task contract
for connectors whose put() returns only once every record is durable.
KIP-1397 keeps that runtime and its durability principle. It adds a second
task contract for connectors whose durable write completes after put()
returns. It also covers poison-pill/DLQ handling, throttling, offset
administration and permissions, which KIP-1302's first version defers. It
builds on the earlier discussion on the KIP-1302 thread and tries to
address some of the points raised there:
https://lists.apache.org/thread/p5zkp20rno8zz55mrjlw9brm6os5gfbv

One thing to settle early: both KIPs use the name ShareSinkTask. Shekhar,
I'm happy to rename mine, or to fold this into KIP-1302 as an additional
contract if you'd prefer. Whatever works best for you.

Feedback on the design, particularly the per-record acknowledgement API and
the DLQ approach, is very welcome. The KIP draft was written with
assistance from Claude; I've reviewed it and take responsibility for its
content.

Thanks,
Ashwin

Reply via email to