Hi, Yash. Thanks for picking up this KIP and discussion.
The KIP includes this rejected alternative:
> 4. Update SinkTask.put in any way to pass the new information outside
> SinkRecord (e.g. a Map or a derived class)
>
> -
>
> Much more disruptive change without considerable pros
>
>
One advantage about doing this is that sink connector implementations can
more easily implement two different "put(...)" methods to handle running in
a variety of runtimes, without having to use try-catch logic around the
newer SinkRecord access methods. That latter logic can get quite ugly.
For example, the existing `put` method has this signature:
public abstract void put(Collection<SinkRecord> records);
If we added an overloaded method that passed in a map of the old
topic+partition for each record (and defined the absence of an entry as
having an unchanged topic and partition):
public void put(Collection<SinkRecord> records, Map<SinkRecord,
TopicPartition> updatedTopicPartitions) {
put(records);
}
then a `SinkTask` implementation that wants to use this new feature could
simply implement both methods:
public void put(Collection<SinkRecord> records) {
// Running in an older runtime, so no tracking of SMT-modified topic names
or partitions
put(records, Map.of());
}
public void put(Collection<SinkRecord> records, Map<SinkRecord,
TopicPartition> updatedTopicPartitions) {
// real logic here
}
This seems a lot easier than having to use try-catch logic, yet still
allows sink connectors to utilize the new functionality and still work with
older Connect runtimes.
WDYT?
Randall
On Thu, Sep 8, 2022 at 7:03 AM Yash Mayya <yash.ma...@gmail.com> wrote:
> Hi all,
>
> I would like to (re)start a new discussion thread on KIP-793 (Kafka
> Connect) which proposes some additions to the public SinkRecord interface
> in order to support topic mutating SMTs for sink connectors that do their
> own offset tracking.
>
> Links:
>
> KIP:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191336830
>
> Older discussion thread:
> https://lists.apache.org/thread/00kcth6057jdcsyzgy1x8nb2s1cymy8h,
> https://lists.apache.org/thread/rzqkm0q5y5v3vdjhg8wqppxbkw7nyopj
>
> Jira: https://issues.apache.org/jira/browse/KAFKA-13431
>
>
> Thanks,
> Yash
>
