I have recently come across several messages that indicate that the
SinkTask.preCommit() method should return the offset + 1.  This seems
incorrect to me as the offset + 1 will not be a valid offset to commit.

My understanding is that PreCommit() is called with a Map of offsets that,
according to the documentation, is the highest offset for each
topic/partition that was sent in the last SinkTask.put() call.  the result
of the call should be the topic/partition offsets that are safe to commit.

Once the offset is committed it should not be sent again.

I have seen several bugs opened on connectors that state the values
returned from precommit should be topic/partition offset+1, otherwise on
restart the the topic/partition offset will be sent again.

If this is true, I think it is a bug in either the Kafka documentation or
the process.  I would expect that the topic/partition offset would be
committed in Kafka and not resent on a restart, but that the next valid
offset would be sent.

Can anyone tell me if my assumption is correct, that the offset returned
from preCommit should be the committed offset from the sink record and not
the offset + 1.

Thank you,
Claude

Reply via email to