I have recently come across several messages that indicate that the SinkTask.preCommit() method should return the offset + 1. This seems incorrect to me as the offset + 1 will not be a valid offset to commit.
My understanding is that PreCommit() is called with a Map of offsets that, according to the documentation, is the highest offset for each topic/partition that was sent in the last SinkTask.put() call. the result of the call should be the topic/partition offsets that are safe to commit. Once the offset is committed it should not be sent again. I have seen several bugs opened on connectors that state the values returned from precommit should be topic/partition offset+1, otherwise on restart the the topic/partition offset will be sent again. If this is true, I think it is a bug in either the Kafka documentation or the process. I would expect that the topic/partition offset would be committed in Kafka and not resent on a restart, but that the next valid offset would be sent. Can anyone tell me if my assumption is correct, that the offset returned from preCommit should be the committed offset from the sink record and not the offset + 1. Thank you, Claude
