C0urante commented on code in PR #13429: URL: https://github.com/apache/kafka/pull/13429#discussion_r1151128638
########## connect/mirror/src/main/java/org/apache/kafka/connect/mirror/OffsetSyncStore.java: ########## @@ -25,17 +25,37 @@ import org.apache.kafka.common.utils.Utils; import org.apache.kafka.connect.util.KafkaBasedLog; import org.apache.kafka.connect.util.TopicAdmin; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import java.util.Arrays; import java.util.Collections; import java.util.Map; import java.util.Optional; import java.util.OptionalLong; import java.util.concurrent.ConcurrentHashMap; -/** Used internally by MirrorMaker. Stores offset syncs and performs offset translation. */ +/** + * Used internally by MirrorMaker. Stores offset syncs and performs offset translation. + * <p>A limited number of offset syncs can be stored per TopicPartition, in a way which provides better translation + * later in the topic, closer to the live end of the topic. + * This maintains the following invariants for each topic-partition in the in-memory sync storage: + * <ul> + * <li>syncs[0] is the latest offset sync from the syncs topic</li> + * <li>For each i,j, i <= j: syncs[j].upstream <= syncs[i].upstream < syncs[j].upstream + 2^j</li> + * </ul> + * <p>Offset translation uses the syncs[i] which most closely precedes the upstream consumer group's current offset. + * For a fixed in-memory state, translation of variable upstream offsets will be monotonic. + * For variable in-memory state, translation of a fixed upstream offset will not be monotonic. + * <p>Translation will be unavailable for all topic-partitions before an initial read-to-end of the offset syncs topic + * is complete. Translation will be unavailable after that if no syncs are present for a topic-partition, or if relevant + * offset syncs for the topic were eligible for compaction at the time of the initial read-to-end. Review Comment: Thanks, about to read the explanation on the `!readToEnd` check 👍 Does the part of the Javadoc on the availability of translation still need to be updated? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
