Efrat19 commented on code in PR #293:
URL:
https://github.com/apache/flink-connector-kafka/pull/293#discussion_r3791337191
##########
flink-connector-kafka/src/test/java/org/apache/flink/connector/kafka/source/reader/KafkaSourceReaderTest.java:
##########
@@ -266,6 +273,340 @@ void testOffsetCommitOnCheckpointComplete() throws
Exception {
}
}
+ @Test
+ void testCommittedOffsetConvergesWithLogEndOffsetForTransactionalTopic()
throws Throwable {
Review Comment:
Do you think we could consider a matrix / reusable method for the
`await-stable-offset → assert-readable-count → read-and-commit →
assert-convergence` for these tests to improve readability?
```
- testCommittedOffsetConvergesWithLogEndOffsetForTransactionalTopic
- testCommittedOffsetConvergesWithLogEndOffsetForNonTransactionalTopic
- testCommittedOffsetConvergesWithLogEndOffsetForMultiRecordTransaction
- testCommittedOffsetConvergesWithLogEndOffsetAfterAbortedTransaction
- testCommittedOffsetConvergesWithLogEndOffsetForMultipleTransactions
- testCommittedOffsetConvergesWithLogEndOffsetForInterleavedTransactions
```
They seem to follow the same pattern
claude example suggestion:
```
@FunctionalInterface
private interface RecordProducer {
void produce(String topic) throws Exception;
}
private static Stream<Arguments> convergenceScenarios() {
return Stream.of(
// scenarioName, expectedStableOffset, expectedReadable,
maxCheckpoints, producer
Arguments.of(
"NonTransactional", 1L, 1, 1,
(RecordProducer)
topic ->
KafkaSourceTestEnv.produceToKafka(
Collections.singletonList(
new ProducerRecord<>(
topic, 0, topic
+ "-key", 0)))),
Arguments.of(
"Transactional", 2L, 1, 1,
(RecordProducer)
topic ->
KafkaSourceTestEnv.produceToKafka(
Collections.singletonList(
new ProducerRecord<>(
topic, 0, topic
kafkaServer.getTransactionalProducerConfig())),
Arguments.of(
"MultiRecordTransaction", 4L, 3, 10,
(RecordProducer) topic -> produceInTransaction(topic, 0,
Arguments.of(
"AbortedTransaction", 7L, 2, 10,
(RecordProducer)
topic -> {
produceInTransaction(topic, 0, 2, true);
produceInTransaction(topic, 0, 3, false);
}),
Arguments.of(
"MultipleTransactions", 6L, 3, 10,
(RecordProducer)
topic -> {
produceInTransaction(topic, 0, 1, true);
produceInTransaction(topic, 0, 1, true);
produceInTransaction(topic, 0, 1, true);
}),
Arguments.of(
"InterleavedTransactions", 6L, 4, 10,
(RecordProducer) topic -> produceInterleavedTransactions
}
@ParameterizedTest(name = "{0}")
@MethodSource("convergenceScenarios")
void testCommittedOffsetConvergesWithLogEndOffset(
String scenarioName,
long expectedStableOffset,
int expectedReadable,
int maxCheckpoints,
RecordProducer produce)
throws Throwable {
final String topic = TOPIC + scenarioName;
final String groupId = scenarioName + "OffsetCommitGroup";
final TopicPartition tp = new TopicPartition(topic, 0);
KafkaSourceTestEnv.createTestTopic(topic, 1, 1);
produce.produce(topic);
final long lastStableOffset = awaitLastStableOffset(tp,
expectedStableOffset);
assertThat(getReadCommittedRecordsCount(tp))
.as("Expected readable record count")
.isEqualTo(expectedReadable);
final long committedOffset =
readAndCommitOffset(tp, groupId, expectedReadable,
maxCheckpoints, lastStableOffset);
assertThat(committedOffset)
.as("The committed offset should converge with the log end
offset") .isEqualTo(lastStableOffset);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
