mjsax commented on code in PR #23815:
URL: https://github.com/apache/kafka/pull/23815#discussion_r4637343636
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/OptimizedKTableIntegrationTest.java:
##########
@@ -181,21 +194,115 @@ public void shouldApplyUpdatesToStandbyStore(final
TestInfo testInfo) throws Exc
}
}
- private void produceValueRange(final int key, final int start, final int
endExclusive) {
+ @Test
+ public void shouldNotFetchBeyondCommittedOffsetForStandby(final TestInfo
testInfo) throws Exception {
+ final String topic = "standby-limit-input-topic";
+ final TopicPartition partition = new TopicPartition(topic, 0);
+ CLUSTER.createTopic(topic, 1, 1);
+ produceValueRange(topic, 1, 0, 1_000);
+
+ final StreamsBuilder builder = new StreamsBuilder();
+ builder.table(topic, Consumed.with(Serdes.Integer(), Serdes.Integer()),
+ Materialized.<Integer, Integer, KeyValueStore<Bytes,
byte[]>>as(TABLE_NAME).withCachingDisabled());
+
+ final String safeTestName = safeUniqueTestName(testInfo);
+ final AtomicLong standbyNextOffset = new AtomicLong();
+ final RestorePolls restorePolls1 = new RestorePolls();
+ final RestorePolls restorePolls2 = new RestorePolls();
+ final KafkaStreams kafkaStreams1 = createKafkaStreams(builder,
streamsConfiguration(safeTestName), restorePolls1, standbyNextOffset);
+ final KafkaStreams kafkaStreams2 = createKafkaStreams(builder,
streamsConfiguration(safeTestName), restorePolls2, standbyNextOffset);
+ startApplicationAndWaitUntilRunning(Arrays.asList(kafkaStreams1,
kafkaStreams2), Duration.ofSeconds(60));
+
+ TestUtils.waitForCondition(
+ () -> standbyNextOffset.get() >= 1_000L,
+ () -> "Standby did not catch up with the active task, next offset:
" + standbyNextOffset.get());
+
+ final boolean firstIsActive =
kafkaStreams1.metadataForLocalThreads().stream().anyMatch(t ->
!t.activeTasks().isEmpty());
+ final KafkaStreams active = firstIsActive ? kafkaStreams1 :
kafkaStreams2;
+ final RestorePolls standbyPolls = firstIsActive ? restorePolls2 :
restorePolls1;
+
+ // the active task stops making progress, so its committed offset (the
limit of the standby) stays at 1,000
+ active.pause();
+ final long polledBefore = standbyPolls.records.get();
+ produceValueRange(topic, 1, 1_000, 2_000);
+
+ // the new records are beyond the limit and cannot be applied, so the
standby must pause the partition
+ // after the first poll instead of fetching all of them
+ TestUtils.waitForCondition(
+ () -> standbyPolls.records.get() > polledBefore &&
standbyPolls.paused.contains(partition),
+ () -> "Standby did not pause the partition, records polled beyond
the limit: " + (standbyPolls.records.get() - polledBefore));
+ final long polled = standbyPolls.records.get() - polledBefore;
+ assertTrue(polled <= 100, "Standby polled " + polled + " records
beyond the committed offset of the active task");
+ }
Review Comment:
This test is overall great. -- I think it would be even better, if we extend
it further, and `resume()` the paused `KafkaStreams` client. This allows us to
test if the standby (ie, the partitions) also resumes as expected (which we
should verify explicitly), and that both active and standby task eventually are
at offset 2_000.
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/OptimizedKTableIntegrationTest.java:
##########
@@ -181,21 +194,115 @@ public void shouldApplyUpdatesToStandbyStore(final
TestInfo testInfo) throws Exc
}
}
- private void produceValueRange(final int key, final int start, final int
endExclusive) {
+ @Test
+ public void shouldNotFetchBeyondCommittedOffsetForStandby(final TestInfo
testInfo) throws Exception {
+ final String topic = "standby-limit-input-topic";
+ final TopicPartition partition = new TopicPartition(topic, 0);
+ CLUSTER.createTopic(topic, 1, 1);
+ produceValueRange(topic, 1, 0, 1_000);
+
+ final StreamsBuilder builder = new StreamsBuilder();
+ builder.table(topic, Consumed.with(Serdes.Integer(), Serdes.Integer()),
+ Materialized.<Integer, Integer, KeyValueStore<Bytes,
byte[]>>as(TABLE_NAME).withCachingDisabled());
+
+ final String safeTestName = safeUniqueTestName(testInfo);
+ final AtomicLong standbyNextOffset = new AtomicLong();
+ final RestorePolls restorePolls1 = new RestorePolls();
+ final RestorePolls restorePolls2 = new RestorePolls();
+ final KafkaStreams kafkaStreams1 = createKafkaStreams(builder,
streamsConfiguration(safeTestName), restorePolls1, standbyNextOffset);
+ final KafkaStreams kafkaStreams2 = createKafkaStreams(builder,
streamsConfiguration(safeTestName), restorePolls2, standbyNextOffset);
+ startApplicationAndWaitUntilRunning(Arrays.asList(kafkaStreams1,
kafkaStreams2), Duration.ofSeconds(60));
+
+ TestUtils.waitForCondition(
+ () -> standbyNextOffset.get() >= 1_000L,
Review Comment:
I am wondering if we could change the test setup, to only update the standby
to offsets 950 instead (we can mess with the restore consumer and put a gate
into it, to only return data to KS up to 950 -- you should throw away any
excess data and just seek back to offset 950 inside the poll() override). After
we pause the active tasks, we would write more input data, and remove this
gate. This allows us to test that even if we run into pausing, the 50 remaining
records are still restored and don't get stuck in the buffer.
This only tests existing behavior, but I think still worth adding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]