mjsax commented on code in PR #23815:
URL: https://github.com/apache/kafka/pull/23815#discussion_r4637343636


##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/OptimizedKTableIntegrationTest.java:
##########
@@ -181,21 +194,115 @@ public void shouldApplyUpdatesToStandbyStore(final 
TestInfo testInfo) throws Exc
         }
     }
 
-    private void produceValueRange(final int key, final int start, final int 
endExclusive) {
+    @Test
+    public void shouldNotFetchBeyondCommittedOffsetForStandby(final TestInfo 
testInfo) throws Exception {
+        final String topic = "standby-limit-input-topic";
+        final TopicPartition partition = new TopicPartition(topic, 0);
+        CLUSTER.createTopic(topic, 1, 1);
+        produceValueRange(topic, 1, 0, 1_000);
+
+        final StreamsBuilder builder = new StreamsBuilder();
+        builder.table(topic, Consumed.with(Serdes.Integer(), Serdes.Integer()),
+            Materialized.<Integer, Integer, KeyValueStore<Bytes, 
byte[]>>as(TABLE_NAME).withCachingDisabled());
+
+        final String safeTestName = safeUniqueTestName(testInfo);
+        final AtomicLong standbyNextOffset = new AtomicLong();
+        final RestorePolls restorePolls1 = new RestorePolls();
+        final RestorePolls restorePolls2 = new RestorePolls();
+        final KafkaStreams kafkaStreams1 = createKafkaStreams(builder, 
streamsConfiguration(safeTestName), restorePolls1, standbyNextOffset);
+        final KafkaStreams kafkaStreams2 = createKafkaStreams(builder, 
streamsConfiguration(safeTestName), restorePolls2, standbyNextOffset);
+        startApplicationAndWaitUntilRunning(Arrays.asList(kafkaStreams1, 
kafkaStreams2), Duration.ofSeconds(60));
+
+        TestUtils.waitForCondition(
+            () -> standbyNextOffset.get() >= 1_000L,
+            () -> "Standby did not catch up with the active task, next offset: 
" + standbyNextOffset.get());
+
+        final boolean firstIsActive = 
kafkaStreams1.metadataForLocalThreads().stream().anyMatch(t -> 
!t.activeTasks().isEmpty());
+        final KafkaStreams active = firstIsActive ? kafkaStreams1 : 
kafkaStreams2;
+        final RestorePolls standbyPolls = firstIsActive ? restorePolls2 : 
restorePolls1;
+
+        // the active task stops making progress, so its committed offset (the 
limit of the standby) stays at 1,000
+        active.pause();
+        final long polledBefore = standbyPolls.records.get();
+        produceValueRange(topic, 1, 1_000, 2_000);
+
+        // the new records are beyond the limit and cannot be applied, so the 
standby must pause the partition
+        // after the first poll instead of fetching all of them
+        TestUtils.waitForCondition(
+            () -> standbyPolls.records.get() > polledBefore && 
standbyPolls.paused.contains(partition),
+            () -> "Standby did not pause the partition, records polled beyond 
the limit: " + (standbyPolls.records.get() - polledBefore));
+        final long polled = standbyPolls.records.get() - polledBefore;
+        assertTrue(polled <= 100, "Standby polled " + polled + " records 
beyond the committed offset of the active task");
+    }

Review Comment:
   This test is overall great. -- I think it would be even better, if we extend 
it further, and `resume()` the paused `KafkaStreams` client. This allows us to 
test if the standby (ie, the partitions) also resumes as expected (which we 
should verify explicitly), and that both active and standby task eventually are 
at offset 2_000.



##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/OptimizedKTableIntegrationTest.java:
##########
@@ -181,21 +194,115 @@ public void shouldApplyUpdatesToStandbyStore(final 
TestInfo testInfo) throws Exc
         }
     }
 
-    private void produceValueRange(final int key, final int start, final int 
endExclusive) {
+    @Test
+    public void shouldNotFetchBeyondCommittedOffsetForStandby(final TestInfo 
testInfo) throws Exception {
+        final String topic = "standby-limit-input-topic";
+        final TopicPartition partition = new TopicPartition(topic, 0);
+        CLUSTER.createTopic(topic, 1, 1);
+        produceValueRange(topic, 1, 0, 1_000);
+
+        final StreamsBuilder builder = new StreamsBuilder();
+        builder.table(topic, Consumed.with(Serdes.Integer(), Serdes.Integer()),
+            Materialized.<Integer, Integer, KeyValueStore<Bytes, 
byte[]>>as(TABLE_NAME).withCachingDisabled());
+
+        final String safeTestName = safeUniqueTestName(testInfo);
+        final AtomicLong standbyNextOffset = new AtomicLong();
+        final RestorePolls restorePolls1 = new RestorePolls();
+        final RestorePolls restorePolls2 = new RestorePolls();
+        final KafkaStreams kafkaStreams1 = createKafkaStreams(builder, 
streamsConfiguration(safeTestName), restorePolls1, standbyNextOffset);
+        final KafkaStreams kafkaStreams2 = createKafkaStreams(builder, 
streamsConfiguration(safeTestName), restorePolls2, standbyNextOffset);
+        startApplicationAndWaitUntilRunning(Arrays.asList(kafkaStreams1, 
kafkaStreams2), Duration.ofSeconds(60));
+
+        TestUtils.waitForCondition(
+            () -> standbyNextOffset.get() >= 1_000L,

Review Comment:
   I am wondering if we could change the test setup, to only update the standby 
to offsets 950 instead (we can mess with the restore consumer and put a gate 
into it, to only return data to KS up to 950 -- you should throw away any 
excess data and just seek back to offset 950 inside the poll() override). After 
we pause the active tasks, we would write more input data, and remove this 
gate. This allows us to test that even if we run into pausing, the 50 remaining 
records are still restored and don't get stuck in the buffer.
   
   This only tests existing behavior, but I think still worth adding.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to