mynameborat commented on a change in pull request #1339: SAMZA-2503 : Adding
retry when getting ssp metadata from kafka
URL: https://github.com/apache/samza/pull/1339#discussion_r405206197
##########
File path:
samza-kafka/src/test/java/org/apache/samza/system/kafka/TestKafkaSystemAdminWithMock.java
##########
@@ -308,4 +309,46 @@ public void
testGetSystemStreamPartitionCountsShouldTerminateAfterFiniteRetriesO
kafkaSystemAdmin.getSystemStreamPartitionCounts(streamNames, cacheTTL);
}
+
+ @Test
+ public void testGetSSPMetadataWithRetry() {
+ SystemStreamPartition oneSSP = new SystemStreamPartition(TEST_SYSTEM,
VALID_TOPIC, new Partition(0));
+ SystemStreamPartition otherSSP = new SystemStreamPartition(TEST_SYSTEM,
"otherTopic", new Partition(1));
+ ImmutableSet<SystemStreamPartition> ssps = ImmutableSet.of(oneSSP,
otherSSP);
+ List<TopicPartition> topicPartitions = ssps.stream()
+ .map(ssp -> new TopicPartition(ssp.getStream(),
ssp.getPartition().getPartitionId()))
+ .collect(Collectors.toList());
+ Map<TopicPartition, Long> testBeginningOffsets =
+ ImmutableMap.of(testTopicPartition0,
KAFKA_BEGINNING_OFFSET_FOR_PARTITION0, testTopicPartition1,
+ KAFKA_BEGINNING_OFFSET_FOR_PARTITION1);
+
+ when(mockKafkaConsumer.beginningOffsets(topicPartitions)).thenThrow(new
RuntimeException())
+ .thenReturn(testBeginningOffsets);
+ Map<SystemStreamPartition,
SystemStreamMetadata.SystemStreamPartitionMetadata> sspMetadata =
+ kafkaSystemAdmin.getSSPMetadata(ssps);
+
+ assertEquals("metadata should return for 2 topics", sspMetadata.size(), 2);
+
+ // retried twice because the first fails and the second succeeds
+ Mockito.verify(mockKafkaConsumer,
Mockito.times(2)).beginningOffsets(topicPartitions);
+ }
+
+ @Test(expected = SamzaException.class)
+ public void testGetSSPMetadataShouldTerminateAfterFiniteRetriesOnException()
throws Exception{
Review comment:
The reason why I suggested to make the retry method package private and
write tests against it is because you can override the policy for tests. With
the current setup, this test is going to always take ~ 16 seconds regardless.
Given our builds are already impacted due to tests, I'd suggest to override
defaults to smaller multipliers and values for tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
