pratyakshsharma commented on a change in pull request #1039: [HUDI-340]: made
max events to read from kafka source configurable
URL: https://github.com/apache/incubator-hudi/pull/1039#discussion_r350581065
##########
File path:
hudi-utilities/src/test/java/org/apache/hudi/utilities/sources/TestKafkaSource.java
##########
@@ -131,6 +140,78 @@ public void testJsonKafkaSource() throws IOException {
assertEquals(Option.empty(), fetch4AsRows.getBatch());
}
+ @Test
+ public void testJsonKafkaSourceWithDefaultUpperCap() throws IOException {
+ // topic setup.
+ testUtils.createTopic(TEST_TOPIC_NAME, 2);
+ HoodieTestDataGenerator dataGenerator = new HoodieTestDataGenerator();
+ TypedProperties props = createPropsForJsonSource(Long.MAX_VALUE);
+
+ Source jsonSource = new JsonKafkaSource(props, jsc, sparkSession,
schemaProvider);
+ SourceFormatAdapter kafkaSource = new SourceFormatAdapter(jsonSource);
+ Config.DEFAULT_MAX_EVENTS_FROM_KAFKA_SOURCE = 500;
+
+ /*
+ 1. Extract without any checkpoint => get all the data, respecting default
upper cap since both sourceLimit and
+ maxEventsFromKafkaSourceProp are set to Long.MAX_VALUE
+ */
+ testUtils.sendMessages(TEST_TOPIC_NAME,
Helpers.jsonifyRecords(dataGenerator.generateInserts("000",1000)));
Review comment:
Done. @leesf
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
