pratyakshsharma commented on a change in pull request #1039: [HUDI-340]: made
max events to read from kafka source configurable
URL: https://github.com/apache/incubator-hudi/pull/1039#discussion_r350014019
##########
File path:
hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/SourceFormatAdapter.java
##########
@@ -126,4 +127,13 @@ public SourceFormatAdapter(Source source) {
throw new IllegalArgumentException("Unknown source type (" +
source.getSourceType() + ")");
}
}
+
+ /**
+ * This method is needed to set Config.DEFAULT_MAX_EVENTS_FROM_KAFKA_SOURCE
to some lower value for test cases,
+ * gets called only from TestKafkaSource.java class
+ * @param maxEventsFromKafkaSource
+ */
+ public void setDefaultMaxEventsFromKafkaSource(long
maxEventsFromKafkaSource) {
Review comment:
@leesf regarding your first point, basically I wanted to show a scenario
where sourceLimit and Config.MAX_EVENTS_FROM_KAFKA_SOURCE_PROP are set to
Long.MAX_VALUE and default value comes into play when the number of events in
kafka topic is greater than that default value (i.e greater than 5M in normal
case). That would require consuming those many events from kafka topic, hence I
added the function to reset DEFAULT_MAX_EVENTS_FROM_KAFKA_SOURCE. IMHO, with
lesser number of events in topic, this scenario is not depicted properly. WDYT?
Regarding second point, my bad, I can directly reset the value in
TestKafkaSource class. Will remove this function.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
