bvaradar commented on code in PR #8376:
URL: https://github.com/apache/hudi/pull/8376#discussion_r1162386526
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/config/KafkaSourceConfig.java:
##########
@@ -63,6 +63,14 @@ public class KafkaSourceConfig extends HoodieConfig {
.defaultValue(5000000L)
.withDocumentation("Maximum number of records obtained in each batch.");
+ public static final ConfigProperty<Long> MAX_EVENTS_PER_KAFKA_PARTITION =
ConfigProperty
Review Comment:
Wouldn't MAX_EVENTS_FROM_KAFKA_SOURCE/minPartitions (what I referred as
parallelism) be the desired num_events_per_spark_partition which controls the
maximum data volume from Kafka per spark partition ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
