Zihan Li created GOBBLIN-1956:
---------------------------------
Summary: Make Kafka streaming pipeline be able to config the max
poll records during runtime
Key: GOBBLIN-1956
URL: https://issues.apache.org/jira/browse/GOBBLIN-1956
Project: Apache Gobblin
Issue Type: Improvement
Reporter: Zihan Li
Kafka max.poll.records affect the performance of the Kafka consumer.
Config it to be a lower number means it take more time to process each record.
While config it to be a higher number means we need more memory to hold the
records. So we should use the avg record size to determine the value for
different topics.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
