jamesyfshao opened a new pull request #3584: add kafka simple consumer buffer and timeout to stream config URL: https://github.com/apache/incubator-pinot/pull/3584 When the kafka broker cluster has max message size config bigger than our current hard-coded value of low-level consumer buffer size (512K), pinot kafka consumer could encounter messages from kafka that are bigger than our buffer size. When this happens, kafka low-level consumer will fail silently and returns an empty message result list to low-level message ingester. From pinot users' point of view, pinot kafka low-level consumer stuck at the large message offset. This change aims to allow users to define their kafka ingestor buffer size so it can handle the failure scenarios above.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
