jamesyfshao commented on issue #3869: add config to control kafka fetcher size and increase default URL: https://github.com/apache/incubator-pinot/pull/3869#issuecomment-471895837 > @jamesyfshao Does Kafka pass any exception when the incoming message size is larger than the buffer size? This still does not fix the issue of silent failure. Logging error or throwing exception when it happens would be a good start. Ideally, we should be able to recover from the such situation and let users to pick on the behavior when such failure happens. > > 1. throw exception and stop consume message until a user restarts server with the new config if they don't want to skip the message. > 2. If their use case can tolerate skipping messages, we need to continue to consume from the next offset. Unfortunately. kafka simple consumer doesn't throw any exception while it encountered this issue. I have tested multiple version of kafka simple consumer library (from 0.9 to 1.1.1), they just simply stuck in a loop trying to read from network channel but failed due to there was no buffer left to host the next message. Eventually, there might be a socketTimeoutException after the consumer timed out from the configured value. However, this exception is fairly generic and it is very hard to tell it apart from just losing the connection from kafka broker (which is a retriable exception and reconnect to kafka should solve it). I feel we should be looking at kafka standard consumer API as simpleConsumer is marked as deprecated in later versions of Kafka (1.0+) and it seems to not getting proper bug-fix like this.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
