tibrewalpratik17 opened a new pull request, #14638: URL: https://github.com/apache/pinot/pull/14638
### Issue During the `segmentCommitEndWithMetadata` call from the server to the controller at commit time, the controller persists information about the committing segment in Zookeeper (ZK). It then attempts to push ZK metadata for the new segments by fetching the `List<PartitionInfo>` of the Kafka topic. If this Kafka call fails due to network issues, the new segment is not created, and consumption halts. While the system has an auto-recover periodic task in the controller that runs every hour to restore consumption, this delay impacts the real-time processing pipeline. ### Root cause Upon investigation, the issue was traced back to the following: - **Kafka Broker Unavailability**: The Kafka broker being queried was down due to issues on its end. Metadata-fetch operations in Kafka are designed to try alternative brokers in a round-robin fashion if the queried broker is unavailable. - **Timeout Mismatch**: Kafka’s `socket.connection.setup.timeout.ms` (default: 10 seconds) specifies the wait time before switching to another broker. Pinot’s timeout for the metadata-fetch call was set to 5 seconds, which is shorter than Kafka's round-robin timeout, leading to Pinot prematurely failing the request. - **Kafka’s Broker List Behavior**: Kafka does not shuffle the list of broker IPs before initiating round-robin retries. Instead, it processes the list as is, potentially retrying the same set of brokers in the same order across successive attempts. ### Solution #### Immediate Hotfix **Increase Pinot’s Timeout**: The default timeout for the metadata-fetch call in Pinot is increased from 5 seconds to 15 seconds. This ensures Kafka has sufficient time to try to connect with at least one more broker in round-robin cycle through the broker list, improving the chances of a successful metadata-fetch. #### Follow-Up Actions: - **Make Timeout Configurable**: Introduce a configurable property for Pinot’s timeout, allowing adjustments based on deployment scenarios and Kafka configurations. - **Adjust Kafka Consumer Configuration**: Investigate setting a lower value for `Kafka’s socket.connection.setup.timeout.ms` to align it with Pinot’s requirements. However, this must be carefully evaluated since the same consumer configuration is used in servers for real-time consumption. - **Improve Kafka Broker Selection (Kafka side improvement)**: A potential enhancement for Kafka could involve shuffling the broker list before initiating round-robin retries. This ensures that retries are evenly distributed across brokers, reducing the likelihood of retrying problematic brokers consecutively. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
