erikbocks opened a new pull request, #14209: URL: https://github.com/apache/cloudstack/pull/14209
### Description Apache CloudStack offers integrations with several [messaging services](https://docs.cloudstack.apache.org/en/4.22.0.0/adminguide/events.html#event-notification), one of which is [Apache Kafka](https://kafka.apache.org/43/getting-started/introduction/). During service configuration on the platform, [properties](https://kafka.apache.org/43/configuration/producer-configs/) are defined that will be passed to the producer, which is responsible for publishing events. Among these properties, the most important are: - `bootstrap.servers`: List of IP addresses and ports of the brokers (services responsible for receiving messages and serving them to consumers); - `topic`: Message storage unit; - `key.serializer` and `value.serializer`: classes responsible for serializing and deserializing the bytes managed by the brokers. Currently, the only properties managed by Apache CloudStack are `topic`, `key.serializer`, and `value.serializer`. The `topic` property has a default value of `cloudstack`, and the `<key|value>.serializer` properties have the value `org.apache.kafka.common.serialization.StringSerializer`. Other properties can be defined in the `kafka.producer.properties` file, and these are passed automatically. However, it was noticed that if one of the brokers present in `bootstrap.server` is unavailable, message publishing is blocked until a timeout occurs. This blockage culminates in the exhaustion of the available processing thread pool, which in turn causes several other errors. Upon analyzing the message publishing flow, it was observed that the error with the unavailable broker is caused by a search for Kafka cluster metadata, which occurs during the first publication. This search is not asynchronous and aims to return data such as the number of available brokers, topics, and cluster partitions, which are cached and used in message sending. Because the configured broker is unavailable, the thread is stuck for 60000 ms (1 minute). This value is defined by the [max.block.ms](https://kafka.apache.org/43/configuration/producer-configs/#producerconfigs_max.block.ms) setting, which has a default value of 60000 ms, and is responsible for defining the maximum time that the metadata retrieval from the cluster can take and how long the producer should wait for free space in the buffer to write the message. To prevent errors like this from occurring, a default timeout of 2500 ms (2.5s) has been defined if the property has not been set in the configuration file. This is a palliative change, in order to prevent the pool exhaustion. In the future, I pretend to find another way to prevent the error without having to change the `max.block.ms` property. ### Types of changes - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [ ] New feature (non-breaking change which adds functionality) - [ ] Bug fix (non-breaking change which fixes an issue) - [X] Enhancement (improves an existing feature and functionality) - [ ] Cleanup (Code refactoring and cleanup, that may add test cases) - [ ] Build/CI - [ ] Test (unit or integration test code) ### Feature/Enhancement Scale or Bug Severity #### Feature/Enhancement Scale - [ ] Major - [X] Minor ### Screenshots (if appropriate): ### How Has This Been Tested? I created a Kafka cluster following the official guide (see [Kafka Docker image](https://kafka.apache.org/43/getting-started/quickstart/#using-jvm-based-apache-kafka-docker-image)), and validated that even using the new default value of the `max.block.ms` configuration, ACS still could communicate correctly with Kafka. After that, I configured a local Kafka cluster with three nodes, and configured them into ACS. Then, I shut one of the cluster nodes down, and executed an operation that generates an event. When ACS tried to reach the unavailable node, it waited for 2.5 seconds and then move on to the next one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
