Hello everyone! We are using the Log4j Kafka Appender to send application LogEvents to a Kafka topic.
We are using the syncSend=false producer attribute to asynchronously send LogEvents to Kafka. This setting works well for already connected producers which are not able to connect to the cluster anymore. For example due to network connectivity issues. Thus, the kafka producer send() method won't wait for an answer from the brokers and exceptions are logged and discarded. However, for applications that can't initially reach the kafka cluster, the send() method will block for every LogEntry, as configured in the producer property max.block.ms. The following exception is thrown: org.apache.kafka.common.errors.TimeoutException: Topic logs not present in metadata after 1000 ms. This is also described in the kafka producer's max.block.ms property documentation: "The configuration controls how long KafkaProducer.send() and KafkaProducer.partitionsFor() will block. These methods can be blocked either because the buffer is full or metadata unavailable." I wonder if the send() call will also block after metadata.max.age has expired? Is there an out-of-the-box workaround ,thus the send() call will not block, without modifying max.block.ms settings? Or is the only solution an independent kafka consumer which checks the connectivity to the cluster. And enables/disables the log4j kafka appender to prevent blocking the application? Kind regards