Hi all,
As some of you are probably aware the Log4j 1.x Kafka Appender was
removed in KAFKA-17860.
The Apache Log4j project currently maintains a Kafka Appender for Log4j
Core 2.x[2], but following a poll on `log4j-user@logging`[3] last year
and a long discussion in the PMC, the appender has been deprecated and
has been removed from the 3.x branch. The reasons behind this decision
are mostly:
* There are longstanding bugs opened against the Kafka Appender ([4] and
[5]) that need to be addressed,
* There are no Apache Log4j devs currently working with Kafka. A
personal interpretation of this fact is that the features implemented in
the Kafka Appender are based on the Kafka documentation and not on what
people actually use.
* There is a mismatch between the lifecycle of the Kafka Appender and
Log4j Core: during the lifetime of Log4j Core 2.x, several major version
of Kafka were released. Our appender is basically using only the
features from Kafka 1.x.
* Last but not least, we are not sure if the Kafka Appender is used at
all. In 2.x the Kafka Appender is included in `log4j-core`, so it is
hard to get usage statistics from downloads.
This is why I would like to ask the Kafka community in general and the
dev team in particular, whether:
* Is there a need for a Kafka Appender at all? Instinctively I would say
yes, because Apache Flume is dormant, so out of the four appenders with
guaranteed delivery[6] only Jakarta JMS and JeroMQ will remain.
* Are there some Kafka devs interested in developing one? The old
appender can be taken as prototype, but I suspect that a lot of work is
required to make it industrial-grade.
* Where such an appender should be hosted? This is just a minor
technical problem. If there is a Kafka dev willing to provide long term
help on Kafka-related issues, we can host it at Apache Logging Services.
If the Apache Kafka project wants to host the appender, I can provide
long term support on Log4j Core-related issues.
Piotr
[1] https://issues.apache.org/jira/browse/KAFKA-17860
[2]
https://logging.apache.org/log4j/2.x/manual/appenders/message-queue.html#KafkaAppender
[3] https://lists.apache.org/thread/76f260b933ol7og757123lyl1ckjvm8y
[4]
https://issues.apache.org/jira/browse/LOG4J2-1650?jql=project%20%3D%20LOG4J2%20AND%20resolution%20%3D%20Unresolved%20AND%20text%20~%20%22kafka%22%20ORDER%20BY%20updated%20DESC%2C%20priority%20DESC
[5]
https://github.com/apache/logging-log4j2/issues?q=is%3Aissue+is%3Aopen+kafka
[6] https://logging.apache.org/log4j/2.x/manual/appenders/message-queue.html
