Logging in Kafka
If Kafka 4.0 is Java 11+, why would you use SLF4J instead of the Java 9+ logging facade System.Logger?
Re: Logging in Kafka
Hi Mickael, I agree it's good to make it clear about what we're going to adopt for the logging library. For keeping reload4j or adopting log4j2, I don't have any preference TBH. But if Viktor is willing to drive log4j2 tasks, then we don't have any reason not to adopt log4j2. (Thanks Viktor!) Thanks Luke On Thu, Jan 11, 2024 at 4:02 AM Mickael Maison wrote: > Hi, > > I think the only thing that would need to be done in 3.8 is the > deprecation of the log4j appender (KIP-719). This was a pre-req for > migrating to log4j2 due to conflicts when having both log4j and log4j2 > in the classpath. I don't know if that's still the case with reload4j > but I think we should take the opportunity of deprecating it before > 4.0 regardless to avoid relying on multiple logging libraries. > > Thanks, > Mickael > > On Wed, Jan 10, 2024 at 7:58 PM Colin McCabe wrote: > > > > Hi Mickael, > > > > Thanks for bringing this up. > > > > If we move to log4j2 in 4.0, is there any work that needs to be done in > 3.8? That's probably what we should focus on. > > > > P.S. My assumption is that if the log4j2 work misses the train, we'll > stick with reload4j in 4.0. Hopefully this won't happen. > > > > best, > > Colin > > > > > > On Wed, Jan 10, 2024, at 09:13, Ismael Juma wrote: > > > Hi Viktor, > > > > > > A logging library that requires Java 17 is a deal breaker since we > need to > > > log from modules that will only require Java 11 in Apache Kafka 4.0. > > > > > > Ismael > > > > > > On Wed, Jan 10, 2024 at 6:43 PM Viktor Somogyi-Vass > > > wrote: > > > > > >> Hi Mickael, > > >> > > >> Reacting to your points: > > >> 1. I think it's somewhat unfortunate that we provide an appender tied > to a > > >> chosen logger implementation. I think that this shouldn't be part of > the > > >> project in its current form. However, there is the sl4fj2 Fluent API > which > > >> may solve our problem and turn KafkaLog4jAppender into a generic > > >> implementation that doesn't depend on a specific library given that > we can > > >> upgrade to slf4j2. That is worth considering. > > >> 2. Since KIP-1013 we'd move to Java17 anyways by 4.0, so I don't feel > it's > > >> a problem if there's a specific dependency that has Java17 as the > minimum > > >> supported version. As I read though from your email thread with the > log4j2 > > >> folks, it'll be supported for years to come and log4j3 isn't yet > stable. > > >> Since we already use log4j2 in our fork, I'm happy to contribute to > this, > > >> review PRs or drive it if needed. > > >> > > >> Thanks, > > >> Viktor > > >> > > >> On Wed, Jan 10, 2024 at 3:58 PM Mickael Maison < > mickael.mai...@gmail.com> > > >> wrote: > > >> > > >> > I asked for details about the future of log4j2 on the logging user > list: > > >> > https://lists.apache.org/thread/6n6bkgwj8tglgdgzz8wxhkx1p1xpwodl > > >> > > > >> > Let's see what they say. > > >> > > > >> > Thanks, > > >> > Mickael > > >> > > > >> > On Wed, Jan 10, 2024 at 3:23 PM Ismael Juma > wrote: > > >> > > > > >> > > Hi Mickael, > > >> > > > > >> > > Thanks for starting the discussion and for summarizing the state > of > > >> > play. I > > >> > > agree with you that it would be important to understand how long > log4j2 > > >> > > will be supported for. An alternative would be sl4fj 2.x and > logback. > > >> > > > > >> > > Ismael > > >> > > > > >> > > On Wed, Jan 10, 2024 at 2:17 PM Mickael Maison < > > >> mickael.mai...@gmail.com > > >> > > > > >> > > wrote: > > >> > > > > >> > > > Hi, > > >> > > > > > >> > > > Starting a new thread to discuss the current logging situation > in > > >> > > > Kafka. I'll restate everything we know but see the [DISCUSS] > Road to > > >> > > > Kafka 4.0 if you are interested in what has already been said. > [0] > > >> > > > > > >> > > > Currently Kafka uses SLF4J and reload4j as the logging backend. > We > > >> had > > >> >
Re: Logging in Kafka
Hi, I think the only thing that would need to be done in 3.8 is the deprecation of the log4j appender (KIP-719). This was a pre-req for migrating to log4j2 due to conflicts when having both log4j and log4j2 in the classpath. I don't know if that's still the case with reload4j but I think we should take the opportunity of deprecating it before 4.0 regardless to avoid relying on multiple logging libraries. Thanks, Mickael On Wed, Jan 10, 2024 at 7:58 PM Colin McCabe wrote: > > Hi Mickael, > > Thanks for bringing this up. > > If we move to log4j2 in 4.0, is there any work that needs to be done in 3.8? > That's probably what we should focus on. > > P.S. My assumption is that if the log4j2 work misses the train, we'll stick > with reload4j in 4.0. Hopefully this won't happen. > > best, > Colin > > > On Wed, Jan 10, 2024, at 09:13, Ismael Juma wrote: > > Hi Viktor, > > > > A logging library that requires Java 17 is a deal breaker since we need to > > log from modules that will only require Java 11 in Apache Kafka 4.0. > > > > Ismael > > > > On Wed, Jan 10, 2024 at 6:43 PM Viktor Somogyi-Vass > > wrote: > > > >> Hi Mickael, > >> > >> Reacting to your points: > >> 1. I think it's somewhat unfortunate that we provide an appender tied to a > >> chosen logger implementation. I think that this shouldn't be part of the > >> project in its current form. However, there is the sl4fj2 Fluent API which > >> may solve our problem and turn KafkaLog4jAppender into a generic > >> implementation that doesn't depend on a specific library given that we can > >> upgrade to slf4j2. That is worth considering. > >> 2. Since KIP-1013 we'd move to Java17 anyways by 4.0, so I don't feel it's > >> a problem if there's a specific dependency that has Java17 as the minimum > >> supported version. As I read though from your email thread with the log4j2 > >> folks, it'll be supported for years to come and log4j3 isn't yet stable. > >> Since we already use log4j2 in our fork, I'm happy to contribute to this, > >> review PRs or drive it if needed. > >> > >> Thanks, > >> Viktor > >> > >> On Wed, Jan 10, 2024 at 3:58 PM Mickael Maison > >> wrote: > >> > >> > I asked for details about the future of log4j2 on the logging user list: > >> > https://lists.apache.org/thread/6n6bkgwj8tglgdgzz8wxhkx1p1xpwodl > >> > > >> > Let's see what they say. > >> > > >> > Thanks, > >> > Mickael > >> > > >> > On Wed, Jan 10, 2024 at 3:23 PM Ismael Juma wrote: > >> > > > >> > > Hi Mickael, > >> > > > >> > > Thanks for starting the discussion and for summarizing the state of > >> > play. I > >> > > agree with you that it would be important to understand how long log4j2 > >> > > will be supported for. An alternative would be sl4fj 2.x and logback. > >> > > > >> > > Ismael > >> > > > >> > > On Wed, Jan 10, 2024 at 2:17 PM Mickael Maison < > >> mickael.mai...@gmail.com > >> > > > >> > > wrote: > >> > > > >> > > > Hi, > >> > > > > >> > > > Starting a new thread to discuss the current logging situation in > >> > > > Kafka. I'll restate everything we know but see the [DISCUSS] Road to > >> > > > Kafka 4.0 if you are interested in what has already been said. [0] > >> > > > > >> > > > Currently Kafka uses SLF4J and reload4j as the logging backend. We > >> had > >> > > > to adopt reload4j in 3.2.0 as log4j was end of life and has a few > >> > > > security issues. > >> > > > > >> > > > In 2020 we adopted KIP-653 to upgrade to log4j2. Due to > >> > > > incompatibilities in the configuration mechanism with log4j/reload4j > >> > > > we decide to delay the upgrade to the next major release, Kafka 4.0. > >> > > > > >> > > > Kafka also currently provides a log4j appender. In 2022, we adopted > >> > > > KIP-719 to deprecate it since we wanted to switch to log4j2. At the > >> > > > time Apache Logging also had a Kafka appender that worked with > >> log4j2. > >> > > > They since deprecated that appender in log4j2 and it is not part of > >> > > > log4j3. [1] > >> > > > > >> &g
Re: Logging in Kafka
Hi Mickael, Thanks for bringing this up. If we move to log4j2 in 4.0, is there any work that needs to be done in 3.8? That's probably what we should focus on. P.S. My assumption is that if the log4j2 work misses the train, we'll stick with reload4j in 4.0. Hopefully this won't happen. best, Colin On Wed, Jan 10, 2024, at 09:13, Ismael Juma wrote: > Hi Viktor, > > A logging library that requires Java 17 is a deal breaker since we need to > log from modules that will only require Java 11 in Apache Kafka 4.0. > > Ismael > > On Wed, Jan 10, 2024 at 6:43 PM Viktor Somogyi-Vass > wrote: > >> Hi Mickael, >> >> Reacting to your points: >> 1. I think it's somewhat unfortunate that we provide an appender tied to a >> chosen logger implementation. I think that this shouldn't be part of the >> project in its current form. However, there is the sl4fj2 Fluent API which >> may solve our problem and turn KafkaLog4jAppender into a generic >> implementation that doesn't depend on a specific library given that we can >> upgrade to slf4j2. That is worth considering. >> 2. Since KIP-1013 we'd move to Java17 anyways by 4.0, so I don't feel it's >> a problem if there's a specific dependency that has Java17 as the minimum >> supported version. As I read though from your email thread with the log4j2 >> folks, it'll be supported for years to come and log4j3 isn't yet stable. >> Since we already use log4j2 in our fork, I'm happy to contribute to this, >> review PRs or drive it if needed. >> >> Thanks, >> Viktor >> >> On Wed, Jan 10, 2024 at 3:58 PM Mickael Maison >> wrote: >> >> > I asked for details about the future of log4j2 on the logging user list: >> > https://lists.apache.org/thread/6n6bkgwj8tglgdgzz8wxhkx1p1xpwodl >> > >> > Let's see what they say. >> > >> > Thanks, >> > Mickael >> > >> > On Wed, Jan 10, 2024 at 3:23 PM Ismael Juma wrote: >> > > >> > > Hi Mickael, >> > > >> > > Thanks for starting the discussion and for summarizing the state of >> > play. I >> > > agree with you that it would be important to understand how long log4j2 >> > > will be supported for. An alternative would be sl4fj 2.x and logback. >> > > >> > > Ismael >> > > >> > > On Wed, Jan 10, 2024 at 2:17 PM Mickael Maison < >> mickael.mai...@gmail.com >> > > >> > > wrote: >> > > >> > > > Hi, >> > > > >> > > > Starting a new thread to discuss the current logging situation in >> > > > Kafka. I'll restate everything we know but see the [DISCUSS] Road to >> > > > Kafka 4.0 if you are interested in what has already been said. [0] >> > > > >> > > > Currently Kafka uses SLF4J and reload4j as the logging backend. We >> had >> > > > to adopt reload4j in 3.2.0 as log4j was end of life and has a few >> > > > security issues. >> > > > >> > > > In 2020 we adopted KIP-653 to upgrade to log4j2. Due to >> > > > incompatibilities in the configuration mechanism with log4j/reload4j >> > > > we decide to delay the upgrade to the next major release, Kafka 4.0. >> > > > >> > > > Kafka also currently provides a log4j appender. In 2022, we adopted >> > > > KIP-719 to deprecate it since we wanted to switch to log4j2. At the >> > > > time Apache Logging also had a Kafka appender that worked with >> log4j2. >> > > > They since deprecated that appender in log4j2 and it is not part of >> > > > log4j3. [1] >> > > > >> > > > Log4j3 is also nearing release but it seems it will require Java 17. >> > > > The website states Java 11 [2] but the artifacts from the latest >> 3.0.0 >> > > > beta are built for Java 17. I was not able to find clear maintenance >> > > > statement about log4j2 once log4j3 gets released. >> > > > >> > > > The question is where do we go from here? >> > > > We can stick with our plans: >> > > > 1. Deprecate the appender in the next 3.x release and plan to remove >> > it in >> > > > 4.0 >> > > > 2. Do the necessary work to switch to log4j2 in 4.0 >> > > > If so we need people to drive these work items. We have PRs for these >> > > > with hopefully the bulk of the code but they need >> > > > rebasing/completing/reviewing. >> > > > >> > > > Otherwise we can reconsider KIP-653 and/or KIP-719. >> > > > >> > > > Assuming log4j2 does not go end of life in the near future (We can >> > > > reach out to Apache Logging to clarify that point.), I think it still >> > > > makes sense to adopt it. I would also go ahead and deprecate our >> > > > appender. >> > > > >> > > > Thanks, >> > > > Mickael >> > > > >> > > > 0: https://lists.apache.org/thread/q0sz910o1y9mhq159oy16w31d6dzh79f >> > > > 1: https://github.com/apache/logging-log4j2/issues/1951 >> > > > 2: https://logging.apache.org/log4j/3.x/#requirements >> > > > >> > >>
Re: Logging in Kafka
Hi Viktor, A logging library that requires Java 17 is a deal breaker since we need to log from modules that will only require Java 11 in Apache Kafka 4.0. Ismael On Wed, Jan 10, 2024 at 6:43 PM Viktor Somogyi-Vass wrote: > Hi Mickael, > > Reacting to your points: > 1. I think it's somewhat unfortunate that we provide an appender tied to a > chosen logger implementation. I think that this shouldn't be part of the > project in its current form. However, there is the sl4fj2 Fluent API which > may solve our problem and turn KafkaLog4jAppender into a generic > implementation that doesn't depend on a specific library given that we can > upgrade to slf4j2. That is worth considering. > 2. Since KIP-1013 we'd move to Java17 anyways by 4.0, so I don't feel it's > a problem if there's a specific dependency that has Java17 as the minimum > supported version. As I read though from your email thread with the log4j2 > folks, it'll be supported for years to come and log4j3 isn't yet stable. > Since we already use log4j2 in our fork, I'm happy to contribute to this, > review PRs or drive it if needed. > > Thanks, > Viktor > > On Wed, Jan 10, 2024 at 3:58 PM Mickael Maison > wrote: > > > I asked for details about the future of log4j2 on the logging user list: > > https://lists.apache.org/thread/6n6bkgwj8tglgdgzz8wxhkx1p1xpwodl > > > > Let's see what they say. > > > > Thanks, > > Mickael > > > > On Wed, Jan 10, 2024 at 3:23 PM Ismael Juma wrote: > > > > > > Hi Mickael, > > > > > > Thanks for starting the discussion and for summarizing the state of > > play. I > > > agree with you that it would be important to understand how long log4j2 > > > will be supported for. An alternative would be sl4fj 2.x and logback. > > > > > > Ismael > > > > > > On Wed, Jan 10, 2024 at 2:17 PM Mickael Maison < > mickael.mai...@gmail.com > > > > > > wrote: > > > > > > > Hi, > > > > > > > > Starting a new thread to discuss the current logging situation in > > > > Kafka. I'll restate everything we know but see the [DISCUSS] Road to > > > > Kafka 4.0 if you are interested in what has already been said. [0] > > > > > > > > Currently Kafka uses SLF4J and reload4j as the logging backend. We > had > > > > to adopt reload4j in 3.2.0 as log4j was end of life and has a few > > > > security issues. > > > > > > > > In 2020 we adopted KIP-653 to upgrade to log4j2. Due to > > > > incompatibilities in the configuration mechanism with log4j/reload4j > > > > we decide to delay the upgrade to the next major release, Kafka 4.0. > > > > > > > > Kafka also currently provides a log4j appender. In 2022, we adopted > > > > KIP-719 to deprecate it since we wanted to switch to log4j2. At the > > > > time Apache Logging also had a Kafka appender that worked with > log4j2. > > > > They since deprecated that appender in log4j2 and it is not part of > > > > log4j3. [1] > > > > > > > > Log4j3 is also nearing release but it seems it will require Java 17. > > > > The website states Java 11 [2] but the artifacts from the latest > 3.0.0 > > > > beta are built for Java 17. I was not able to find clear maintenance > > > > statement about log4j2 once log4j3 gets released. > > > > > > > > The question is where do we go from here? > > > > We can stick with our plans: > > > > 1. Deprecate the appender in the next 3.x release and plan to remove > > it in > > > > 4.0 > > > > 2. Do the necessary work to switch to log4j2 in 4.0 > > > > If so we need people to drive these work items. We have PRs for these > > > > with hopefully the bulk of the code but they need > > > > rebasing/completing/reviewing. > > > > > > > > Otherwise we can reconsider KIP-653 and/or KIP-719. > > > > > > > > Assuming log4j2 does not go end of life in the near future (We can > > > > reach out to Apache Logging to clarify that point.), I think it still > > > > makes sense to adopt it. I would also go ahead and deprecate our > > > > appender. > > > > > > > > Thanks, > > > > Mickael > > > > > > > > 0: https://lists.apache.org/thread/q0sz910o1y9mhq159oy16w31d6dzh79f > > > > 1: https://github.com/apache/logging-log4j2/issues/1951 > > > > 2: https://logging.apache.org/log4j/3.x/#requirements > > > > > > >
Re: Logging in Kafka
Hi Mickael, Reacting to your points: 1. I think it's somewhat unfortunate that we provide an appender tied to a chosen logger implementation. I think that this shouldn't be part of the project in its current form. However, there is the sl4fj2 Fluent API which may solve our problem and turn KafkaLog4jAppender into a generic implementation that doesn't depend on a specific library given that we can upgrade to slf4j2. That is worth considering. 2. Since KIP-1013 we'd move to Java17 anyways by 4.0, so I don't feel it's a problem if there's a specific dependency that has Java17 as the minimum supported version. As I read though from your email thread with the log4j2 folks, it'll be supported for years to come and log4j3 isn't yet stable. Since we already use log4j2 in our fork, I'm happy to contribute to this, review PRs or drive it if needed. Thanks, Viktor On Wed, Jan 10, 2024 at 3:58 PM Mickael Maison wrote: > I asked for details about the future of log4j2 on the logging user list: > https://lists.apache.org/thread/6n6bkgwj8tglgdgzz8wxhkx1p1xpwodl > > Let's see what they say. > > Thanks, > Mickael > > On Wed, Jan 10, 2024 at 3:23 PM Ismael Juma wrote: > > > > Hi Mickael, > > > > Thanks for starting the discussion and for summarizing the state of > play. I > > agree with you that it would be important to understand how long log4j2 > > will be supported for. An alternative would be sl4fj 2.x and logback. > > > > Ismael > > > > On Wed, Jan 10, 2024 at 2:17 PM Mickael Maison > > > wrote: > > > > > Hi, > > > > > > Starting a new thread to discuss the current logging situation in > > > Kafka. I'll restate everything we know but see the [DISCUSS] Road to > > > Kafka 4.0 if you are interested in what has already been said. [0] > > > > > > Currently Kafka uses SLF4J and reload4j as the logging backend. We had > > > to adopt reload4j in 3.2.0 as log4j was end of life and has a few > > > security issues. > > > > > > In 2020 we adopted KIP-653 to upgrade to log4j2. Due to > > > incompatibilities in the configuration mechanism with log4j/reload4j > > > we decide to delay the upgrade to the next major release, Kafka 4.0. > > > > > > Kafka also currently provides a log4j appender. In 2022, we adopted > > > KIP-719 to deprecate it since we wanted to switch to log4j2. At the > > > time Apache Logging also had a Kafka appender that worked with log4j2. > > > They since deprecated that appender in log4j2 and it is not part of > > > log4j3. [1] > > > > > > Log4j3 is also nearing release but it seems it will require Java 17. > > > The website states Java 11 [2] but the artifacts from the latest 3.0.0 > > > beta are built for Java 17. I was not able to find clear maintenance > > > statement about log4j2 once log4j3 gets released. > > > > > > The question is where do we go from here? > > > We can stick with our plans: > > > 1. Deprecate the appender in the next 3.x release and plan to remove > it in > > > 4.0 > > > 2. Do the necessary work to switch to log4j2 in 4.0 > > > If so we need people to drive these work items. We have PRs for these > > > with hopefully the bulk of the code but they need > > > rebasing/completing/reviewing. > > > > > > Otherwise we can reconsider KIP-653 and/or KIP-719. > > > > > > Assuming log4j2 does not go end of life in the near future (We can > > > reach out to Apache Logging to clarify that point.), I think it still > > > makes sense to adopt it. I would also go ahead and deprecate our > > > appender. > > > > > > Thanks, > > > Mickael > > > > > > 0: https://lists.apache.org/thread/q0sz910o1y9mhq159oy16w31d6dzh79f > > > 1: https://github.com/apache/logging-log4j2/issues/1951 > > > 2: https://logging.apache.org/log4j/3.x/#requirements > > > >
Re: Logging in Kafka
Hi, A couple of PMC members from Apache Logging replied and they said they plan to keep supporting log4j2 for several years. https://lists.apache.org/thread/6n6bkgwj8tglgdgzz8wxhkx1p1xpwodl Thanks, Mickael On Wed, Jan 10, 2024 at 3:57 PM Mickael Maison wrote: > > I asked for details about the future of log4j2 on the logging user list: > https://lists.apache.org/thread/6n6bkgwj8tglgdgzz8wxhkx1p1xpwodl > > Let's see what they say. > > Thanks, > Mickael > > On Wed, Jan 10, 2024 at 3:23 PM Ismael Juma wrote: > > > > Hi Mickael, > > > > Thanks for starting the discussion and for summarizing the state of play. I > > agree with you that it would be important to understand how long log4j2 > > will be supported for. An alternative would be sl4fj 2.x and logback. > > > > Ismael > > > > On Wed, Jan 10, 2024 at 2:17 PM Mickael Maison > > wrote: > > > > > Hi, > > > > > > Starting a new thread to discuss the current logging situation in > > > Kafka. I'll restate everything we know but see the [DISCUSS] Road to > > > Kafka 4.0 if you are interested in what has already been said. [0] > > > > > > Currently Kafka uses SLF4J and reload4j as the logging backend. We had > > > to adopt reload4j in 3.2.0 as log4j was end of life and has a few > > > security issues. > > > > > > In 2020 we adopted KIP-653 to upgrade to log4j2. Due to > > > incompatibilities in the configuration mechanism with log4j/reload4j > > > we decide to delay the upgrade to the next major release, Kafka 4.0. > > > > > > Kafka also currently provides a log4j appender. In 2022, we adopted > > > KIP-719 to deprecate it since we wanted to switch to log4j2. At the > > > time Apache Logging also had a Kafka appender that worked with log4j2. > > > They since deprecated that appender in log4j2 and it is not part of > > > log4j3. [1] > > > > > > Log4j3 is also nearing release but it seems it will require Java 17. > > > The website states Java 11 [2] but the artifacts from the latest 3.0.0 > > > beta are built for Java 17. I was not able to find clear maintenance > > > statement about log4j2 once log4j3 gets released. > > > > > > The question is where do we go from here? > > > We can stick with our plans: > > > 1. Deprecate the appender in the next 3.x release and plan to remove it in > > > 4.0 > > > 2. Do the necessary work to switch to log4j2 in 4.0 > > > If so we need people to drive these work items. We have PRs for these > > > with hopefully the bulk of the code but they need > > > rebasing/completing/reviewing. > > > > > > Otherwise we can reconsider KIP-653 and/or KIP-719. > > > > > > Assuming log4j2 does not go end of life in the near future (We can > > > reach out to Apache Logging to clarify that point.), I think it still > > > makes sense to adopt it. I would also go ahead and deprecate our > > > appender. > > > > > > Thanks, > > > Mickael > > > > > > 0: https://lists.apache.org/thread/q0sz910o1y9mhq159oy16w31d6dzh79f > > > 1: https://github.com/apache/logging-log4j2/issues/1951 > > > 2: https://logging.apache.org/log4j/3.x/#requirements > > >
Re: Logging in Kafka
I asked for details about the future of log4j2 on the logging user list: https://lists.apache.org/thread/6n6bkgwj8tglgdgzz8wxhkx1p1xpwodl Let's see what they say. Thanks, Mickael On Wed, Jan 10, 2024 at 3:23 PM Ismael Juma wrote: > > Hi Mickael, > > Thanks for starting the discussion and for summarizing the state of play. I > agree with you that it would be important to understand how long log4j2 > will be supported for. An alternative would be sl4fj 2.x and logback. > > Ismael > > On Wed, Jan 10, 2024 at 2:17 PM Mickael Maison > wrote: > > > Hi, > > > > Starting a new thread to discuss the current logging situation in > > Kafka. I'll restate everything we know but see the [DISCUSS] Road to > > Kafka 4.0 if you are interested in what has already been said. [0] > > > > Currently Kafka uses SLF4J and reload4j as the logging backend. We had > > to adopt reload4j in 3.2.0 as log4j was end of life and has a few > > security issues. > > > > In 2020 we adopted KIP-653 to upgrade to log4j2. Due to > > incompatibilities in the configuration mechanism with log4j/reload4j > > we decide to delay the upgrade to the next major release, Kafka 4.0. > > > > Kafka also currently provides a log4j appender. In 2022, we adopted > > KIP-719 to deprecate it since we wanted to switch to log4j2. At the > > time Apache Logging also had a Kafka appender that worked with log4j2. > > They since deprecated that appender in log4j2 and it is not part of > > log4j3. [1] > > > > Log4j3 is also nearing release but it seems it will require Java 17. > > The website states Java 11 [2] but the artifacts from the latest 3.0.0 > > beta are built for Java 17. I was not able to find clear maintenance > > statement about log4j2 once log4j3 gets released. > > > > The question is where do we go from here? > > We can stick with our plans: > > 1. Deprecate the appender in the next 3.x release and plan to remove it in > > 4.0 > > 2. Do the necessary work to switch to log4j2 in 4.0 > > If so we need people to drive these work items. We have PRs for these > > with hopefully the bulk of the code but they need > > rebasing/completing/reviewing. > > > > Otherwise we can reconsider KIP-653 and/or KIP-719. > > > > Assuming log4j2 does not go end of life in the near future (We can > > reach out to Apache Logging to clarify that point.), I think it still > > makes sense to adopt it. I would also go ahead and deprecate our > > appender. > > > > Thanks, > > Mickael > > > > 0: https://lists.apache.org/thread/q0sz910o1y9mhq159oy16w31d6dzh79f > > 1: https://github.com/apache/logging-log4j2/issues/1951 > > 2: https://logging.apache.org/log4j/3.x/#requirements > >
Re: Logging in Kafka
Hi Mickael, Thanks for starting the discussion and for summarizing the state of play. I agree with you that it would be important to understand how long log4j2 will be supported for. An alternative would be sl4fj 2.x and logback. Ismael On Wed, Jan 10, 2024 at 2:17 PM Mickael Maison wrote: > Hi, > > Starting a new thread to discuss the current logging situation in > Kafka. I'll restate everything we know but see the [DISCUSS] Road to > Kafka 4.0 if you are interested in what has already been said. [0] > > Currently Kafka uses SLF4J and reload4j as the logging backend. We had > to adopt reload4j in 3.2.0 as log4j was end of life and has a few > security issues. > > In 2020 we adopted KIP-653 to upgrade to log4j2. Due to > incompatibilities in the configuration mechanism with log4j/reload4j > we decide to delay the upgrade to the next major release, Kafka 4.0. > > Kafka also currently provides a log4j appender. In 2022, we adopted > KIP-719 to deprecate it since we wanted to switch to log4j2. At the > time Apache Logging also had a Kafka appender that worked with log4j2. > They since deprecated that appender in log4j2 and it is not part of > log4j3. [1] > > Log4j3 is also nearing release but it seems it will require Java 17. > The website states Java 11 [2] but the artifacts from the latest 3.0.0 > beta are built for Java 17. I was not able to find clear maintenance > statement about log4j2 once log4j3 gets released. > > The question is where do we go from here? > We can stick with our plans: > 1. Deprecate the appender in the next 3.x release and plan to remove it in > 4.0 > 2. Do the necessary work to switch to log4j2 in 4.0 > If so we need people to drive these work items. We have PRs for these > with hopefully the bulk of the code but they need > rebasing/completing/reviewing. > > Otherwise we can reconsider KIP-653 and/or KIP-719. > > Assuming log4j2 does not go end of life in the near future (We can > reach out to Apache Logging to clarify that point.), I think it still > makes sense to adopt it. I would also go ahead and deprecate our > appender. > > Thanks, > Mickael > > 0: https://lists.apache.org/thread/q0sz910o1y9mhq159oy16w31d6dzh79f > 1: https://github.com/apache/logging-log4j2/issues/1951 > 2: https://logging.apache.org/log4j/3.x/#requirements >
Logging in Kafka
Hi, Starting a new thread to discuss the current logging situation in Kafka. I'll restate everything we know but see the [DISCUSS] Road to Kafka 4.0 if you are interested in what has already been said. [0] Currently Kafka uses SLF4J and reload4j as the logging backend. We had to adopt reload4j in 3.2.0 as log4j was end of life and has a few security issues. In 2020 we adopted KIP-653 to upgrade to log4j2. Due to incompatibilities in the configuration mechanism with log4j/reload4j we decide to delay the upgrade to the next major release, Kafka 4.0. Kafka also currently provides a log4j appender. In 2022, we adopted KIP-719 to deprecate it since we wanted to switch to log4j2. At the time Apache Logging also had a Kafka appender that worked with log4j2. They since deprecated that appender in log4j2 and it is not part of log4j3. [1] Log4j3 is also nearing release but it seems it will require Java 17. The website states Java 11 [2] but the artifacts from the latest 3.0.0 beta are built for Java 17. I was not able to find clear maintenance statement about log4j2 once log4j3 gets released. The question is where do we go from here? We can stick with our plans: 1. Deprecate the appender in the next 3.x release and plan to remove it in 4.0 2. Do the necessary work to switch to log4j2 in 4.0 If so we need people to drive these work items. We have PRs for these with hopefully the bulk of the code but they need rebasing/completing/reviewing. Otherwise we can reconsider KIP-653 and/or KIP-719. Assuming log4j2 does not go end of life in the near future (We can reach out to Apache Logging to clarify that point.), I think it still makes sense to adopt it. I would also go ahead and deprecate our appender. Thanks, Mickael 0: https://lists.apache.org/thread/q0sz910o1y9mhq159oy16w31d6dzh79f 1: https://github.com/apache/logging-log4j2/issues/1951 2: https://logging.apache.org/log4j/3.x/#requirements
[jira] [Resolved] (KAFKA-12454) Add ERROR logging on kafka-log-dirs when given brokerIds do not exist in current kafka cluster
[ https://issues.apache.org/jira/browse/KAFKA-12454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai resolved KAFKA-12454. Resolution: Fixed > Add ERROR logging on kafka-log-dirs when given brokerIds do not exist in > current kafka cluster > --- > > Key: KAFKA-12454 > URL: https://issues.apache.org/jira/browse/KAFKA-12454 > Project: Kafka > Issue Type: Improvement >Reporter: Wenbing Shen >Assignee: Wenbing Shen >Priority: Minor > Fix For: 3.0.0 > > > When non-existent brokerIds value are given, the kafka-log-dirs tool will > have a timeout error: > Exception in thread "main" java.util.concurrent.ExecutionException: > org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node > assignment. Call: describeLogDirs > at > org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45) > at > org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32) > at > org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89) > at > org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260) > at kafka.admin.LogDirsCommand$.describe(LogDirsCommand.scala:50) > at kafka.admin.LogDirsCommand$.main(LogDirsCommand.scala:36) > at kafka.admin.LogDirsCommand.main(LogDirsCommand.scala) > Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting > for a node assignment. Call: describeLogDirs > > When the brokerId entered by the user does not exist, an error message > indicating that the node is not present should be printed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-12454) Add ERROR logging on kafka-log-dirs when given brokerIds do not exist in current kafka cluster
Wenbing Shen created KAFKA-12454: Summary: Add ERROR logging on kafka-log-dirs when given brokerIds do not exist in current kafka cluster Key: KAFKA-12454 URL: https://issues.apache.org/jira/browse/KAFKA-12454 Project: Kafka Issue Type: Improvement Reporter: Wenbing Shen When non-existent brokerIds value are given, the kafka-log-dirs tool will have a timeout error: Exception in thread "main" java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: describeLogDirs at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45) at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32) at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89) at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260) at kafka.admin.LogDirsCommand$.describe(LogDirsCommand.scala:50) at kafka.admin.LogDirsCommand$.main(LogDirsCommand.scala:36) at kafka.admin.LogDirsCommand.main(LogDirsCommand.scala) Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: describeLogDirs When the brokerId entered by the user does not exist, an error message indicating that the node is not present should be printed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-8839) Improve logging in Kafka Streams around debugging task lifecycle
[ https://issues.apache.org/jira/browse/KAFKA-8839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinoth Chandar resolved KAFKA-8839. --- Resolution: Fixed Closing since the PR has been merged > Improve logging in Kafka Streams around debugging task lifecycle > - > > Key: KAFKA-8839 > URL: https://issues.apache.org/jira/browse/KAFKA-8839 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Vinoth Chandar >Assignee: Vinoth Chandar >Priority: Major > Fix For: 2.4.0 > > > As a follow up to KAFKA-8831, this Jira will track efforts around improving > logging/docs around > > * Being able to follow state of tasks from assignment to restoration > * Better detection of misconfigured state store dir > * Docs giving guidance for rebalance time and state store config -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-8839) Improve logging in Kafka Streams around debugging task lifecycle
Vinoth Chandar created KAFKA-8839: - Summary: Improve logging in Kafka Streams around debugging task lifecycle Key: KAFKA-8839 URL: https://issues.apache.org/jira/browse/KAFKA-8839 Project: Kafka Issue Type: Improvement Components: streams Reporter: Vinoth Chandar As a follow up to KAFKA-8831, this Jira will track efforts around improving logging/docs around * Being able to follow state of tasks from assignment to restoration * Better detection of misconfigured state store dir * Docs giving guidance for rebalance time and state store config -- This message was sent by Atlassian Jira (v8.3.2#803003)
Re: [DISCUSSION] Error Handling and Logging at Kafka
The wiki is great place to record the results or make suggestions that require formatting and graphics to explain, but I think I prefer mailing lists for discussion. So, let me repeat here a question I raised in the wiki, since I don't want it to get lost :) Is there a JIRA for the issues mentioned in the first background example? (closing socket and connection unsuccessful). I'll be more than happy to pick those up as they've made my life pretty painful. Gwen On Tue, Aug 26, 2014 at 7:21 PM, Joe Stein joe.st...@stealth.ly wrote: Hi Guozhang thanks for kicking this off. I made some comments in the Wiki (and we can continue the discussion there) but think this type of collaborative mailing list discussion and confluence writeup is a great way for different discussions about the same thing in different organizations to coalesce together in code. This may also be a good place for folks looking to get their feet wet contributing code to-do so. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Tue, Aug 26, 2014 at 6:22 PM, Guozhang Wang wangg...@gmail.com wrote: Hello all, We want to kick off some discussions about error handling and logging conventions. With a number of great patch contributions to Kafka recently, it is good time for us to sit down and think a little bit more about the coding style guidelines we have (http://kafka.apache.org/coding-guide.html ). People at LinkedIn have discussed a bit about some observed issues about the error handling and logging verboseness, and here is a summary of the bullet points covered: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Error+Handling+and+Logging We would like to collect as much comments as possible before we move ahead to those logging cleaning JIRAs mentioned in the wiki, so please feel free to shoot any thoughts and suggestions around this topic. -- Guozhang
Re: [DISCUSSION] Error Handling and Logging at Kafka
Thanks Joe / Gwen for the great input. I have commented on the page accordingly also. There is a ticket for the background connection issue: KAFKA-1592 https://issues.apache.org/jira/browse/KAFKA-1592. On Wed, Aug 27, 2014 at 10:25 AM, Gwen Shapira gshap...@cloudera.com wrote: The wiki is great place to record the results or make suggestions that require formatting and graphics to explain, but I think I prefer mailing lists for discussion. So, let me repeat here a question I raised in the wiki, since I don't want it to get lost :) Is there a JIRA for the issues mentioned in the first background example? (closing socket and connection unsuccessful). I'll be more than happy to pick those up as they've made my life pretty painful. Gwen On Tue, Aug 26, 2014 at 7:21 PM, Joe Stein joe.st...@stealth.ly wrote: Hi Guozhang thanks for kicking this off. I made some comments in the Wiki (and we can continue the discussion there) but think this type of collaborative mailing list discussion and confluence writeup is a great way for different discussions about the same thing in different organizations to coalesce together in code. This may also be a good place for folks looking to get their feet wet contributing code to-do so. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Tue, Aug 26, 2014 at 6:22 PM, Guozhang Wang wangg...@gmail.com wrote: Hello all, We want to kick off some discussions about error handling and logging conventions. With a number of great patch contributions to Kafka recently, it is good time for us to sit down and think a little bit more about the coding style guidelines we have ( http://kafka.apache.org/coding-guide.html ). People at LinkedIn have discussed a bit about some observed issues about the error handling and logging verboseness, and here is a summary of the bullet points covered: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Error+Handling+and+Logging We would like to collect as much comments as possible before we move ahead to those logging cleaning JIRAs mentioned in the wiki, so please feel free to shoot any thoughts and suggestions around this topic. -- Guozhang -- -- Guozhang
[DISCUSSION] Error Handling and Logging at Kafka
Hello all, We want to kick off some discussions about error handling and logging conventions. With a number of great patch contributions to Kafka recently, it is good time for us to sit down and think a little bit more about the coding style guidelines we have (http://kafka.apache.org/coding-guide.html). People at LinkedIn have discussed a bit about some observed issues about the error handling and logging verboseness, and here is a summary of the bullet points covered: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Error+Handling+and+Logging We would like to collect as much comments as possible before we move ahead to those logging cleaning JIRAs mentioned in the wiki, so please feel free to shoot any thoughts and suggestions around this topic. -- Guozhang
Re: [DISCUSSION] Error Handling and Logging at Kafka
Hi Guozhang thanks for kicking this off. I made some comments in the Wiki (and we can continue the discussion there) but think this type of collaborative mailing list discussion and confluence writeup is a great way for different discussions about the same thing in different organizations to coalesce together in code. This may also be a good place for folks looking to get their feet wet contributing code to-do so. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Tue, Aug 26, 2014 at 6:22 PM, Guozhang Wang wangg...@gmail.com wrote: Hello all, We want to kick off some discussions about error handling and logging conventions. With a number of great patch contributions to Kafka recently, it is good time for us to sit down and think a little bit more about the coding style guidelines we have (http://kafka.apache.org/coding-guide.html ). People at LinkedIn have discussed a bit about some observed issues about the error handling and logging verboseness, and here is a summary of the bullet points covered: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Error+Handling+and+Logging We would like to collect as much comments as possible before we move ahead to those logging cleaning JIRAs mentioned in the wiki, so please feel free to shoot any thoughts and suggestions around this topic. -- Guozhang
