Re: [DISCUSS] KIP-719: Add Log4J2 Appender

[Dongjin Lee]

Hi Tom,

1. > I think you mean tools-log4j.properties there, right?

Thanks for reading carefully. Yes, you are right. It was an error.

2. > Can the log4j-appender log a warning about its own deprecation?

That's a good suggestion. I hadn't thought of this part - adding a WARNing
message to the Log4jAppender would help, like:

> log4j-appender is deprecated and will be removed in a future release. For
migration, please refer to the latest documentation.

I just updated the proposal and the PR accordingly.

Best,
Dongjin

On Fri, Jan 21, 2022 at 9:32 PM Tom Bentley  wrote:

> Hi Dongjin,
>
> Many thanks for the KIP. I have just a couple of minor points.
>
> 1. In the Proposed Changes section there is the sentence "For backward
> compatibility, these tools will use the log4j configuration
> file (tools-log4j2.properties) by default.", I think you mean
> tools-log4j.properties there, right?
>
> 2. Can the log4j-appender log a warning about its own deprecation? The
> "DEPRECATED: using log4j 1.x configuration. To use log4j 2.x
> configuration," message is a bit confusing since it suggests that it's just
> the use of 1.x configuration that's deprecated (which it is), but it's also
> the use of the appender itself which is deprecated.
>
> Kind regards,
>
> Tom
>
> On Thu, 20 Jan 2022 at 05:01, Dongjin Lee  wrote:
>
> > Hi all,
> >
> > I just filed the issue based on the discussion here:
> >
> > KAFKA-13604: Add pluggable logging framework support (
> > https://issues.apache.org/jira/browse/KAFKA-13604)
> >
> > Thanks for posing a great issue, Viktor and Haruki!
> >
> > Best,
> > Dongjin
> >
> > On Thu, Jan 20, 2022 at 2:54 AM Viktor Somogyi-Vass
> >  wrote:
> >
> > > Hi folks,
> > >
> > > >a) I think this makes sense. But I think this shouldn't block this
> > > KIP, it can be done after moving to log4j2.
> > >
> > > Absolutely.
> > >
> > > >However, this limitation may change in the future - slf4j will support
> > > dynamic logging levels in 2.0:
> > >
> > > This would be awesome, let's keep an eye on this.
> > >
> > > >Kafka avoided all the log4shell vulnerabilities exactly because we
> > DIDN'T
> > > move to log4j 2.x.
> > >
> > > My first reaction was how lucky we are (and all Kafka users) that we
> > didn't
> > > have to do anything on that weekend! Blessed are the lazy. :)
> > >
> > > Viktor
> > >
> > > On Tue, Jan 18, 2022 at 5:16 PM Colin McCabe 
> wrote:
> > >
> > > > On Wed, Jan 12, 2022, at 02:37, Viktor Somogyi-Vass wrote:
> > > > > Hi Dongjin,
> > > > >
> > > > > We're also looking into this matter as our stack was also affected
> by
> > > all
> > > > > the log4j hell and users increasingly pushing us to upgrade to
> log4j2
> > > or
> > > > > logback because of the existing vulnerabilities in log4j1.
> > > > > Regarding the points raised by Haruki:
> > > > >
> > > >
> > > > I've heard the same thing from other people -- that there is now more
> > > > interest in moving from log4j1.x to log4j2.x, after all the recent
> > > > vulnerabilities in the latter. I found this a bit strange. Kafka
> > avoided
> > > > all the log4shell vulnerabilities exactly because we DIDN'T move to
> > log4j
> > > > 2.x. (Yes, I am aware that there is a longstanding vulnerability in
> > that
> > > > one log sink in log4j 1.x, but you can just not use that one.)
> > > >
> > > > I haven't thought about this very hard. Maybe it's still a good idea
> to
> > > > move to log4j2. But it's odd that nobody is commenting about how in
> > this
> > > > case, not updating actually prevented a major security incident for
> > > Kafka.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > >
> > > > > a) In my opinion the best would be to make the dynamic logger
> support
> > > > > (Log4jController and LoggingResource) pluggable for log4j2 and
> > logback
> > > > (so
> > > > > an interface could be used to define the dynamic logging control
> > > methods
> > > > > and a config to specify the implementation). That way we're not
> bound
> > > to
> > > > > either logback or log4j and seems like a low-effort thing to do.
> > > > > Additionally this could be used in Connect too in LoggingResource.
> > > > >
> > > > > b) I think testing dependencies aren't that important from the user
> > > > > perspective, it's fine to either use log4j2 or logback, whichever
> is
> > > > > easier. Kafka is either used from the distribution (tgz) or pulled
> in
> > > > > through maven, but test dependencies shouldn't be exposed to the
> > world.
> > > > >
> > > > > c) I would support deprecating the appender in favor of the log4j2
> > > Kafka
> > > > > appender. VerifiableLog4jAppender is intended as a testing tool
> > anyway,
> > > > so
> > > > > I think it's less important to change this to logback.
> > > > >
> > > > > Future vulnerabilities will always be found in either logback or
> > log4j2
> > > > or
> > > > > any other logging framework, so I think the safest approach is to
> > allow
> > > > > users to choose their implementation, while in tests I think 

Re: [DISCUSS] KIP-719: Add Log4J2 Appender

[Tom Bentley]

Hi Dongjin,

Many thanks for the KIP. I have just a couple of minor points.

1. In the Proposed Changes section there is the sentence "For backward
compatibility, these tools will use the log4j configuration
file (tools-log4j2.properties) by default.", I think you mean
tools-log4j.properties there, right?

2. Can the log4j-appender log a warning about its own deprecation? The
"DEPRECATED: using log4j 1.x configuration. To use log4j 2.x
configuration," message is a bit confusing since it suggests that it's just
the use of 1.x configuration that's deprecated (which it is), but it's also
the use of the appender itself which is deprecated.

Kind regards,

Tom

On Thu, 20 Jan 2022 at 05:01, Dongjin Lee  wrote:

> Hi all,
>
> I just filed the issue based on the discussion here:
>
> KAFKA-13604: Add pluggable logging framework support (
> https://issues.apache.org/jira/browse/KAFKA-13604)
>
> Thanks for posing a great issue, Viktor and Haruki!
>
> Best,
> Dongjin
>
> On Thu, Jan 20, 2022 at 2:54 AM Viktor Somogyi-Vass
>  wrote:
>
> > Hi folks,
> >
> > >a) I think this makes sense. But I think this shouldn't block this
> > KIP, it can be done after moving to log4j2.
> >
> > Absolutely.
> >
> > >However, this limitation may change in the future - slf4j will support
> > dynamic logging levels in 2.0:
> >
> > This would be awesome, let's keep an eye on this.
> >
> > >Kafka avoided all the log4shell vulnerabilities exactly because we
> DIDN'T
> > move to log4j 2.x.
> >
> > My first reaction was how lucky we are (and all Kafka users) that we
> didn't
> > have to do anything on that weekend! Blessed are the lazy. :)
> >
> > Viktor
> >
> > On Tue, Jan 18, 2022 at 5:16 PM Colin McCabe  wrote:
> >
> > > On Wed, Jan 12, 2022, at 02:37, Viktor Somogyi-Vass wrote:
> > > > Hi Dongjin,
> > > >
> > > > We're also looking into this matter as our stack was also affected by
> > all
> > > > the log4j hell and users increasingly pushing us to upgrade to log4j2
> > or
> > > > logback because of the existing vulnerabilities in log4j1.
> > > > Regarding the points raised by Haruki:
> > > >
> > >
> > > I've heard the same thing from other people -- that there is now more
> > > interest in moving from log4j1.x to log4j2.x, after all the recent
> > > vulnerabilities in the latter. I found this a bit strange. Kafka
> avoided
> > > all the log4shell vulnerabilities exactly because we DIDN'T move to
> log4j
> > > 2.x. (Yes, I am aware that there is a longstanding vulnerability in
> that
> > > one log sink in log4j 1.x, but you can just not use that one.)
> > >
> > > I haven't thought about this very hard. Maybe it's still a good idea to
> > > move to log4j2. But it's odd that nobody is commenting about how in
> this
> > > case, not updating actually prevented a major security incident for
> > Kafka.
> > >
> > > best,
> > > Colin
> > >
> > > >
> > > > a) In my opinion the best would be to make the dynamic logger support
> > > > (Log4jController and LoggingResource) pluggable for log4j2 and
> logback
> > > (so
> > > > an interface could be used to define the dynamic logging control
> > methods
> > > > and a config to specify the implementation). That way we're not bound
> > to
> > > > either logback or log4j and seems like a low-effort thing to do.
> > > > Additionally this could be used in Connect too in LoggingResource.
> > > >
> > > > b) I think testing dependencies aren't that important from the user
> > > > perspective, it's fine to either use log4j2 or logback, whichever is
> > > > easier. Kafka is either used from the distribution (tgz) or pulled in
> > > > through maven, but test dependencies shouldn't be exposed to the
> world.
> > > >
> > > > c) I would support deprecating the appender in favor of the log4j2
> > Kafka
> > > > appender. VerifiableLog4jAppender is intended as a testing tool
> anyway,
> > > so
> > > > I think it's less important to change this to logback.
> > > >
> > > > Future vulnerabilities will always be found in either logback or
> log4j2
> > > or
> > > > any other logging framework, so I think the safest approach is to
> allow
> > > > users to choose their implementation, while in tests I think we're
> free
> > > to
> > > > use whatever we want as that shouldn't be constrained by
> > vulnerabilities.
> > > >
> > > > Viktor
> > > >
> > > > On Thu, Dec 23, 2021 at 9:37 AM Haruki Okada 
> > > wrote:
> > > >
> > > >> Thanks for the clarification.
> > > >>
> > > >> About 2, I wan't aware of those concerns.
> > > >> Let me check them first.
> > > >>
> > > >>
> > > >> Thanks,
> > > >>
> > > >> 2021年12月23日(木) 13:37 Dongjin Lee :
> > > >>
> > > >> > Hi Haruki,
> > > >> >
> > > >> >
> > > >> > Thanks for organizing the issue.
> > > >> >
> > > >> >
> > > >> > If the community prefers logback, I will gladly change the
> > dependency
> > > and
> > > >> > update the PR. However, it has the following issues:
> > > >> >
> > > >> >
> > > >> > 1. The log4j2 vulnerabilities seem mostly fixed, and KIP-653 +
> > KIP-719
> > > >> are
> 

Re: [DISCUSS] KIP-719: Add Log4J2 Appender

[Dongjin Lee]

Hi all,

I just filed the issue based on the discussion here:

KAFKA-13604: Add pluggable logging framework support (
https://issues.apache.org/jira/browse/KAFKA-13604)

Thanks for posing a great issue, Viktor and Haruki!

Best,
Dongjin

On Thu, Jan 20, 2022 at 2:54 AM Viktor Somogyi-Vass
 wrote:

> Hi folks,
>
> >a) I think this makes sense. But I think this shouldn't block this
> KIP, it can be done after moving to log4j2.
>
> Absolutely.
>
> >However, this limitation may change in the future - slf4j will support
> dynamic logging levels in 2.0:
>
> This would be awesome, let's keep an eye on this.
>
> >Kafka avoided all the log4shell vulnerabilities exactly because we DIDN'T
> move to log4j 2.x.
>
> My first reaction was how lucky we are (and all Kafka users) that we didn't
> have to do anything on that weekend! Blessed are the lazy. :)
>
> Viktor
>
> On Tue, Jan 18, 2022 at 5:16 PM Colin McCabe  wrote:
>
> > On Wed, Jan 12, 2022, at 02:37, Viktor Somogyi-Vass wrote:
> > > Hi Dongjin,
> > >
> > > We're also looking into this matter as our stack was also affected by
> all
> > > the log4j hell and users increasingly pushing us to upgrade to log4j2
> or
> > > logback because of the existing vulnerabilities in log4j1.
> > > Regarding the points raised by Haruki:
> > >
> >
> > I've heard the same thing from other people -- that there is now more
> > interest in moving from log4j1.x to log4j2.x, after all the recent
> > vulnerabilities in the latter. I found this a bit strange. Kafka avoided
> > all the log4shell vulnerabilities exactly because we DIDN'T move to log4j
> > 2.x. (Yes, I am aware that there is a longstanding vulnerability in that
> > one log sink in log4j 1.x, but you can just not use that one.)
> >
> > I haven't thought about this very hard. Maybe it's still a good idea to
> > move to log4j2. But it's odd that nobody is commenting about how in this
> > case, not updating actually prevented a major security incident for
> Kafka.
> >
> > best,
> > Colin
> >
> > >
> > > a) In my opinion the best would be to make the dynamic logger support
> > > (Log4jController and LoggingResource) pluggable for log4j2 and logback
> > (so
> > > an interface could be used to define the dynamic logging control
> methods
> > > and a config to specify the implementation). That way we're not bound
> to
> > > either logback or log4j and seems like a low-effort thing to do.
> > > Additionally this could be used in Connect too in LoggingResource.
> > >
> > > b) I think testing dependencies aren't that important from the user
> > > perspective, it's fine to either use log4j2 or logback, whichever is
> > > easier. Kafka is either used from the distribution (tgz) or pulled in
> > > through maven, but test dependencies shouldn't be exposed to the world.
> > >
> > > c) I would support deprecating the appender in favor of the log4j2
> Kafka
> > > appender. VerifiableLog4jAppender is intended as a testing tool anyway,
> > so
> > > I think it's less important to change this to logback.
> > >
> > > Future vulnerabilities will always be found in either logback or log4j2
> > or
> > > any other logging framework, so I think the safest approach is to allow
> > > users to choose their implementation, while in tests I think we're free
> > to
> > > use whatever we want as that shouldn't be constrained by
> vulnerabilities.
> > >
> > > Viktor
> > >
> > > On Thu, Dec 23, 2021 at 9:37 AM Haruki Okada 
> > wrote:
> > >
> > >> Thanks for the clarification.
> > >>
> > >> About 2, I wan't aware of those concerns.
> > >> Let me check them first.
> > >>
> > >>
> > >> Thanks,
> > >>
> > >> 2021年12月23日(木) 13:37 Dongjin Lee :
> > >>
> > >> > Hi Haruki,
> > >> >
> > >> >
> > >> > Thanks for organizing the issue.
> > >> >
> > >> >
> > >> > If the community prefers logback, I will gladly change the
> dependency
> > and
> > >> > update the PR. However, it has the following issues:
> > >> >
> > >> >
> > >> > 1. The log4j2 vulnerabilities seem mostly fixed, and KIP-653 +
> KIP-719
> > >> are
> > >> > not released yet. So, using log4j2 (whose recent update pace is so
> > high)
> > >> > will not affect the users.
> > >> >
> > >> >
> > >> > 2. To switch to logback, the following features should be reworked:
> > >> >
> > >> >
> > >> >   a. Dynamic logger level configuration (core, connect)
> > >> >
> > >> >   b. Logging tests (streams)
> > >> >
> > >> >   c. Kafka Appender (tools)
> > >> >
> > >> >
> > >> > a and b are the most challenging ones since there is little
> > documentation
> > >> > on how to do this, so it requires analyzing the implementation
> itself.
> > >> > (what I actually did with log4j2) About c, logback does not provide
> a
> > >> Kafka
> > >> > Appender so we have to provide an equivalent.
> > >> >
> > >> >
> > >> > It is why I prefer to use log4j2. How do you think?
> > >> >
> > >> >
> > >> > Thanks,
> > >> >
> > >> > Dongjin
> > >> >
> > >> >
> > >> > On Thu, Dec 23, 2021 at 9:01 AM Haruki Okada 
> > >> wrote:
> > >> >
> > >> > > Hi, 

Re: [DISCUSS] KIP-719: Add Log4J2 Appender

[Viktor Somogyi-Vass]

Hi folks,

>a) I think this makes sense. But I think this shouldn't block this
KIP, it can be done after moving to log4j2.

Absolutely.

>However, this limitation may change in the future - slf4j will support
dynamic logging levels in 2.0:

This would be awesome, let's keep an eye on this.

>Kafka avoided all the log4shell vulnerabilities exactly because we DIDN'T
move to log4j 2.x.

My first reaction was how lucky we are (and all Kafka users) that we didn't
have to do anything on that weekend! Blessed are the lazy. :)

Viktor

On Tue, Jan 18, 2022 at 5:16 PM Colin McCabe  wrote:

> On Wed, Jan 12, 2022, at 02:37, Viktor Somogyi-Vass wrote:
> > Hi Dongjin,
> >
> > We're also looking into this matter as our stack was also affected by all
> > the log4j hell and users increasingly pushing us to upgrade to log4j2 or
> > logback because of the existing vulnerabilities in log4j1.
> > Regarding the points raised by Haruki:
> >
>
> I've heard the same thing from other people -- that there is now more
> interest in moving from log4j1.x to log4j2.x, after all the recent
> vulnerabilities in the latter. I found this a bit strange. Kafka avoided
> all the log4shell vulnerabilities exactly because we DIDN'T move to log4j
> 2.x. (Yes, I am aware that there is a longstanding vulnerability in that
> one log sink in log4j 1.x, but you can just not use that one.)
>
> I haven't thought about this very hard. Maybe it's still a good idea to
> move to log4j2. But it's odd that nobody is commenting about how in this
> case, not updating actually prevented a major security incident for Kafka.
>
> best,
> Colin
>
> >
> > a) In my opinion the best would be to make the dynamic logger support
> > (Log4jController and LoggingResource) pluggable for log4j2 and logback
> (so
> > an interface could be used to define the dynamic logging control methods
> > and a config to specify the implementation). That way we're not bound to
> > either logback or log4j and seems like a low-effort thing to do.
> > Additionally this could be used in Connect too in LoggingResource.
> >
> > b) I think testing dependencies aren't that important from the user
> > perspective, it's fine to either use log4j2 or logback, whichever is
> > easier. Kafka is either used from the distribution (tgz) or pulled in
> > through maven, but test dependencies shouldn't be exposed to the world.
> >
> > c) I would support deprecating the appender in favor of the log4j2 Kafka
> > appender. VerifiableLog4jAppender is intended as a testing tool anyway,
> so
> > I think it's less important to change this to logback.
> >
> > Future vulnerabilities will always be found in either logback or log4j2
> or
> > any other logging framework, so I think the safest approach is to allow
> > users to choose their implementation, while in tests I think we're free
> to
> > use whatever we want as that shouldn't be constrained by vulnerabilities.
> >
> > Viktor
> >
> > On Thu, Dec 23, 2021 at 9:37 AM Haruki Okada 
> wrote:
> >
> >> Thanks for the clarification.
> >>
> >> About 2, I wan't aware of those concerns.
> >> Let me check them first.
> >>
> >>
> >> Thanks,
> >>
> >> 2021年12月23日(木) 13:37 Dongjin Lee :
> >>
> >> > Hi Haruki,
> >> >
> >> >
> >> > Thanks for organizing the issue.
> >> >
> >> >
> >> > If the community prefers logback, I will gladly change the dependency
> and
> >> > update the PR. However, it has the following issues:
> >> >
> >> >
> >> > 1. The log4j2 vulnerabilities seem mostly fixed, and KIP-653 + KIP-719
> >> are
> >> > not released yet. So, using log4j2 (whose recent update pace is so
> high)
> >> > will not affect the users.
> >> >
> >> >
> >> > 2. To switch to logback, the following features should be reworked:
> >> >
> >> >
> >> >   a. Dynamic logger level configuration (core, connect)
> >> >
> >> >   b. Logging tests (streams)
> >> >
> >> >   c. Kafka Appender (tools)
> >> >
> >> >
> >> > a and b are the most challenging ones since there is little
> documentation
> >> > on how to do this, so it requires analyzing the implementation itself.
> >> > (what I actually did with log4j2) About c, logback does not provide a
> >> Kafka
> >> > Appender so we have to provide an equivalent.
> >> >
> >> >
> >> > It is why I prefer to use log4j2. How do you think?
> >> >
> >> >
> >> > Thanks,
> >> >
> >> > Dongjin
> >> >
> >> >
> >> > On Thu, Dec 23, 2021 at 9:01 AM Haruki Okada 
> >> wrote:
> >> >
> >> > > Hi, Dongjin,
> >> > >
> >> > > Sorry for interrupting the discussion.
> >> > > And thank you for your hard work about KIP-653, KIP-719.
> >> > >
> >> > > I understand that KIP-653 is already accepted so log4j2 is the
> choice
> >> of
> >> > > the Kafka community though, I'm now feeling that logback is a better
> >> > choice
> >> > > here.
> >> > >
> >> > > Reasons:
> >> > >
> >> > > - even after "log4shell", several vulnerabilities found on log4j2 so
> >> new
> >> > > versions are released and users have to update in high-pace
> >> > > * actually, a CVE was also reported for 

Re: [DISCUSS] KIP-719: Add Log4J2 Appender

[Colin McCabe]

On Wed, Jan 12, 2022, at 02:37, Viktor Somogyi-Vass wrote:
> Hi Dongjin,
>
> We're also looking into this matter as our stack was also affected by all
> the log4j hell and users increasingly pushing us to upgrade to log4j2 or
> logback because of the existing vulnerabilities in log4j1.
> Regarding the points raised by Haruki:
>

I've heard the same thing from other people -- that there is now more interest 
in moving from log4j1.x to log4j2.x, after all the recent vulnerabilities in 
the latter. I found this a bit strange. Kafka avoided all the log4shell 
vulnerabilities exactly because we DIDN'T move to log4j 2.x. (Yes, I am aware 
that there is a longstanding vulnerability in that one log sink in log4j 1.x, 
but you can just not use that one.)

I haven't thought about this very hard. Maybe it's still a good idea to move to 
log4j2. But it's odd that nobody is commenting about how in this case, not 
updating actually prevented a major security incident for Kafka.

best,
Colin

>
> a) In my opinion the best would be to make the dynamic logger support
> (Log4jController and LoggingResource) pluggable for log4j2 and logback (so
> an interface could be used to define the dynamic logging control methods
> and a config to specify the implementation). That way we're not bound to
> either logback or log4j and seems like a low-effort thing to do.
> Additionally this could be used in Connect too in LoggingResource.
>
> b) I think testing dependencies aren't that important from the user
> perspective, it's fine to either use log4j2 or logback, whichever is
> easier. Kafka is either used from the distribution (tgz) or pulled in
> through maven, but test dependencies shouldn't be exposed to the world.
>
> c) I would support deprecating the appender in favor of the log4j2 Kafka
> appender. VerifiableLog4jAppender is intended as a testing tool anyway, so
> I think it's less important to change this to logback.
>
> Future vulnerabilities will always be found in either logback or log4j2 or
> any other logging framework, so I think the safest approach is to allow
> users to choose their implementation, while in tests I think we're free to
> use whatever we want as that shouldn't be constrained by vulnerabilities.
>
> Viktor
>
> On Thu, Dec 23, 2021 at 9:37 AM Haruki Okada  wrote:
>
>> Thanks for the clarification.
>>
>> About 2, I wan't aware of those concerns.
>> Let me check them first.
>>
>>
>> Thanks,
>>
>> 2021年12月23日(木) 13:37 Dongjin Lee :
>>
>> > Hi Haruki,
>> >
>> >
>> > Thanks for organizing the issue.
>> >
>> >
>> > If the community prefers logback, I will gladly change the dependency and
>> > update the PR. However, it has the following issues:
>> >
>> >
>> > 1. The log4j2 vulnerabilities seem mostly fixed, and KIP-653 + KIP-719
>> are
>> > not released yet. So, using log4j2 (whose recent update pace is so high)
>> > will not affect the users.
>> >
>> >
>> > 2. To switch to logback, the following features should be reworked:
>> >
>> >
>> >   a. Dynamic logger level configuration (core, connect)
>> >
>> >   b. Logging tests (streams)
>> >
>> >   c. Kafka Appender (tools)
>> >
>> >
>> > a and b are the most challenging ones since there is little documentation
>> > on how to do this, so it requires analyzing the implementation itself.
>> > (what I actually did with log4j2) About c, logback does not provide a
>> Kafka
>> > Appender so we have to provide an equivalent.
>> >
>> >
>> > It is why I prefer to use log4j2. How do you think?
>> >
>> >
>> > Thanks,
>> >
>> > Dongjin
>> >
>> >
>> > On Thu, Dec 23, 2021 at 9:01 AM Haruki Okada 
>> wrote:
>> >
>> > > Hi, Dongjin,
>> > >
>> > > Sorry for interrupting the discussion.
>> > > And thank you for your hard work about KIP-653, KIP-719.
>> > >
>> > > I understand that KIP-653 is already accepted so log4j2 is the choice
>> of
>> > > the Kafka community though, I'm now feeling that logback is a better
>> > choice
>> > > here.
>> > >
>> > > Reasons:
>> > >
>> > > - even after "log4shell", several vulnerabilities found on log4j2 so
>> new
>> > > versions are released and users have to update in high-pace
>> > > * actually, a CVE was also reported for logback (CVE-2021-42550)
>> but
>> > it
>> > > requires edit-permission of the config file for an attacker so it's
>> much
>> > > less threatening
>> > > - log4j1.x and logback are made by same developer (ceki), so
>> > substantially
>> > > the successor of log4j1 is logback rather than log4j2
>> > > - in Hadoop project, seems similar suggestion was made from a PMC
>> > > * https://issues.apache.org/jira/browse/HADOOP-12956
>> > >
>> > >
>> > > What do you think about adopting logback instead?
>> > >
>> > >
>> > > Thanks,
>> > >
>> > > 2021年12月21日(火) 18:02 Dongjin Lee :
>> > >
>> > > > Hi Mickael,
>> > > >
>> > > > > In the meantime, you may want to bump the VOTE thread too.
>> > > >
>> > > > Sure, I just reset the voting thread with a brief context.
>> > > >
>> > > > Thanks,
>> > > > Dongjin
>> > > >
>> > > > On Tue, 

Re: [DISCUSS] KIP-719: Add Log4J2 Appender

[Dongjin Lee]

Hi Mickael and Viktor,

Recently, I also had research on pluggable logging dependencies. As far as
I experienced in KIP-653, the issues related to the logging dependency are
organized into like the following:

1. Append the desired logging messages.
2. Testing the logging messages. (especially Kafka Streams)
3. Support dynamic logging level change feature in Kafka and Kafka Connect.

As Viktor pointed out, 1 is easy. It seems like 2 is also (relatively)
easy, but the real problem is 3; Presently, we need to use slf4j to make
Kafka and Kafka Connect to be the logging implementation-neutral (like
Kafka Streams currently does). But there is no way to change the logging
level dynamically only with slf4j API yet; All cases I found were making
use of the actual implementation's API, like the following:

-
https://prateep.info/2015/12/12/dynamically-change-log-level-in-slf4j-log4j-with-standalone-java-class/
- https://gist.github.com/nkcoder/cd74919fd80594c56e09b448a2d1ba31

However, this limitation may change in the future - slf4j will support
dynamic logging levels in 2.0:

-
https://stackoverflow.com/questions/2621701/setting-log-level-of-message-at-runtime-in-slf4j
- https://jira.qos.ch/browse/SLF4J-124

So, how about this? *I will keep my eyes on slf4j's update progress, and as
soon as they stabilize their 2.0 implementation, I will file another KIP
for the logger implementation neutrality.* Since there is no precise timing
of slf4j 2.0's stabilization, I think this plan would be more reasonable.

I am looking forward to your valuable opinions.

Thanks,
Dongjin

On Mon, Jan 17, 2022 at 9:28 PM Mickael Maison 
wrote:

> Hi Viktor,
>
> a) I think this makes sense. But I think this shouldn't block this
> KIP, it can be done after moving to log4j2.
>
> b) Yes, tests dependencies are for tests only and shouldn't prevent us
> from making other changes.
>
> c) This is what the KIP is now proposing
>
> Thanks,
> Mickael
>
> On Wed, Jan 12, 2022 at 11:37 AM Viktor Somogyi-Vass
>  wrote:
> >
> > Hi Dongjin,
> >
> > We're also looking into this matter as our stack was also affected by all
> > the log4j hell and users increasingly pushing us to upgrade to log4j2 or
> > logback because of the existing vulnerabilities in log4j1.
> > Regarding the points raised by Haruki:
> >
> > a) In my opinion the best would be to make the dynamic logger support
> > (Log4jController and LoggingResource) pluggable for log4j2 and logback
> (so
> > an interface could be used to define the dynamic logging control methods
> > and a config to specify the implementation). That way we're not bound to
> > either logback or log4j and seems like a low-effort thing to do.
> > Additionally this could be used in Connect too in LoggingResource.
> >
> > b) I think testing dependencies aren't that important from the user
> > perspective, it's fine to either use log4j2 or logback, whichever is
> > easier. Kafka is either used from the distribution (tgz) or pulled in
> > through maven, but test dependencies shouldn't be exposed to the world.
> >
> > c) I would support deprecating the appender in favor of the log4j2 Kafka
> > appender. VerifiableLog4jAppender is intended as a testing tool anyway,
> so
> > I think it's less important to change this to logback.
> >
> > Future vulnerabilities will always be found in either logback or log4j2
> or
> > any other logging framework, so I think the safest approach is to allow
> > users to choose their implementation, while in tests I think we're free
> to
> > use whatever we want as that shouldn't be constrained by vulnerabilities.
> >
> > Viktor
> >
> > On Thu, Dec 23, 2021 at 9:37 AM Haruki Okada 
> wrote:
> >
> > > Thanks for the clarification.
> > >
> > > About 2, I wan't aware of those concerns.
> > > Let me check them first.
> > >
> > >
> > > Thanks,
> > >
> > > 2021年12月23日(木) 13:37 Dongjin Lee :
> > >
> > > > Hi Haruki,
> > > >
> > > >
> > > > Thanks for organizing the issue.
> > > >
> > > >
> > > > If the community prefers logback, I will gladly change the
> dependency and
> > > > update the PR. However, it has the following issues:
> > > >
> > > >
> > > > 1. The log4j2 vulnerabilities seem mostly fixed, and KIP-653 +
> KIP-719
> > > are
> > > > not released yet. So, using log4j2 (whose recent update pace is so
> high)
> > > > will not affect the users.
> > > >
> > > >
> > > > 2. To switch to logback, the following features should be reworked:
> > > >
> > > >
> > > >   a. Dynamic logger level configuration (core, connect)
> > > >
> > > >   b. Logging tests (streams)
> > > >
> > > >   c. Kafka Appender (tools)
> > > >
> > > >
> > > > a and b are the most challenging ones since there is little
> documentation
> > > > on how to do this, so it requires analyzing the implementation
> itself.
> > > > (what I actually did with log4j2) About c, logback does not provide a
> > > Kafka
> > > > Appender so we have to provide an equivalent.
> > > >
> > > >
> > > > It is why I prefer to use log4j2. How do you think?
> > 

Re: [DISCUSS] KIP-719: Add Log4J2 Appender

[Mickael Maison]

Hi Viktor,

a) I think this makes sense. But I think this shouldn't block this
KIP, it can be done after moving to log4j2.

b) Yes, tests dependencies are for tests only and shouldn't prevent us
from making other changes.

c) This is what the KIP is now proposing

Thanks,
Mickael

On Wed, Jan 12, 2022 at 11:37 AM Viktor Somogyi-Vass
 wrote:
>
> Hi Dongjin,
>
> We're also looking into this matter as our stack was also affected by all
> the log4j hell and users increasingly pushing us to upgrade to log4j2 or
> logback because of the existing vulnerabilities in log4j1.
> Regarding the points raised by Haruki:
>
> a) In my opinion the best would be to make the dynamic logger support
> (Log4jController and LoggingResource) pluggable for log4j2 and logback (so
> an interface could be used to define the dynamic logging control methods
> and a config to specify the implementation). That way we're not bound to
> either logback or log4j and seems like a low-effort thing to do.
> Additionally this could be used in Connect too in LoggingResource.
>
> b) I think testing dependencies aren't that important from the user
> perspective, it's fine to either use log4j2 or logback, whichever is
> easier. Kafka is either used from the distribution (tgz) or pulled in
> through maven, but test dependencies shouldn't be exposed to the world.
>
> c) I would support deprecating the appender in favor of the log4j2 Kafka
> appender. VerifiableLog4jAppender is intended as a testing tool anyway, so
> I think it's less important to change this to logback.
>
> Future vulnerabilities will always be found in either logback or log4j2 or
> any other logging framework, so I think the safest approach is to allow
> users to choose their implementation, while in tests I think we're free to
> use whatever we want as that shouldn't be constrained by vulnerabilities.
>
> Viktor
>
> On Thu, Dec 23, 2021 at 9:37 AM Haruki Okada  wrote:
>
> > Thanks for the clarification.
> >
> > About 2, I wan't aware of those concerns.
> > Let me check them first.
> >
> >
> > Thanks,
> >
> > 2021年12月23日(木) 13:37 Dongjin Lee :
> >
> > > Hi Haruki,
> > >
> > >
> > > Thanks for organizing the issue.
> > >
> > >
> > > If the community prefers logback, I will gladly change the dependency and
> > > update the PR. However, it has the following issues:
> > >
> > >
> > > 1. The log4j2 vulnerabilities seem mostly fixed, and KIP-653 + KIP-719
> > are
> > > not released yet. So, using log4j2 (whose recent update pace is so high)
> > > will not affect the users.
> > >
> > >
> > > 2. To switch to logback, the following features should be reworked:
> > >
> > >
> > >   a. Dynamic logger level configuration (core, connect)
> > >
> > >   b. Logging tests (streams)
> > >
> > >   c. Kafka Appender (tools)
> > >
> > >
> > > a and b are the most challenging ones since there is little documentation
> > > on how to do this, so it requires analyzing the implementation itself.
> > > (what I actually did with log4j2) About c, logback does not provide a
> > Kafka
> > > Appender so we have to provide an equivalent.
> > >
> > >
> > > It is why I prefer to use log4j2. How do you think?
> > >
> > >
> > > Thanks,
> > >
> > > Dongjin
> > >
> > >
> > > On Thu, Dec 23, 2021 at 9:01 AM Haruki Okada 
> > wrote:
> > >
> > > > Hi, Dongjin,
> > > >
> > > > Sorry for interrupting the discussion.
> > > > And thank you for your hard work about KIP-653, KIP-719.
> > > >
> > > > I understand that KIP-653 is already accepted so log4j2 is the choice
> > of
> > > > the Kafka community though, I'm now feeling that logback is a better
> > > choice
> > > > here.
> > > >
> > > > Reasons:
> > > >
> > > > - even after "log4shell", several vulnerabilities found on log4j2 so
> > new
> > > > versions are released and users have to update in high-pace
> > > > * actually, a CVE was also reported for logback (CVE-2021-42550)
> > but
> > > it
> > > > requires edit-permission of the config file for an attacker so it's
> > much
> > > > less threatening
> > > > - log4j1.x and logback are made by same developer (ceki), so
> > > substantially
> > > > the successor of log4j1 is logback rather than log4j2
> > > > - in Hadoop project, seems similar suggestion was made from a PMC
> > > > * https://issues.apache.org/jira/browse/HADOOP-12956
> > > >
> > > >
> > > > What do you think about adopting logback instead?
> > > >
> > > >
> > > > Thanks,
> > > >
> > > > 2021年12月21日(火) 18:02 Dongjin Lee :
> > > >
> > > > > Hi Mickael,
> > > > >
> > > > > > In the meantime, you may want to bump the VOTE thread too.
> > > > >
> > > > > Sure, I just reset the voting thread with a brief context.
> > > > >
> > > > > Thanks,
> > > > > Dongjin
> > > > >
> > > > > On Tue, Dec 21, 2021 at 2:13 AM Mickael Maison <
> > > mickael.mai...@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Thanks Dongjin!
> > > > > >
> > > > > > I'll take a look soon.
> > > > > > In the meantime, you may want to bump the VOTE thread too.
> > > > > >
> > > > > > 

Re: [DISCUSS] KIP-719: Add Log4J2 Appender

[Viktor Somogyi-Vass]

Hi Dongjin,

We're also looking into this matter as our stack was also affected by all
the log4j hell and users increasingly pushing us to upgrade to log4j2 or
logback because of the existing vulnerabilities in log4j1.
Regarding the points raised by Haruki:

a) In my opinion the best would be to make the dynamic logger support
(Log4jController and LoggingResource) pluggable for log4j2 and logback (so
an interface could be used to define the dynamic logging control methods
and a config to specify the implementation). That way we're not bound to
either logback or log4j and seems like a low-effort thing to do.
Additionally this could be used in Connect too in LoggingResource.

b) I think testing dependencies aren't that important from the user
perspective, it's fine to either use log4j2 or logback, whichever is
easier. Kafka is either used from the distribution (tgz) or pulled in
through maven, but test dependencies shouldn't be exposed to the world.

c) I would support deprecating the appender in favor of the log4j2 Kafka
appender. VerifiableLog4jAppender is intended as a testing tool anyway, so
I think it's less important to change this to logback.

Future vulnerabilities will always be found in either logback or log4j2 or
any other logging framework, so I think the safest approach is to allow
users to choose their implementation, while in tests I think we're free to
use whatever we want as that shouldn't be constrained by vulnerabilities.

Viktor

On Thu, Dec 23, 2021 at 9:37 AM Haruki Okada  wrote:

> Thanks for the clarification.
>
> About 2, I wan't aware of those concerns.
> Let me check them first.
>
>
> Thanks,
>
> 2021年12月23日(木) 13:37 Dongjin Lee :
>
> > Hi Haruki,
> >
> >
> > Thanks for organizing the issue.
> >
> >
> > If the community prefers logback, I will gladly change the dependency and
> > update the PR. However, it has the following issues:
> >
> >
> > 1. The log4j2 vulnerabilities seem mostly fixed, and KIP-653 + KIP-719
> are
> > not released yet. So, using log4j2 (whose recent update pace is so high)
> > will not affect the users.
> >
> >
> > 2. To switch to logback, the following features should be reworked:
> >
> >
> >   a. Dynamic logger level configuration (core, connect)
> >
> >   b. Logging tests (streams)
> >
> >   c. Kafka Appender (tools)
> >
> >
> > a and b are the most challenging ones since there is little documentation
> > on how to do this, so it requires analyzing the implementation itself.
> > (what I actually did with log4j2) About c, logback does not provide a
> Kafka
> > Appender so we have to provide an equivalent.
> >
> >
> > It is why I prefer to use log4j2. How do you think?
> >
> >
> > Thanks,
> >
> > Dongjin
> >
> >
> > On Thu, Dec 23, 2021 at 9:01 AM Haruki Okada 
> wrote:
> >
> > > Hi, Dongjin,
> > >
> > > Sorry for interrupting the discussion.
> > > And thank you for your hard work about KIP-653, KIP-719.
> > >
> > > I understand that KIP-653 is already accepted so log4j2 is the choice
> of
> > > the Kafka community though, I'm now feeling that logback is a better
> > choice
> > > here.
> > >
> > > Reasons:
> > >
> > > - even after "log4shell", several vulnerabilities found on log4j2 so
> new
> > > versions are released and users have to update in high-pace
> > > * actually, a CVE was also reported for logback (CVE-2021-42550)
> but
> > it
> > > requires edit-permission of the config file for an attacker so it's
> much
> > > less threatening
> > > - log4j1.x and logback are made by same developer (ceki), so
> > substantially
> > > the successor of log4j1 is logback rather than log4j2
> > > - in Hadoop project, seems similar suggestion was made from a PMC
> > > * https://issues.apache.org/jira/browse/HADOOP-12956
> > >
> > >
> > > What do you think about adopting logback instead?
> > >
> > >
> > > Thanks,
> > >
> > > 2021年12月21日(火) 18:02 Dongjin Lee :
> > >
> > > > Hi Mickael,
> > > >
> > > > > In the meantime, you may want to bump the VOTE thread too.
> > > >
> > > > Sure, I just reset the voting thread with a brief context.
> > > >
> > > > Thanks,
> > > > Dongjin
> > > >
> > > > On Tue, Dec 21, 2021 at 2:13 AM Mickael Maison <
> > mickael.mai...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Thanks Dongjin!
> > > > >
> > > > > I'll take a look soon.
> > > > > In the meantime, you may want to bump the VOTE thread too.
> > > > >
> > > > > Best,
> > > > > Mickael
> > > > >
> > > > >
> > > > > On Sat, Dec 18, 2021 at 10:00 AM Dongjin Lee 
> > > wrote:
> > > > > >
> > > > > > Hi Mickael,
> > > > > >
> > > > > > Finally, I did it! As you can see at the PR
> > > > > > , KIP-719 now uses
> > > > log4j2's
> > > > > > Kafka appender, and log4j-appender is not used by the other
> modules
> > > > > > anymore. You can see how it will work with KIP-653 at this
> preview
> > > > > > <
> http://home.apache.org/~dongjin/post/apache-kafka-log4j2-support/
> > >,
> > > > > based
> > > > > > on Apache Kafka 3.0.0. The 

Re: [DISCUSS] KIP-719: Add Log4J2 Appender

[Haruki Okada]

Thanks for the clarification.

About 2, I wan't aware of those concerns.
Let me check them first.


Thanks,

2021年12月23日(木) 13:37 Dongjin Lee :

> Hi Haruki,
>
>
> Thanks for organizing the issue.
>
>
> If the community prefers logback, I will gladly change the dependency and
> update the PR. However, it has the following issues:
>
>
> 1. The log4j2 vulnerabilities seem mostly fixed, and KIP-653 + KIP-719 are
> not released yet. So, using log4j2 (whose recent update pace is so high)
> will not affect the users.
>
>
> 2. To switch to logback, the following features should be reworked:
>
>
>   a. Dynamic logger level configuration (core, connect)
>
>   b. Logging tests (streams)
>
>   c. Kafka Appender (tools)
>
>
> a and b are the most challenging ones since there is little documentation
> on how to do this, so it requires analyzing the implementation itself.
> (what I actually did with log4j2) About c, logback does not provide a Kafka
> Appender so we have to provide an equivalent.
>
>
> It is why I prefer to use log4j2. How do you think?
>
>
> Thanks,
>
> Dongjin
>
>
> On Thu, Dec 23, 2021 at 9:01 AM Haruki Okada  wrote:
>
> > Hi, Dongjin,
> >
> > Sorry for interrupting the discussion.
> > And thank you for your hard work about KIP-653, KIP-719.
> >
> > I understand that KIP-653 is already accepted so log4j2 is the choice of
> > the Kafka community though, I'm now feeling that logback is a better
> choice
> > here.
> >
> > Reasons:
> >
> > - even after "log4shell", several vulnerabilities found on log4j2 so new
> > versions are released and users have to update in high-pace
> > * actually, a CVE was also reported for logback (CVE-2021-42550) but
> it
> > requires edit-permission of the config file for an attacker so it's much
> > less threatening
> > - log4j1.x and logback are made by same developer (ceki), so
> substantially
> > the successor of log4j1 is logback rather than log4j2
> > - in Hadoop project, seems similar suggestion was made from a PMC
> > * https://issues.apache.org/jira/browse/HADOOP-12956
> >
> >
> > What do you think about adopting logback instead?
> >
> >
> > Thanks,
> >
> > 2021年12月21日(火) 18:02 Dongjin Lee :
> >
> > > Hi Mickael,
> > >
> > > > In the meantime, you may want to bump the VOTE thread too.
> > >
> > > Sure, I just reset the voting thread with a brief context.
> > >
> > > Thanks,
> > > Dongjin
> > >
> > > On Tue, Dec 21, 2021 at 2:13 AM Mickael Maison <
> mickael.mai...@gmail.com
> > >
> > > wrote:
> > >
> > > > Thanks Dongjin!
> > > >
> > > > I'll take a look soon.
> > > > In the meantime, you may want to bump the VOTE thread too.
> > > >
> > > > Best,
> > > > Mickael
> > > >
> > > >
> > > > On Sat, Dec 18, 2021 at 10:00 AM Dongjin Lee 
> > wrote:
> > > > >
> > > > > Hi Mickael,
> > > > >
> > > > > Finally, I did it! As you can see at the PR
> > > > > , KIP-719 now uses
> > > log4j2's
> > > > > Kafka appender, and log4j-appender is not used by the other modules
> > > > > anymore. You can see how it will work with KIP-653 at this preview
> > > > >  >,
> > > > based
> > > > > on Apache Kafka 3.0.0. The proposal document
> > > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-719%3A+Deprecate+Log4J+Appender
> > > > >
> > > > > is also updated accordingly, with its title.
> > > > >
> > > > > There is a minor issue on log4j2
> > > > > , but it seems
> > like
> > > > it
> > > > > will be resolved soon.
> > > > >
> > > > > Best,
> > > > > Dongjin
> > > > >
> > > > > On Wed, Dec 15, 2021 at 9:28 PM Dongjin Lee 
> > > wrote:
> > > > >
> > > > > > Hi Mickael,
> > > > > >
> > > > > > > Can we do step 3 without breaking any compatibility? If so then
> > > that
> > > > > > sounds like a good idea.
> > > > > >
> > > > > > As far as I know, the answer is yes; I am now updating my PR, so
> I
> > > will
> > > > > > notify you as soon as I complete the work.
> > > > > >
> > > > > > Best,
> > > > > > Dongjin
> > > > > >
> > > > > > On Wed, Dec 15, 2021 at 2:00 AM Mickael Maison <
> > > > mickael.mai...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Hi Dongjin,
> > > > > >>
> > > > > >> Sorry for the late reply. Can we do step 3 without breaking any
> > > > > >> compatibility? If so then that sounds like a good idea.
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Mickael
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On Tue, Nov 23, 2021 at 2:08 PM Dongjin Lee  >
> > > > wrote:
> > > > > >> >
> > > > > >> > Hi Mickael,
> > > > > >> >
> > > > > >> > I also thought over the issue thoroughly and would like to
> > > propose a
> > > > > >> minor
> > > > > >> > change to your proposal:
> > > > > >> >
> > > > > >> > 1. Deprecate log4j-appender now
> > > > > >> > 2. Document how to migrate into logging-log4j2
> > > > > >> > 3. (Changed) Replace the log4j-appender (in turn log4j 1.x)
> > > 

Re: [DISCUSS] KIP-719: Add Log4J2 Appender

[Dongjin Lee]

Hi Haruki,


Thanks for organizing the issue.


If the community prefers logback, I will gladly change the dependency and
update the PR. However, it has the following issues:


1. The log4j2 vulnerabilities seem mostly fixed, and KIP-653 + KIP-719 are
not released yet. So, using log4j2 (whose recent update pace is so high)
will not affect the users.


2. To switch to logback, the following features should be reworked:


  a. Dynamic logger level configuration (core, connect)

  b. Logging tests (streams)

  c. Kafka Appender (tools)


a and b are the most challenging ones since there is little documentation
on how to do this, so it requires analyzing the implementation itself.
(what I actually did with log4j2) About c, logback does not provide a Kafka
Appender so we have to provide an equivalent.


It is why I prefer to use log4j2. How do you think?


Thanks,

Dongjin


On Thu, Dec 23, 2021 at 9:01 AM Haruki Okada  wrote:

> Hi, Dongjin,
>
> Sorry for interrupting the discussion.
> And thank you for your hard work about KIP-653, KIP-719.
>
> I understand that KIP-653 is already accepted so log4j2 is the choice of
> the Kafka community though, I'm now feeling that logback is a better choice
> here.
>
> Reasons:
>
> - even after "log4shell", several vulnerabilities found on log4j2 so new
> versions are released and users have to update in high-pace
> * actually, a CVE was also reported for logback (CVE-2021-42550) but it
> requires edit-permission of the config file for an attacker so it's much
> less threatening
> - log4j1.x and logback are made by same developer (ceki), so substantially
> the successor of log4j1 is logback rather than log4j2
> - in Hadoop project, seems similar suggestion was made from a PMC
> * https://issues.apache.org/jira/browse/HADOOP-12956
>
>
> What do you think about adopting logback instead?
>
>
> Thanks,
>
> 2021年12月21日(火) 18:02 Dongjin Lee :
>
> > Hi Mickael,
> >
> > > In the meantime, you may want to bump the VOTE thread too.
> >
> > Sure, I just reset the voting thread with a brief context.
> >
> > Thanks,
> > Dongjin
> >
> > On Tue, Dec 21, 2021 at 2:13 AM Mickael Maison  >
> > wrote:
> >
> > > Thanks Dongjin!
> > >
> > > I'll take a look soon.
> > > In the meantime, you may want to bump the VOTE thread too.
> > >
> > > Best,
> > > Mickael
> > >
> > >
> > > On Sat, Dec 18, 2021 at 10:00 AM Dongjin Lee 
> wrote:
> > > >
> > > > Hi Mickael,
> > > >
> > > > Finally, I did it! As you can see at the PR
> > > > , KIP-719 now uses
> > log4j2's
> > > > Kafka appender, and log4j-appender is not used by the other modules
> > > > anymore. You can see how it will work with KIP-653 at this preview
> > > > ,
> > > based
> > > > on Apache Kafka 3.0.0. The proposal document
> > > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-719%3A+Deprecate+Log4J+Appender
> > > >
> > > > is also updated accordingly, with its title.
> > > >
> > > > There is a minor issue on log4j2
> > > > , but it seems
> like
> > > it
> > > > will be resolved soon.
> > > >
> > > > Best,
> > > > Dongjin
> > > >
> > > > On Wed, Dec 15, 2021 at 9:28 PM Dongjin Lee 
> > wrote:
> > > >
> > > > > Hi Mickael,
> > > > >
> > > > > > Can we do step 3 without breaking any compatibility? If so then
> > that
> > > > > sounds like a good idea.
> > > > >
> > > > > As far as I know, the answer is yes; I am now updating my PR, so I
> > will
> > > > > notify you as soon as I complete the work.
> > > > >
> > > > > Best,
> > > > > Dongjin
> > > > >
> > > > > On Wed, Dec 15, 2021 at 2:00 AM Mickael Maison <
> > > mickael.mai...@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Hi Dongjin,
> > > > >>
> > > > >> Sorry for the late reply. Can we do step 3 without breaking any
> > > > >> compatibility? If so then that sounds like a good idea.
> > > > >>
> > > > >> Thanks,
> > > > >> Mickael
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Tue, Nov 23, 2021 at 2:08 PM Dongjin Lee 
> > > wrote:
> > > > >> >
> > > > >> > Hi Mickael,
> > > > >> >
> > > > >> > I also thought over the issue thoroughly and would like to
> > propose a
> > > > >> minor
> > > > >> > change to your proposal:
> > > > >> >
> > > > >> > 1. Deprecate log4j-appender now
> > > > >> > 2. Document how to migrate into logging-log4j2
> > > > >> > 3. (Changed) Replace the log4j-appender (in turn log4j 1.x)
> > > > >> dependencies in
> > > > >> > tools, trogdor, and shell and upgrade to log4j2 in 3.x, removing
> > > log4j
> > > > >> 1.x
> > > > >> > dependencies.
> > > > >> > 4. (Changed) Remove log4j-appender in Kafka 4.0
> > > > >> >
> > > > >> > What we need to do for the log4j2 upgrade is just removing the
> > log4j
> > > > >> > dependencies only, for they can cause a classpath error. And
> > > actually,
> > > > >> we
> > > > >> > can do it without discontinuing publishing the log4j-appender
> > 

Re: [DISCUSS] KIP-719: Add Log4J2 Appender

[Haruki Okada]

Hi, Dongjin,

Sorry for interrupting the discussion.
And thank you for your hard work about KIP-653, KIP-719.

I understand that KIP-653 is already accepted so log4j2 is the choice of
the Kafka community though, I'm now feeling that logback is a better choice
here.

Reasons:

- even after "log4shell", several vulnerabilities found on log4j2 so new
versions are released and users have to update in high-pace
* actually, a CVE was also reported for logback (CVE-2021-42550) but it
requires edit-permission of the config file for an attacker so it's much
less threatening
- log4j1.x and logback are made by same developer (ceki), so substantially
the successor of log4j1 is logback rather than log4j2
- in Hadoop project, seems similar suggestion was made from a PMC
* https://issues.apache.org/jira/browse/HADOOP-12956


What do you think about adopting logback instead?


Thanks,

2021年12月21日(火) 18:02 Dongjin Lee :

> Hi Mickael,
>
> > In the meantime, you may want to bump the VOTE thread too.
>
> Sure, I just reset the voting thread with a brief context.
>
> Thanks,
> Dongjin
>
> On Tue, Dec 21, 2021 at 2:13 AM Mickael Maison 
> wrote:
>
> > Thanks Dongjin!
> >
> > I'll take a look soon.
> > In the meantime, you may want to bump the VOTE thread too.
> >
> > Best,
> > Mickael
> >
> >
> > On Sat, Dec 18, 2021 at 10:00 AM Dongjin Lee  wrote:
> > >
> > > Hi Mickael,
> > >
> > > Finally, I did it! As you can see at the PR
> > > , KIP-719 now uses
> log4j2's
> > > Kafka appender, and log4j-appender is not used by the other modules
> > > anymore. You can see how it will work with KIP-653 at this preview
> > > ,
> > based
> > > on Apache Kafka 3.0.0. The proposal document
> > > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-719%3A+Deprecate+Log4J+Appender
> > >
> > > is also updated accordingly, with its title.
> > >
> > > There is a minor issue on log4j2
> > > , but it seems like
> > it
> > > will be resolved soon.
> > >
> > > Best,
> > > Dongjin
> > >
> > > On Wed, Dec 15, 2021 at 9:28 PM Dongjin Lee 
> wrote:
> > >
> > > > Hi Mickael,
> > > >
> > > > > Can we do step 3 without breaking any compatibility? If so then
> that
> > > > sounds like a good idea.
> > > >
> > > > As far as I know, the answer is yes; I am now updating my PR, so I
> will
> > > > notify you as soon as I complete the work.
> > > >
> > > > Best,
> > > > Dongjin
> > > >
> > > > On Wed, Dec 15, 2021 at 2:00 AM Mickael Maison <
> > mickael.mai...@gmail.com>
> > > > wrote:
> > > >
> > > >> Hi Dongjin,
> > > >>
> > > >> Sorry for the late reply. Can we do step 3 without breaking any
> > > >> compatibility? If so then that sounds like a good idea.
> > > >>
> > > >> Thanks,
> > > >> Mickael
> > > >>
> > > >>
> > > >>
> > > >> On Tue, Nov 23, 2021 at 2:08 PM Dongjin Lee 
> > wrote:
> > > >> >
> > > >> > Hi Mickael,
> > > >> >
> > > >> > I also thought over the issue thoroughly and would like to
> propose a
> > > >> minor
> > > >> > change to your proposal:
> > > >> >
> > > >> > 1. Deprecate log4j-appender now
> > > >> > 2. Document how to migrate into logging-log4j2
> > > >> > 3. (Changed) Replace the log4j-appender (in turn log4j 1.x)
> > > >> dependencies in
> > > >> > tools, trogdor, and shell and upgrade to log4j2 in 3.x, removing
> > log4j
> > > >> 1.x
> > > >> > dependencies.
> > > >> > 4. (Changed) Remove log4j-appender in Kafka 4.0
> > > >> >
> > > >> > What we need to do for the log4j2 upgrade is just removing the
> log4j
> > > >> > dependencies only, for they can cause a classpath error. And
> > actually,
> > > >> we
> > > >> > can do it without discontinuing publishing the log4j-appender
> > artifact.
> > > >> So,
> > > >> > I suggest separating the upgrade to log4j2 and removing the
> > > >> log4j-appender
> > > >> > module.
> > > >> >
> > > >> > How do you think? If you agree, I will update the KIP and the PR
> > > >> > accordingly ASAP.
> > > >> >
> > > >> > Thanks,
> > > >> > Dongjin
> > > >> >
> > > >> > On Mon, Nov 15, 2021 at 8:06 PM Mickael Maison <
> > > >> mickael.mai...@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> > > Hi Dongjin,
> > > >> > >
> > > >> > > Thanks for the clarifications.
> > > >> > >
> > > >> > > I wonder if a simpler course of action could be:
> > > >> > > - Deprecate log4j-appender now
> > > >> > > - Document how to use logging-log4j2
> > > >> > > - Remove log4j-appender and all the log4j dependencies in Kafka
> > 4.0
> > > >> > >
> > > >> > > This delays KIP-653 till Kafka 4.0 but (so far) Kafka is not
> > directly
> > > >> > > affected by the log4j CVEs. At least this gives us a clear and
> > simple
> > > >> > > roadmap to follow.
> > > >> > >
> > > >> > > What do you think?
> > > >> > >
> > > >> > > On Tue, Nov 9, 2021 at 12:12 PM Dongjin Lee  >
> > > >> wrote:
> > > >> > > >
> > > >> > > > Hi Mickael,
> > > >> > > >
> > > >> > > > 

Re: [DISCUSS] KIP-719: Add Log4J2 Appender

[Dongjin Lee]

Hi Mickael,

> In the meantime, you may want to bump the VOTE thread too.

Sure, I just reset the voting thread with a brief context.

Thanks,
Dongjin

On Tue, Dec 21, 2021 at 2:13 AM Mickael Maison 
wrote:

> Thanks Dongjin!
>
> I'll take a look soon.
> In the meantime, you may want to bump the VOTE thread too.
>
> Best,
> Mickael
>
>
> On Sat, Dec 18, 2021 at 10:00 AM Dongjin Lee  wrote:
> >
> > Hi Mickael,
> >
> > Finally, I did it! As you can see at the PR
> > , KIP-719 now uses log4j2's
> > Kafka appender, and log4j-appender is not used by the other modules
> > anymore. You can see how it will work with KIP-653 at this preview
> > ,
> based
> > on Apache Kafka 3.0.0. The proposal document
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-719%3A+Deprecate+Log4J+Appender
> >
> > is also updated accordingly, with its title.
> >
> > There is a minor issue on log4j2
> > , but it seems like
> it
> > will be resolved soon.
> >
> > Best,
> > Dongjin
> >
> > On Wed, Dec 15, 2021 at 9:28 PM Dongjin Lee  wrote:
> >
> > > Hi Mickael,
> > >
> > > > Can we do step 3 without breaking any compatibility? If so then that
> > > sounds like a good idea.
> > >
> > > As far as I know, the answer is yes; I am now updating my PR, so I will
> > > notify you as soon as I complete the work.
> > >
> > > Best,
> > > Dongjin
> > >
> > > On Wed, Dec 15, 2021 at 2:00 AM Mickael Maison <
> mickael.mai...@gmail.com>
> > > wrote:
> > >
> > >> Hi Dongjin,
> > >>
> > >> Sorry for the late reply. Can we do step 3 without breaking any
> > >> compatibility? If so then that sounds like a good idea.
> > >>
> > >> Thanks,
> > >> Mickael
> > >>
> > >>
> > >>
> > >> On Tue, Nov 23, 2021 at 2:08 PM Dongjin Lee 
> wrote:
> > >> >
> > >> > Hi Mickael,
> > >> >
> > >> > I also thought over the issue thoroughly and would like to propose a
> > >> minor
> > >> > change to your proposal:
> > >> >
> > >> > 1. Deprecate log4j-appender now
> > >> > 2. Document how to migrate into logging-log4j2
> > >> > 3. (Changed) Replace the log4j-appender (in turn log4j 1.x)
> > >> dependencies in
> > >> > tools, trogdor, and shell and upgrade to log4j2 in 3.x, removing
> log4j
> > >> 1.x
> > >> > dependencies.
> > >> > 4. (Changed) Remove log4j-appender in Kafka 4.0
> > >> >
> > >> > What we need to do for the log4j2 upgrade is just removing the log4j
> > >> > dependencies only, for they can cause a classpath error. And
> actually,
> > >> we
> > >> > can do it without discontinuing publishing the log4j-appender
> artifact.
> > >> So,
> > >> > I suggest separating the upgrade to log4j2 and removing the
> > >> log4j-appender
> > >> > module.
> > >> >
> > >> > How do you think? If you agree, I will update the KIP and the PR
> > >> > accordingly ASAP.
> > >> >
> > >> > Thanks,
> > >> > Dongjin
> > >> >
> > >> > On Mon, Nov 15, 2021 at 8:06 PM Mickael Maison <
> > >> mickael.mai...@gmail.com>
> > >> > wrote:
> > >> >
> > >> > > Hi Dongjin,
> > >> > >
> > >> > > Thanks for the clarifications.
> > >> > >
> > >> > > I wonder if a simpler course of action could be:
> > >> > > - Deprecate log4j-appender now
> > >> > > - Document how to use logging-log4j2
> > >> > > - Remove log4j-appender and all the log4j dependencies in Kafka
> 4.0
> > >> > >
> > >> > > This delays KIP-653 till Kafka 4.0 but (so far) Kafka is not
> directly
> > >> > > affected by the log4j CVEs. At least this gives us a clear and
> simple
> > >> > > roadmap to follow.
> > >> > >
> > >> > > What do you think?
> > >> > >
> > >> > > On Tue, Nov 9, 2021 at 12:12 PM Dongjin Lee 
> > >> wrote:
> > >> > > >
> > >> > > > Hi Mickael,
> > >> > > >
> > >> > > > I greatly appreciate you for reading the proposal so carefully!
> I
> > >> wrote
> > >> > > it
> > >> > > > quite a while ago and rechecked it today.
> > >> > > >
> > >> > > > > Is the KIP proposing to replace the existing log4-appender or
> > >> simply
> > >> > > add
> > >> > > > a new one for log4j2? Reading the KIP and with its current
> title,
> > >> it's
> > >> > > not
> > >> > > > entirely explicit.
> > >> > > >
> > >> > > > Oh, After re-reading it, I realized that this is not clear. Let
> me
> > >> > > clarify;
> > >> > > >
> > >> > > > 1. Provide a lo4j2 equivalent of traditional log4j-appender,
> > >> > > > log4j2-appender.
> > >> > > > 2. Migrate the modules depending on log4j-appender (i.e., tools,
> > >> trogdor,
> > >> > > > shell) into log4j2-appender, removing log4j-appender from
> > >> dependencies.
> > >> > > > 3. Entirely remove log4j-appender from the project dependencies,
> > >> along
> > >> > > with
> > >> > > > log4j.
> > >> > > >
> > >> > > > I think log4j-appender may be published for every new release
> like
> > >> > > before,
> > >> > > > but the committee should make a decision on the policy.
> > >> > > >
> > >> > > > > Under Rejected Alternative, the KIP states: 

Re: [DISCUSS] KIP-719: Add Log4J2 Appender

[Mickael Maison]

Thanks Dongjin!

I'll take a look soon.
In the meantime, you may want to bump the VOTE thread too.

Best,
Mickael


On Sat, Dec 18, 2021 at 10:00 AM Dongjin Lee  wrote:
>
> Hi Mickael,
>
> Finally, I did it! As you can see at the PR
> , KIP-719 now uses log4j2's
> Kafka appender, and log4j-appender is not used by the other modules
> anymore. You can see how it will work with KIP-653 at this preview
> , based
> on Apache Kafka 3.0.0. The proposal document
> 
> is also updated accordingly, with its title.
>
> There is a minor issue on log4j2
> , but it seems like it
> will be resolved soon.
>
> Best,
> Dongjin
>
> On Wed, Dec 15, 2021 at 9:28 PM Dongjin Lee  wrote:
>
> > Hi Mickael,
> >
> > > Can we do step 3 without breaking any compatibility? If so then that
> > sounds like a good idea.
> >
> > As far as I know, the answer is yes; I am now updating my PR, so I will
> > notify you as soon as I complete the work.
> >
> > Best,
> > Dongjin
> >
> > On Wed, Dec 15, 2021 at 2:00 AM Mickael Maison 
> > wrote:
> >
> >> Hi Dongjin,
> >>
> >> Sorry for the late reply. Can we do step 3 without breaking any
> >> compatibility? If so then that sounds like a good idea.
> >>
> >> Thanks,
> >> Mickael
> >>
> >>
> >>
> >> On Tue, Nov 23, 2021 at 2:08 PM Dongjin Lee  wrote:
> >> >
> >> > Hi Mickael,
> >> >
> >> > I also thought over the issue thoroughly and would like to propose a
> >> minor
> >> > change to your proposal:
> >> >
> >> > 1. Deprecate log4j-appender now
> >> > 2. Document how to migrate into logging-log4j2
> >> > 3. (Changed) Replace the log4j-appender (in turn log4j 1.x)
> >> dependencies in
> >> > tools, trogdor, and shell and upgrade to log4j2 in 3.x, removing log4j
> >> 1.x
> >> > dependencies.
> >> > 4. (Changed) Remove log4j-appender in Kafka 4.0
> >> >
> >> > What we need to do for the log4j2 upgrade is just removing the log4j
> >> > dependencies only, for they can cause a classpath error. And actually,
> >> we
> >> > can do it without discontinuing publishing the log4j-appender artifact.
> >> So,
> >> > I suggest separating the upgrade to log4j2 and removing the
> >> log4j-appender
> >> > module.
> >> >
> >> > How do you think? If you agree, I will update the KIP and the PR
> >> > accordingly ASAP.
> >> >
> >> > Thanks,
> >> > Dongjin
> >> >
> >> > On Mon, Nov 15, 2021 at 8:06 PM Mickael Maison <
> >> mickael.mai...@gmail.com>
> >> > wrote:
> >> >
> >> > > Hi Dongjin,
> >> > >
> >> > > Thanks for the clarifications.
> >> > >
> >> > > I wonder if a simpler course of action could be:
> >> > > - Deprecate log4j-appender now
> >> > > - Document how to use logging-log4j2
> >> > > - Remove log4j-appender and all the log4j dependencies in Kafka 4.0
> >> > >
> >> > > This delays KIP-653 till Kafka 4.0 but (so far) Kafka is not directly
> >> > > affected by the log4j CVEs. At least this gives us a clear and simple
> >> > > roadmap to follow.
> >> > >
> >> > > What do you think?
> >> > >
> >> > > On Tue, Nov 9, 2021 at 12:12 PM Dongjin Lee 
> >> wrote:
> >> > > >
> >> > > > Hi Mickael,
> >> > > >
> >> > > > I greatly appreciate you for reading the proposal so carefully! I
> >> wrote
> >> > > it
> >> > > > quite a while ago and rechecked it today.
> >> > > >
> >> > > > > Is the KIP proposing to replace the existing log4-appender or
> >> simply
> >> > > add
> >> > > > a new one for log4j2? Reading the KIP and with its current title,
> >> it's
> >> > > not
> >> > > > entirely explicit.
> >> > > >
> >> > > > Oh, After re-reading it, I realized that this is not clear. Let me
> >> > > clarify;
> >> > > >
> >> > > > 1. Provide a lo4j2 equivalent of traditional log4j-appender,
> >> > > > log4j2-appender.
> >> > > > 2. Migrate the modules depending on log4j-appender (i.e., tools,
> >> trogdor,
> >> > > > shell) into log4j2-appender, removing log4j-appender from
> >> dependencies.
> >> > > > 3. Entirely remove log4j-appender from the project dependencies,
> >> along
> >> > > with
> >> > > > log4j.
> >> > > >
> >> > > > I think log4j-appender may be published for every new release like
> >> > > before,
> >> > > > but the committee should make a decision on the policy.
> >> > > >
> >> > > > > Under Rejected Alternative, the KIP states: "the Kafka appender
> >> > > provided
> >> > > > by log4j2 community stores log message in the Record key". Looking
> >> at the
> >> > > > code, it looks like the log message is stored in the Record value:
> >> > > >
> >> > >
> >> https://github.com/apache/logging-log4j2/blob/master/log4j-kafka/src/main/java/org/apache/logging/log4j/kafka/appender/KafkaManager.java#L135
> >> > > > Am I missing something?
> >> > > >
> >> > > > It's totally my fault; I confused it with another appender. The
> >> > > > compatibility problem in the logging-log4j2 

Re: [DISCUSS] KIP-719: Add Log4J2 Appender

[Dongjin Lee]

Hi Mickael,

Finally, I did it! As you can see at the PR
, KIP-719 now uses log4j2's
Kafka appender, and log4j-appender is not used by the other modules
anymore. You can see how it will work with KIP-653 at this preview
, based
on Apache Kafka 3.0.0. The proposal document

is also updated accordingly, with its title.

There is a minor issue on log4j2
, but it seems like it
will be resolved soon.

Best,
Dongjin

On Wed, Dec 15, 2021 at 9:28 PM Dongjin Lee  wrote:

> Hi Mickael,
>
> > Can we do step 3 without breaking any compatibility? If so then that
> sounds like a good idea.
>
> As far as I know, the answer is yes; I am now updating my PR, so I will
> notify you as soon as I complete the work.
>
> Best,
> Dongjin
>
> On Wed, Dec 15, 2021 at 2:00 AM Mickael Maison 
> wrote:
>
>> Hi Dongjin,
>>
>> Sorry for the late reply. Can we do step 3 without breaking any
>> compatibility? If so then that sounds like a good idea.
>>
>> Thanks,
>> Mickael
>>
>>
>>
>> On Tue, Nov 23, 2021 at 2:08 PM Dongjin Lee  wrote:
>> >
>> > Hi Mickael,
>> >
>> > I also thought over the issue thoroughly and would like to propose a
>> minor
>> > change to your proposal:
>> >
>> > 1. Deprecate log4j-appender now
>> > 2. Document how to migrate into logging-log4j2
>> > 3. (Changed) Replace the log4j-appender (in turn log4j 1.x)
>> dependencies in
>> > tools, trogdor, and shell and upgrade to log4j2 in 3.x, removing log4j
>> 1.x
>> > dependencies.
>> > 4. (Changed) Remove log4j-appender in Kafka 4.0
>> >
>> > What we need to do for the log4j2 upgrade is just removing the log4j
>> > dependencies only, for they can cause a classpath error. And actually,
>> we
>> > can do it without discontinuing publishing the log4j-appender artifact.
>> So,
>> > I suggest separating the upgrade to log4j2 and removing the
>> log4j-appender
>> > module.
>> >
>> > How do you think? If you agree, I will update the KIP and the PR
>> > accordingly ASAP.
>> >
>> > Thanks,
>> > Dongjin
>> >
>> > On Mon, Nov 15, 2021 at 8:06 PM Mickael Maison <
>> mickael.mai...@gmail.com>
>> > wrote:
>> >
>> > > Hi Dongjin,
>> > >
>> > > Thanks for the clarifications.
>> > >
>> > > I wonder if a simpler course of action could be:
>> > > - Deprecate log4j-appender now
>> > > - Document how to use logging-log4j2
>> > > - Remove log4j-appender and all the log4j dependencies in Kafka 4.0
>> > >
>> > > This delays KIP-653 till Kafka 4.0 but (so far) Kafka is not directly
>> > > affected by the log4j CVEs. At least this gives us a clear and simple
>> > > roadmap to follow.
>> > >
>> > > What do you think?
>> > >
>> > > On Tue, Nov 9, 2021 at 12:12 PM Dongjin Lee 
>> wrote:
>> > > >
>> > > > Hi Mickael,
>> > > >
>> > > > I greatly appreciate you for reading the proposal so carefully! I
>> wrote
>> > > it
>> > > > quite a while ago and rechecked it today.
>> > > >
>> > > > > Is the KIP proposing to replace the existing log4-appender or
>> simply
>> > > add
>> > > > a new one for log4j2? Reading the KIP and with its current title,
>> it's
>> > > not
>> > > > entirely explicit.
>> > > >
>> > > > Oh, After re-reading it, I realized that this is not clear. Let me
>> > > clarify;
>> > > >
>> > > > 1. Provide a lo4j2 equivalent of traditional log4j-appender,
>> > > > log4j2-appender.
>> > > > 2. Migrate the modules depending on log4j-appender (i.e., tools,
>> trogdor,
>> > > > shell) into log4j2-appender, removing log4j-appender from
>> dependencies.
>> > > > 3. Entirely remove log4j-appender from the project dependencies,
>> along
>> > > with
>> > > > log4j.
>> > > >
>> > > > I think log4j-appender may be published for every new release like
>> > > before,
>> > > > but the committee should make a decision on the policy.
>> > > >
>> > > > > Under Rejected Alternative, the KIP states: "the Kafka appender
>> > > provided
>> > > > by log4j2 community stores log message in the Record key". Looking
>> at the
>> > > > code, it looks like the log message is stored in the Record value:
>> > > >
>> > >
>> https://github.com/apache/logging-log4j2/blob/master/log4j-kafka/src/main/java/org/apache/logging/log4j/kafka/appender/KafkaManager.java#L135
>> > > > Am I missing something?
>> > > >
>> > > > It's totally my fault; I confused it with another appender. The
>> > > > compatibility problem in the logging-log4j2 Kafka appender is not
>> the
>> > > > format but the configuration. logging-log4j2 Kafka appender supports
>> > > > `properties` configuration, which will be directly used to
>> instantiate a
>> > > > Kafka producer. However, log4j-appender has been using non-producer
>> > > config
>> > > > names like brokerList (=bootstrap.servers), requiredNumAcks (=acks).
>> > > > Instead, logging-log4j2 Kafka appender supports retryCount,
>> > > > 

Re: [DISCUSS] KIP-719: Add Log4J2 Appender

[Dongjin Lee]

Hi Mickael,

> Can we do step 3 without breaking any compatibility? If so then that
sounds like a good idea.

As far as I know, the answer is yes; I am now updating my PR, so I will
notify you as soon as I complete the work.

Best,
Dongjin

On Wed, Dec 15, 2021 at 2:00 AM Mickael Maison 
wrote:

> Hi Dongjin,
>
> Sorry for the late reply. Can we do step 3 without breaking any
> compatibility? If so then that sounds like a good idea.
>
> Thanks,
> Mickael
>
>
>
> On Tue, Nov 23, 2021 at 2:08 PM Dongjin Lee  wrote:
> >
> > Hi Mickael,
> >
> > I also thought over the issue thoroughly and would like to propose a
> minor
> > change to your proposal:
> >
> > 1. Deprecate log4j-appender now
> > 2. Document how to migrate into logging-log4j2
> > 3. (Changed) Replace the log4j-appender (in turn log4j 1.x) dependencies
> in
> > tools, trogdor, and shell and upgrade to log4j2 in 3.x, removing log4j
> 1.x
> > dependencies.
> > 4. (Changed) Remove log4j-appender in Kafka 4.0
> >
> > What we need to do for the log4j2 upgrade is just removing the log4j
> > dependencies only, for they can cause a classpath error. And actually, we
> > can do it without discontinuing publishing the log4j-appender artifact.
> So,
> > I suggest separating the upgrade to log4j2 and removing the
> log4j-appender
> > module.
> >
> > How do you think? If you agree, I will update the KIP and the PR
> > accordingly ASAP.
> >
> > Thanks,
> > Dongjin
> >
> > On Mon, Nov 15, 2021 at 8:06 PM Mickael Maison  >
> > wrote:
> >
> > > Hi Dongjin,
> > >
> > > Thanks for the clarifications.
> > >
> > > I wonder if a simpler course of action could be:
> > > - Deprecate log4j-appender now
> > > - Document how to use logging-log4j2
> > > - Remove log4j-appender and all the log4j dependencies in Kafka 4.0
> > >
> > > This delays KIP-653 till Kafka 4.0 but (so far) Kafka is not directly
> > > affected by the log4j CVEs. At least this gives us a clear and simple
> > > roadmap to follow.
> > >
> > > What do you think?
> > >
> > > On Tue, Nov 9, 2021 at 12:12 PM Dongjin Lee 
> wrote:
> > > >
> > > > Hi Mickael,
> > > >
> > > > I greatly appreciate you for reading the proposal so carefully! I
> wrote
> > > it
> > > > quite a while ago and rechecked it today.
> > > >
> > > > > Is the KIP proposing to replace the existing log4-appender or
> simply
> > > add
> > > > a new one for log4j2? Reading the KIP and with its current title,
> it's
> > > not
> > > > entirely explicit.
> > > >
> > > > Oh, After re-reading it, I realized that this is not clear. Let me
> > > clarify;
> > > >
> > > > 1. Provide a lo4j2 equivalent of traditional log4j-appender,
> > > > log4j2-appender.
> > > > 2. Migrate the modules depending on log4j-appender (i.e., tools,
> trogdor,
> > > > shell) into log4j2-appender, removing log4j-appender from
> dependencies.
> > > > 3. Entirely remove log4j-appender from the project dependencies,
> along
> > > with
> > > > log4j.
> > > >
> > > > I think log4j-appender may be published for every new release like
> > > before,
> > > > but the committee should make a decision on the policy.
> > > >
> > > > > Under Rejected Alternative, the KIP states: "the Kafka appender
> > > provided
> > > > by log4j2 community stores log message in the Record key". Looking
> at the
> > > > code, it looks like the log message is stored in the Record value:
> > > >
> > >
> https://github.com/apache/logging-log4j2/blob/master/log4j-kafka/src/main/java/org/apache/logging/log4j/kafka/appender/KafkaManager.java#L135
> > > > Am I missing something?
> > > >
> > > > It's totally my fault; I confused it with another appender. The
> > > > compatibility problem in the logging-log4j2 Kafka appender is not the
> > > > format but the configuration. logging-log4j2 Kafka appender supports
> > > > `properties` configuration, which will be directly used to
> instantiate a
> > > > Kafka producer. However, log4j-appender has been using non-producer
> > > config
> > > > names like brokerList (=bootstrap.servers), requiredNumAcks (=acks).
> > > > Instead, logging-log4j2 Kafka appender supports retryCount,
> > > > sendEventTimestamp.
> > > >
> > > > On second thought, using logging-log4j2 Kafka appender internally and
> > > > making log4j2-appender to focus on compatibility facade only would
> be a
> > > > better approach; As I described above, the goal of this module is
> just
> > > > keeping the backward-compatibility, and (as you pointed out) the
> current
> > > > implementation has little value. Since
> > > org.apache.logging.log4j:log4j-core
> > > > already includes Kafka appender, we can make use of the 'proven
> wheel'
> > > > without adding more dependencies. I have not tried it yet, but I
> think it
> > > > is well worth it. (One additional advantage of this approach is
> > > providing a
> > > > bridge to the users who hope to move from/into logging-log4j2 Kafka
> > > > appender.)
> > > >
> > > > > As the current log4j-appender is not even deprecated yet, in
> theory we
> > > > can't remove it 

Re: [DISCUSS] KIP-719: Add Log4J2 Appender

[Mickael Maison]

Hi Dongjin,

Sorry for the late reply. Can we do step 3 without breaking any
compatibility? If so then that sounds like a good idea.

Thanks,
Mickael



On Tue, Nov 23, 2021 at 2:08 PM Dongjin Lee  wrote:
>
> Hi Mickael,
>
> I also thought over the issue thoroughly and would like to propose a minor
> change to your proposal:
>
> 1. Deprecate log4j-appender now
> 2. Document how to migrate into logging-log4j2
> 3. (Changed) Replace the log4j-appender (in turn log4j 1.x) dependencies in
> tools, trogdor, and shell and upgrade to log4j2 in 3.x, removing log4j 1.x
> dependencies.
> 4. (Changed) Remove log4j-appender in Kafka 4.0
>
> What we need to do for the log4j2 upgrade is just removing the log4j
> dependencies only, for they can cause a classpath error. And actually, we
> can do it without discontinuing publishing the log4j-appender artifact. So,
> I suggest separating the upgrade to log4j2 and removing the log4j-appender
> module.
>
> How do you think? If you agree, I will update the KIP and the PR
> accordingly ASAP.
>
> Thanks,
> Dongjin
>
> On Mon, Nov 15, 2021 at 8:06 PM Mickael Maison 
> wrote:
>
> > Hi Dongjin,
> >
> > Thanks for the clarifications.
> >
> > I wonder if a simpler course of action could be:
> > - Deprecate log4j-appender now
> > - Document how to use logging-log4j2
> > - Remove log4j-appender and all the log4j dependencies in Kafka 4.0
> >
> > This delays KIP-653 till Kafka 4.0 but (so far) Kafka is not directly
> > affected by the log4j CVEs. At least this gives us a clear and simple
> > roadmap to follow.
> >
> > What do you think?
> >
> > On Tue, Nov 9, 2021 at 12:12 PM Dongjin Lee  wrote:
> > >
> > > Hi Mickael,
> > >
> > > I greatly appreciate you for reading the proposal so carefully! I wrote
> > it
> > > quite a while ago and rechecked it today.
> > >
> > > > Is the KIP proposing to replace the existing log4-appender or simply
> > add
> > > a new one for log4j2? Reading the KIP and with its current title, it's
> > not
> > > entirely explicit.
> > >
> > > Oh, After re-reading it, I realized that this is not clear. Let me
> > clarify;
> > >
> > > 1. Provide a lo4j2 equivalent of traditional log4j-appender,
> > > log4j2-appender.
> > > 2. Migrate the modules depending on log4j-appender (i.e., tools, trogdor,
> > > shell) into log4j2-appender, removing log4j-appender from dependencies.
> > > 3. Entirely remove log4j-appender from the project dependencies, along
> > with
> > > log4j.
> > >
> > > I think log4j-appender may be published for every new release like
> > before,
> > > but the committee should make a decision on the policy.
> > >
> > > > Under Rejected Alternative, the KIP states: "the Kafka appender
> > provided
> > > by log4j2 community stores log message in the Record key". Looking at the
> > > code, it looks like the log message is stored in the Record value:
> > >
> > https://github.com/apache/logging-log4j2/blob/master/log4j-kafka/src/main/java/org/apache/logging/log4j/kafka/appender/KafkaManager.java#L135
> > > Am I missing something?
> > >
> > > It's totally my fault; I confused it with another appender. The
> > > compatibility problem in the logging-log4j2 Kafka appender is not the
> > > format but the configuration. logging-log4j2 Kafka appender supports
> > > `properties` configuration, which will be directly used to instantiate a
> > > Kafka producer. However, log4j-appender has been using non-producer
> > config
> > > names like brokerList (=bootstrap.servers), requiredNumAcks (=acks).
> > > Instead, logging-log4j2 Kafka appender supports retryCount,
> > > sendEventTimestamp.
> > >
> > > On second thought, using logging-log4j2 Kafka appender internally and
> > > making log4j2-appender to focus on compatibility facade only would be a
> > > better approach; As I described above, the goal of this module is just
> > > keeping the backward-compatibility, and (as you pointed out) the current
> > > implementation has little value. Since
> > org.apache.logging.log4j:log4j-core
> > > already includes Kafka appender, we can make use of the 'proven wheel'
> > > without adding more dependencies. I have not tried it yet, but I think it
> > > is well worth it. (One additional advantage of this approach is
> > providing a
> > > bridge to the users who hope to move from/into logging-log4j2 Kafka
> > > appender.)
> > >
> > > > As the current log4j-appender is not even deprecated yet, in theory we
> > > can't remove it till Kafka 4. If we want to speed up the process, I
> > wonder
> > > if the lack of documentation and a migration guide could help us. What do
> > > you think?
> > >
> > > In fact, this is what I am doing nowadays. While working with
> > > log4j-appender, I found that despite a lack of documentation,
> > considerable
> > > users are already using it[^1][^2][^3][^4][^5]. So, I think providing a
> > > documentation to those who are already using log4j-appender is
> > > indispensable. It should include:
> > >
> > > - What is the difference between log4j-appender 

Re: [DISCUSS] KIP-719: Add Log4J2 Appender

[Dongjin Lee]

Hi Mickael,

I also thought over the issue thoroughly and would like to propose a minor
change to your proposal:

1. Deprecate log4j-appender now
2. Document how to migrate into logging-log4j2
3. (Changed) Replace the log4j-appender (in turn log4j 1.x) dependencies in
tools, trogdor, and shell and upgrade to log4j2 in 3.x, removing log4j 1.x
dependencies.
4. (Changed) Remove log4j-appender in Kafka 4.0

What we need to do for the log4j2 upgrade is just removing the log4j
dependencies only, for they can cause a classpath error. And actually, we
can do it without discontinuing publishing the log4j-appender artifact. So,
I suggest separating the upgrade to log4j2 and removing the log4j-appender
module.

How do you think? If you agree, I will update the KIP and the PR
accordingly ASAP.

Thanks,
Dongjin

On Mon, Nov 15, 2021 at 8:06 PM Mickael Maison 
wrote:

> Hi Dongjin,
>
> Thanks for the clarifications.
>
> I wonder if a simpler course of action could be:
> - Deprecate log4j-appender now
> - Document how to use logging-log4j2
> - Remove log4j-appender and all the log4j dependencies in Kafka 4.0
>
> This delays KIP-653 till Kafka 4.0 but (so far) Kafka is not directly
> affected by the log4j CVEs. At least this gives us a clear and simple
> roadmap to follow.
>
> What do you think?
>
> On Tue, Nov 9, 2021 at 12:12 PM Dongjin Lee  wrote:
> >
> > Hi Mickael,
> >
> > I greatly appreciate you for reading the proposal so carefully! I wrote
> it
> > quite a while ago and rechecked it today.
> >
> > > Is the KIP proposing to replace the existing log4-appender or simply
> add
> > a new one for log4j2? Reading the KIP and with its current title, it's
> not
> > entirely explicit.
> >
> > Oh, After re-reading it, I realized that this is not clear. Let me
> clarify;
> >
> > 1. Provide a lo4j2 equivalent of traditional log4j-appender,
> > log4j2-appender.
> > 2. Migrate the modules depending on log4j-appender (i.e., tools, trogdor,
> > shell) into log4j2-appender, removing log4j-appender from dependencies.
> > 3. Entirely remove log4j-appender from the project dependencies, along
> with
> > log4j.
> >
> > I think log4j-appender may be published for every new release like
> before,
> > but the committee should make a decision on the policy.
> >
> > > Under Rejected Alternative, the KIP states: "the Kafka appender
> provided
> > by log4j2 community stores log message in the Record key". Looking at the
> > code, it looks like the log message is stored in the Record value:
> >
> https://github.com/apache/logging-log4j2/blob/master/log4j-kafka/src/main/java/org/apache/logging/log4j/kafka/appender/KafkaManager.java#L135
> > Am I missing something?
> >
> > It's totally my fault; I confused it with another appender. The
> > compatibility problem in the logging-log4j2 Kafka appender is not the
> > format but the configuration. logging-log4j2 Kafka appender supports
> > `properties` configuration, which will be directly used to instantiate a
> > Kafka producer. However, log4j-appender has been using non-producer
> config
> > names like brokerList (=bootstrap.servers), requiredNumAcks (=acks).
> > Instead, logging-log4j2 Kafka appender supports retryCount,
> > sendEventTimestamp.
> >
> > On second thought, using logging-log4j2 Kafka appender internally and
> > making log4j2-appender to focus on compatibility facade only would be a
> > better approach; As I described above, the goal of this module is just
> > keeping the backward-compatibility, and (as you pointed out) the current
> > implementation has little value. Since
> org.apache.logging.log4j:log4j-core
> > already includes Kafka appender, we can make use of the 'proven wheel'
> > without adding more dependencies. I have not tried it yet, but I think it
> > is well worth it. (One additional advantage of this approach is
> providing a
> > bridge to the users who hope to move from/into logging-log4j2 Kafka
> > appender.)
> >
> > > As the current log4j-appender is not even deprecated yet, in theory we
> > can't remove it till Kafka 4. If we want to speed up the process, I
> wonder
> > if the lack of documentation and a migration guide could help us. What do
> > you think?
> >
> > In fact, this is what I am doing nowadays. While working with
> > log4j-appender, I found that despite a lack of documentation,
> considerable
> > users are already using it[^1][^2][^3][^4][^5]. So, I think providing a
> > documentation to those who are already using log4j-appender is
> > indispensable. It should include:
> >
> > - What is the difference between log4j-appender vs. log4j2-appender.
> > - Which options are supported and deprecated.
> > - Exemplar configurations that show how to migrate.
> >
> > Here is the summary:
> >
> > 1. The goal of this proposal is to replace the traditional log4j-appender
> > for compatibility concerns. But log4j-appender may be published after the
> > deprecation.
> > 2. As of present, the description about logging-log4j2 Kafka appender is
> > entirely wrong. The 

Re: [DISCUSS] KIP-719: Add Log4J2 Appender

[Mickael Maison]

Hi Dongjin,

Thanks for the clarifications.

I wonder if a simpler course of action could be:
- Deprecate log4j-appender now
- Document how to use logging-log4j2
- Remove log4j-appender and all the log4j dependencies in Kafka 4.0

This delays KIP-653 till Kafka 4.0 but (so far) Kafka is not directly
affected by the log4j CVEs. At least this gives us a clear and simple
roadmap to follow.

What do you think?

On Tue, Nov 9, 2021 at 12:12 PM Dongjin Lee  wrote:
>
> Hi Mickael,
>
> I greatly appreciate you for reading the proposal so carefully! I wrote it
> quite a while ago and rechecked it today.
>
> > Is the KIP proposing to replace the existing log4-appender or simply add
> a new one for log4j2? Reading the KIP and with its current title, it's not
> entirely explicit.
>
> Oh, After re-reading it, I realized that this is not clear. Let me clarify;
>
> 1. Provide a lo4j2 equivalent of traditional log4j-appender,
> log4j2-appender.
> 2. Migrate the modules depending on log4j-appender (i.e., tools, trogdor,
> shell) into log4j2-appender, removing log4j-appender from dependencies.
> 3. Entirely remove log4j-appender from the project dependencies, along with
> log4j.
>
> I think log4j-appender may be published for every new release like before,
> but the committee should make a decision on the policy.
>
> > Under Rejected Alternative, the KIP states: "the Kafka appender provided
> by log4j2 community stores log message in the Record key". Looking at the
> code, it looks like the log message is stored in the Record value:
> https://github.com/apache/logging-log4j2/blob/master/log4j-kafka/src/main/java/org/apache/logging/log4j/kafka/appender/KafkaManager.java#L135
> Am I missing something?
>
> It's totally my fault; I confused it with another appender. The
> compatibility problem in the logging-log4j2 Kafka appender is not the
> format but the configuration. logging-log4j2 Kafka appender supports
> `properties` configuration, which will be directly used to instantiate a
> Kafka producer. However, log4j-appender has been using non-producer config
> names like brokerList (=bootstrap.servers), requiredNumAcks (=acks).
> Instead, logging-log4j2 Kafka appender supports retryCount,
> sendEventTimestamp.
>
> On second thought, using logging-log4j2 Kafka appender internally and
> making log4j2-appender to focus on compatibility facade only would be a
> better approach; As I described above, the goal of this module is just
> keeping the backward-compatibility, and (as you pointed out) the current
> implementation has little value. Since org.apache.logging.log4j:log4j-core
> already includes Kafka appender, we can make use of the 'proven wheel'
> without adding more dependencies. I have not tried it yet, but I think it
> is well worth it. (One additional advantage of this approach is providing a
> bridge to the users who hope to move from/into logging-log4j2 Kafka
> appender.)
>
> > As the current log4j-appender is not even deprecated yet, in theory we
> can't remove it till Kafka 4. If we want to speed up the process, I wonder
> if the lack of documentation and a migration guide could help us. What do
> you think?
>
> In fact, this is what I am doing nowadays. While working with
> log4j-appender, I found that despite a lack of documentation, considerable
> users are already using it[^1][^2][^3][^4][^5]. So, I think providing a
> documentation to those who are already using log4j-appender is
> indispensable. It should include:
>
> - What is the difference between log4j-appender vs. log4j2-appender.
> - Which options are supported and deprecated.
> - Exemplar configurations that show how to migrate.
>
> Here is the summary:
>
> 1. The goal of this proposal is to replace the traditional log4j-appender
> for compatibility concerns. But log4j-appender may be published after the
> deprecation.
> 2. As of present, the description about logging-log4j2 Kafka appender is
> entirely wrong. The problem is interface compatibility, not record format.
> Focusing on the compatibility facade is a good approach.
> 3. A documentation focus on migration should be provided.
>
> If you have any questions or suggestions, don't hesitate to tell me. Thanks
> again for your comments!
>
> Best,
> Dongjin
>
> [^1]:
> https://docs.cloudera.com/csa/1.2.0/monitoring/topics/csa-kafka-logging.html
> [^2]:
> https://stackoverflow.com/questions/22034895/how-to-use-kafka-0-8-log4j-appender
> [^3]:
> https://stackoverflow.com/questions/32402405/delay-in-kafka-log4j-appender
> [^4]:
> https://stackoverflow.com/questions/32301129/kafka-log4j-appender-not-sending-messages
> [^5]:
> https://stackoverflow.com/questions/35628706/kafka-log4j-appender-0-9-does-not-work
>
> On Mon, Nov 8, 2021 at 9:04 PM Mickael Maison 
> wrote:
>
> > Hi Dongjin,
> >
> > Thanks for working on the update to log4j2, it's definitively
> > something we should complete.
> > I have a couple of comments:
> >
> > 1) Is the KIP proposing to replace the existing log4-appender or
> > simply add 

Re: [DISCUSS] KIP-719: Add Log4J2 Appender

[Dongjin Lee]

Hi Mickael,

I greatly appreciate you for reading the proposal so carefully! I wrote it
quite a while ago and rechecked it today.

> Is the KIP proposing to replace the existing log4-appender or simply add
a new one for log4j2? Reading the KIP and with its current title, it's not
entirely explicit.

Oh, After re-reading it, I realized that this is not clear. Let me clarify;

1. Provide a lo4j2 equivalent of traditional log4j-appender,
log4j2-appender.
2. Migrate the modules depending on log4j-appender (i.e., tools, trogdor,
shell) into log4j2-appender, removing log4j-appender from dependencies.
3. Entirely remove log4j-appender from the project dependencies, along with
log4j.

I think log4j-appender may be published for every new release like before,
but the committee should make a decision on the policy.

> Under Rejected Alternative, the KIP states: "the Kafka appender provided
by log4j2 community stores log message in the Record key". Looking at the
code, it looks like the log message is stored in the Record value:
https://github.com/apache/logging-log4j2/blob/master/log4j-kafka/src/main/java/org/apache/logging/log4j/kafka/appender/KafkaManager.java#L135
Am I missing something?

It's totally my fault; I confused it with another appender. The
compatibility problem in the logging-log4j2 Kafka appender is not the
format but the configuration. logging-log4j2 Kafka appender supports
`properties` configuration, which will be directly used to instantiate a
Kafka producer. However, log4j-appender has been using non-producer config
names like brokerList (=bootstrap.servers), requiredNumAcks (=acks).
Instead, logging-log4j2 Kafka appender supports retryCount,
sendEventTimestamp.

On second thought, using logging-log4j2 Kafka appender internally and
making log4j2-appender to focus on compatibility facade only would be a
better approach; As I described above, the goal of this module is just
keeping the backward-compatibility, and (as you pointed out) the current
implementation has little value. Since org.apache.logging.log4j:log4j-core
already includes Kafka appender, we can make use of the 'proven wheel'
without adding more dependencies. I have not tried it yet, but I think it
is well worth it. (One additional advantage of this approach is providing a
bridge to the users who hope to move from/into logging-log4j2 Kafka
appender.)

> As the current log4j-appender is not even deprecated yet, in theory we
can't remove it till Kafka 4. If we want to speed up the process, I wonder
if the lack of documentation and a migration guide could help us. What do
you think?

In fact, this is what I am doing nowadays. While working with
log4j-appender, I found that despite a lack of documentation, considerable
users are already using it[^1][^2][^3][^4][^5]. So, I think providing a
documentation to those who are already using log4j-appender is
indispensable. It should include:

- What is the difference between log4j-appender vs. log4j2-appender.
- Which options are supported and deprecated.
- Exemplar configurations that show how to migrate.

Here is the summary:

1. The goal of this proposal is to replace the traditional log4j-appender
for compatibility concerns. But log4j-appender may be published after the
deprecation.
2. As of present, the description about logging-log4j2 Kafka appender is
entirely wrong. The problem is interface compatibility, not record format.
Focusing on the compatibility facade is a good approach.
3. A documentation focus on migration should be provided.

If you have any questions or suggestions, don't hesitate to tell me. Thanks
again for your comments!

Best,
Dongjin

[^1]:
https://docs.cloudera.com/csa/1.2.0/monitoring/topics/csa-kafka-logging.html
[^2]:
https://stackoverflow.com/questions/22034895/how-to-use-kafka-0-8-log4j-appender
[^3]:
https://stackoverflow.com/questions/32402405/delay-in-kafka-log4j-appender
[^4]:
https://stackoverflow.com/questions/32301129/kafka-log4j-appender-not-sending-messages
[^5]:
https://stackoverflow.com/questions/35628706/kafka-log4j-appender-0-9-does-not-work

On Mon, Nov 8, 2021 at 9:04 PM Mickael Maison 
wrote:

> Hi Dongjin,
>
> Thanks for working on the update to log4j2, it's definitively
> something we should complete.
> I have a couple of comments:
>
> 1) Is the KIP proposing to replace the existing log4-appender or
> simply add a new one for log4j2? Reading the KIP and with its current
> title, it's not entirely explicit. For example I don't see a statement
> under the proposed changes section. The PR seems to only add a new
> appender but the KIP mentions we want to fully remove dependencies to
> log4j.
>
> 2) Under Rejected Alternative, the KIP states: "the Kafka appender
> provided by log4j2 community stores log message in the Record key".
> Looking at the code, it looks like the log message is stored in the
> Record value:
> 

Re: [DISCUSS] KIP-719: Add Log4J2 Appender

[Mickael Maison]

Hi Dongjin,

Thanks for working on the update to log4j2, it's definitively
something we should complete.
I have a couple of comments:

1) Is the KIP proposing to replace the existing log4-appender or
simply add a new one for log4j2? Reading the KIP and with its current
title, it's not entirely explicit. For example I don't see a statement
under the proposed changes section. The PR seems to only add a new
appender but the KIP mentions we want to fully remove dependencies to
log4j.

2) Under Rejected Alternative, the KIP states: "the Kafka appender
provided by log4j2 community stores log message in the Record key".
Looking at the code, it looks like the log message is stored in the
Record value: 
https://github.com/apache/logging-log4j2/blob/master/log4j-kafka/src/main/java/org/apache/logging/log4j/kafka/appender/KafkaManager.java#L135
Am I missing something?
Comparing it with the proposed new appender, apart from their
configuration format (hence the backwards compatibility issues), they
both work pretty much the same way, so it's not clear it would add a
ton a value.

At a glance, _I've not extensively looked at it_, it does not look
very hard to migrate to the appender from the logging team. I was
wondering if we should mention it in our documentation but I was not
able to find any references to the log4j-appender in the Kafka docs:
https://github.com/apache/kafka-site/search?q=KafkaLog4jAppender

As the current log4j-appender is not even deprecated yet, in theory we
can't remove it till Kafka 4. If we want to speed up the process, I
wonder if the lack of documentation and a migration guide could help
us. What do you think?

Thanks,
Mickael




On Fri, Jun 11, 2021 at 4:57 PM Boojapho O  wrote:
>
> Continuing to use log4j would leave several known security vulnerabilities in 
> Apache Kafka, including https://nvd.nist.gov/vuln/detail/CVE-2019-17571.  The 
> Apache log4j team will not fix this vulnerability and is urging an upgrade to 
> log4j2.  See https://logging.apache.org/log4j/1.2/ for further information.
>
> This is desperately needed in Apache 3.0 to keep the software secure.
>
> On 2021/05/26 12:31:20, Dongjin Lee  wrote:
> > CC'd the +1ers of KIP-653 with detailed context:
> >
> > When I submitted and got the approval of KIP-653: Upgrade log4j to log4j2
> > ,
> > I thought the log4j2-appender should not be the scope of the work. But it
> > was wrong.
> >
> > Since the VerifiableLog4jAppender tool is built upon log4j-appender, log4j
> > 1.x artifact will co-exist with log4j2 artifact in the classpath within
> > this scheme. Since the log4j 1.x code is not called anymore, I thought it
> > is not problematic but actually, it was not - when I started to provide a
> > preview of KIP-653
> > , some
> > users reported that sometimes slf4j fails to find the appropriate binding
> > within the classpath, resulting fail to append the log message.
> >
> > To resolve this problem, I subtly adjusted the scope of the work; I
> > excluded Tools and Trogdor from KIP-653 and extended KIP-719 to take care
> > of them instead, along with providing log4j2-appender. It is why the
> > current WIP implementations include some classpath logic in the shell
> > script and *why KIP-653 only can't complete the log4j2 migration*.
> >
> > I hope you will check this proposal out.
> >
> > Best,
> > Dongjin
> >
> > On Tue, May 25, 2021 at 10:43 PM Dongjin Lee  wrote:
> >
> > > Bumping up the discussion thread.
> > >
> > > Recently, I updated the document of KIP-653: Upgrade log4j to log4j2
> > > 
> > >  (accepted)
> > > and KIP-719: Add Log4J2 Appender
> > > 
> > >  (under
> > > discussion) reflecting the recent changes to our codebase. Especially:
> > >
> > > 1. KIP-653 document
> > > 
> > >  now
> > > explains which modules will be migrated and why.
> > > 2. KIP-719 document
> > > 
> > >  now
> > > explains not only the log4j2-appender plan but also upgrading the omitted
> > > modules in KIP-653 into log4j2.
> > >
> > > As you can see here, those two KIPs are the different parts of the same
> > > problem. I believe the community will have a good grasp on why both KIPs
> > > are best if released altogether.
> > >
> > > I will open the voting thread now, and please leave a vote if you are
> > > interested in this issue.
> > >
> > > Best,
> > > Dongjin
> > >
> > > On Tue, Mar 2, 2021 at 5:00 PM Dongjin Lee  wrote:
> > >
> > >> Hi Kafka dev,
> > >>
> > >> I would like to start the discussion of KIP-719: Add Log4J2 Appender.
> > >>
> > >>
> > >> 

Re: [DISCUSS] KIP-719: Add Log4J2 Appender

[Boojapho O]

Continuing to use log4j would leave several known security vulnerabilities in 
Apache Kafka, including https://nvd.nist.gov/vuln/detail/CVE-2019-17571.  The 
Apache log4j team will not fix this vulnerability and is urging an upgrade to 
log4j2.  See https://logging.apache.org/log4j/1.2/ for further information.

This is desperately needed in Apache 3.0 to keep the software secure.

On 2021/05/26 12:31:20, Dongjin Lee  wrote: 
> CC'd the +1ers of KIP-653 with detailed context:
> 
> When I submitted and got the approval of KIP-653: Upgrade log4j to log4j2
> ,
> I thought the log4j2-appender should not be the scope of the work. But it
> was wrong.
> 
> Since the VerifiableLog4jAppender tool is built upon log4j-appender, log4j
> 1.x artifact will co-exist with log4j2 artifact in the classpath within
> this scheme. Since the log4j 1.x code is not called anymore, I thought it
> is not problematic but actually, it was not - when I started to provide a
> preview of KIP-653
> , some
> users reported that sometimes slf4j fails to find the appropriate binding
> within the classpath, resulting fail to append the log message.
> 
> To resolve this problem, I subtly adjusted the scope of the work; I
> excluded Tools and Trogdor from KIP-653 and extended KIP-719 to take care
> of them instead, along with providing log4j2-appender. It is why the
> current WIP implementations include some classpath logic in the shell
> script and *why KIP-653 only can't complete the log4j2 migration*.
> 
> I hope you will check this proposal out.
> 
> Best,
> Dongjin
> 
> On Tue, May 25, 2021 at 10:43 PM Dongjin Lee  wrote:
> 
> > Bumping up the discussion thread.
> >
> > Recently, I updated the document of KIP-653: Upgrade log4j to log4j2
> > 
> >  (accepted)
> > and KIP-719: Add Log4J2 Appender
> > 
> >  (under
> > discussion) reflecting the recent changes to our codebase. Especially:
> >
> > 1. KIP-653 document
> > 
> >  now
> > explains which modules will be migrated and why.
> > 2. KIP-719 document
> > 
> >  now
> > explains not only the log4j2-appender plan but also upgrading the omitted
> > modules in KIP-653 into log4j2.
> >
> > As you can see here, those two KIPs are the different parts of the same
> > problem. I believe the community will have a good grasp on why both KIPs
> > are best if released altogether.
> >
> > I will open the voting thread now, and please leave a vote if you are
> > interested in this issue.
> >
> > Best,
> > Dongjin
> >
> > On Tue, Mar 2, 2021 at 5:00 PM Dongjin Lee  wrote:
> >
> >> Hi Kafka dev,
> >>
> >> I would like to start the discussion of KIP-719: Add Log4J2 Appender.
> >>
> >>
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-719%3A+Add+Log4J2+Appender
> >>
> >> All kinds of feedbacks are greatly appreciated!
> >>
> >> Best,
> >> Dongjin
> >>
> >> --
> >> *Dongjin Lee*
> >>
> >> *A hitchhiker in the mathematical world.*
> >>
> >>
> >>
> >> *github:  github.com/dongjinleekr
> >> keybase: https://keybase.io/dongjinleekr
> >> linkedin: kr.linkedin.com/in/dongjinleekr
> >> speakerdeck: 
> >> speakerdeck.com/dongjin
> >> *
> >>
> >
> >
> > --
> > *Dongjin Lee*
> >
> > *A hitchhiker in the mathematical world.*
> >
> >
> >
> > *github:  github.com/dongjinleekr
> > keybase: https://keybase.io/dongjinleekr
> > linkedin: kr.linkedin.com/in/dongjinleekr
> > speakerdeck: 
> > speakerdeck.com/dongjin
> > *
> >
> 
> 
> -- 
> *Dongjin Lee*
> 
> *A hitchhiker in the mathematical world.*
> 
> 
> 
> *github:  github.com/dongjinleekr
> keybase: https://keybase.io/dongjinleekr
> linkedin: kr.linkedin.com/in/dongjinleekr
> speakerdeck: speakerdeck.com/dongjin
> *
> 

Re: [DISCUSS] KIP-719: Add Log4J2 Appender

[Dongjin Lee]

CC'd the +1ers of KIP-653 with detailed context:

When I submitted and got the approval of KIP-653: Upgrade log4j to log4j2
,
I thought the log4j2-appender should not be the scope of the work. But it
was wrong.

Since the VerifiableLog4jAppender tool is built upon log4j-appender, log4j
1.x artifact will co-exist with log4j2 artifact in the classpath within
this scheme. Since the log4j 1.x code is not called anymore, I thought it
is not problematic but actually, it was not - when I started to provide a
preview of KIP-653
, some
users reported that sometimes slf4j fails to find the appropriate binding
within the classpath, resulting fail to append the log message.

To resolve this problem, I subtly adjusted the scope of the work; I
excluded Tools and Trogdor from KIP-653 and extended KIP-719 to take care
of them instead, along with providing log4j2-appender. It is why the
current WIP implementations include some classpath logic in the shell
script and *why KIP-653 only can't complete the log4j2 migration*.

I hope you will check this proposal out.

Best,
Dongjin

On Tue, May 25, 2021 at 10:43 PM Dongjin Lee  wrote:

> Bumping up the discussion thread.
>
> Recently, I updated the document of KIP-653: Upgrade log4j to log4j2
> 
>  (accepted)
> and KIP-719: Add Log4J2 Appender
> 
>  (under
> discussion) reflecting the recent changes to our codebase. Especially:
>
> 1. KIP-653 document
> 
>  now
> explains which modules will be migrated and why.
> 2. KIP-719 document
> 
>  now
> explains not only the log4j2-appender plan but also upgrading the omitted
> modules in KIP-653 into log4j2.
>
> As you can see here, those two KIPs are the different parts of the same
> problem. I believe the community will have a good grasp on why both KIPs
> are best if released altogether.
>
> I will open the voting thread now, and please leave a vote if you are
> interested in this issue.
>
> Best,
> Dongjin
>
> On Tue, Mar 2, 2021 at 5:00 PM Dongjin Lee  wrote:
>
>> Hi Kafka dev,
>>
>> I would like to start the discussion of KIP-719: Add Log4J2 Appender.
>>
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-719%3A+Add+Log4J2+Appender
>>
>> All kinds of feedbacks are greatly appreciated!
>>
>> Best,
>> Dongjin
>>
>> --
>> *Dongjin Lee*
>>
>> *A hitchhiker in the mathematical world.*
>>
>>
>>
>> *github:  github.com/dongjinleekr
>> keybase: https://keybase.io/dongjinleekr
>> linkedin: kr.linkedin.com/in/dongjinleekr
>> speakerdeck: speakerdeck.com/dongjin
>> *
>>
>
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
>
>
> *github:  github.com/dongjinleekr
> keybase: https://keybase.io/dongjinleekr
> linkedin: kr.linkedin.com/in/dongjinleekr
> speakerdeck: speakerdeck.com/dongjin
> *
>


-- 
*Dongjin Lee*

*A hitchhiker in the mathematical world.*



*github:  github.com/dongjinleekr
keybase: https://keybase.io/dongjinleekr
linkedin: kr.linkedin.com/in/dongjinleekr
speakerdeck: speakerdeck.com/dongjin
*

Re: [DISCUSS] KIP-719: Add Log4J2 Appender

[Dongjin Lee]

Bumping up the discussion thread.

Recently, I updated the document of KIP-653: Upgrade log4j to log4j2

(accepted)
and KIP-719: Add Log4J2 Appender

(under
discussion) reflecting the recent changes to our codebase. Especially:

1. KIP-653 document

now
explains which modules will be migrated and why.
2. KIP-719 document

now
explains not only the log4j2-appender plan but also upgrading the omitted
modules in KIP-653 into log4j2.

As you can see here, those two KIPs are the different parts of the same
problem. I believe the community will have a good grasp on why both KIPs
are best if released altogether.

I will open the voting thread now, and please leave a vote if you are
interested in this issue.

Best,
Dongjin

On Tue, Mar 2, 2021 at 5:00 PM Dongjin Lee  wrote:

> Hi Kafka dev,
>
> I would like to start the discussion of KIP-719: Add Log4J2 Appender.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-719%3A+Add+Log4J2+Appender
>
> All kinds of feedbacks are greatly appreciated!
>
> Best,
> Dongjin
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
>
>
> *github:  github.com/dongjinleekr
> keybase: https://keybase.io/dongjinleekr
> linkedin: kr.linkedin.com/in/dongjinleekr
> speakerdeck: speakerdeck.com/dongjin
> *
>


-- 
*Dongjin Lee*

*A hitchhiker in the mathematical world.*



*github:  github.com/dongjinleekr
keybase: https://keybase.io/dongjinleekr
linkedin: kr.linkedin.com/in/dongjinleekr
speakerdeck: speakerdeck.com/dongjin
*

[DISCUSS] KIP-719: Add Log4J2 Appender

[Dongjin Lee]

Hi Kafka dev,

I would like to start the discussion of KIP-719: Add Log4J2 Appender.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-719%3A+Add+Log4J2+Appender

All kinds of feedbacks are greatly appreciated!

Best,
Dongjin

-- 
*Dongjin Lee*

*A hitchhiker in the mathematical world.*



*github:  github.com/dongjinleekr
keybase: https://keybase.io/dongjinleekr
linkedin: kr.linkedin.com/in/dongjinleekr
speakerdeck: speakerdeck.com/dongjin
*