Hi, So to summarize, it seems the plan is to: - adopt reload4j in Kafka 3.2.0 - delay the switch to log4j2 to Kafka 4.0.0
Reload4j is supposed to be a fully compatible drop-in replacement for log4j. Do we still want to do a vote for the switch? It looks like there's already a JIRA and a PR ready: https://issues.apache.org/jira/browse/KAFKA-13660 https://github.com/apache/kafka/pull/11743 Do we also want to switch for 3.1.1? Adding Bruno and Tom on CC as they are the release managers for 3.2.0 and 3.1.1. Thanks, Mickael On Thu, Mar 24, 2022 at 7:06 AM Dongjin Lee <dong...@apache.org> wrote: > > No. It is why I have no firm position. I will follow the community's > decision. > > Thanks, > Dongjin > > On Thu, Mar 24, 2022 at 12:34 AM Ismael Juma <ism...@juma.me.uk> wrote: > > > Hi Dongjin, > > > > We really appreciate the super valuable work you've been doing here. Do we > > have evidence that customers don't use custom filters/layouts? > > > > Ismael > > > > On Wed, Mar 23, 2022 at 7:53 AM Dongjin Lee <dong...@apache.org> wrote: > > > > > Hi Mikael, Edoardo and Ismael, > > > > > > Sorry for being late. Frankly, I thought KIP-653 is not a breaking change > > > since (as Edoardo stated) unless the user uses custom filters or layouts, > > > log4j-1.2-api.jar 'bridge' jar can handle the cases. It is why the > > > 'Compatibility, Deprecation, and Migration Plan' section of the document > > is > > > so brief. (As far as I know, it is such a rare case, and I thought it > > would > > > not be so problematic.) > > > > > > I have no firm position on the release plan of this feature. Regardless > > of > > > whether the community decides to put off the adoption of log4j2 to 4.0, I > > > will maintain the PR and the preview releases up-to-date as far as > > possible > > > - although it can be significantly late for my main job. The decision is > > > totally up to the community - No matter how it is decided, I will follow. > > > > > > Best, > > > Dongjin > > > > > > p.s. @Edoardo I'm HIM. (HAHA) > > > > > > On Wed, Mar 23, 2022 at 10:12 PM Ismael Juma <ism...@juma.me.uk> wrote: > > > > > > > Hi Mickael, > > > > > > > > Thanks for your feedback. I agree with the importance of fixing the > > CVEs > > > > and also of not breaking compatibility in a critical layer. Regarding > > > > Apache Kafka 4.0, you suggested it would include: > > > > > > > > - log4j2 migration > > > > - idempotency enablement cleanups > > > > - removal of Java 8 and Scala 2.12 support > > > > - removal of MirrorMaker1 > > > > > > > > It's too soon to remove Java 8/Scala 2.12 support, so I don't think > > that > > > > would work. The other things hardly justify a major release so soon. > > Have > > > > we considered adjusting the existing log4j 2 PR so that both libraries > > > > versions are supported for a period of time? Since reload4j doesn't > > have > > > > the CVEs, this would be acceptable and would avoid a premature 4.0 > > > release. > > > > I expect 4.0 to be the release after 3.4 or 3.5 given where we are > > right > > > > now. > > > > > > > > Ismael > > > > > > > > On Wed, Mar 23, 2022 at 4:43 AM Mickael Maison < > > mickael.mai...@gmail.com > > > > > > > > wrote: > > > > > > > > > Hi Ismael, > > > > > > > > > > About 2) > > > > > We can't keep shipping new releases with dependencies that have CVEs. > > > > > This is negatively impacting the project and eroding the hard earned > > > > > trust we have from our users. Kafka is known to be a robust, reliable > > > > > and up to date project. > > > > > > > > > > With that in mind, and since clearly at this point we're not going to > > > > > update to log4j2 in 3.2.0, I too would be in favor of tactically > > > > > adopting reload4j in 3.2.0. This would allow 3.2.0 to release without > > > > > any known CVEs and surely make the life of many users better! > > > > > > > > > > Now regarding log4j2. I still consider there's value in adopting > > > > > log4j2 (Apache project, plugin ecosystem, reconfiguration support) > > and > > > > > I'd like to see it happen as soon as possible. If unfortunately there > > > > > are compatibility issues, I agree that we can't force breakage in a > > > > > minor release. We've always put a lot of attention into preserving > > > > > compatibility, we should not suddenly stop doing it. So it makes > > sense > > > > > to wait for the next major release. > > > > > > > > > > Currently in many minds, 4.0 is kind of associated with the removal > > of > > > > > ZooKeeper. At this stage, it's still unclear when this will be ready > > > > > and even if I'm optimistic it's still at the very least 6 to 9 months > > > > > away. The code changes to migrate to log4j2 are not trivial and > > > > > there's certainly a high cost in maintaining then outside of trunk > > for > > > > > many months. Dongjin has done a stellar work so far in regularly > > > > > updating his PRs since this KIP was started back in 2020, but we > > can't > > > > > ask him to just keep doing it for another unknown amount of time. > > > > > > > > > > What about if the next release is 4.0? Even if it's light on > > features, > > > > > it would enable us to do quite a few cleanups and migrate to log4j2. > > > > > Then the removal of ZooKeeper can happen in a future major release > > > > > when it's ready. > > > > > > > > > > 4.0 would include: > > > > > - log4j2 migration > > > > > - idempotency enablement cleanups > > > > > - removal of Java 8 and Scala 2.12 support > > > > > - removal of MirrorMaker1 > > > > > > > > > > So I propose to adopt reload4j in Kafka 3.2 and make the next release > > > > > 4.0. Let me know what you think. > > > > > > > > > > Thanks, > > > > > Mickael > > > > > > > > > > > > > > > > > > > > On Mon, Mar 21, 2022 at 4:33 PM Ismael Juma <ism...@juma.me.uk> > > wrote: > > > > > > > > > > > > Hi Edoardo, > > > > > > > > > > > > Thanks for the information. That's definitely useful. A couple of > > > > > questions > > > > > > for you and the rest of the group: > > > > > > > > > > > > 1. Did you test the branch using log4j 1.x configs? > > > > > > 2. Given the release of https://github.com/qos-ch/reload4j, does > > it > > > > > really > > > > > > make sense to force breakage on users in a minor release? Would it > > > not > > > > be > > > > > > better to use reload4j in Kafka 3.2 and log4j 2 in Kafka 4.0? > > > > > > > > > > > > Thanks, > > > > > > Ismael > > > > > > > > > > > > On Mon, Mar 21, 2022 at 8:16 AM Edoardo Comar <eco...@uk.ibm.com> > > > > wrote: > > > > > > > > > > > > > Hi Ismael and Luke, > > > > > > > we've tested Dongjin code - porting her preview releases and PR > > to > > > > > > > different Kafka code levels (2.8.1+, 3.1.0+, trunk). > > > > > > > We're happy with it and would love it if her PR was merged in > > > 3.2.0. > > > > > > > > > > > > > > To chime in on the issue of compatibility, as we have experienced > > > it, > > > > > the > > > > > > > main limitation of the log4j-1.2-api.jar 'bridge' jar is in the > > > > > support for > > > > > > > custom Appenders, Filters and Layouts. > > > > > > > If you're using such components, they may need to be rewritten to > > > the > > > > > > > Log4j2 spec and correspondingly use the configuration file in > > > log4j2 > > > > > format > > > > > > > (and referenced with the log4j2 system property). > > > > > > > Details at > > > > > > > > > > > > > > > > > > > > > https://logging.apache.org/log4j/2.x/manual/migration.html#ConfigurationCompatibility > > > > > > > and > > > > > > > > > > > > > > > > > > > > > https://logging.apache.org/log4j/2.x/manual/migration.html#Log4j1.2BridgeLimitations > > > > > > > > > > > > > > I think that the above information should find its way in the > > KIP's > > > > > > > compatibility section. > > > > > > > > > > > > > > HTH > > > > > > > Edo > > > > > > > -------------------------------------------------- > > > > > > > Edoardo Comar > > > > > > > Event Streams for IBM Cloud > > > > > > > > > > > > > > > > > > > > > ________________________________ > > > > > > > From: Luke Chen <show...@gmail.com> > > > > > > > Sent: 18 March 2022 07:57 > > > > > > > To: dev <dev@kafka.apache.org> > > > > > > > Subject: [EXTERNAL] Re: [VOTE] KIP-653: Upgrade log4j to log4j2 > > > > > > > > > > > > > > Hi Dongjin, > > > > > > > > > > > > > > I know there are some discussions about the compatibility issue. > > > > > > > Could you help answer this question? > > > > > > > > > > > > > > Thank you. > > > > > > > Luke > > > > > > > > > > > > > > On Fri, Mar 18, 2022 at 3:32 AM Ismael Juma <ism...@juma.me.uk> > > > > wrote: > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > The KIP compatibility section does not include enough detail. I > > > am > > > > > > > puzzled > > > > > > > > how we voted +1 given that. I noticed that Colin indicated it > > > would > > > > > only > > > > > > > be > > > > > > > > acceptable in a major release unless the new version was fully > > > > > compatible > > > > > > > > (which it is not). Can we clarify what we actually voted for > > > here? > > > > > > > > > > > > > > > > Ismael > > > > > > > > > > > > > > > > On Wed, Oct 21, 2020 at 6:41 PM Dongjin Lee < > > dong...@apache.org> > > > > > wrote: > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > As of present: > > > > > > > > > > > > > > > > > > - Binding: +3 (Gwen, John, Colin) > > > > > > > > > - Non-binding: +1 (David, Tom) > > > > > > > > > > > > > > > > > > This KIP is now accepted. Thanks for your votes! > > > > > > > > > > > > > > > > > > @Colin Sure, I have some plan for providing a compatibility > > > > > preview. > > > > > > > > Let's > > > > > > > > > continue in the discussion thread. > > > > > > > > > > > > > > > > > > All other voters not in KIP-676 Vote thread: KIP-676 (by Tom) > > > is > > > > a > > > > > > > > > prerequisite of this KIP. Please have a look at that proposal > > > and > > > > > vote > > > > > > > > for > > > > > > > > > it. > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > Dongjin > > > > > > > > > > > > > > > > > > On Wed, Oct 21, 2020 at 9:17 PM Colin McCabe < > > > cmcc...@apache.org > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > +1 (binding). I think we should consider doing this in 3.0 > > > > > rather > > > > > > > than > > > > > > > > > > 2.8, though, unless we are really confident that it is 100% > > > > > > > compatible. > > > > > > > > > > > > > > > > > > > > I wasn't able to find much information on how compatible > > the > > > > new > > > > > API > > > > > > > > > > bridge is, but the log4j website does have this: > > > > > > > > > > > > > > > > > > > > > Basic compatibility with Log4j 1.x is provided through > > the > > > > > > > > log4j12-api > > > > > > > > > > component, > > > > > > > > > > > however it does not implement some of the very > > > implementation > > > > > > > > specific > > > > > > > > > > > classes and methods > > > > > > > > > > > > > > > > > > > > best, > > > > > > > > > > Colin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Oct 9, 2020, at 02:51, Tom Bentley wrote: > > > > > > > > > > > +1 non-binding. > > > > > > > > > > > > > > > > > > > > > > Thanks for your efforts on this Dongjin. > > > > > > > > > > > > > > > > > > > > > > Tom > > > > > > > > > > > > > > > > > > > > > > On Wed, Oct 7, 2020 at 6:45 AM Dongjin Lee < > > > > dong...@apache.org > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > As of present: > > > > > > > > > > > > > > > > > > > > > > > > - Binding: +2 (Gwen, John) > > > > > > > > > > > > - Non-binding: +1 (David) > > > > > > > > > > > > > > > > > > > > > > > > Now we need one more binding +1. > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Dongjin > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Oct 7, 2020 at 1:37 AM David Jacot < > > > > > > > david.ja...@gmail.com> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for driving this, Dongjin! > > > > > > > > > > > > > > > > > > > > > > > > > > The KIP looks good to me. I’m +1 (non-binding). > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > David > > > > > > > > > > > > > > > > > > > > > > > > > > Le mar. 6 oct. 2020 à 17:23, Dongjin Lee < > > > > > dong...@apache.org> > > > > > > > a > > > > > > > > > > écrit : > > > > > > > > > > > > > > > > > > > > > > > > > > > As of present: > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Binding: +2 (Gwen, John) > > > > > > > > > > > > > > - Non-binding: 0 > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Dongjin > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Sat, Oct 3, 2020 at 10:51 AM John Roesler < > > > > > > > > > vvcep...@apache.org> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks for the KIP, Dongjin! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I’ve just reviewed the KIP document, and it looks > > > > good > > > > > to > > > > > > > me. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I’m +1 (binding) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > John > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Oct 2, 2020, at 19:11, Gwen Shapira > > wrote: > > > > > > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > A very welcome update :) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 22, 2020 at 9:09 AM Dongjin Lee < > > > > > > > > > > dong...@apache.org> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi devs, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Here I open the vote for KIP-653: Upgrade > > log4j > > > > to > > > > > > > > log4j2. > > > > > > > > > It > > > > > > > > > > > > > > replaces > > > > > > > > > > > > > > > the > > > > > > > > > > > > > > > > > obsolete log4j logging library into the > > current > > > > > > > standard, > > > > > > > > > > log4j2, > > > > > > > > > > > > > > with > > > > > > > > > > > > > > > > > maintaining backward-compatibility. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > Dongjin > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > *Dongjin Lee* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *github: <http://goog_969573159/ > > > > > > > > > github.com/dongjinleekr > > > > > > > > > > > > > > > > > <https://github.com/dongjinleekr >keybase: > > > > > > > > > > > > > > > https://keybase.io/dongjinleekr > > > > > > > > > > > > > > > > > <https://keybase.io/dongjinleekr >linkedin: > > > > > > > > > > > > > > > kr.linkedin.com/in/dongjinleekr > > > > > > > > > > > > > > > > > <https://kr.linkedin.com/in/dongjinleekr > > > > > >speakerdeck: > > > > > > > > > > > > > > > speakerdeck.com/dongjin > > > > > > > > > > > > > > > > > <https://speakerdeck.com/dongjin >* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > Gwen Shapira > > > > > > > > > > > > > > > > Engineering Manager | Confluent > > > > > > > > > > > > > > > > 650.450.2760 | @gwenshap > > > > > > > > > > > > > > > > Follow us: Twitter | blog > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > *Dongjin Lee* > > > > > > > > > > > > > > > > > > > > > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *github: <http://goog_969573159/ > > > > > > github.com/dongjinleekr > > > > > > > > > > > > > > <https://github.com/dongjinleekr >keybase: > > > > > > > > > > > > > https://keybase.io/dongjinleekr > > > > > > > > > > > > > > <https://keybase.io/dongjinleekr >linkedin: > > > > > > > > > > > > > kr.linkedin.com/in/dongjinleekr > > > > > > > > > > > > > > <https://kr.linkedin.com/in/dongjinleekr > > > >speakerdeck: > > > > > > > > > > > > > > speakerdeck.com/dongjin > > > > > > > > > > > > > > <https://speakerdeck.com/dongjin >* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > *Dongjin Lee* > > > > > > > > > > > > > > > > > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *github: <http://goog_969573159/ > > > > github.com/dongjinleekr > > > > > > > > > > > > <https://github.com/dongjinleekr >keybase: > > > > > > > > > > https://keybase.io/dongjinleekr > > > > > > > > > > > > <https://keybase.io/dongjinleekr >linkedin: > > > > > > > > > > kr.linkedin.com/in/dongjinleekr > > > > > > > > > > > > <https://kr.linkedin.com/in/dongjinleekr >speakerdeck: > > > > > > > > > > > > speakerdeck.com/dongjin > > > > > > > > > > > > <https://speakerdeck.com/dongjin >* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > *Dongjin Lee* > > > > > > > > > > > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *github: <http://goog_969573159/ >github.com/dongjinleekr > > > > > > > > > <https://github.com/dongjinleekr >keybase: > > > > > > > > https://keybase.io/dongjinleekr > > > > > > > > > <https://keybase.io/dongjinleekr >linkedin: > > > > > > > > kr.linkedin.com/in/dongjinleekr > > > > > > > > > <https://kr.linkedin.com/in/dongjinleekr >speakerdeck: > > > > > > > > > speakerdeck.com/dongjin > > > > > > > > > <https://speakerdeck.com/dongjin >* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > *Dongjin Lee* > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > *github: <http://goog_969573159/>github.com/dongjinleekr > > > <https://github.com/dongjinleekr>keybase: > > https://keybase.io/dongjinleekr > > > <https://keybase.io/dongjinleekr>linkedin: > > kr.linkedin.com/in/dongjinleekr > > > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: > > > speakerdeck.com/dongjin > > > <https://speakerdeck.com/dongjin>* > > > > > > > > -- > *Dongjin Lee* > > *A hitchhiker in the mathematical world.* > > > > *github: <http://goog_969573159/>github.com/dongjinleekr > <https://github.com/dongjinleekr>keybase: https://keybase.io/dongjinleekr > <https://keybase.io/dongjinleekr>linkedin: kr.linkedin.com/in/dongjinleekr > <https://kr.linkedin.com/in/dongjinleekr>speakerdeck: speakerdeck.com/dongjin > <https://speakerdeck.com/dongjin>*
