Hi Mickael, Thanks for your feedback. I agree with the importance of fixing the CVEs and also of not breaking compatibility in a critical layer. Regarding Apache Kafka 4.0, you suggested it would include:
- log4j2 migration - idempotency enablement cleanups - removal of Java 8 and Scala 2.12 support - removal of MirrorMaker1 It's too soon to remove Java 8/Scala 2.12 support, so I don't think that would work. The other things hardly justify a major release so soon. Have we considered adjusting the existing log4j 2 PR so that both libraries versions are supported for a period of time? Since reload4j doesn't have the CVEs, this would be acceptable and would avoid a premature 4.0 release. I expect 4.0 to be the release after 3.4 or 3.5 given where we are right now. Ismael On Wed, Mar 23, 2022 at 4:43 AM Mickael Maison <mickael.mai...@gmail.com> wrote: > Hi Ismael, > > About 2) > We can't keep shipping new releases with dependencies that have CVEs. > This is negatively impacting the project and eroding the hard earned > trust we have from our users. Kafka is known to be a robust, reliable > and up to date project. > > With that in mind, and since clearly at this point we're not going to > update to log4j2 in 3.2.0, I too would be in favor of tactically > adopting reload4j in 3.2.0. This would allow 3.2.0 to release without > any known CVEs and surely make the life of many users better! > > Now regarding log4j2. I still consider there's value in adopting > log4j2 (Apache project, plugin ecosystem, reconfiguration support) and > I'd like to see it happen as soon as possible. If unfortunately there > are compatibility issues, I agree that we can't force breakage in a > minor release. We've always put a lot of attention into preserving > compatibility, we should not suddenly stop doing it. So it makes sense > to wait for the next major release. > > Currently in many minds, 4.0 is kind of associated with the removal of > ZooKeeper. At this stage, it's still unclear when this will be ready > and even if I'm optimistic it's still at the very least 6 to 9 months > away. The code changes to migrate to log4j2 are not trivial and > there's certainly a high cost in maintaining then outside of trunk for > many months. Dongjin has done a stellar work so far in regularly > updating his PRs since this KIP was started back in 2020, but we can't > ask him to just keep doing it for another unknown amount of time. > > What about if the next release is 4.0? Even if it's light on features, > it would enable us to do quite a few cleanups and migrate to log4j2. > Then the removal of ZooKeeper can happen in a future major release > when it's ready. > > 4.0 would include: > - log4j2 migration > - idempotency enablement cleanups > - removal of Java 8 and Scala 2.12 support > - removal of MirrorMaker1 > > So I propose to adopt reload4j in Kafka 3.2 and make the next release > 4.0. Let me know what you think. > > Thanks, > Mickael > > > > On Mon, Mar 21, 2022 at 4:33 PM Ismael Juma <ism...@juma.me.uk> wrote: > > > > Hi Edoardo, > > > > Thanks for the information. That's definitely useful. A couple of > questions > > for you and the rest of the group: > > > > 1. Did you test the branch using log4j 1.x configs? > > 2. Given the release of https://github.com/qos-ch/reload4j, does it > really > > make sense to force breakage on users in a minor release? Would it not be > > better to use reload4j in Kafka 3.2 and log4j 2 in Kafka 4.0? > > > > Thanks, > > Ismael > > > > On Mon, Mar 21, 2022 at 8:16 AM Edoardo Comar <eco...@uk.ibm.com> wrote: > > > > > Hi Ismael and Luke, > > > we've tested Dongjin code - porting her preview releases and PR to > > > different Kafka code levels (2.8.1+, 3.1.0+, trunk). > > > We're happy with it and would love it if her PR was merged in 3.2.0. > > > > > > To chime in on the issue of compatibility, as we have experienced it, > the > > > main limitation of the log4j-1.2-api.jar 'bridge' jar is in the > support for > > > custom Appenders, Filters and Layouts. > > > If you're using such components, they may need to be rewritten to the > > > Log4j2 spec and correspondingly use the configuration file in log4j2 > format > > > (and referenced with the log4j2 system property). > > > Details at > > > > https://logging.apache.org/log4j/2.x/manual/migration.html#ConfigurationCompatibility > > > and > > > > https://logging.apache.org/log4j/2.x/manual/migration.html#Log4j1.2BridgeLimitations > > > > > > I think that the above information should find its way in the KIP's > > > compatibility section. > > > > > > HTH > > > Edo > > > -------------------------------------------------- > > > Edoardo Comar > > > Event Streams for IBM Cloud > > > > > > > > > ________________________________ > > > From: Luke Chen <show...@gmail.com> > > > Sent: 18 March 2022 07:57 > > > To: dev <dev@kafka.apache.org> > > > Subject: [EXTERNAL] Re: [VOTE] KIP-653: Upgrade log4j to log4j2 > > > > > > Hi Dongjin, > > > > > > I know there are some discussions about the compatibility issue. > > > Could you help answer this question? > > > > > > Thank you. > > > Luke > > > > > > On Fri, Mar 18, 2022 at 3:32 AM Ismael Juma <ism...@juma.me.uk> wrote: > > > > > > > Hi all, > > > > > > > > The KIP compatibility section does not include enough detail. I am > > > puzzled > > > > how we voted +1 given that. I noticed that Colin indicated it would > only > > > be > > > > acceptable in a major release unless the new version was fully > compatible > > > > (which it is not). Can we clarify what we actually voted for here? > > > > > > > > Ismael > > > > > > > > On Wed, Oct 21, 2020 at 6:41 PM Dongjin Lee <dong...@apache.org> > wrote: > > > > > > > > > Hi All, > > > > > > > > > > As of present: > > > > > > > > > > - Binding: +3 (Gwen, John, Colin) > > > > > - Non-binding: +1 (David, Tom) > > > > > > > > > > This KIP is now accepted. Thanks for your votes! > > > > > > > > > > @Colin Sure, I have some plan for providing a compatibility > preview. > > > > Let's > > > > > continue in the discussion thread. > > > > > > > > > > All other voters not in KIP-676 Vote thread: KIP-676 (by Tom) is a > > > > > prerequisite of this KIP. Please have a look at that proposal and > vote > > > > for > > > > > it. > > > > > > > > > > Best, > > > > > Dongjin > > > > > > > > > > On Wed, Oct 21, 2020 at 9:17 PM Colin McCabe <cmcc...@apache.org> > > > wrote: > > > > > > > > > > > +1 (binding). I think we should consider doing this in 3.0 > rather > > > than > > > > > > 2.8, though, unless we are really confident that it is 100% > > > compatible. > > > > > > > > > > > > I wasn't able to find much information on how compatible the new > API > > > > > > bridge is, but the log4j website does have this: > > > > > > > > > > > > > Basic compatibility with Log4j 1.x is provided through the > > > > log4j12-api > > > > > > component, > > > > > > > however it does not implement some of the very implementation > > > > specific > > > > > > > classes and methods > > > > > > > > > > > > best, > > > > > > Colin > > > > > > > > > > > > > > > > > > On Fri, Oct 9, 2020, at 02:51, Tom Bentley wrote: > > > > > > > +1 non-binding. > > > > > > > > > > > > > > Thanks for your efforts on this Dongjin. > > > > > > > > > > > > > > Tom > > > > > > > > > > > > > > On Wed, Oct 7, 2020 at 6:45 AM Dongjin Lee <dong...@apache.org > > > > > > wrote: > > > > > > > > > > > > > > > As of present: > > > > > > > > > > > > > > > > - Binding: +2 (Gwen, John) > > > > > > > > - Non-binding: +1 (David) > > > > > > > > > > > > > > > > Now we need one more binding +1. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Dongjin > > > > > > > > > > > > > > > > On Wed, Oct 7, 2020 at 1:37 AM David Jacot < > > > david.ja...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > > > > > Thanks for driving this, Dongjin! > > > > > > > > > > > > > > > > > > The KIP looks good to me. I’m +1 (non-binding). > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > David > > > > > > > > > > > > > > > > > > Le mar. 6 oct. 2020 à 17:23, Dongjin Lee < > dong...@apache.org> > > > a > > > > > > écrit : > > > > > > > > > > > > > > > > > > > As of present: > > > > > > > > > > > > > > > > > > > > - Binding: +2 (Gwen, John) > > > > > > > > > > - Non-binding: 0 > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Dongjin > > > > > > > > > > > > > > > > > > > > On Sat, Oct 3, 2020 at 10:51 AM John Roesler < > > > > > vvcep...@apache.org> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Thanks for the KIP, Dongjin! > > > > > > > > > > > > > > > > > > > > > > I’ve just reviewed the KIP document, and it looks good > to > > > me. > > > > > > > > > > > > > > > > > > > > > > I’m +1 (binding) > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > John > > > > > > > > > > > > > > > > > > > > > > On Fri, Oct 2, 2020, at 19:11, Gwen Shapira wrote: > > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > > > > > > > > > > > > > A very welcome update :) > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Sep 22, 2020 at 9:09 AM Dongjin Lee < > > > > > > dong...@apache.org> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > Hi devs, > > > > > > > > > > > > > > > > > > > > > > > > > > Here I open the vote for KIP-653: Upgrade log4j to > > > > log4j2. > > > > > It > > > > > > > > > > replaces > > > > > > > > > > > the > > > > > > > > > > > > > obsolete log4j logging library into the current > > > standard, > > > > > > log4j2, > > > > > > > > > > with > > > > > > > > > > > > > maintaining backward-compatibility. > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > Dongjin > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > *Dongjin Lee* > > > > > > > > > > > > > > > > > > > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *github: <http://goog_969573159/ > > > > > github.com/dongjinleekr > > > > > > > > > > > > > <https://github.com/dongjinleekr >keybase: > > > > > > > > > > > https://keybase.io/dongjinleekr > > > > > > > > > > > > > <https://keybase.io/dongjinleekr >linkedin: > > > > > > > > > > > kr.linkedin.com/in/dongjinleekr > > > > > > > > > > > > > <https://kr.linkedin.com/in/dongjinleekr > >speakerdeck: > > > > > > > > > > > speakerdeck.com/dongjin > > > > > > > > > > > > > <https://speakerdeck.com/dongjin >* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Gwen Shapira > > > > > > > > > > > > Engineering Manager | Confluent > > > > > > > > > > > > 650.450.2760 | @gwenshap > > > > > > > > > > > > Follow us: Twitter | blog > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > *Dongjin Lee* > > > > > > > > > > > > > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *github: <http://goog_969573159/ > > github.com/dongjinleekr > > > > > > > > > > <https://github.com/dongjinleekr >keybase: > > > > > > > > > https://keybase.io/dongjinleekr > > > > > > > > > > <https://keybase.io/dongjinleekr >linkedin: > > > > > > > > > kr.linkedin.com/in/dongjinleekr > > > > > > > > > > <https://kr.linkedin.com/in/dongjinleekr >speakerdeck: > > > > > > > > > > speakerdeck.com/dongjin > > > > > > > > > > <https://speakerdeck.com/dongjin >* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > *Dongjin Lee* > > > > > > > > > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *github: <http://goog_969573159/ >github.com/dongjinleekr > > > > > > > > <https://github.com/dongjinleekr >keybase: > > > > > > https://keybase.io/dongjinleekr > > > > > > > > <https://keybase.io/dongjinleekr >linkedin: > > > > > > kr.linkedin.com/in/dongjinleekr > > > > > > > > <https://kr.linkedin.com/in/dongjinleekr >speakerdeck: > > > > > > > > speakerdeck.com/dongjin > > > > > > > > <https://speakerdeck.com/dongjin >* > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > *Dongjin Lee* > > > > > > > > > > *A hitchhiker in the mathematical world.* > > > > > > > > > > > > > > > > > > > > > > > > > *github: <http://goog_969573159/ >github.com/dongjinleekr > > > > > <https://github.com/dongjinleekr >keybase: > > > > https://keybase.io/dongjinleekr > > > > > <https://keybase.io/dongjinleekr >linkedin: > > > > kr.linkedin.com/in/dongjinleekr > > > > > <https://kr.linkedin.com/in/dongjinleekr >speakerdeck: > > > > > speakerdeck.com/dongjin > > > > > <https://speakerdeck.com/dongjin >* > > > > > > > > > > > > >
