[jira] [Resolved] (KAFKA-13545) Workaround for mitigating CVE-2021-4104 Kafka

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-13545.
---
Resolution: Won't Fix

> Workaround for mitigating CVE-2021-4104 Kafka 
> --
>
> Key: KAFKA-13545
> URL: https://issues.apache.org/jira/browse/KAFKA-13545
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 2.8.1
>Reporter: Akansh Shandilya
>Priority: Major
>
> A new vulnerability is published today :
> https://nvd.nist.gov/vuln/detail/CVE-2021-4104
>  
> Kafka v2.8.1 uses log4j v1.x . Please review following information :
> Is Kafka v2.8.1 impacted by  CVE-2021-4104?
> If yes, is there any workaround/recommendation available for Kafka  v2.8.1 to 
> mitigate CVE-2021-4104



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[GitHub] [kafka-site] showuon commented on pull request #389: Added CVE-2021-45046

[GitBox]

showuon commented on pull request #389:
URL: https://github.com/apache/kafka-site/pull/389#issuecomment-994229870


   Thanks for your quick response! Hope there will be no more vulnerabilities! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] junrao merged pull request #389: Added CVE-2021-45046

[GitBox]

junrao merged pull request #389:
URL: https://github.com/apache/kafka-site/pull/389


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] mkcops opened a new pull request #389: Added CVE-2021-45046

[GitBox]

mkcops opened a new pull request #389:
URL: https://github.com/apache/kafka-site/pull/389


   Adding new CVE-2021-45046 to the list 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] junrao merged pull request #388: adding new CVEs to list

[GitBox]

junrao merged pull request #388:
URL: https://github.com/apache/kafka-site/pull/388


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] junrao commented on pull request #388: adding new CVEs to list

[GitBox]

junrao commented on pull request #388:
URL: https://github.com/apache/kafka-site/pull/388#issuecomment-994163284


   @scott-confluent : Thanks for the updated PR. LGTM


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] scott-confluent commented on pull request #388: adding new CVEs to list

[GitBox]

scott-confluent commented on pull request #388:
URL: https://github.com/apache/kafka-site/pull/388#issuecomment-994151250


   @junrao @rhauch Does adding it here make sense?
   https://user-images.githubusercontent.com/66280178/146098304-984b64f1-4a58-4e53-b69a-4fb0d1fa1197.png;>
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] junrao commented on pull request #388: adding new CVEs to list

[GitBox]

junrao commented on pull request #388:
URL: https://github.com/apache/kafka-site/pull/388#issuecomment-994141826


   Thanks, @rhauch. The simpler version is probably enough. @scott-confluent : 
Could you add that?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] rhauch commented on pull request #388: adding new CVEs to list

[GitBox]

rhauch commented on pull request #388:
URL: https://github.com/apache/kafka-site/pull/388#issuecomment-994136975


   Or, if we want something even simpler, maybe we could just add a sentence to 
the `CVE-2021-44228` section that says:
   
   >  Check with the vendor of any connector plugin that includes a Log4J 2.x 
JAR file.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] rhauch edited a comment on pull request #388: adding new CVEs to list

[GitBox]

rhauch edited a comment on pull request #388:
URL: https://github.com/apache/kafka-site/pull/388#issuecomment-994134387


   As background for Connect: 
   * Connect runtime puts all JARs from each connector plugin on a separate 
classloader, and the Connect runtime nor other connector plugins have access to 
a plugin's JARs. This is why a connector plugin that includes a Log4J 2.x JAR 
   * Most connector implementations simply use the logging provided by the 
Connect runtime, which is Log4J 1.x regardless of the JARs included by 
connector plugins.
   * However, if a connector plugins does include the Log4J 2.x JAR files, 
those JAR files will only be used if the connector implementation explicitly 
uses those APIs. There isn't a need to do this, but connectors are custom code 
and can do quite a bit.
   
   We might consider adding something like this under the `CVE-2021-44228` 
section, which I hope conveys the limited scope of the risk:
   
   > The Connect runtime of Apache Kafka allows users to install third party 
connector plugins. These connector plugins will use Connect runtime's Log4J 1.x 
by default, even when Log4J 1.x or 2.x JARs are inadvertently shipped with the 
connector plugin. Check with the vendor of any connector plugin that includes a 
Log4J 2.x JAR file.
   
   Basically, AK is not responsible for third party connectors that users add 
to their Connect installations. But users should consult with the vendor of 
those third party connectors.
   
   As for `CVE-2021-4104`, I think the existing wording applies to Connect just 
as well as every other part of AK, so IMO no changes are necessary to that 
section specifically for Connect.
   
   Feel free to wordsmith as needed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] rhauch edited a comment on pull request #388: adding new CVEs to list

[GitBox]

rhauch edited a comment on pull request #388:
URL: https://github.com/apache/kafka-site/pull/388#issuecomment-994134387


   As background for Connect: 
   * Connect runtime puts all JARs from each connector plugin on a separate 
classloader, and the Connect runtime nor other connector plugins have access to 
a plugin's JARs. This is why a connector plugin that includes a Log4J 2.x JAR 
   * Most connector implementations simply use the logging provided by the 
Connect runtime, which is Log4J 1.x regardless of the JARs included by 
connector plugins.
   * However, if a connector plugins does include the Log4J 2.x JAR files, 
those JAR files will only be used if the connector implementation explicitly 
uses those APIs. There isn't a need to do this, but connectors are custom code 
and can do quite a bit.
   
   We might consider adding something like this under the CVE-2021-44228 
section, which I hope conveys the limited scope of the risk:
   
   > The Connect runtime of Apache Kafka allows users to install third party 
connector plugins. These connector plugins will use Connect runtime's Log4J 1.x 
by default, even when Log4J 1.x or 2.x JARs are inadvertently shipped with the 
connector plugin. Check with the vendor of any connector plugin that includes a 
Log4J 2.x JAR file.
   
   Basically, AK is not responsible for third party connectors that users add 
to their Connect installations. But users should consult with the vendor of 
those third party connectors.
   
   As for CVE-2021-4104, I think the existing wording applies to Connect just 
as well as every other part of AK, so IMO no changes are necessary to that 
section specifically for Connect.
   
   Feel free to wordsmith as needed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] rhauch commented on pull request #388: adding new CVEs to list

[GitBox]

rhauch commented on pull request #388:
URL: https://github.com/apache/kafka-site/pull/388#issuecomment-994134387


   As background for Connect: 
   * Connect runtime puts all JARs from each connector plugin on a separate 
classloader, and the Connect runtime nor other connector plugins have access to 
a plugin's JARs. This is why a connector plugin that includes a Log4J 2.x JAR 
   * Most connector implementations simply use the logging provided by the 
Connect runtime, which is Log4J 1.x regardless of the JARs included by 
connector plugins.
   * However, if a connector plugins does include the Log4J 2.x JAR files, 
those JAR files will only be used if the connector implementation explicitly 
uses those APIs. There isn't a need to do this, but connectors are custom code 
and can do quite a bit.
   
   We might consider adding something like this, which I hope conveys the 
limited scope of the risk:
   
   > The Connect runtime of Apache Kafka allows users to install third party 
connector plugins. These connector plugins will use Connect runtime's Log4J 1.x 
by default, even when Log4J 1.x or 2.x JARs are inadvertently shipped with the 
connector plugin. Check with the vendor of any connector plugin that includes a 
Log4J 2.x JAR file.
   
   Basically, AK is not responsible for third party connectors that users add 
to their Connect installations. But users should consult with the vendor of 
those third party connectors.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] scott-confluent commented on pull request #388: adding new CVEs to list

[GitBox]

scott-confluent commented on pull request #388:
URL: https://github.com/apache/kafka-site/pull/388#issuecomment-994133552


   
![screencapture-localhost-cve-list-html-2021-12-14-18_22_24](https://user-images.githubusercontent.com/66280178/146095095-1f2999bf-4930-4091-ac4d-ecfd066cbd0d.png)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] scott-confluent removed a comment on pull request #388: adding new CVEs to list

[GitBox]

scott-confluent removed a comment on pull request #388:
URL: https://github.com/apache/kafka-site/pull/388#issuecomment-994121512


   
![screencapture-localhost-cve-list-html-2021-12-14-17_54_58](https://user-images.githubusercontent.com/66280178/146093132-8eb8d32d-bcfb-4542-a0d9-86a545459ac6.png)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] scott-confluent commented on pull request #388: adding new CVEs to list

[GitBox]

scott-confluent commented on pull request #388:
URL: https://github.com/apache/kafka-site/pull/388#issuecomment-994121512


   
![screencapture-localhost-cve-list-html-2021-12-14-17_54_58](https://user-images.githubusercontent.com/66280178/146093132-8eb8d32d-bcfb-4542-a0d9-86a545459ac6.png)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] scott-confluent removed a comment on pull request #388: adding new CVEs to list

[GitBox]

scott-confluent removed a comment on pull request #388:
URL: https://github.com/apache/kafka-site/pull/388#issuecomment-993980560


   @junrao I have pushed all updates from the doc as well as your comments here:
   
   
![screencapture-localhost-cve-list-html-2021-12-14-15_54_20](https://user-images.githubusercontent.com/66280178/146077971-c3afc146-723c-472a-a285-3d7586e172aa.png)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] scott-confluent commented on a change in pull request #388: adding new CVEs to list

[GitBox]

scott-confluent commented on a change in pull request #388:
URL: https://github.com/apache/kafka-site/pull/388#discussion_r769105156



##
File path: cve-list.html
##
@@ -9,6 +9,70 @@ Apache Kafka Security Vulnerabilities
 
 This page lists all security vulnerabilities fixed in released versions of 
Apache Kafka.
 
+https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44228;>CVE-2021-44228
+  Flaw in Apache Log4j logging library in versions from 2.0.0 and before 
2.15.0
+
+  Some components in Apache Kafka use Log4j-v1.2.17  there is 
no dependence on Log4j v2.*
+  
+  https://logging.apache.org/log4j/2.x/manual/lookups.html;>Lookups 
feature was introduced in Log4j v2.x in order to allow specifying Log4j 
configuration parameters in arbitrary locations (even outside of the 
configuration files). Log4j v1.x does not offer the same functionality and thus 
is not vulnerable to https://access.redhat.com/security/cve/cve-2021-44228;>CVE-2021-44228.
+  Users should NOT be impacted by this vulnerability
+  
+  
+  
+
+  Versions affected
+  org.apache.logging.log4j:log4j-core =2.0-beta9 and  
2.15.0
+
+
+  Fixed versions
+  NA
+
+
+  Impact
+  NA
+
+
+  Issue announced
+  09 Dec 2021
+
+  
+  
+
+https://access.redhat.com/security/cve/CVE-2021-4104;>CVE-2021-4104
+  Flaw in Apache Log4j logging library in versions 1.x
+  
+  The following components in Apache Kafka use Log4j-v1.2.17: 
broker, controller, zookeeper, connect, mirrormaker and tools. Clients may also 
be configured to use Log4j-v1.2.17.

Review comment:
   Updated. Thanks for clarifying




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] junrao commented on a change in pull request #388: adding new CVEs to list

[GitBox]

junrao commented on a change in pull request #388:
URL: https://github.com/apache/kafka-site/pull/388#discussion_r769082582



##
File path: cve-list.html
##
@@ -9,6 +9,70 @@ Apache Kafka Security Vulnerabilities
 
 This page lists all security vulnerabilities fixed in released versions of 
Apache Kafka.
 
+https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44228;>CVE-2021-44228
+  Flaw in Apache Log4j logging library in versions from 2.0.0 and before 
2.15.0
+
+  Some components in Apache Kafka use Log4j-v1.2.17  there is 
no dependence on Log4j v2.*
+  
+  https://logging.apache.org/log4j/2.x/manual/lookups.html;>Lookups 
feature was introduced in Log4j v2.x in order to allow specifying Log4j 
configuration parameters in arbitrary locations (even outside of the 
configuration files). Log4j v1.x does not offer the same functionality and thus 
is not vulnerable to https://access.redhat.com/security/cve/cve-2021-44228;>CVE-2021-44228.
+  Users should NOT be impacted by this vulnerability
+  
+  
+  
+
+  Versions affected
+  org.apache.logging.log4j:log4j-core =2.0-beta9 and  
2.15.0
+
+
+  Fixed versions
+  NA
+
+
+  Impact
+  NA
+
+
+  Issue announced
+  09 Dec 2021
+
+  
+  
+
+https://access.redhat.com/security/cve/CVE-2021-4104;>CVE-2021-4104
+  Flaw in Apache Log4j logging library in versions 1.x
+  
+  The following components in Apache Kafka use Log4j-v1.2.17: 
broker, controller, zookeeper, connect, mirrormaker and tools. Clients may also 
be configured to use Log4j-v1.2.17.

Review comment:
   Clients may also be configured to use Log4j-v1.2.17 => Clients may 
also be configured to use Log4j-v1.x




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] scott-confluent commented on a change in pull request #388: adding new CVEs to list

[GitBox]

scott-confluent commented on a change in pull request #388:
URL: https://github.com/apache/kafka-site/pull/388#discussion_r769068111



##
File path: cve-list.html
##
@@ -9,6 +9,63 @@ Apache Kafka Security Vulnerabilities
 
 This page lists all security vulnerabilities fixed in released versions of 
Apache Kafka.
 
+https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44228;>CVE-2021-44228
+  Flaw in Apache Log4j logging library in versions from 2.0.0 and before 
2.15.0
+
+  Some components in Apache Kafka use Log4j-v1.2.17  there is 
no dependence on Log4j v2.*
+  
+  https://logging.apache.org/log4j/2.x/manual/lookups.html;>Lookups 
feature was introduced in Log4j v2.x in order to allow specifying Log4j 
configuration parameters in arbitrary locations (even outside of the 
configuration files). Log4j v1.x does not offer the same functionality and thus 
is not vulnerable to https://access.redhat.com/security/cve/cve-2021-44228;>CVE-2021-44228.
+  Users should NOT be impacted by this vulnerability
+  
+  
+  
+
+  Versions affected
+  NA
+
+
+  Fixed versions
+  NA
+
+
+  Impact
+  NA
+
+
+  Issue announced
+  09 Dec 2021
+
+  
+  
+
+https://access.redhat.com/security/cve/CVE-2021-4104;>CVE-2021-4104
+  Flaw in Apache Log4j logging library in versions 1.x
+  
+  Some components in Apache Kafka use Log4j-v1.2.17

Review comment:
   Is this to update the last sentence of the first paragraph?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] KIP-801: Implement an Authorizer that stores metadata in __cluster_metadata

[Colin McCabe]

On Tue, Dec 14, 2021, at 10:13, David Arthur wrote:
> Thanks for the KIP, Colin!
>
> 4) For the various "Type" fields in the AccessControlEntityRecord, is it
> worth explicitly enumerating the allowed types in the KIP? E.g.,
> PermissionType = {Any, Deny, Allow}. If these are listed out in another
> KIP, maybe we can just reference that.
>

Hi David,

Good question. The values should be the same as those used in the ACL RPCs 
(CreateAclsRequest, etc.) I will include this information in the KIP.

> 5) You mention "StandardAuthorizer must load all the records atomically as
> a group" when loading from a snapshot. I was under the impression that
> snapshot loads were already effectively atomic operations. That is, we
> recalculate the whole MetadataImage from the snapshot and publish it to
> components. Can you clarify what you mean here? Is this to do with how
> StandardAuthorizer handles the published metadata?
>

Snapshots are still replayed one record at a time, though. I was pointing out 
that we don't want to do this for ACLs since we want to encounter only valid 
states.

> 6) When we handle Create/Delete ACL RPCs on the controller, I think
> the operations should be written as atomic batches to the metadata log.
> Should we mention this here?

Hmm... I'm not sure if batching has an impact since these are single records. I 
hope I'm not issing something.

best,
Colin

>
> Thanks!
> David
>
>
>
>
>
>
>
> On Tue, Dec 14, 2021 at 11:27 AM José Armando García Sancio
>  wrote:
>
>> Thanks for the additional information Colin.
>>
>> On Mon, Dec 13, 2021 at 4:43 PM Colin McCabe  wrote:
>> >
>> > Hi José,
>> >
>> > I think these are good questions. We have a few situations like this
>> where there is something brokers have to know before they can contact the
>> controller quorum -- or something controllers have to know before they can
>> accept broker connections. Basically, the bootstrapping problem.
>> >
>> > Offhand, I can think of a few scenarios like this:
>> > 1. If you need certain ACLs to be present, you need a way of setting
>> those up on the controller before starting the controller quorum for the
>> first time.
>> > 2. If you are using SCRAM for the broker user, you need some way of
>> setting that up before starting up the cluster for the first time.
>> > 3. If you are using KIP-226 dynamic broker configurations to configure
>> the SSL settings for the connection to the controller, you need a way of
>> setting those up before starting the broker.
>>
>> It sounds to me like KIP-801 is assuming that this "bootstrapping KIP"
>> will at least generate a snapshot with this information in all of the
>> controllers. I would like to understand this a bit better. Do you
>> think that we need to write this "bootstrapping KIP" as soon as
>> possible?
>>
>> Thanks
>> -José
>>
>
>
> -- 
> -David

Re: [DISCUSS] KIP-801: Implement an Authorizer that stores metadata in __cluster_metadata

[Colin McCabe]

On Tue, Dec 14, 2021, at 08:27, José Armando García Sancio wrote:
> Thanks for the additional information Colin.
>
...
>
> It sounds to me like KIP-801 is assuming that this "bootstrapping KIP"
> will at least generate a snapshot with this information in all of the
> controllers. I would like to understand this a bit better. Do you
> think that we need to write this "bootstrapping KIP" as soon as
> possible?
>

Hi José,

I don't know about "as soon as possible." The authorizer is useful even without 
the bootstrapping KIP, as I mentioned (just using super.users). But I do think 
we'll need the bootstrapping KIP before KRaft goes GA.

best,
Colin

[GitHub] [kafka-site] scott-confluent removed a comment on pull request #388: adding new CVEs to list

[GitBox]

scott-confluent removed a comment on pull request #388:
URL: https://github.com/apache/kafka-site/pull/388#issuecomment-993854475


   We can't get a preview up, but here's the full page screenshot from my local:
   
![screencapture-localhost-cve-list-html-2021-12-14-12_14_14](https://user-images.githubusercontent.com/66280178/146056994-ceac3624-eda9-40a6-9d79-6c51c112cf7a.png)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] scott-confluent edited a comment on pull request #388: adding new CVEs to list

[GitBox]

scott-confluent edited a comment on pull request #388:
URL: https://github.com/apache/kafka-site/pull/388#issuecomment-993980560


   @junrao I have pushed all updates from the doc as well as your comments here:
   
   
![screencapture-localhost-cve-list-html-2021-12-14-15_54_20](https://user-images.githubusercontent.com/66280178/146077971-c3afc146-723c-472a-a285-3d7586e172aa.png)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] scott-confluent edited a comment on pull request #388: adding new CVEs to list

[GitBox]

scott-confluent edited a comment on pull request #388:
URL: https://github.com/apache/kafka-site/pull/388#issuecomment-993980560


   @junrao I have pushed all updates from the doc as well as your comments here


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] scott-confluent edited a comment on pull request #388: adding new CVEs to list

[GitBox]

scott-confluent edited a comment on pull request #388:
URL: https://github.com/apache/kafka-site/pull/388#issuecomment-993980560


   @junrao I have pushed all updates from the doc as well as your comments here
   
   
![screencapture-localhost-cve-list-html-2021-12-14-12_14_14](https://user-images.githubusercontent.com/66280178/146077833-d2453703-e1fa-4aa2-854a-86fd1183978b.png)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] scott-confluent commented on pull request #388: adding new CVEs to list

[GitBox]

scott-confluent commented on pull request #388:
URL: https://github.com/apache/kafka-site/pull/388#issuecomment-993980560


   @junrao I have pushed all updates from the doc as well as your comments here


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] scott-confluent commented on pull request #388: adding new CVEs to list

[GitBox]

scott-confluent commented on pull request #388:
URL: https://github.com/apache/kafka-site/pull/388#issuecomment-993960076


   > Do we want to add a disclaimer that users need to check their connectors 
to see if it uses log4j2?
   > 
   > Though the core library does not use this dependency, it is possible 
external connectors that use it could introduce vulnerabilities if they depend 
on the affected log4j2 version
   
   @junrao what do you think?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] izzyacademy commented on pull request #388: adding new CVEs to list

[GitBox]

izzyacademy commented on pull request #388:
URL: https://github.com/apache/kafka-site/pull/388#issuecomment-993942492


   Do we want to add a disclaimer that users need to check their connectors to 
see if it uses log4j2?
   
   Though the core library does not use this dependency, it is possible 
external connectors that use it could introduce vulnerabilities if they depend 
on the affected log4j2 version


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] junrao commented on pull request #388: adding new CVEs to list

[GitBox]

junrao commented on pull request #388:
URL: https://github.com/apache/kafka-site/pull/388#issuecomment-993922371


   @ijuma : Does the content look good to you? Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] junrao commented on pull request #388: adding new CVEs to list

[GitBox]

junrao commented on pull request #388:
URL: https://github.com/apache/kafka-site/pull/388#issuecomment-993920823


   @scott-confluent : Have you pushed the new changes?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] junrao commented on a change in pull request #388: adding new CVEs to list

[GitBox]

junrao commented on a change in pull request #388:
URL: https://github.com/apache/kafka-site/pull/388#discussion_r768987825



##
File path: cve-list.html
##
@@ -9,6 +9,63 @@ Apache Kafka Security Vulnerabilities
 
 This page lists all security vulnerabilities fixed in released versions of 
Apache Kafka.
 
+https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44228;>CVE-2021-44228
+  Flaw in Apache Log4j logging library in versions from 2.0.0 and before 
2.15.0
+
+  Some components in Apache Kafka use Log4j-v1.2.17  there is 
no dependence on Log4j v2.*
+  
+  https://logging.apache.org/log4j/2.x/manual/lookups.html;>Lookups 
feature was introduced in Log4j v2.x in order to allow specifying Log4j 
configuration parameters in arbitrary locations (even outside of the 
configuration files). Log4j v1.x does not offer the same functionality and thus 
is not vulnerable to https://access.redhat.com/security/cve/cve-2021-44228;>CVE-2021-44228.
+  Users should NOT be impacted by this vulnerability
+  
+  
+  
+
+  Versions affected
+  NA
+
+
+  Fixed versions
+  NA
+
+
+  Impact
+  NA
+
+
+  Issue announced
+  09 Dec 2021
+
+  
+  
+
+https://access.redhat.com/security/cve/CVE-2021-4104;>CVE-2021-4104
+  Flaw in Apache Log4j logging library in versions 1.x
+  
+  Some components in Apache Kafka use Log4j-v1.2.17

Review comment:
   A minor change. "Client applications may also be configured to use 
Log4j-v1.x."




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Resolved] (KAFKA-13532) Flaky test KafkaStreamsTest.testInitializesAndDestroysMetricsReporters

[John Roesler (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Roesler resolved KAFKA-13532.
--
Resolution: Cannot Reproduce

I looked in every branch build I could find, and didn't see any failures for 
this test. I think it's more likely that there was actually something wrong 
with whatever PR this was on (i.e., the failure was legit and not flaky).

 

Rather than leaving this hanging around, I'll go ahead and close it. We can 
re-open if it does start to pop up in a flaky fashion.

> Flaky test KafkaStreamsTest.testInitializesAndDestroysMetricsReporters
> --
>
> Key: KAFKA-13532
> URL: https://issues.apache.org/jira/browse/KAFKA-13532
> Project: Kafka
>  Issue Type: Test
>  Components: streams, unit tests
>Reporter: Matthias J. Sax
>Priority: Critical
>  Labels: flaky-test
>
> {quote}java.lang.AssertionError: expected:<26> but was:<27> at 
> org.junit.Assert.fail(Assert.java:89) at 
> org.junit.Assert.failNotEquals(Assert.java:835) at 
> org.junit.Assert.assertEquals(Assert.java:647) at 
> org.junit.Assert.assertEquals(Assert.java:633) at 
> org.apache.kafka.streams.KafkaStreamsTest.testInitializesAndDestroysMetricsReporters(KafkaStreamsTest.java:556){quote}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[GitHub] [kafka-site] scott-confluent commented on a change in pull request #388: adding new CVEs to list

[GitBox]

scott-confluent commented on a change in pull request #388:
URL: https://github.com/apache/kafka-site/pull/388#discussion_r768958081



##
File path: cve-list.html
##
@@ -9,6 +9,63 @@ Apache Kafka Security Vulnerabilities
 
 This page lists all security vulnerabilities fixed in released versions of 
Apache Kafka.
 
+https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44228;>CVE-2021-44228
+  Flaw in Apache Log4j logging library in versions from 2.0.0 and before 
2.15.0
+
+  Some components in Apache Kafka use Log4j-v1.2.17  there is 
no dependence on Log4j v2.*
+  
+  https://logging.apache.org/log4j/2.x/manual/lookups.html;>Lookups 
feature was introduced in Log4j v2.x in order to allow specifying Log4j 
configuration parameters in arbitrary locations (even outside of the 
configuration files). Log4j v1.x does not offer the same functionality and thus 
is not vulnerable to https://access.redhat.com/security/cve/cve-2021-44228;>CVE-2021-44228.
+  Users should NOT be impacted by this vulnerability
+  
+  
+  
+
+  Versions affected
+  NA
+
+
+  Fixed versions
+  NA
+
+
+  Impact
+  NA
+
+
+  Issue announced
+  09 Dec 2021
+
+  
+  
+
+https://access.redhat.com/security/cve/CVE-2021-4104;>CVE-2021-4104
+  Flaw in Apache Log4j logging library in versions 1.x
+  
+  Some components in Apache Kafka use Log4j-v1.2.17

Review comment:
   Here it is with both updates:
   https://user-images.githubusercontent.com/66280178/146062124-bc90d370-91ed-40e7-a0c6-771c0bf9d856.png;>

##
File path: cve-list.html
##
@@ -9,6 +9,63 @@ Apache Kafka Security Vulnerabilities
 
 This page lists all security vulnerabilities fixed in released versions of 
Apache Kafka.
 
+https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44228;>CVE-2021-44228
+  Flaw in Apache Log4j logging library in versions from 2.0.0 and before 
2.15.0
+
+  Some components in Apache Kafka use Log4j-v1.2.17  there is 
no dependence on Log4j v2.*
+  
+  https://logging.apache.org/log4j/2.x/manual/lookups.html;>Lookups 
feature was introduced in Log4j v2.x in order to allow specifying Log4j 
configuration parameters in arbitrary locations (even outside of the 
configuration files). Log4j v1.x does not offer the same functionality and thus 
is not vulnerable to https://access.redhat.com/security/cve/cve-2021-44228;>CVE-2021-44228.
+  Users should NOT be impacted by this vulnerability
+  
+  
+  
+
+  Versions affected
+  NA
+
+
+  Fixed versions
+  NA
+
+
+  Impact
+  NA
+
+
+  Issue announced
+  09 Dec 2021
+
+  
+  
+
+https://access.redhat.com/security/cve/CVE-2021-4104;>CVE-2021-4104
+  Flaw in Apache Log4j logging library in versions 1.x
+  
+  Some components in Apache Kafka use Log4j-v1.2.17
+
+  Version 1.x of Log4J can be configured to use JMS Appender, which 
publishes log events to a JMS Topic. Log4j 1.x is vulnerable if the deployed 
application is configured to use JMSAppender.
+  
+  
+  
+
+  Versions affected
+  All versions
+
+
+  Fixed versions
+  NA

Review comment:
   Here it is with both updates:
   https://user-images.githubusercontent.com/66280178/146062124-bc90d370-91ed-40e7-a0c6-771c0bf9d856.png;>




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] junrao commented on a change in pull request #388: adding new CVEs to list

[GitBox]

junrao commented on a change in pull request #388:
URL: https://github.com/apache/kafka-site/pull/388#discussion_r768953613



##
File path: cve-list.html
##
@@ -9,6 +9,63 @@ Apache Kafka Security Vulnerabilities
 
 This page lists all security vulnerabilities fixed in released versions of 
Apache Kafka.
 
+https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44228;>CVE-2021-44228
+  Flaw in Apache Log4j logging library in versions from 2.0.0 and before 
2.15.0
+
+  Some components in Apache Kafka use Log4j-v1.2.17  there is 
no dependence on Log4j v2.*
+  
+  https://logging.apache.org/log4j/2.x/manual/lookups.html;>Lookups 
feature was introduced in Log4j v2.x in order to allow specifying Log4j 
configuration parameters in arbitrary locations (even outside of the 
configuration files). Log4j v1.x does not offer the same functionality and thus 
is not vulnerable to https://access.redhat.com/security/cve/cve-2021-44228;>CVE-2021-44228.
+  Users should NOT be impacted by this vulnerability
+  
+  
+  
+
+  Versions affected
+  NA
+
+
+  Fixed versions
+  NA
+
+
+  Impact
+  NA
+
+
+  Issue announced
+  09 Dec 2021
+
+  
+  
+
+https://access.redhat.com/security/cve/CVE-2021-4104;>CVE-2021-4104
+  Flaw in Apache Log4j logging library in versions 1.x
+  
+  Some components in Apache Kafka use Log4j-v1.2.17

Review comment:
   How about the following?
   
   "The following components in Apache Kafka use Log4j-v1.2.17: broker, 
controller, zookeeper, connect, mirrormaker and tools. Clients may also be 
configured to use Log4j-v1.2.17."




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] junrao commented on a change in pull request #388: adding new CVEs to list

[GitBox]

junrao commented on a change in pull request #388:
URL: https://github.com/apache/kafka-site/pull/388#discussion_r768947897



##
File path: cve-list.html
##
@@ -9,6 +9,63 @@ Apache Kafka Security Vulnerabilities
 
 This page lists all security vulnerabilities fixed in released versions of 
Apache Kafka.
 
+https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44228;>CVE-2021-44228
+  Flaw in Apache Log4j logging library in versions from 2.0.0 and before 
2.15.0
+
+  Some components in Apache Kafka use Log4j-v1.2.17  there is 
no dependence on Log4j v2.*
+  
+  https://logging.apache.org/log4j/2.x/manual/lookups.html;>Lookups 
feature was introduced in Log4j v2.x in order to allow specifying Log4j 
configuration parameters in arbitrary locations (even outside of the 
configuration files). Log4j v1.x does not offer the same functionality and thus 
is not vulnerable to https://access.redhat.com/security/cve/cve-2021-44228;>CVE-2021-44228.
+  Users should NOT be impacted by this vulnerability
+  
+  
+  
+
+  Versions affected
+  NA
+
+
+  Fixed versions
+  NA
+
+
+  Impact
+  NA
+
+
+  Issue announced
+  09 Dec 2021
+
+  
+  
+
+https://access.redhat.com/security/cve/CVE-2021-4104;>CVE-2021-4104
+  Flaw in Apache Log4j logging library in versions 1.x
+  
+  Some components in Apache Kafka use Log4j-v1.2.17
+
+  Version 1.x of Log4J can be configured to use JMS Appender, which 
publishes log events to a JMS Topic. Log4j 1.x is vulnerable if the deployed 
application is configured to use JMSAppender.
+  
+  
+  
+
+  Versions affected
+  All versions
+
+
+  Fixed versions
+  NA

Review comment:
   Since we don't have a fix yet, perhaps we could add the following.
   
   In the absence of a new log4j 1.x release, one can remove JMSAppender
   from the *log4j-1.2.17.jar* artifact. Commands are listed in the
   page .
   
   We also recommend that configuration files be protected against write access 
as stated in .




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] scott-confluent commented on a change in pull request #388: adding new CVEs to list

[GitBox]

scott-confluent commented on a change in pull request #388:
URL: https://github.com/apache/kafka-site/pull/388#discussion_r768941484



##
File path: cve-list.html
##
@@ -9,6 +9,63 @@ Apache Kafka Security Vulnerabilities
 
 This page lists all security vulnerabilities fixed in released versions of 
Apache Kafka.
 
+https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44228;>CVE-2021-44228
+  Flaw in Apache Log4j logging library in versions from 2.0.0 and before 
2.15.0
+
+  Some components in Apache Kafka use Log4j-v1.2.17  there is 
no dependence on Log4j v2.*
+  
+  https://logging.apache.org/log4j/2.x/manual/lookups.html;>Lookups 
feature was introduced in Log4j v2.x in order to allow specifying Log4j 
configuration parameters in arbitrary locations (even outside of the 
configuration files). Log4j v1.x does not offer the same functionality and thus 
is not vulnerable to https://access.redhat.com/security/cve/cve-2021-44228;>CVE-2021-44228.
+  Users should NOT be impacted by this vulnerability
+  
+  
+  
+
+  Versions affected
+  NA
+
+
+  Fixed versions
+  NA
+
+
+  Impact
+  NA
+
+
+  Issue announced
+  09 Dec 2021
+
+  
+  
+
+https://access.redhat.com/security/cve/CVE-2021-4104;>CVE-2021-4104
+  Flaw in Apache Log4j logging library in versions 1.x
+  
+  Some components in Apache Kafka use Log4j-v1.2.17

Review comment:
   How does something like this sound?
   
   "Some components, including but not limited to: broker, controller, 
zookeeper, connect, mirrormaker, tools, and clients configured with log4j, use 
`Log4j-v1.2.17`"




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] junrao commented on a change in pull request #388: adding new CVEs to list

[GitBox]

junrao commented on a change in pull request #388:
URL: https://github.com/apache/kafka-site/pull/388#discussion_r768937548



##
File path: cve-list.html
##
@@ -9,6 +9,63 @@ Apache Kafka Security Vulnerabilities
 
 This page lists all security vulnerabilities fixed in released versions of 
Apache Kafka.
 
+https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-44228;>CVE-2021-44228
+  Flaw in Apache Log4j logging library in versions from 2.0.0 and before 
2.15.0
+
+  Some components in Apache Kafka use Log4j-v1.2.17  there is 
no dependence on Log4j v2.*
+  
+  https://logging.apache.org/log4j/2.x/manual/lookups.html;>Lookups 
feature was introduced in Log4j v2.x in order to allow specifying Log4j 
configuration parameters in arbitrary locations (even outside of the 
configuration files). Log4j v1.x does not offer the same functionality and thus 
is not vulnerable to https://access.redhat.com/security/cve/cve-2021-44228;>CVE-2021-44228.
+  Users should NOT be impacted by this vulnerability
+  
+  
+  
+
+  Versions affected
+  NA
+
+
+  Fixed versions
+  NA
+
+
+  Impact
+  NA
+
+
+  Issue announced
+  09 Dec 2021
+
+  
+  
+
+https://access.redhat.com/security/cve/CVE-2021-4104;>CVE-2021-4104
+  Flaw in Apache Log4j logging library in versions 1.x
+  
+  Some components in Apache Kafka use Log4j-v1.2.17

Review comment:
   Could we spell out all components? I think the components include the 
following.
   
   broker
   controller
   zookeeper
   connect
   mirrormaker
   tools
   clients configured with log4j




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] scott-confluent commented on pull request #388: adding new CVEs to list

[GitBox]

scott-confluent commented on pull request #388:
URL: https://github.com/apache/kafka-site/pull/388#issuecomment-993854475


   We can't get a preview up, but here's the full page screenshot from my local:
   
![screencapture-localhost-cve-list-html-2021-12-14-12_14_14](https://user-images.githubusercontent.com/66280178/146056994-ceac3624-eda9-40a6-9d79-6c51c112cf7a.png)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (KAFKA-13547) Kafka - 1.0.0 | Remove log4j.jar

[masood (Jira)]

masood created KAFKA-13547:
--

 Summary: Kafka - 1.0.0 | Remove log4j.jar
 Key: KAFKA-13547
 URL: https://issues.apache.org/jira/browse/KAFKA-13547
 Project: Kafka
  Issue Type: Bug
Reporter: masood


We wanted to remove the log4j.jar but ended up with a dependency on the 
kafka.producer.ProducerConfig. 

Caused by: java.lang.NoClassDefFoundError: org/apache/log4j/Logger
    at kafka.utils.Logging.logger(Logging.scala:24)
    at kafka.utils.Logging.logger$(Logging.scala:24)
    at 
kafka.utils.VerifiableProperties.logger$lzycompute(VerifiableProperties.scala:27)
    at kafka.utils.VerifiableProperties.logger(VerifiableProperties.scala:27)
    at kafka.utils.Logging.info(Logging.scala:71)
    at kafka.utils.Logging.info$(Logging.scala:70)
    at kafka.utils.VerifiableProperties.info(VerifiableProperties.scala:27)
    at kafka.utils.VerifiableProperties.verify(VerifiableProperties.scala:218)
    at kafka.producer.ProducerConfig.(ProducerConfig.scala:61)

Is there any configuration available which can resolve this error.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] KIP-801: Implement an Authorizer that stores metadata in __cluster_metadata

[David Arthur]

Thanks for the KIP, Colin!

4) For the various "Type" fields in the AccessControlEntityRecord, is it
worth explicitly enumerating the allowed types in the KIP? E.g.,
PermissionType = {Any, Deny, Allow}. If these are listed out in another
KIP, maybe we can just reference that.

5) You mention "StandardAuthorizer must load all the records atomically as
a group" when loading from a snapshot. I was under the impression that
snapshot loads were already effectively atomic operations. That is, we
recalculate the whole MetadataImage from the snapshot and publish it to
components. Can you clarify what you mean here? Is this to do with how
StandardAuthorizer handles the published metadata?

6) When we handle Create/Delete ACL RPCs on the controller, I think
the operations should be written as atomic batches to the metadata log.
Should we mention this here?

Thanks!
David







On Tue, Dec 14, 2021 at 11:27 AM José Armando García Sancio
 wrote:

> Thanks for the additional information Colin.
>
> On Mon, Dec 13, 2021 at 4:43 PM Colin McCabe  wrote:
> >
> > Hi José,
> >
> > I think these are good questions. We have a few situations like this
> where there is something brokers have to know before they can contact the
> controller quorum -- or something controllers have to know before they can
> accept broker connections. Basically, the bootstrapping problem.
> >
> > Offhand, I can think of a few scenarios like this:
> > 1. If you need certain ACLs to be present, you need a way of setting
> those up on the controller before starting the controller quorum for the
> first time.
> > 2. If you are using SCRAM for the broker user, you need some way of
> setting that up before starting up the cluster for the first time.
> > 3. If you are using KIP-226 dynamic broker configurations to configure
> the SSL settings for the connection to the controller, you need a way of
> setting those up before starting the broker.
>
> It sounds to me like KIP-801 is assuming that this "bootstrapping KIP"
> will at least generate a snapshot with this information in all of the
> controllers. I would like to understand this a bit better. Do you
> think that we need to write this "bootstrapping KIP" as soon as
> possible?
>
> Thanks
> -José
>


-- 
-David

[jira] [Created] (KAFKA-13546) Explicitly specifying default topic creation groups should not make connector fail

[venkat teki (Jira)]

venkat teki created KAFKA-13546:
---

 Summary: Explicitly specifying default topic creation groups 
should not make connector fail
 Key: KAFKA-13546
 URL: https://issues.apache.org/jira/browse/KAFKA-13546
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Affects Versions: 2.8.1, 2.7.2, 2.6.3, 2.6.2, 2.7.1, 2.8.0, 2.6.1, 2.7.0, 
2.6.0
Reporter: venkat teki


[KIP-158|https://cwiki.apache.org/confluence/display/KAFKA/KIP-158%3A+Kafka+Connect+should+allow+source+connectors+to+set+topic-specific+settings+for+new+topics]
 introduced support for Connect worker to allow source connector configurations 
to define topic creation settings.

A new source connector configuration {{topic.creation.groups}} was introduced, 
which takes a list of groups.  


h4. Expected behavior

According to KIP-158, specifying value "default" in {{topic.creation.groups}} 
configration should throw a warning, but not let connector fail.

*Actual behavior*

Specifying "default" will make a connector fail



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] KIP-719: Add Log4J2 Appender

[Mickael Maison]

Hi Dongjin,

Sorry for the late reply. Can we do step 3 without breaking any
compatibility? If so then that sounds like a good idea.

Thanks,
Mickael



On Tue, Nov 23, 2021 at 2:08 PM Dongjin Lee  wrote:
>
> Hi Mickael,
>
> I also thought over the issue thoroughly and would like to propose a minor
> change to your proposal:
>
> 1. Deprecate log4j-appender now
> 2. Document how to migrate into logging-log4j2
> 3. (Changed) Replace the log4j-appender (in turn log4j 1.x) dependencies in
> tools, trogdor, and shell and upgrade to log4j2 in 3.x, removing log4j 1.x
> dependencies.
> 4. (Changed) Remove log4j-appender in Kafka 4.0
>
> What we need to do for the log4j2 upgrade is just removing the log4j
> dependencies only, for they can cause a classpath error. And actually, we
> can do it without discontinuing publishing the log4j-appender artifact. So,
> I suggest separating the upgrade to log4j2 and removing the log4j-appender
> module.
>
> How do you think? If you agree, I will update the KIP and the PR
> accordingly ASAP.
>
> Thanks,
> Dongjin
>
> On Mon, Nov 15, 2021 at 8:06 PM Mickael Maison 
> wrote:
>
> > Hi Dongjin,
> >
> > Thanks for the clarifications.
> >
> > I wonder if a simpler course of action could be:
> > - Deprecate log4j-appender now
> > - Document how to use logging-log4j2
> > - Remove log4j-appender and all the log4j dependencies in Kafka 4.0
> >
> > This delays KIP-653 till Kafka 4.0 but (so far) Kafka is not directly
> > affected by the log4j CVEs. At least this gives us a clear and simple
> > roadmap to follow.
> >
> > What do you think?
> >
> > On Tue, Nov 9, 2021 at 12:12 PM Dongjin Lee  wrote:
> > >
> > > Hi Mickael,
> > >
> > > I greatly appreciate you for reading the proposal so carefully! I wrote
> > it
> > > quite a while ago and rechecked it today.
> > >
> > > > Is the KIP proposing to replace the existing log4-appender or simply
> > add
> > > a new one for log4j2? Reading the KIP and with its current title, it's
> > not
> > > entirely explicit.
> > >
> > > Oh, After re-reading it, I realized that this is not clear. Let me
> > clarify;
> > >
> > > 1. Provide a lo4j2 equivalent of traditional log4j-appender,
> > > log4j2-appender.
> > > 2. Migrate the modules depending on log4j-appender (i.e., tools, trogdor,
> > > shell) into log4j2-appender, removing log4j-appender from dependencies.
> > > 3. Entirely remove log4j-appender from the project dependencies, along
> > with
> > > log4j.
> > >
> > > I think log4j-appender may be published for every new release like
> > before,
> > > but the committee should make a decision on the policy.
> > >
> > > > Under Rejected Alternative, the KIP states: "the Kafka appender
> > provided
> > > by log4j2 community stores log message in the Record key". Looking at the
> > > code, it looks like the log message is stored in the Record value:
> > >
> > https://github.com/apache/logging-log4j2/blob/master/log4j-kafka/src/main/java/org/apache/logging/log4j/kafka/appender/KafkaManager.java#L135
> > > Am I missing something?
> > >
> > > It's totally my fault; I confused it with another appender. The
> > > compatibility problem in the logging-log4j2 Kafka appender is not the
> > > format but the configuration. logging-log4j2 Kafka appender supports
> > > `properties` configuration, which will be directly used to instantiate a
> > > Kafka producer. However, log4j-appender has been using non-producer
> > config
> > > names like brokerList (=bootstrap.servers), requiredNumAcks (=acks).
> > > Instead, logging-log4j2 Kafka appender supports retryCount,
> > > sendEventTimestamp.
> > >
> > > On second thought, using logging-log4j2 Kafka appender internally and
> > > making log4j2-appender to focus on compatibility facade only would be a
> > > better approach; As I described above, the goal of this module is just
> > > keeping the backward-compatibility, and (as you pointed out) the current
> > > implementation has little value. Since
> > org.apache.logging.log4j:log4j-core
> > > already includes Kafka appender, we can make use of the 'proven wheel'
> > > without adding more dependencies. I have not tried it yet, but I think it
> > > is well worth it. (One additional advantage of this approach is
> > providing a
> > > bridge to the users who hope to move from/into logging-log4j2 Kafka
> > > appender.)
> > >
> > > > As the current log4j-appender is not even deprecated yet, in theory we
> > > can't remove it till Kafka 4. If we want to speed up the process, I
> > wonder
> > > if the lack of documentation and a migration guide could help us. What do
> > > you think?
> > >
> > > In fact, this is what I am doing nowadays. While working with
> > > log4j-appender, I found that despite a lack of documentation,
> > considerable
> > > users are already using it[^1][^2][^3][^4][^5]. So, I think providing a
> > > documentation to those who are already using log4j-appender is
> > > indispensable. It should include:
> > >
> > > - What is the difference between log4j-appender 

[GitHub] [kafka-site] scott-confluent opened a new pull request #388: adding new CVEs to list

[GitBox]

scott-confluent opened a new pull request #388:
URL: https://github.com/apache/kafka-site/pull/388


   Per 
https://docs.google.com/document/d/1zar4nlhIQnDB7qm9i4RqFUFXM5pWP7_a0WERcLlR0Wo/edit#heading=h.jp7xjzp1clqe


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] KIP-801: Implement an Authorizer that stores metadata in __cluster_metadata

[José Armando García Sancio]

Thanks for the additional information Colin.

On Mon, Dec 13, 2021 at 4:43 PM Colin McCabe  wrote:
>
> Hi José,
>
> I think these are good questions. We have a few situations like this where 
> there is something brokers have to know before they can contact the 
> controller quorum -- or something controllers have to know before they can 
> accept broker connections. Basically, the bootstrapping problem.
>
> Offhand, I can think of a few scenarios like this:
> 1. If you need certain ACLs to be present, you need a way of setting those up 
> on the controller before starting the controller quorum for the first time.
> 2. If you are using SCRAM for the broker user, you need some way of setting 
> that up before starting up the cluster for the first time.
> 3. If you are using KIP-226 dynamic broker configurations to configure the 
> SSL settings for the connection to the controller, you need a way of setting 
> those up before starting the broker.

It sounds to me like KIP-801 is assuming that this "bootstrapping KIP"
will at least generate a snapshot with this information in all of the
controllers. I would like to understand this a bit better. Do you
think that we need to write this "bootstrapping KIP" as soon as
possible?

Thanks
-José

Re: [DISCUSS] Enable links from Github commits to Jira Issues

[Josep Prat]

Sounds like a good idea Mickael!

———
Josep Prat

Aiven Deutschland GmbH

Immanuelkirchstraße 26, 10405 Berlin

Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen

m: +491715557497

w: aiven.io

e: josep.p...@aiven.io

On Tue, Dec 14, 2021, 17:07 Tom Bentley  wrote:

> +1
>
> I didn't know this was possible, thanks Mickael!
>
>
> On Tue, Dec 14, 2021 at 3:57 PM Bill Bejeck  wrote:
>
> > Hi Mickael,
> >
> > Thanks for highlighting this, it seems very useful.
> >
> > It's a +1 for me.
> >
> > -Bill
> >
> > On Tue, Dec 14, 2021 at 9:58 AM Bruno Cadonna 
> wrote:
> >
> > > Hi Mickael,
> > >
> > > I agree that your proposal would be useful.
> > >
> > > Thank you for starting this.
> > >
> > > Best,
> > > Bruno
> > >
> > > On 14.12.21 15:55, Luke Chen wrote:
> > > > Hi Mickael,
> > > >
> > > > I love this idea.
> > > > That would be very helpful for reviewers to check the original
> problem
> > > > description in JIRA.
> > > >
> > > > Thank you for raising this!
> > > >
> > > > Luke
> > > >
> > > > On Tue, Dec 14, 2021 at 10:36 PM Mickael Maison  >
> > > wrote:
> > > >
> > > >> Hi all,
> > > >>
> > > >> A number of Apache projects have enabled JIRA links in Github
> commits.
> > > >> You can see it in action in the following repositories for example:
> > > >> - https://github.com/apache/camel
> > > >> - https://github.com/apache/spark
> > > >> where the CAMEL-XXX and SPARK-XXX prefixes link to individual JIRAs
> > > >>
> > > >> I find this feature pretty useful and I propose enabling it on the
> > > >> Kafka repository.
> > > >> Are there any concerns/objections or are people interested in having
> > > this?
> > > >>
> > > >> Thanks,
> > > >> Mickael
> > > >>
> > > >
> > >
> >
>

Re: [DISCUSS] Apache Kafka 3.1.0 release

[David Jacot]

Hi Jason,

Thanks for bringing this up. I do agree with you that it makes sense
to include this follow up to correctly fix the entire issue in 3.1.

Thanks,
David

On Mon, Dec 13, 2021 at 9:01 PM Jason Gustafson
 wrote:
>
> Hi David,
>
> I think we should get https://issues.apache.org/jira/browse/KAFKA-13488.
> This is a follow-up to https://issues.apache.org/jira/browse/KAFKA-12257,
> which was previously considered a 3.0 blocker. Without the additional
> patch, the bug causing the consumer to get stuck can still occur in a
> common scenario.
>
> Thanks,
> Jason
>
> On Tue, Dec 7, 2021 at 11:19 PM David Jacot 
> wrote:
>
> > Hi Justine,
> >
> > I am +1 on getting this in the 3.1 release as it is a serious
> > regression in clusters with a high number of partitions.
> >
> > Thanks,
> > David
> >
> > On Tue, Dec 7, 2021 at 10:39 PM Justine Olshan
> >  wrote:
> > >
> > > Hi all,
> > > I've filed a bug for an extra map allocation that is used in the fetch
> > > path. https://issues.apache.org/jira/browse/KAFKA-13512
> > > I think it qualifies as a blocker since this path is used pretty
> > frequently
> > > and it looks to be a regression.
> > >
> > > I also have a PR open to fix the issue. With this change, the performance
> > > looks much better. https://github.com/apache/kafka/pull/11576
> > > Thanks,
> > > Justine
> > >
> > > On Fri, Dec 3, 2021 at 5:29 AM David Jacot 
> > > wrote:
> > >
> > > > Hi Rajini,
> > > >
> > > > Interesting bug. The patch seems to be low risk so I suppose that
> > > > it is fine to keep it in 3.1.0.
> > > >
> > > > Thanks,
> > > > David
> > > >
> > > > On Fri, Dec 3, 2021 at 2:26 PM David Jacot 
> > wrote:
> > > > >
> > > > > Hi Colin,
> > > > >
> > > > > Thanks for the heads up. It makes sense to include it in order
> > > > > to keep the KRaft inline with ZK behavior.
> > > > >
> > > > > Thanks,
> > > > > David
> > > > >
> > > > > On Fri, Dec 3, 2021 at 9:44 AM Rajini Sivaram <
> > rajinisiva...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > Hi David,
> > > > > >
> > > > > > Sorry, I had completely forgotten about code freeze and merged
> > > > > > https://issues.apache.org/jira/browse/KAFKA-13461 to 3.1 branch
> > > > yesterday.
> > > > > > Can you take a look and see if we want it in 3.1.0? It is not a
> > > > regression
> > > > > > in 3.1, but we see this issue in tests and when it happens, the
> > > > controller
> > > > > > no longer operates as a controller.
> > > > > >
> > > > > > Thank you,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > > > > On Thu, Dec 2, 2021 at 10:56 PM Colin McCabe 
> > > > wrote:
> > > > > >
> > > > > > > Hi David,
> > > > > > >
> > > > > > > We'd like to include "KAFKA-13490: Fix createTopics and
> > > > > > > incrementalAlterConfigs for KRaft mode #11416" in the upcoming
> > > > release.
> > > > > > > This fixes some bugs in how createTopics and
> > incrementalAlterConfigs
> > > > are
> > > > > > > handled by the controller. It is specific to KRaft, so will not
> > > > affect ZK
> > > > > > > mode.
> > > > > > >
> > > > > > > best,
> > > > > > > Colin
> > > > > > >
> > > > > > > On Wed, Nov 24, 2021, at 01:20, David Jacot wrote:
> > > > > > > > Hi Mickael,
> > > > > > > >
> > > > > > > > Thanks for reporting it. It makes sense to include it in the
> > 3.1
> > > > release
> > > > > > > > as well as it is a regression.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > David
> > > > > > > >
> > > > > > > > On Tue, Nov 23, 2021 at 6:52 PM Mickael Maison <
> > > > mickael.mai...@gmail.com>
> > > > > > > wrote:
> > > > > > > >>
> > > > > > > >> Hi David,
> > > > > > > >>
> > > > > > > >> Can we also consider
> > > > https://issues.apache.org/jira/browse/KAFKA-13397?
> > > > > > > >> It's essentially a regression but in a very specific case. To
> > hit
> > > > it,
> > > > > > > >> you must be running MirrorMaker in dedicated mode and have
> > > > changed the
> > > > > > > >> separator of the default replication policy.
> > > > > > > >>
> > > > > > > >> Thanks,
> > > > > > > >> Mickael
> > > > > > > >>
> > > > > > > >> On Tue, Nov 23, 2021 at 4:58 PM David Jacot
> > > > 
> > > > > > > wrote:
> > > > > > > >> >
> > > > > > > >> > Hi Ron,
> > > > > > > >> >
> > > > > > > >> > Thank you for reaching out about this. While this is clearly
> > > > not a
> > > > > > > >> > regression, I agree with including it in 3.1 in order to
> > have
> > > > proper
> > > > > > > >> > and correct configuration constraints for KRaft. You can
> > > > proceed.
> > > > > > > >> >
> > > > > > > >> > Cheers,
> > > > > > > >> > David
> > > > > > > >> >
> > > > > > > >> > On Tue, Nov 23, 2021 at 2:55 PM Ron Dagostino <
> > > > rndg...@gmail.com>
> > > > > > > wrote:
> > > > > > > >> > >
> > > > > > > >> > > Hi David.  I would like to nominate
> > > > > > > >> > >
> > > > https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-13456
> > > > > > > >> > > "Tighten KRaft config checks/constraints" as a 3.1.0
> > > > blocker.  The
> > > > > > > >> > > existing configuration 

Re: [DISCUSS] Enable links from Github commits to Jira Issues

[Tom Bentley]

+1

I didn't know this was possible, thanks Mickael!


On Tue, Dec 14, 2021 at 3:57 PM Bill Bejeck  wrote:

> Hi Mickael,
>
> Thanks for highlighting this, it seems very useful.
>
> It's a +1 for me.
>
> -Bill
>
> On Tue, Dec 14, 2021 at 9:58 AM Bruno Cadonna  wrote:
>
> > Hi Mickael,
> >
> > I agree that your proposal would be useful.
> >
> > Thank you for starting this.
> >
> > Best,
> > Bruno
> >
> > On 14.12.21 15:55, Luke Chen wrote:
> > > Hi Mickael,
> > >
> > > I love this idea.
> > > That would be very helpful for reviewers to check the original problem
> > > description in JIRA.
> > >
> > > Thank you for raising this!
> > >
> > > Luke
> > >
> > > On Tue, Dec 14, 2021 at 10:36 PM Mickael Maison 
> > wrote:
> > >
> > >> Hi all,
> > >>
> > >> A number of Apache projects have enabled JIRA links in Github commits.
> > >> You can see it in action in the following repositories for example:
> > >> - https://github.com/apache/camel
> > >> - https://github.com/apache/spark
> > >> where the CAMEL-XXX and SPARK-XXX prefixes link to individual JIRAs
> > >>
> > >> I find this feature pretty useful and I propose enabling it on the
> > >> Kafka repository.
> > >> Are there any concerns/objections or are people interested in having
> > this?
> > >>
> > >> Thanks,
> > >> Mickael
> > >>
> > >
> >
>

Re: [DISCUSS] Enable links from Github commits to Jira Issues

[Bill Bejeck]

Hi Mickael,

Thanks for highlighting this, it seems very useful.

It's a +1 for me.

-Bill

On Tue, Dec 14, 2021 at 9:58 AM Bruno Cadonna  wrote:

> Hi Mickael,
>
> I agree that your proposal would be useful.
>
> Thank you for starting this.
>
> Best,
> Bruno
>
> On 14.12.21 15:55, Luke Chen wrote:
> > Hi Mickael,
> >
> > I love this idea.
> > That would be very helpful for reviewers to check the original problem
> > description in JIRA.
> >
> > Thank you for raising this!
> >
> > Luke
> >
> > On Tue, Dec 14, 2021 at 10:36 PM Mickael Maison 
> wrote:
> >
> >> Hi all,
> >>
> >> A number of Apache projects have enabled JIRA links in Github commits.
> >> You can see it in action in the following repositories for example:
> >> - https://github.com/apache/camel
> >> - https://github.com/apache/spark
> >> where the CAMEL-XXX and SPARK-XXX prefixes link to individual JIRAs
> >>
> >> I find this feature pretty useful and I propose enabling it on the
> >> Kafka repository.
> >> Are there any concerns/objections or are people interested in having
> this?
> >>
> >> Thanks,
> >> Mickael
> >>
> >
>

[jira] [Created] (KAFKA-13545) Workaround for mitigating CVE-2021-4104 Kafka

[Akansh Shandilya (Jira)]

Akansh Shandilya created KAFKA-13545:


 Summary: Workaround for mitigating CVE-2021-4104 Kafka 
 Key: KAFKA-13545
 URL: https://issues.apache.org/jira/browse/KAFKA-13545
 Project: Kafka
  Issue Type: Bug
Affects Versions: 2.8.1
Reporter: Akansh Shandilya


A new vulnerability is published today :

https://nvd.nist.gov/vuln/detail/CVE-2021-4104
 

Kafka v2.8.1 uses log4j v1.x . Please review following information :

Is Kafka v2.8.1 impacted by  CVE-2021-4104?

If yes, is there any workaround/recommendation available for Kafka  v2.8.1 to 
mitigate CVE-2021-4104



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] Enable links from Github commits to Jira Issues

[Bruno Cadonna]

Hi Mickael,

I agree that your proposal would be useful.

Thank you for starting this.

Best,
Bruno

On 14.12.21 15:55, Luke Chen wrote:

Hi Mickael,

I love this idea.
That would be very helpful for reviewers to check the original problem
description in JIRA.

Thank you for raising this!

Luke

On Tue, Dec 14, 2021 at 10:36 PM Mickael Maison  wrote:


Hi all,

A number of Apache projects have enabled JIRA links in Github commits.
You can see it in action in the following repositories for example:
- https://github.com/apache/camel
- https://github.com/apache/spark
where the CAMEL-XXX and SPARK-XXX prefixes link to individual JIRAs

I find this feature pretty useful and I propose enabling it on the
Kafka repository.
Are there any concerns/objections or are people interested in having this?

Thanks,
Mickael

Re: [DISCUSS] Enable links from Github commits to Jira Issues

[Luke Chen]

Hi Mickael,

I love this idea.
That would be very helpful for reviewers to check the original problem
description in JIRA.

Thank you for raising this!

Luke

On Tue, Dec 14, 2021 at 10:36 PM Mickael Maison  wrote:

> Hi all,
>
> A number of Apache projects have enabled JIRA links in Github commits.
> You can see it in action in the following repositories for example:
> - https://github.com/apache/camel
> - https://github.com/apache/spark
> where the CAMEL-XXX and SPARK-XXX prefixes link to individual JIRAs
>
> I find this feature pretty useful and I propose enabling it on the
> Kafka repository.
> Are there any concerns/objections or are people interested in having this?
>
> Thanks,
> Mickael
>

[DISCUSS] Enable links from Github commits to Jira Issues

[Mickael Maison]

Hi all,

A number of Apache projects have enabled JIRA links in Github commits.
You can see it in action in the following repositories for example:
- https://github.com/apache/camel
- https://github.com/apache/spark
where the CAMEL-XXX and SPARK-XXX prefixes link to individual JIRAs

I find this feature pretty useful and I propose enabling it on the
Kafka repository.
Are there any concerns/objections or are people interested in having this?

Thanks,
Mickael

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #579

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-13539) Improve propagation and processing of SSL handshake failures

[Rajini Sivaram (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajini Sivaram resolved KAFKA-13539.

  Reviewer: Manikumar
Resolution: Fixed

> Improve propagation and processing of SSL handshake failures
> 
>
> Key: KAFKA-13539
> URL: https://issues.apache.org/jira/browse/KAFKA-13539
> Project: Kafka
>  Issue Type: Bug
>  Components: security
>Affects Versions: 3.1.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
>Priority: Major
> Fix For: 3.2.0
>
>
> {color:#172b4d}When server fails SSL handshake and closes its connection, we 
> attempt to report this to clients on a best-effort basis. However, our tests 
> assume that peer always detects the failure. This may not be the case when 
> there are delays. It will be good to improve reliability of handshake failure 
> reporting. {color}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (KAFKA-13544) Deadlock during shutting down kafka broker because of connectivity problem with zookeeper

[Andrei Lakhmanets (Jira)]

Andrei Lakhmanets created KAFKA-13544:
-

 Summary: Deadlock during shutting down kafka broker because of 
connectivity problem with zookeeper 
 Key: KAFKA-13544
 URL: https://issues.apache.org/jira/browse/KAFKA-13544
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 2.8.1
Reporter: Andrei Lakhmanets
 Attachments: kafka_broker_logs.log, kafka_broker_stackdump.txt

Hi team,

*Kafka version:* 2.8.1
*Configuration:* 3 kafka brokers in different availability zones and 3 
zookeeper brokers in different availability zones.

I faced with deadlock in kafka. I've attached stack dump of the kafka state to 
this ticket. The locked threads are "feature-zk-node-event-process-thread" and 
"kafka-shutdown-hook".

*Context:*
My kafka cluster had connectivity problems with zookeeper and in the logs I saw 
the next exception:

The stacktrace:
{code:java}
[2021-12-06 18:31:14,629] WARN Unable to reconnect to ZooKeeper service, 
session 0x1039563000f has expired (org.apache.zookeeper.ClientCnxn)
[2021-12-06 18:31:14,629] INFO Unable to reconnect to ZooKeeper service, 
session 0x1039563000f has expired, closing socket connection 
(org.apache.zookeeper.ClientCnxn)
[2021-12-06 18:31:14,629] INFO EventThread shut down for session: 
0x1039563000f (org.apache.zookeeper.ClientCnxn)
[2021-12-06 18:31:14,631] INFO [ZooKeeperClient Kafka server] Session expired. 
(kafka.zookeeper.ZooKeeperClient)
[2021-12-06 18:31:14,632] ERROR [feature-zk-node-event-process-thread]: Failed 
to process feature ZK node change event. The broker will eventually exit. 
(kafka.server.FinalizedFeatureChangeListener$ChangeNotificationProcessorThread)
kafka.zookeeper.ZooKeeperClientExpiredException: Session expired either before 
or while waiting for connection
    at 
kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:279)
    at 
kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$1(ZooKeeperClient.scala:261)
    at 
kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:261)
    at 
kafka.zk.KafkaZkClient.retryRequestsUntilConnected(KafkaZkClient.scala:1797)
    at 
kafka.zk.KafkaZkClient.retryRequestsUntilConnected(KafkaZkClient.scala:1767)
    at 
kafka.zk.KafkaZkClient.retryRequestUntilConnected(KafkaZkClient.scala:1762)
    at kafka.zk.KafkaZkClient.getDataAndStat(KafkaZkClient.scala:771)
    at kafka.zk.KafkaZkClient.getDataAndVersion(KafkaZkClient.scala:755)
    at 
kafka.server.FinalizedFeatureChangeListener$FeatureCacheUpdater.updateLatestOrThrow(FinalizedFeatureChangeListener.scala:74)
    at 
kafka.server.FinalizedFeatureChangeListener$ChangeNotificationProcessorThread.doWork(FinalizedFeatureChangeListener.scala:147)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96) {code}
The exception is thrown in feature-zk-node-event-process-thread thread and it 
is catched in method 
FinalizedFeatureChangeListener.ChangeNotificationProcessorThread.doWork and 
then doWork method throws FatalExitError(1).
The FatalExitError catched in ShutdownableThread.run method and call 
Exit.exit(e.statusCode()) which calls System.exit under the hood.

The stackdump of "feature-zk-node-event-process-thread" thread:
{code:java}
"feature-zk-node-event-process-thread" #23 prio=5 os_prio=0 cpu=163.19ms 
elapsed=1563046.32s tid=0x7fd0dcdec800 nid=0x2088 in Object.wait()  
[0x7fd07e2c1000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(java.base@11.0.11/Native Method)
    - waiting on 
    at java.lang.Thread.join(java.base@11.0.11/Thread.java:1300)
    - waiting to re-lock in wait() <0x88b9d3c8> (a 
org.apache.kafka.common.utils.KafkaThread)
    at java.lang.Thread.join(java.base@11.0.11/Thread.java:1375)
    at 
java.lang.ApplicationShutdownHooks.runHooks(java.base@11.0.11/ApplicationShutdownHooks.java:107)
    at 
java.lang.ApplicationShutdownHooks$1.run(java.base@11.0.11/ApplicationShutdownHooks.java:46)
    at java.lang.Shutdown.runHooks(java.base@11.0.11/Shutdown.java:130)
    at java.lang.Shutdown.exit(java.base@11.0.11/Shutdown.java:174)
    - locked <0x806872f8> (a java.lang.Class for java.lang.Shutdown)
    at java.lang.Runtime.exit(java.base@11.0.11/Runtime.java:116)
    at java.lang.System.exit(java.base@11.0.11/System.java:1752)
    at org.apache.kafka.common.utils.Exit$2.execute(Exit.java:43)
    at org.apache.kafka.common.utils.Exit.exit(Exit.java:66)
    at kafka.utils.Exit$.exit(Exit.scala:28)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:102) {code}
System.exit method before shutting down virtual machine calls shutdown hooks 
and spawns the thread "kafka-shutdown-hook".
The thread "kafka-shutdown-hook" calls kafka.server.KafkaServer.shutdown which 
during work calls
{code:java}
if (featureChangeListener != null)