[jira] [Created] (KAFKA-15165) Handle Kafka client certificate failures without impacting brokers
Sandeep created KAFKA-15165: --- Summary: Handle Kafka client certificate failures without impacting brokers Key: KAFKA-15165 URL: https://issues.apache.org/jira/browse/KAFKA-15165 Project: Kafka Issue Type: Improvement Components: core, security Affects Versions: 2.8.1 Environment: production Reporter: Sandeep Following situation is observed in production: Consumer or Producer SSL Certificates have expired due to mis-management of extending the certs. When these clients to connect to either read or publish messages, they get authentication failures. These clients keep on retrying and this impacts broker CPUs utilisation, which impacts other healthy clients connected to brokers. CPU increase observed from 35% to 85-90%. Clients which are healthy see a spike in publish and consumer latencies upwards to multiply seconds. This kind of situation creates a denial of service kind of attack on Kafka cluster. We must gracefully handle this, but either: 1) Not allowing clients to connect or retry or do exponential retries after it fails to authenticate using SSL certs 2) Broker side changes, where it can blacklist clients for certain duration, which can be overwritten after certs are renewed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-12412) Group Coordinator followers are failing with OffsetsOutOfOrderException
[ https://issues.apache.org/jira/browse/KAFKA-12412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep resolved KAFKA-12412. - Resolution: Fixed > Group Coordinator followers are failing with OffsetsOutOfOrderException > --- > > Key: KAFKA-12412 > URL: https://issues.apache.org/jira/browse/KAFKA-12412 > Project: Kafka > Issue Type: Bug >Reporter: Sandeep >Priority: Major > Attachments: replica_logs > > > Upon failure of group coordinator, the followers of newly elected group > coordinator are failing with OffsetsOutOfOrderException > > Kafka Broker Version: 2.6.0 > Zookeeper version: 3.0.7 > consumer API: 1.6.0 > producer: libdirkafka: 0.9.1 > PFA: follower logs -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (KAFKA-12416) Group Coordinator followers are failing with OffsetsOutOfOrderException
[ https://issues.apache.org/jira/browse/KAFKA-12416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep reopened KAFKA-12416: - > Group Coordinator followers are failing with OffsetsOutOfOrderException > --- > > Key: KAFKA-12416 > URL: https://issues.apache.org/jira/browse/KAFKA-12416 > Project: Kafka > Issue Type: Bug > Components: replication >Reporter: Sandeep >Priority: Major > Attachments: replica_logs > > > Upon failure of group coordinator, the followers of newly elected group > coordinator are failing with OffsetsOutOfOrderException > Kafka Broker Version: 2.6.0 > Zookeeper version: 3.0.7 > consumer API: 1.6.0 > producer: libdirkafka: 0.9.1 > PFA: [^replica_logs] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-12414) Group Coordinator followers are failing with OffsetsOutOfOrderException
[ https://issues.apache.org/jira/browse/KAFKA-12414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep resolved KAFKA-12414. - Resolution: Duplicate > Group Coordinator followers are failing with OffsetsOutOfOrderException > --- > > Key: KAFKA-12414 > URL: https://issues.apache.org/jira/browse/KAFKA-12414 > Project: Kafka > Issue Type: Bug > Components: replication >Affects Versions: 2.6.0 >Reporter: Sandeep >Priority: Major > Attachments: replica_logs > > > Upon failure of group coordinator, the followers of newly elected group > coordinator are failing with OffsetsOutOfOrderException > Kafka Broker Version: 2.6.0 > Zookeeper version: 3.0.7 > consumer API: 1.6.0 > producer: libdirkafka: 0.9.1 > PFA: follower logs -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-12413) Group Coordinator followers are failing with OffsetsOutOfOrderException
[ https://issues.apache.org/jira/browse/KAFKA-12413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep resolved KAFKA-12413. - Resolution: Duplicate > Group Coordinator followers are failing with OffsetsOutOfOrderException > --- > > Key: KAFKA-12413 > URL: https://issues.apache.org/jira/browse/KAFKA-12413 > Project: Kafka > Issue Type: Bug >Reporter: Sandeep >Priority: Major > Attachments: replica_logs > > > Upon failure of group coordinator, the followers of newly elected group > coordinator are failing with OffsetsOutOfOrderException > Kafka Broker Version: 2.6.0 > Zookeeper version: 3.0.7 > consumer API: 1.6.0 > producer: libdirkafka: 0.9.1 > PFA: follower logs -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-12416) Group Coordinator followers are failing with OffsetsOutOfOrderException
[ https://issues.apache.org/jira/browse/KAFKA-12416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep resolved KAFKA-12416. - Resolution: Duplicate > Group Coordinator followers are failing with OffsetsOutOfOrderException > --- > > Key: KAFKA-12416 > URL: https://issues.apache.org/jira/browse/KAFKA-12416 > Project: Kafka > Issue Type: Bug > Components: replication >Reporter: Sandeep >Priority: Major > Attachments: replica_logs > > > Upon failure of group coordinator, the followers of newly elected group > coordinator are failing with OffsetsOutOfOrderException > Kafka Broker Version: 2.6.0 > Zookeeper version: 3.0.7 > consumer API: 1.6.0 > producer: libdirkafka: 0.9.1 > PFA: [^replica_logs] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-12411) Group Coordinator Followers failing with OutOfOrderOffsetException
[ https://issues.apache.org/jira/browse/KAFKA-12411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep resolved KAFKA-12411. - Resolution: Duplicate > Group Coordinator Followers failing with OutOfOrderOffsetException > -- > > Key: KAFKA-12411 > URL: https://issues.apache.org/jira/browse/KAFKA-12411 > Project: Kafka > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Sandeep >Priority: Major > Attachments: replica_logs > > > Post group coordinator failure and new leader election the followers are > failing with OffsetsOutOfOrderException. > clearing follower log directory and restarting did not help. > > Broker Version: 2.6.0 > Zookeeper: 3.0.7 > PFA for follower logs -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-12414) Group Coordinator followers are failing with OffsetsOutOfOrderException
Sandeep created KAFKA-12414: --- Summary: Group Coordinator followers are failing with OffsetsOutOfOrderException Key: KAFKA-12414 URL: https://issues.apache.org/jira/browse/KAFKA-12414 Project: Kafka Issue Type: Bug Components: replication Affects Versions: 2.6.0 Reporter: Sandeep Attachments: replica_logs Upon failure of group coordinator, the followers of newly elected group coordinator are failing with OffsetsOutOfOrderException Kafka Broker Version: 2.6.0 Zookeeper version: 3.0.7 consumer API: 1.6.0 producer: libdirkafka: 0.9.1 PFA: follower logs -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-12416) Group Coordinator followers are failing with OffsetsOutOfOrderException
Sandeep created KAFKA-12416: --- Summary: Group Coordinator followers are failing with OffsetsOutOfOrderException Key: KAFKA-12416 URL: https://issues.apache.org/jira/browse/KAFKA-12416 Project: Kafka Issue Type: Bug Components: replication Reporter: Sandeep Attachments: replica_logs Upon failure of group coordinator, the followers of newly elected group coordinator are failing with OffsetsOutOfOrderException Kafka Broker Version: 2.6.0 Zookeeper version: 3.0.7 consumer API: 1.6.0 producer: libdirkafka: 0.9.1 PFA: [^replica_logs] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-12412) Group Coordinator followers are failing with OffsetsOutOfOrderException
Sandeep created KAFKA-12412: --- Summary: Group Coordinator followers are failing with OffsetsOutOfOrderException Key: KAFKA-12412 URL: https://issues.apache.org/jira/browse/KAFKA-12412 Project: Kafka Issue Type: Bug Reporter: Sandeep Attachments: replica_logs Upon failure of group coordinator, the followers of newly elected group coordinator are failing with OffsetsOutOfOrderException Kafka Broker Version: 2.6.0 Zookeeper version: 3.0.7 consumer API: 1.6.0 producer: libdirkafka: 0.9.1 PFA: follower logs -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-12413) Group Coordinator followers are failing with OffsetsOutOfOrderException
Sandeep created KAFKA-12413: --- Summary: Group Coordinator followers are failing with OffsetsOutOfOrderException Key: KAFKA-12413 URL: https://issues.apache.org/jira/browse/KAFKA-12413 Project: Kafka Issue Type: Bug Reporter: Sandeep Attachments: replica_logs Upon failure of group coordinator, the followers of newly elected group coordinator are failing with OffsetsOutOfOrderException Kafka Broker Version: 2.6.0 Zookeeper version: 3.0.7 consumer API: 1.6.0 producer: libdirkafka: 0.9.1 PFA: follower logs -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-12411) Group Coordinator Followers failing with OutOfOrderOffsetException
Sandeep created KAFKA-12411: --- Summary: Group Coordinator Followers failing with OutOfOrderOffsetException Key: KAFKA-12411 URL: https://issues.apache.org/jira/browse/KAFKA-12411 Project: Kafka Issue Type: Bug Affects Versions: 2.6.0 Reporter: Sandeep Attachments: replica_logs Post group coordinator failure and new leader election the followers are failing with OffsetsOutOfOrderException. clearing follower log directory and restarting did not help. Broker Version: 2.6.0 Zookeeper: 3.0.7 PFA for follower logs -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-8669) Add java security providers in Kafka Security config
Sai Sandeep created KAFKA-8669: -- Summary: Add java security providers in Kafka Security config Key: KAFKA-8669 URL: https://issues.apache.org/jira/browse/KAFKA-8669 Project: Kafka Issue Type: Improvement Reporter: Sai Sandeep Currently kafka supports ssl.keymanager.algorithm and ssl.trustmanager.algorithm parameters as part of secure config. These parameters can be configured to load the key manager and trust managers which provide keys and certificates for ssl handshakes with the clients/server. The algorithms configured by parameters need to be registered by Java security provider classes. These provider classes are configured as JVM properties through java.security file. An example file given below ``` $ cat /usr/lib/jvm/jdk-8-oracle-x64/jre/lib/security/java.security ... security.provider.1=sun.security.provider.Sun security.provider.2=sun.security.rsa.SunRsaSign security.provider.3=sun.security.ec.SunEC … ``` Custom keymanager and trustmanager algorithms can be used to supply the kafka brokers with keys and certificates, these algorithms can be used to replace the traditional, non-scalable static keystore and truststore jks files. To take advantage of these custom algorithms, we want to support java security provider parameter in security config. This param can be used by kafka brokers or kafka clients(when connecting to the kafka brokers). The security providers can also be used for configuring security in SASL based communication too. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (KAFKA-8191) Add pluggability of KeyManager to generate the broker Private Keys and Certificates
Sai Sandeep created KAFKA-8191: -- Summary: Add pluggability of KeyManager to generate the broker Private Keys and Certificates Key: KAFKA-8191 URL: https://issues.apache.org/jira/browse/KAFKA-8191 Project: Kafka Issue Type: Bug Components: security Affects Versions: 1.1.1, 1.1.0 Reporter: Sai Sandeep Fix For: 1.1.1, 1.1.0 *Context:* Currently, in SslFactory.java, if the keystore is created null (caused by passing an empty config value to ssl.keystore.location), the default Sun KeyManager is used ignoring the 'ssl.keymanager.algorithm' provided. We need changes to fetch KeyManager from the KeyManagerFactory based on the provided keymanager algorithm, populated by 'ssl.keymanager.algorithm' if the keystore is found empty *Background and Use Case:* Kafka allows users to configure truststore and keystore to enable TLS connections from clients to brokers. Often this means during deployment, one needs to pre-provision keystores to enable clients to communicate with brokers on TLS port. Most of the time users end up configuring a long-lived certificate which is not good for security. Although KAFKA-4701 introduced the reload of keystores it still a cumbersome to distribute these files onto compute system for clients. There are several projects that allows one to distribute the certificates through a local agent, example [Spiffe|[https://spiffe.io/]]. To take advantage of such systems we need changes to consider 'ssl.keymanager.algorithm' for KeyManagerFactory creation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
