[jira] [Resolved] (KAFKA-16473) KafkaDockerWrapper uses wrong cluster ID when formatting log dir
[
https://issues.apache.org/jira/browse/KAFKA-16473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-16473.
---
Fix Version/s: 3.8.0
3.7.1
Resolution: Fixed
> KafkaDockerWrapper uses wrong cluster ID when formatting log dir
>
>
> Key: KAFKA-16473
> URL: https://issues.apache.org/jira/browse/KAFKA-16473
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.7.0
>Reporter: Sebastian Marsching
>Priority: Major
> Fix For: 3.8.0, 3.7.1
>
>
> There is a bug in {{{}KafkaDockerWrapper{}}}, that causes {{Some( CLUSTER_ID environment variable>)}} to be used when formatting the log dir
> when Kafka is started for the first time inside a Docker container.
> More specifically, the problem is in {{{}formatStorageCmd{}}}: The code uses
> {{{}env.get("CLUSTER_ID"){}}}, but this returns an {{Option}} not a
> {{{}String{}}}.
> The code should instead check whether the environment variable is set,
> raising an exception if it is not set.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Reopened] (KAFKA-16310) ListOffsets doesn't report the offset with maxTimestamp anymore
[
https://issues.apache.org/jira/browse/KAFKA-16310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar reopened KAFKA-16310:
---
> ListOffsets doesn't report the offset with maxTimestamp anymore
> ---
>
> Key: KAFKA-16310
> URL: https://issues.apache.org/jira/browse/KAFKA-16310
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 3.7.0
>Reporter: Emanuele Sabellico
>Assignee: Chia-Ping Tsai
>Priority: Blocker
> Fix For: 3.6.2, 3.8.0, 3.7.1
>
>
> Updated: This is confirmed a regression issue in v3.7.0.
> The impact of this issue is that when there is a batch containing records
> with timestamp not in order, the offset of the timestamp will be wrong.(ex:
> the timestamp for t0 should be mapping to offset 10, but will get offset 12..
> etc). It'll cause the time index is putting the wrong offset, so the result
> will be unexpected.
> ===
> The last offset is reported instead.
> A test in librdkafka (0081/do_test_ListOffsets) is failing an it's checking
> that the offset with the max timestamp is the middle one and not the last
> one. The tests is passing with 3.6.0 and previous versions
> This is the test:
> [https://github.com/confluentinc/librdkafka/blob/a6d85bdbc1023b1a5477b8befe516242c3e182f6/tests/0081-admin.c#L4989]
>
> there are three messages, with timestamps:
> {noformat}
> t0 + 100
> t0 + 400
> t0 + 250{noformat}
> and indices 0,1,2.
> then a ListOffsets with RD_KAFKA_OFFSET_SPEC_MAX_TIMESTAMP is done.
> it should return offset 1 but in 3.7.0 and trunk is returning offset 2
> Even after 5 seconds from producing it's still returning 2 as the offset with
> max timestamp.
> ProduceRequest and ListOffsets were sent to the same broker (2), the leader
> didn't change.
> {code:java}
> %7|1709134230.019|SEND|0081_admin#producer-3|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Sent ProduceRequest (v7,
> 206 bytes @ 0, CorrId 2) %7|1709134230.020|RECV|0081_admin#producer-3|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Received ProduceResponse
> (v7, 95 bytes, CorrId 2, rtt 1.18ms)
> %7|1709134230.020|MSGSET|0081_admin#producer-3|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2:
> rdkafkatest_rnd22e8d8ec45b53f98_do_test_ListOffsets [0]: MessageSet with 3
> message(s) (MsgId 0, BaseSeq -1) delivered {code}
> {code:java}
> %7|1709134235.021|SEND|0081_admin#producer-2|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Sent ListOffsetsRequest
> (v7, 103 bytes @ 0, CorrId 7) %7|1709134235.022|RECV|0081_admin#producer-2|
> [thrd:localhost:39951/bootstrap]: localhost:39951/2: Received
> ListOffsetsResponse (v7, 88 bytes, CorrId 7, rtt 0.54ms){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-15738) KRaft support in ConsumerWithLegacyMessageFormatIntegrationTest
[
https://issues.apache.org/jira/browse/KAFKA-15738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-15738.
---
Fix Version/s: 3.8.0
Resolution: Fixed
> KRaft support in ConsumerWithLegacyMessageFormatIntegrationTest
> ---
>
> Key: KAFKA-15738
> URL: https://issues.apache.org/jira/browse/KAFKA-15738
> Project: Kafka
> Issue Type: Task
> Components: core
>Reporter: Sameer Tejani
>Assignee: Abhinav Dixit
>Priority: Minor
> Labels: kraft, kraft-test, newbie
> Fix For: 3.8.0
>
>
> The following tests in ConsumerWithLegacyMessageFormatIntegrationTest in
> core/src/test/scala/integration/kafka/api/ConsumerWithLegacyMessageFormatIntegrationTest.scala
> need to be updated to support KRaft
> 0 : def testOffsetsForTimes(): Unit = {
> 102 : def testEarliestOrLatestOffsets(): Unit = {
> Scanned 132 lines. Found 0 KRaft tests out of 2 tests
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-15735) KRaft support in SaslMultiMechanismConsumerTest
[
https://issues.apache.org/jira/browse/KAFKA-15735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-15735.
---
Fix Version/s: 3.8.0
Resolution: Fixed
> KRaft support in SaslMultiMechanismConsumerTest
> ---
>
> Key: KAFKA-15735
> URL: https://issues.apache.org/jira/browse/KAFKA-15735
> Project: Kafka
> Issue Type: Task
> Components: core
>Reporter: Sameer Tejani
>Assignee: Sanskar Jhajharia
>Priority: Minor
> Labels: kraft, kraft-test, newbie
> Fix For: 3.8.0
>
>
> The following tests in SaslMultiMechanismConsumerTest in
> core/src/test/scala/integration/kafka/api/SaslMultiMechanismConsumerTest.scala
> need to be updated to support KRaft
> 45 : def testMultipleBrokerMechanisms(): Unit = {
> Scanned 94 lines. Found 0 KRaft tests out of 1 tests
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-15726) KRaft support in ProduceRequestTest
[
https://issues.apache.org/jira/browse/KAFKA-15726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-15726.
---
Fix Version/s: 3.8.0
Resolution: Fixed
> KRaft support in ProduceRequestTest
> ---
>
> Key: KAFKA-15726
> URL: https://issues.apache.org/jira/browse/KAFKA-15726
> Project: Kafka
> Issue Type: Task
> Components: core
>Reporter: Sameer Tejani
>Priority: Minor
> Labels: kraft, kraft-test, newbie
> Fix For: 3.8.0
>
>
> The following tests in ProduceRequestTest in
> core/src/test/scala/unit/kafka/server/ProduceRequestTest.scala need to be
> updated to support KRaft
> 45 : def testSimpleProduceRequest(): Unit = {
> 82 : def testProduceWithInvalidTimestamp(): Unit = {
> 128 : def testProduceToNonReplica(): Unit = {
> 170 : def testCorruptLz4ProduceRequest(): Unit = {
> 204 : def testZSTDProduceRequest(): Unit = {
> Scanned 253 lines. Found 0 KRaft tests out of 5 tests
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-15904) Downgrade tests are failing with directory.id
Manikumar created KAFKA-15904:
-
Summary: Downgrade tests are failing with directory.id
Key: KAFKA-15904
URL: https://issues.apache.org/jira/browse/KAFKA-15904
Project: Kafka
Issue Type: Bug
Reporter: Manikumar
Fix For: 3.7.0
{{kafkatest.tests.core.downgrade_test.TestDowngrade}} tests are failing after
[https://github.com/apache/kafka/pull/14628.]
We have added {{directory.id}} to metadata.properties. This means
{{metadata.properties}} will be different for different log directories.
Cluster downgrades will fail with below error if we have multiple log
directories . This looks blocker or requires additional downgrade steps from AK
3.7.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-14927) Dynamic configs not validated when using kafka-configs and --add-config-file
[
https://issues.apache.org/jira/browse/KAFKA-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-14927.
---
Fix Version/s: 3.7.0
Assignee: Aman Singh (was: José Armando García Sancio)
Resolution: Fixed
> Dynamic configs not validated when using kafka-configs and --add-config-file
>
>
> Key: KAFKA-14927
> URL: https://issues.apache.org/jira/browse/KAFKA-14927
> Project: Kafka
> Issue Type: Bug
> Components: tools
>Affects Versions: 3.3.2
>Reporter: Justin Daines
>Assignee: Aman Singh
>Priority: Minor
> Labels: 4.0-blocker
> Fix For: 3.7.0
>
>
> Using {{kafka-configs}} should validate dynamic configurations before
> applying. It is possible to send a file with invalid configurations.
> For example a file containing the following:
> {code:java}
> {
> "routes": {
> "crn:///kafka=*": {
> "management": {
> "allowed": "confluent-audit-log-events_audit",
> "denied": "confluent-audit-log-events-denied"
> },
> "describe": {
> "allowed": "",
> "denied": "confluent-audit-log-events-denied"
> },
> "authentication": {
> "allowed": "confluent-audit-log-events_audit",
> "denied": "confluent-audit-log-events-denied-authn"
> },
> "authorize": {
> "allowed": "confluent-audit-log-events_audit",
> "denied": "confluent-audit-log-events-denied-authz"
> },
> "interbroker": {
> "allowed": "",
> "denied": ""
> }
> },
> "crn:///kafka=*/group=*": {
> "consume": {
> "allowed": "confluent-audit-log-events_audit",
> "denied": "confluent-audit-log-events"
> }
> },
> "crn:///kafka=*/topic=*": {
> "produce": {
> "allowed": "confluent-audit-log-events_audit",
> "denied": "confluent-audit-log-events"
> },
> "consume": {
> "allowed": "confluent-audit-log-events_audit",
> "denied": "confluent-audit-log-events"
> }
> }
> },
> "destinations": {
> "topics": {
> "confluent-audit-log-events": {
> "retention_ms": 777600
> },
> "confluent-audit-log-events-denied": {
> "retention_ms": 777600
> },
> "confluent-audit-log-events-denied-authn": {
> "retention_ms": 777600
> },
> "confluent-audit-log-events-denied-authz": {
> "retention_ms": 777600
> },
> "confluent-audit-log-events_audit": {
> "retention_ms": 777600
> }
> }
> },
> "default_topics": {
> "allowed": "confluent-audit-log-events_audit",
> "denied": "confluent-audit-log-events"
> },
> "excluded_principals": [
> "User:schemaregistryUser",
> "User:ANONYMOUS",
> "User:appSA",
> "User:admin",
> "User:connectAdmin",
> "User:connectorSubmitter",
> "User:connectorSA",
> "User:schemaregistryUser",
> "User:ksqlDBAdmin",
> "User:ksqlDBUser",
> "User:controlCenterAndKsqlDBServer",
> "User:controlcenterAdmin",
> "User:restAdmin",
> "User:appSA",
> "User:clientListen",
> "User:superUser"
> ]
> } {code}
> {code:java}
> kafka-configs --bootstrap-server $KAFKA_BOOTSTRAP --entity-type brokers
> --entity-default --alter --add-config-file audit-log.json {code}
> Yields the following dynamic configs:
> {code:java}
> Default configs for brokers in the cluster are:
> "destinations"=null sensitive=true
> synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:"destinations"=null}
> "confluent-audit-log-events-denied-authn"=null sensitive=true
> synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:"confluent-audit-log-events-denied-authn"=null}
> "routes"=null sensitive=true
> synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:"routes"=null}
> "User=null sensitive=true
> synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:"User=null}
> },=null sensitive=true synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:},=null}
> "excluded_principals"=null sensitive=true
> synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:"excluded_principals"=null}
> "confluent-audit-log-events_audit"=null sensitive=true
> synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:"confluent-audit-log-events_audit"=null}
> "authorize"=null sensitive=true
> synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:"authorize"=null}
> "default_topics"=null sensitive=true
> synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:"default_topics"=null}
> "topics"=null sensitive=true
> synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:"topics"=null}
> ]=null sensitive=true synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:]=null}
> "interbroker"=null sensitive=true
> synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:"interbroker"=null}
> "produce"=null sensitive=true
>
[jira] [Resolved] (KAFKA-15502) Handle large keystores in SslEngineValidator
[ https://issues.apache.org/jira/browse/KAFKA-15502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-15502. --- Fix Version/s: 3.4.2 3.5.2 3.7.0 3.6.1 Resolution: Fixed > Handle large keystores in SslEngineValidator > > > Key: KAFKA-15502 > URL: https://issues.apache.org/jira/browse/KAFKA-15502 > Project: Kafka > Issue Type: Bug >Affects Versions: 3.6.0 >Reporter: Manikumar >Assignee: Manikumar >Priority: Major > Fix For: 3.4.2, 3.5.2, 3.7.0, 3.6.1 > > > We have observed an issue where inter broker SSL listener is not coming up > for large keystores (size >16K) > 1. Currently validator code doesn't work well with large stores. Right now, > WRAP returns if there is already data in the buffer. But if we need more data > to be wrapped for UNWRAP to succeed, we end up looping forever. > 2. Observed large TLSv3 post handshake messages are not getting read and > causing validator code loop forever. This is observed with JDK17+ > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15502) Handle large keystores in SslEngineValidator
Manikumar created KAFKA-15502: - Summary: Handle large keystores in SslEngineValidator Key: KAFKA-15502 URL: https://issues.apache.org/jira/browse/KAFKA-15502 Project: Kafka Issue Type: Bug Affects Versions: 3.6.0 Reporter: Manikumar Assignee: Manikumar We have observed an issue where inter broker SSL listener is not coming up for large keystores (size >16K) 1. Currently validator code doesn't work well with large stores. Right now, WRAP returns if there is already data in the buffer. But if we need more data to be wrapped for UNWRAP to succeed, we end up looping forever. 2. Observed large TLSv3 post handshake messages are not getting read and causing UNWRAP loop forever. This is observed with JDK17+ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15273) Log common name of expired client certificate
[ https://issues.apache.org/jira/browse/KAFKA-15273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-15273. --- Fix Version/s: 3.7.0 Resolution: Fixed > Log common name of expired client certificate > - > > Key: KAFKA-15273 > URL: https://issues.apache.org/jira/browse/KAFKA-15273 > Project: Kafka > Issue Type: Improvement > Components: clients, core, security >Affects Versions: 3.6.0 >Reporter: Eike Thaden >Assignee: Eike Thaden >Priority: Minor > Labels: PatchAvailable > Fix For: 3.7.0 > > > If a client tries to authenticate via mTLS with an expired certificate, the > connection is closed and the IP address of the connection attempt is logged. > However, in complex enterprise IT environments it might be very hard or even > impossible to identify which client tried to connect if only the IP address > is known (e.g. due to complex virtualization/containerization/NAT). This > results in significant effort for the Kafka platform teams to identify the > developmers responsible for such a misconfigured client. > As a possible solution I propose to log the common name used in the client > certificate in addition to the IP address. Due to security considerations, > this should only be done if that certificate is just expired and would be > valid otherwise (e.g. signed by a known, non-expired root/intermediate CA). > The way Kafka should handle any valid/invalid/expired certificate must be > exactly the same as before, except for the creation of a log message in case > it is expired. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15243) User creation mismatch
[
https://issues.apache.org/jira/browse/KAFKA-15243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-15243.
---
Fix Version/s: 3.6.0
Resolution: Fixed
> User creation mismatch
> --
>
> Key: KAFKA-15243
> URL: https://issues.apache.org/jira/browse/KAFKA-15243
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 3.3.2
>Reporter: Sergio Troiano
>Assignee: Sergio Troiano
>Priority: Major
> Labels: kafka-source
> Fix For: 3.6.0
>
>
> We found the Kafka users were not created properly, so let's suppose we
> create the user [myu...@myuser.com|mailto:myu...@myuser.com]
>
> COMMAND:
> {code:java}
> /etc/new_kafka/bin/kafka-configs.sh --bootstrap-server localhost:9092
> --alter --add-config
> 'SCRAM-SHA-256=[iterations=4096,password=blabla],SCRAM-SHA-256=[password=blabla]'
> --entity-type users --entity-name myu...@myuser.com{code}
> RESPONSE:
> {code:java}
> Completed updating config for user myu...@myuser.com{code}
> When listing the users I see the user was created as an encoded string
> COMMAND
> {code:java}
> kafka-configs.sh --bootstrap-server localhost:9092 --describe --entity-type
> users|grep myuser {code}
> RESPONSE
> {code:java}
> SCRAM credential configs for user-principal 'myuser%40myuser.com' are
> SCRAM-SHA-256=iterations=8192, SCRAM-SHA-512=iterations=4096 {code}
>
> So basically the user is being "sanitized" and giving a false OK to the user
> requester. The user requested does not exist as it should, it creates the
> encoded one instead.
>
> I dug deep in the code until I found this is happening in the
> ZkAdminManager.scala in this line
>
> {code:java}
> adminZkClient.changeConfigs(ConfigType.User, Sanitizer.sanitize(user),
> configsByPotentiallyValidUser(user)) {code}
> So removing the Sanitizer fix the problem, but I have a couple of doubts
> I checked we Sanitize because of some JMX metrics, but in this case I don't
> know if this is really needed, supossing this is needed I think we should
> forbid to create users with characters that will be encoded.
> Even worse after creating an user in general we create ACLs and they are
> created properly without encoding the characters, this creates a mismatch
> between the user and the ACLs.
>
>
> So I can work on fixing this, but I think we need to decide :
>
> A) We forbid to create users with characters that will be encoded, so we fail
> in the user creation step.
>
> B) We allow the user creation with special characters and remove the
> Sanitizer.sanitize(user) from the 2 places where it shows up in the file
> ZkAdminManager.scala
>
>
> And of course if we go for B we need to create the tests.
> Please let me know what you think and i can work on it
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-15077) FileTokenRetriever doesn't trim the token before returning it.
[
https://issues.apache.org/jira/browse/KAFKA-15077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-15077.
---
Resolution: Fixed
> FileTokenRetriever doesn't trim the token before returning it.
> --
>
> Key: KAFKA-15077
> URL: https://issues.apache.org/jira/browse/KAFKA-15077
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Reporter: Sushant Mahajan
>Assignee: Sushant Mahajan
>Priority: Minor
> Fix For: 3.6.0
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The {{FileTokenRetriever}} class is used to read the access_token from a file
> on the clients system and then the info is passed along with jaas config to
> the {{{}OAuthBearerSaslServer{}}}.
> The server uses the class {{OAuthBearerClientInitialResponse}} to validate
> the token format.
> In case the token was sent using {{FileTokenRetriever}} on the client side,
> some EOL character is getting appended to the token, causing authentication
> to fail with the message (in case to topic create):
> {{ERROR org.apache.kafka.common.errors.SaslAuthenticationException:
> Authentication failed during authentication due to invalid credentials with
> SASL mechanism OAUTHBEARER}}
>
> On the server side the following line
> [https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/internals/OAuthBearerClientInitialResponse.java#L68]
> with throw an exception failing the request.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-14320) Upgrade Jackson for CVE fix
[ https://issues.apache.org/jira/browse/KAFKA-14320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-14320. --- Resolution: Fixed > Upgrade Jackson for CVE fix > --- > > Key: KAFKA-14320 > URL: https://issues.apache.org/jira/browse/KAFKA-14320 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 3.2.0 >Reporter: Javier Li Sam >Assignee: Thomas Cooper >Priority: Minor > Labels: security > Fix For: 3.4.0, 3.3.2 > > > There is a CVE for Jackson: > Jackson: [CVE-2020-36518|https://nvd.nist.gov/vuln/detail/CVE-2020-36518] - > Fixed by upgrading to 2.14.0+ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-13518) Update gson dependency
[ https://issues.apache.org/jira/browse/KAFKA-13518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-13518. --- Fix Version/s: 3.4.0 Resolution: Fixed > Update gson dependency > -- > > Key: KAFKA-13518 > URL: https://issues.apache.org/jira/browse/KAFKA-13518 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 3.0.0 >Reporter: Pavel Kuznetsov >Assignee: Dongjin Lee >Priority: Major > Labels: security > Fix For: 3.4.0 > > > *Describe the bug* > I checked kafka_2.13-3.0.0.tgz distribution with WhiteSource and find out > that some libraries have vulnerabilities. > Here they are: > * gson-2.8.6.jar has WS-2021-0419 vulnerability. The way to fix it is to > upgrade to com.google.code.gson:gson:2.8.9 > * netty-codec-4.1.65.Final.jar has CVE-2021-37136 and CVE-2021-37137 > vulnerabilities. The way to fix it is to upgrade to > io.netty:netty-codec:4.1.68.Final > *To Reproduce* > Download kafka_2.13-3.0.0.tgz and find jars, listed above. > Check that these jars with corresponding versions are mentioned in > corresponding vulnerability description. > *Expected behavior* > * gson upgraded to 2.8.9 or higher > * netty-codec upgraded to 4.1.68.Final or higher > *Actual behaviour* > * gson is 2.8.6 > * netty-codec is 4.1.65.Final -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-14212) Fetch error response when hitting public OAuth/OIDC provider
[ https://issues.apache.org/jira/browse/KAFKA-14212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-14212. --- Fix Version/s: 3.4.0 Resolution: Fixed > Fetch error response when hitting public OAuth/OIDC provider > > > Key: KAFKA-14212 > URL: https://issues.apache.org/jira/browse/KAFKA-14212 > Project: Kafka > Issue Type: Improvement >Reporter: Sushant Mahajan >Assignee: Sushant Mahajan >Priority: Minor > Fix For: 3.4.0 > > > The class > org.apache.kafka.common.security.oauthbearer.secured.HttpAccessTokenRetriever > is used to send client creds to public OAuth/OIDC provider and fetch the > response, possibly including the access token. > However, if there is an error - the exact error message from the provider is > not currently being retrieved. > The error message can help the client easily diagnose if failure to fetch > token is due to some misconfiguration on their side. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-14063) Kafka message parsing can cause ooms with small antagonistic payloads
[ https://issues.apache.org/jira/browse/KAFKA-14063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-14063. --- Resolution: Fixed > Kafka message parsing can cause ooms with small antagonistic payloads > - > > Key: KAFKA-14063 > URL: https://issues.apache.org/jira/browse/KAFKA-14063 > Project: Kafka > Issue Type: Bug > Components: generator >Affects Versions: 3.2.0 >Reporter: Daniel Collins >Priority: Major > Fix For: 2.8.2, 3.2.3, 3.1.2, 3.0.2 > > > When parsing code receives a payload for a variable length field where the > length is specified in the code as some arbitrarily large number (assume > INT32_MAX for example) this will immediately try to allocate an ArrayList to > hold this many elements, before checking whether this is a reasonable array > size given the available data. > The fix for this is to instead throw a runtime exception if the length of a > variably sized container exceeds the amount of remaining data. Then, the > worst a user can do is force the server to allocate 8x the size of the actual > delivered data (if they claim there are N elements for a container of Objects > (i.e. not a byte string) and each Object bottoms out in an 8 byte pointer in > the ArrayList's backing array). > This was identified by fuzzing the kafka request parsing code. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-13730) OAuth access token validation fails if it does not contain the "sub" claim
[
https://issues.apache.org/jira/browse/KAFKA-13730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-13730.
---
Fix Version/s: 3.4.0
Resolution: Fixed
> OAuth access token validation fails if it does not contain the "sub" claim
> --
>
> Key: KAFKA-13730
> URL: https://issues.apache.org/jira/browse/KAFKA-13730
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Affects Versions: 3.1.0
>Reporter: Daniel Fonai
>Assignee: Kirk True
>Priority: Minor
> Fix For: 3.4.0
>
>
> Client authentication fails, when configured to use OAuth and the JWT access
> token does {*}not contain the sub claim{*}. This issue was discovered while
> testing Kafka integration with Ping Identity OAuth server. According to
> Ping's
> [documentation|https://apidocs.pingidentity.com/pingone/devguide/v1/api/#access-tokens-and-id-tokens]:
> {quote}sub – A string that specifies the identifier for the authenticated
> user. This claim is not present for client_credentials tokens.
> {quote}
> In this case Kafka broker rejects the token regardless of the
> [sasl.oauthbearer.sub.claim.name|https://kafka.apache.org/documentation/#brokerconfigs_sasl.oauthbearer.sub.claim.name]
> property value.
>
>
>
> Steps to reproduce:
> 1. Client configuration:
> {noformat}
> security.protocol=SASL_PLAINTEXT
> sasl.mechanism=OAUTHBEARER
> sasl.login.callback.handler.class=org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerLoginCallbackHandler
> sasl.oauthbearer.token.endpoint.url=https://oauth.server.fqdn/token/endpoint
> sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule
> required\
> clientId="kafka-client"\
> clientSecret="kafka-client-secret";
> sasl.oauthbearer.sub.claim.name=client_id # claim name for the principal to
> be extracted from, needed for client side validation too
> {noformat}
> 2. Broker configuration:
> {noformat}
> sasl.enabled.mechanisms=...,OAUTHBEARER
> listener.name.sasl_plaintext.oauthbearer.sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule
> required;
> listener.name.sasl_plaintext.oauthbearer.sasl.server.callback.handler.class=org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerValidatorCallbackHandler
> sasl.oauthbearer.jwks.endpoint.url=https://oauth.server.fqdn/jwks/endpoint
> sasl.oauthbearer.expected.audience=oauth-audience # based on OAuth server
> setup
> sasl.oauthbearer.sub.claim.name=client_id # claim name for the principal to
> be extracted from
> {noformat}
> 3. Try to perform some client operation:
> {noformat}
> kafka-topics --bootstrap-server `hostname`:9092 --list --command-config
> oauth-client.properties
> {noformat}
> Result:
> Client authentication fails due to invalid access token.
> - client log:
> {noformat}
> [2022-03-11 16:21:20,461] ERROR [AdminClient clientId=adminclient-1]
> Connection to node -1 (localhost/127.0.0.1:9092) failed authentication due
> to: {"status":"invalid_token"} (org.apache.kafka.clients.NetworkClient)
> [2022-03-11 16:21:20,463] WARN [AdminClient clientId=adminclient-1] Metadata
> update failed due to authentication error
> (org.apache.kafka.clients.admin.internals.AdminMetadataManager)
> org.apache.kafka.common.errors.SaslAuthenticationException:
> {"status":"invalid_token"}
> Error while executing topic command : {"status":"invalid_token"}
> [2022-03-11 16:21:20,468] ERROR
> org.apache.kafka.common.errors.SaslAuthenticationException:
> {"status":"invalid_token"}
> (kafka.admin.TopicCommand$)
> {noformat}
> - broker log:
> {noformat}
> [2022-03-11 16:21:20,150] WARN Could not validate the access token: JWT
> (claims->{"client_id":"...","iss":"...","iat":1647012079,"exp":1647015679,"aud":[...],"env":"...","org":"..."})
> rejected due to invalid claims or other invalid content. Additional details:
> [[14] No Subject (sub) claim is present.]
> (org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerValidatorCallbackHandler)
> org.apache.kafka.common.security.oauthbearer.secured.ValidateException: Could
> not validate the access token: JWT
> (claims->{"client_id":"...","iss":"...","iat":1647012079,"exp":1647015679,"aud":[...],"env":"...","org":"..."})
> rejected due to invalid claims or other invalid content. Additional details:
> [[14] No Subject (sub) claim is present.]
> at
> org.apache.kafka.common.security.oauthbearer.secured.ValidatorAccessTokenValidator.validate(ValidatorAccessTokenValidator.java:159)
> at
> org.apache.kafka.common.security.oauthbearer.secured.OAuthBearerValidatorCallbackHandler.handleValidatorCallback(OAuthBearerValidatorCallbackHandler.java:184)
> at
>
[jira] [Resolved] (KAFKA-13983) Support special character in Resource name in ACLs operation by sanitizing
[
https://issues.apache.org/jira/browse/KAFKA-13983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-13983.
---
Fix Version/s: 3.3.0
Reviewer: Manikumar
Resolution: Fixed
> Support special character in Resource name in ACLs operation by sanitizing
> ---
>
> Key: KAFKA-13983
> URL: https://issues.apache.org/jira/browse/KAFKA-13983
> Project: Kafka
> Issue Type: Bug
> Components: core
>Reporter: Aman Singh
>Assignee: Aman Singh
>Priority: Minor
> Fix For: 3.3.0
>
>
> Currently, resource names in ACLS can contain any special characters, but
> resource names with some special characters are not a valid zookeeper path
> entry.
> For example resource name {color:#de350b}{{test/true}} {color}is not a valid
> zookeeper path entry.
> Zookeeper will create a child node, name as {color:#de350b}{{true}}{color}
> inside the {color:#de350b}{{test}}{color} node.
> This will create two problems:-
> # If there is *one* ACL with a resource name {color:#de350b}{{test}}{color}
> it can't be deleted because if there is only one, Kafka tries to delete the
> node as well by thinking it will be empty which is not true it has the child
> node {{{color:#de350b}true{color}.}}
> # When broker restarts {color:#de350b}{{ACL cache}}{color}(which is used for
> ACL operations like describe, authorization etc) update from zookeeper and
> Kafka only looks for ACLs that are direct child nodes of resource type in
> the ACL tree.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-12985) CVE-2021-28169 - Upgrade jetty to 9.4.41
[ https://issues.apache.org/jira/browse/KAFKA-12985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-12985. --- Fix Version/s: 2.8.1 2.7.2 3.0.0 Resolution: Fixed > CVE-2021-28169 - Upgrade jetty to 9.4.41 > > > Key: KAFKA-12985 > URL: https://issues.apache.org/jira/browse/KAFKA-12985 > Project: Kafka > Issue Type: Task > Components: security >Reporter: Dongjin Lee >Assignee: Dongjin Lee >Priority: Minor > Fix For: 3.0.0, 2.7.2, 2.8.1 > > > CVE-2021-28169 vulnerability affects Jetty versions up to 9.4.40. For more > information see https://nvd.nist.gov/vuln/detail/CVE-2021-28169 > Upgrading to Jetty version 9.4.41 should address this issue > (https://github.com/eclipse/jetty.project/security/advisories/GHSA-gwcr-j4wh-j3cq). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-13041) Support debugging system tests with ducker-ak
[ https://issues.apache.org/jira/browse/KAFKA-13041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-13041. --- Fix Version/s: 3.0.0 Resolution: Fixed > Support debugging system tests with ducker-ak > - > > Key: KAFKA-13041 > URL: https://issues.apache.org/jira/browse/KAFKA-13041 > Project: Kafka > Issue Type: Improvement > Components: system tests >Reporter: Stanislav Vodetskyi >Priority: Major > Fix For: 3.0.0 > > > Currently when you're using ducker-ak to run system tests locally, your only > debug option is to add print/log messages. > It should be possible to connect to a ducker-ak test with a remote debugger. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-12866) Kafka requires ZK root access even when using a chroot
[
https://issues.apache.org/jira/browse/KAFKA-12866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-12866.
---
Resolution: Fixed
> Kafka requires ZK root access even when using a chroot
> --
>
> Key: KAFKA-12866
> URL: https://issues.apache.org/jira/browse/KAFKA-12866
> Project: Kafka
> Issue Type: Bug
> Components: core, zkclient
>Affects Versions: 2.6.1, 2.8.0, 2.7.1, 2.6.2
>Reporter: Igor Soarez
>Assignee: Igor Soarez
>Priority: Major
> Fix For: 3.0.0
>
>
> When a Zookeeper chroot is configured, users do not expect Kafka to need
> Zookeeper access outside of that chroot.
> h1. Why is this important?
> A zookeeper cluster may be shared with other Kafka clusters or even other
> applications. It is an expected security practice to restrict each
> cluster/application's access to it's own Zookeeper chroot.
> h1. Steps to reproduce
> h2. Zookeeper setup
> Using the zkCli, create a chroot for Kafka, make it available to Kafka but
> lock the root znode.
>
> {code:java}
> [zk: localhost:2181(CONNECTED) 1] create /somechroot
> Created /some
> [zk: localhost:2181(CONNECTED) 2] setAcl /somechroot world:anyone:cdrwa
> [zk: localhost:2181(CONNECTED) 3] addauth digest test:12345
> [zk: localhost:2181(CONNECTED) 4] setAcl /
> digest:test:Mx1uO9GLtm1qaVAQ20Vh9ODgACg=:cdrwa{code}
>
> h2. Kafka setup
> Configure the chroot in broker.properties:
>
> {code:java}
> zookeeper.connect=localhost:2181/somechroot{code}
>
>
> h2. Expected behavior
> The expected behavior here is that Kafka will use the chroot without issues.
> h2. Actual result
> Kafka fails to start with a fatal exception:
> {code:java}
> org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode =
> NoAuth for /chroot
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:120)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
> at kafka.zookeeper.AsyncResponse.maybeThrow(ZooKeeperClient.scala:583)
> at kafka.zk.KafkaZkClient.createRecursive(KafkaZkClient.scala:1729)
> at
> kafka.zk.KafkaZkClient.makeSurePersistentPathExists(KafkaZkClient.scala:1627)
> at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1957)
> at
> kafka.zk.ZkClientAclTest.testChrootExistsAndRootIsLocked(ZkClientAclTest.scala:60)
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KAFKA-12865) Documentation error for Admin Client API in describe ACLs
[ https://issues.apache.org/jira/browse/KAFKA-12865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-12865. --- Fix Version/s: 3.0.0 Resolution: Fixed > Documentation error for Admin Client API in describe ACLs > - > > Key: KAFKA-12865 > URL: https://issues.apache.org/jira/browse/KAFKA-12865 > Project: Kafka > Issue Type: Bug > Components: clients >Affects Versions: 2.8.0 >Reporter: Rohit Sachan >Priority: Major > Fix For: 3.0.0 > > > There is a documentation bug in *Admin.java's* `describeAcls` and its > overloaded variation, function's return type shows `*DeleteAclsResult*` > instead of `*DescribeAclResult*`. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-12820) Upgrade maven-artifact dependency to resolve CVE-2021-26291
[ https://issues.apache.org/jira/browse/KAFKA-12820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-12820. --- Fix Version/s: 2.8.1 2.7.2 2.6.3 3.0.0 Resolution: Fixed > Upgrade maven-artifact dependency to resolve CVE-2021-26291 > --- > > Key: KAFKA-12820 > URL: https://issues.apache.org/jira/browse/KAFKA-12820 > Project: Kafka > Issue Type: Task > Components: build >Affects Versions: 2.6.1, 2.8.0, 2.7.1 >Reporter: Boojapho >Assignee: Dongjin Lee >Priority: Major > Fix For: 3.0.0, 2.6.3, 2.7.2, 2.8.1 > > > Current Gradle builds of Kafka contain a dependency of `maven-artifact` > version 3.6.3, which contains CVE-2021-26291 > ([https://nvd.nist.gov/vuln/detail/CVE-2021-26291).] This vulnerability has > been fixed in Maven 3.8.1 > ([https://maven.apache.org/docs/3.8.1/release-notes.html]). Apache Kafka > should update `dependencies.gradle` to use the latest `maven-artifact` > library to eliminate this vulnerability. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-12752) CVE-2021-28168 upgrade jersey to 2.34 or 3.02
[ https://issues.apache.org/jira/browse/KAFKA-12752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-12752. --- Fix Version/s: 2.8.1 2.7.2 3.0.0 Reviewer: Manikumar Resolution: Fixed > CVE-2021-28168 upgrade jersey to 2.34 or 3.02 > - > > Key: KAFKA-12752 > URL: https://issues.apache.org/jira/browse/KAFKA-12752 > Project: Kafka > Issue Type: Bug >Affects Versions: 2.8.0 >Reporter: John Stacy >Assignee: Dongjin Lee >Priority: Major > Labels: CVE, security > Fix For: 3.0.0, 2.7.2, 2.8.1 > > > [https://nvd.nist.gov/vuln/detail/CVE-2021-28168] > CVE-2021-28168 affects jersey versions <=2.33, <=3.0.1. Upgrading to 2.34 or > 3.02 should resolve the issue. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-12400) Upgrade jetty to fix CVE-2020-27223
[ https://issues.apache.org/jira/browse/KAFKA-12400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-12400. --- Resolution: Fixed Issue resolved by pull request 10245 [https://github.com/apache/kafka/pull/10245] > Upgrade jetty to fix CVE-2020-27223 > --- > > Key: KAFKA-12400 > URL: https://issues.apache.org/jira/browse/KAFKA-12400 > Project: Kafka > Issue Type: Improvement >Reporter: Dongjin Lee >Assignee: Dongjin Lee >Priority: Major > Fix For: 2.7.1, 2.6.2, 2.8.0 > > > h3. CVE-2020-27223 Detail > In Eclipse Jetty 9.4.6.v20170531 to 9.4.36.v20210114 (inclusive), 10.0.0, and > 11.0.0 when Jetty handles a request containing multiple Accept headers with a > large number of quality (i.e. q) parameters, the server may enter a denial of > service (DoS) state due to high CPU usage processing those quality values, > resulting in minutes of CPU time exhausted processing those quality values. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-12389) Upgrade of netty-codec due to CVE-2021-21290
[ https://issues.apache.org/jira/browse/KAFKA-12389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-12389. --- Fix Version/s: 2.8.0 2.6.2 2.7.1 Resolution: Fixed Issue resolved by pull request 10235 [https://github.com/apache/kafka/pull/10235] > Upgrade of netty-codec due to CVE-2021-21290 > > > Key: KAFKA-12389 > URL: https://issues.apache.org/jira/browse/KAFKA-12389 > Project: Kafka > Issue Type: Bug > Components: security >Affects Versions: 2.7.0 >Reporter: Dominique Mongelli >Assignee: Dongjin Lee >Priority: Major > Fix For: 2.7.1, 2.6.2, 2.8.0 > > > Our security tool raised the following security flaw on kafka 2.7: > [https://nvd.nist.gov/vuln/detail/CVE-2021-21290] > It is a vulnerability related to jar *netty-codec-4.1.51.Final.jar*. > Looking at source code, the netty-codec in trunk and 2.7.0 branches are still > vulnerable. > Based on netty issue tracker, the vulnerability is fixed in 4.1.59.Final: > https://github.com/netty/netty/security/advisories/GHSA-5mcr-gq6c-3hq2 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-7188) Avoid reverse DNS lookup in SASL channel builder
[ https://issues.apache.org/jira/browse/KAFKA-7188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-7188. -- Assignee: (was: Rajini Sivaram) Resolution: Duplicate > Avoid reverse DNS lookup in SASL channel builder > > > Key: KAFKA-7188 > URL: https://issues.apache.org/jira/browse/KAFKA-7188 > Project: Kafka > Issue Type: Bug > Components: network >Reporter: Rajini Sivaram >Priority: Major > > SaslChannelBuilder uses InetAddress.getHostName which may perform reverse DNS > lookup, causing delays in some environments. We should replace these with > SocketAddress.getHostString if possible. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-12324) Upgrade jetty to fix CVE-2020-27218
[ https://issues.apache.org/jira/browse/KAFKA-12324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-12324. --- Fix Version/s: 2.8.0 2.6.2 2.7.1 Resolution: Fixed Issue resolved by pull request 10177 [https://github.com/apache/kafka/pull/10177] > Upgrade jetty to fix CVE-2020-27218 > --- > > Key: KAFKA-12324 > URL: https://issues.apache.org/jira/browse/KAFKA-12324 > Project: Kafka > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: John Stacy >Assignee: Dongjin Lee >Priority: Major > Fix For: 2.7.1, 2.6.2, 2.8.0 > > > h3. CVE-2020-27218 Detail > In Eclipse Jetty version 9.4.0.RC0 to 9.4.34.v20201102, 10.0.0.alpha0 to > 10.0.0.beta2, and 11.0.0.alpha0 to 11.0.0.beta2, if GZIP request body > inflation is enabled and requests from different clients are multiplexed onto > a single connection, and if an attacker can send a request with a body that > is received entirely but not consumed by the application, then a subsequent > request on the same connection will see that body prepended to its body. The > attacker will not see any data but may inject data into the body of the > subsequent request. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-12297) Implementation of MockProducer contradicts documentation of Callback for async send
[
https://issues.apache.org/jira/browse/KAFKA-12297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-12297.
---
Fix Version/s: 3.0.0
Reviewer: Manikumar
Resolution: Fixed
> Implementation of MockProducer contradicts documentation of Callback for
> async send
> ---
>
> Key: KAFKA-12297
> URL: https://issues.apache.org/jira/browse/KAFKA-12297
> Project: Kafka
> Issue Type: Bug
> Components: producer , unit tests
>Affects Versions: 2.7.0
>Reporter: Olaf Gottschalk
>Priority: Major
> Fix For: 3.0.0
>
>
> In Unit tests, a MockProducer is used to imitate a real producer.
> Using the errorNext(RuntimeException e) method, it is possible to indicate
> failures.
> BUT: the asynchronous send method with a callback has a clear documentation
> of that callback interface, stating that Metadata will always be set, and
> never null.
> {{The metadata for the record that was sent (i.e. the partition and offset).
> An empty metadata with -1 value for all fields except for topicPartition will
> be returned if an error occurred.}}
>
> The bug is, that in MockProducer's Completion implementation the following
> happens:
> {{if (e == null)}}
> {{ callback.onCompletion(metadata, null);}}
> {{else}}
> {{ callback.onCompletion(null, e);}}
>
> Behaving against the own documentation leads to very subtle bugs: tests that
> implement the error condition checking metadata != null will be fine, but in
> real life fail horribly.
>
> A MockProducer should at all times behave exactly like the real thing and
> adhere to the documentation of the Callback!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KAFKA-10669) ListOffsetRequest: make CurrentLeaderEpoch field ignorable and set MaxNumOffsets field to 1
[ https://issues.apache.org/jira/browse/KAFKA-10669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-10669. --- Resolution: Fixed Issue resolved by pull request 9540 [https://github.com/apache/kafka/pull/9540] > ListOffsetRequest: make CurrentLeaderEpoch field ignorable and set > MaxNumOffsets field to 1 > --- > > Key: KAFKA-10669 > URL: https://issues.apache.org/jira/browse/KAFKA-10669 > Project: Kafka > Issue Type: Task >Affects Versions: 2.7.0 >Reporter: Manikumar >Priority: Blocker > Fix For: 2.7.0 > > > Couple of failures observed after KAFKA-9627: Replace ListOffset > request/response with automated protocol > ([https://github.com/apache/kafka/pull/8295]) > 1. Latest consumer fails to consume from 0.10.0.1 brokers. Below system tests > are failing > > kafkatest.tests.client.client_compatibility_features_test.ClientCompatibilityFeaturesTest > > kafkatest.tests.client.client_compatibility_produce_consume_test.ClientCompatibilityProduceConsumeTest > 2. In some scenarios, latest consumer fails with below error when connecting > to a Kafka cluster which consists of newer and older (<=2.0) Kafka brokers > org.apache.kafka.common.errors.UnsupportedVersionException: Attempted to > write a non-default currentLeaderEpoch at version 3 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-10669) ListOffsetRequest: make CurrentLeaderEpoch field ignorable and set MaxNumOffsets field to 1
Manikumar created KAFKA-10669: - Summary: ListOffsetRequest: make CurrentLeaderEpoch field ignorable and set MaxNumOffsets field to 1 Key: KAFKA-10669 URL: https://issues.apache.org/jira/browse/KAFKA-10669 Project: Kafka Issue Type: Task Affects Versions: 2.7.0 Reporter: Manikumar Fix For: 2.7.0 Couple of failures observed after KAFKA-9627: Replace ListOffset request/response with automated protocol ([https://github.com/apache/kafka/pull/8295]) 1. Latest consumer fails to consume from 0.10.0.1 brokers. Below system tests are failing kafkatest.tests.client.client_compatibility_features_test.ClientCompatibilityFeaturesTest system test failing for "0.10.0.1" broker kafkatest.tests.client.client_compatibility_produce_consume_test.ClientCompatibilityProduceConsumeTest 2. In some scenarios, latest consumer fails with below error when connecting to a Kafka cluster which consists of newer and older (<=2.0) Kafka brokers org.apache.kafka.common.errors.UnsupportedVersionException: Attempted to write a non-default currentLeaderEpoch at version 3 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-10109) kafka-acls.sh/AclCommand opens multiple AdminClients
[
https://issues.apache.org/jira/browse/KAFKA-10109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-10109.
---
Fix Version/s: 2.7.0
Resolution: Fixed
Issue resolved by pull request 8808
[https://github.com/apache/kafka/pull/8808]
> kafka-acls.sh/AclCommand opens multiple AdminClients
>
>
> Key: KAFKA-10109
> URL: https://issues.apache.org/jira/browse/KAFKA-10109
> Project: Kafka
> Issue Type: Bug
> Components: tools
>Reporter: Tom Bentley
>Assignee: Tom Bentley
>Priority: Minor
> Fix For: 2.7.0
>
>
> {{AclCommand.AclCommandService}} uses {{withAdminClient(opts:
> AclCommandOptions)(f: Admin => Unit)}} to abstract the execution of an action
> using an {{AdminClient}} instance. Unfortunately the use of this method in
> implemeting {{addAcls()}} and {{removeAcls()}} calls {{listAcls()}}. This
> causes the creation of a second {{AdminClient}} instance. When the
> {{--command-config}} option has been used to specify a {{client.id}} for the
> Admin client, the second instance fails to register an MBean, resulting in a
> warning being logged.
> {code}
> ./bin/kafka-acls.sh --bootstrap-server localhost:9092 --command-config
> config/broker_connection.conf.reproducing --add --allow-principal User:alice
> --operation Describe --topic 'test' --resource-pattern-type prefixed
> Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=test,
> patternType=PREFIXED)`:
> (principal=User:alice, host=*, operation=DESCRIBE,
> permissionType=ALLOW)
> [2020-06-03 18:43:12,190] WARN Error registering AppInfo mbean
> (org.apache.kafka.common.utils.AppInfoParser)
> javax.management.InstanceAlreadyExistsException:
> kafka.admin.client:type=app-info,id=administrator_data
> at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
> at
> org.apache.kafka.common.utils.AppInfoParser.registerAppInfo(AppInfoParser.java:64)
> at
> org.apache.kafka.clients.admin.KafkaAdminClient.(KafkaAdminClient.java:500)
> at
> org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:444)
> at org.apache.kafka.clients.admin.Admin.create(Admin.java:59)
> at
> org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:39)
> at
> kafka.admin.AclCommand$AdminClientService.withAdminClient(AclCommand.scala:105)
> at
> kafka.admin.AclCommand$AdminClientService.listAcls(AclCommand.scala:146)
> at
> kafka.admin.AclCommand$AdminClientService.$anonfun$addAcls$1(AclCommand.scala:123)
> at
> kafka.admin.AclCommand$AdminClientService.$anonfun$addAcls$1$adapted(AclCommand.scala:116)
> at
> kafka.admin.AclCommand$AdminClientService.withAdminClient(AclCommand.scala:108)
> at
> kafka.admin.AclCommand$AdminClientService.addAcls(AclCommand.scala:116)
> at kafka.admin.AclCommand$.main(AclCommand.scala:78)
> at kafka.admin.AclCommand.main(AclCommand.scala)
> Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=test,
> patternType=PREFIXED)`:
> (principal=User:alice, host=*, operation=DESCRIBE, permissionType=ALLOW)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KAFKA-10214) fix flaky zookeeper_tls_test.py
[ https://issues.apache.org/jira/browse/KAFKA-10214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-10214. --- Fix Version/s: 2.6.0 Resolution: Fixed Issue resolved by pull request 8949 [https://github.com/apache/kafka/pull/8949] > fix flaky zookeeper_tls_test.py > --- > > Key: KAFKA-10214 > URL: https://issues.apache.org/jira/browse/KAFKA-10214 > Project: Kafka > Issue Type: Bug >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai >Priority: Minor > Fix For: 2.6.0 > > > After > https://github.com/apache/kafka/commit/3661f981fff2653aaf1d5ee0b6dde3410b5498db, > security_config is cached. Hence, the later changes to security flag can't > impact the security_config used by later tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-10171) Please ignore
[ https://issues.apache.org/jira/browse/KAFKA-10171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-10171. --- Resolution: Invalid > Please ignore > - > > Key: KAFKA-10171 > URL: https://issues.apache.org/jira/browse/KAFKA-10171 > Project: Kafka > Issue Type: Bug > Components: clients >Reporter: Wang Ge >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-10033) AdminClient should throw UnknownTopicOrPartitionException instead of UnknownServerException if altering configs of non-existing topic
[
https://issues.apache.org/jira/browse/KAFKA-10033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-10033.
---
Fix Version/s: 2.7.0
Resolution: Fixed
> AdminClient should throw UnknownTopicOrPartitionException instead of
> UnknownServerException if altering configs of non-existing topic
> -
>
> Key: KAFKA-10033
> URL: https://issues.apache.org/jira/browse/KAFKA-10033
> Project: Kafka
> Issue Type: Improvement
> Components: admin
>Affects Versions: 2.4.0
>Reporter: Gregory Koshelev
>Assignee: Brian Byrne
>Priority: Major
> Labels: pull-request-available
> Fix For: 2.7.0
>
>
> Currently, altering configs of non-existing topic leads to
> {{UnknownServerException}}:
> {code}
> java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.UnknownServerException: Topic "kgn_test" does
> not exist.
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
> at
> ru.kontur.vostok.hercules.stream.manager.kafka.KafkaManager.changeTtl(KafkaManager.java:130)
> ... 10 common frames omitted
> Caused by: org.apache.kafka.common.errors.UnknownServerException: Topic
> "kgn_test" does not exist.
> {code}
> The output above is produced due to {{AdminZkClient.validateTopicConfig}}
> method:
> {code}
> def validateTopicConfig(topic: String, configs: Properties): Unit = {
> Topic.validate(topic)
> if (!zkClient.topicExists(topic))
> throw new AdminOperationException("Topic \"%s\" does not
> exist.".format(topic))
> // remove the topic overrides
> LogConfig.validate(configs)
> }
> {code}
> {{UnknownServerException}} is common exception but in this case cause is
> pretty clear. So this can be fixed easily by using
> {{UnknownTopicOrPartitionException}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-10112) Consider making the number of threads configurable for offset/group metadata cache loading
Manikumar created KAFKA-10112: - Summary: Consider making the number of threads configurable for offset/group metadata cache loading Key: KAFKA-10112 URL: https://issues.apache.org/jira/browse/KAFKA-10112 Project: Kafka Issue Type: Task Components: core Reporter: Manikumar Currently we use [single-thread scheduler|https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala#L84] to handle offset/group metadata cache loading and unloading. If there are leadership changes for multiple offset topic partitions, overall loading time will be high. So if you have to load 10 partitions, the 10th one will have to wait for the previous ones. We can consider making the number of threads configurable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-9034) kafka-run-class.sh will fail if JAVA_HOME has space
[
https://issues.apache.org/jira/browse/KAFKA-9034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-9034.
--
Fix Version/s: 2.6.0
Resolution: Fixed
Issue resolved by pull request 8481
[https://github.com/apache/kafka/pull/8481]
> kafka-run-class.sh will fail if JAVA_HOME has space
> ---
>
> Key: KAFKA-9034
> URL: https://issues.apache.org/jira/browse/KAFKA-9034
> Project: Kafka
> Issue Type: Bug
> Components: tools
> Environment: macOS
>Reporter: Fenjin Wang
>Assignee: Manasvi Gupta
>Priority: Minor
> Labels: easyfix, newbie
> Fix For: 2.6.0
>
>
> If set JAVA_HOME to IntelliJ's java, bin/zookeeper-server-start.sh can't work
> because the path has space in it.
>
> {quote}export JAVA_HOME="/Applications/IntelliJ
> IDEA.app/Contents/jbr/Contents/Home/"
> {quote}
>
> We can fix this by quote "$JAVA" in the shell script according to:
> [https://stackoverflow.com/a/7740746/1203241]
>
> I can send a PR if you like.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KAFKA-9612) CLI Dynamic Configuration with file input
[ https://issues.apache.org/jira/browse/KAFKA-9612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-9612. -- Fix Version/s: 2.6.0 Reviewer: Manikumar Resolution: Fixed > CLI Dynamic Configuration with file input > - > > Key: KAFKA-9612 > URL: https://issues.apache.org/jira/browse/KAFKA-9612 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: Aneel Nazareth >Assignee: Aneel Nazareth >Priority: Minor > Fix For: 2.6.0 > > > Add --add-config-file options to kafka-configs.sh > More details here: > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-574%3A+CLI+Dynamic+Configuration+with+file+input] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-9729) Shrink inWriteLock time in SimpleAuthorizer
[
https://issues.apache.org/jira/browse/KAFKA-9729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-9729.
--
Resolution: Fixed
> Shrink inWriteLock time in SimpleAuthorizer
> ---
>
> Key: KAFKA-9729
> URL: https://issues.apache.org/jira/browse/KAFKA-9729
> Project: Kafka
> Issue Type: Improvement
> Components: security
>Affects Versions: 1.1.0
>Reporter: Jiao Zhang
>Assignee: Lucas Bradstreet
>Priority: Minor
>
> Current SimpleAuthorizer needs 'inWriteLock' when processing add/remove acls
> requests, while getAcls in authorize() needs 'inReadLock'.
> That means handling add/remove acls requests would block all other requests
> for example produce and fetch requests.
> When processing add/remove acls, updateResourceAcls() access zk to update
> acls, which could be long in the case like network glitch.
> We did the simulation for zk delay.
> When adding 100ms delay on zk side, 'inWriteLock' in addAcls()/removeAcls
> lasts for 400ms~500ms.
> When adding 500ms delay on zk side, 'inWriteLock' in addAcls()/removeAcls
> lasts for 2000ms~2500ms.
> {code:java}
> override def addAcls(acls: Set[Acl], resource: Resource) {
> if (acls != null && acls.nonEmpty) {
> inWriteLock(lock) {
> val startMs = Time.SYSTEM.milliseconds()
> updateResourceAcls(resource) { currentAcls =>
> currentAcls ++ acls
> }
> warn(s"inWriteLock in addAcls consumes ${Time.SYSTEM.milliseconds() -
> startMs} milliseconds.")
> }
> }
> }{code}
> Blocking produce/fetch requests for 2s would cause apparent performance
> degradation for the whole cluster.
> So considering is it possible to remove 'inWriteLock' in addAcls/removeAcls
> and only put 'inWriteLock' inside updateCache, which is called by
> addAcls/removeAcls.
> {code:java}
> // code placeholder
> private def updateCache(resource: Resource, versionedAcls: VersionedAcls) {
> if (versionedAcls.acls.nonEmpty) {
> aclCache.put(resource, versionedAcls)
> } else {
> aclCache.remove(resource)
> }
> }
> {code}
> If do this, block time is only the time for updating local cache, which isn't
> influenced by network glitch. But don't know if there were special concerns
> to have current strict write lock and not sure if there are side effects if
> only put lock to updateCache.
> Btw, the latest version uses 'inWriteLock' at same places as version 1.1.0.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KAFKA-9685) Solve Set concatenation perf issue in AclAuthorizer
[
https://issues.apache.org/jira/browse/KAFKA-9685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-9685.
--
Fix Version/s: 2.6.0
Resolution: Fixed
Issue resolved by pull request 8261
[https://github.com/apache/kafka/pull/8261]
> Solve Set concatenation perf issue in AclAuthorizer
> ---
>
> Key: KAFKA-9685
> URL: https://issues.apache.org/jira/browse/KAFKA-9685
> Project: Kafka
> Issue Type: Improvement
> Components: security
>Affects Versions: 1.1.0
>Reporter: Jiao Zhang
>Priority: Minor
> Fix For: 2.6.0
>
>
> In version 1.1,
> [https://github.com/apache/kafka/blob/71b1e19fc60b5e1f9bba33025737ec2b7fb1c2aa/core/src/main/scala/kafka/security/auth/SimpleAclAuthorizer.scala#L110]
> the logic for checking acls is preparing a merged acl Set with
> {code:java}
> acls = getAcls(new Resource(resource.resourceType,
> Resource.WildCardResource)) ++ getAcls(resource);{code}
> and then pass it as aclMatch's parameter.
> We found scala's Set ++ operation is very slow for example in the case that
> the Set on right hand of ++ has more than 100 entries.
> And the bad performance of ++ is due to iterating every entry of the Set on
> right hand of ++, in which the calculation of HashCode seems heavy.
> The performance of 'authorize' is important as each request delivered to
> broker goes through the logic, that's the reason we can't leave it as-is
> although the change for this proposal seems trivial.
> Here is the approach. We propose to solve this issue by introducing a new
> class 'AclSets' which takes multiple Sets as parameters and do 'find' against
> them one by one.
> {code:java}
> class AclSets(sets: Set[Acl]*){
> def find(p: Acl => Boolean): Option[Acl] =
> sets.flatMap(_.find(p)).headOption
> def isEmpty: Boolean = !sets.exists(_.nonEmpty)
> }
> {code}
> This approach avoid the Set ++ operation like following,
> {code:java}
> val acls = new AclSets(getAcls(new Resource(resource.resourceType,
> Resource.WildCardResource)), getAcls(resource)){code}
> and thus outperforms a lot compared to old logic.
> The benchmark result(we did the test with kafka version 1.1) shows notable
> difference under the condition:
> 1. set on left consists of 60 entries
> 2. set of right consists of 30 entries
> 3. search for absent entry (so that all entries are iterated)
> Benchmark Results is as following.
> Mode Cnt Score
> Error Units
> ScalaSetConcatination.Set thrpt 3 281.974 ± 140.029 ops/ms
> ScalaSetConcatination.AclSets thrpt 3 887.426 ± 40.261 ops/ms
> As the upstream also use the similar ++ operation,
> [https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/security/authorizer/AclAuthorizer.scala#L360]
> we think it's necessary to fix this issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KAFKA-9644) incrementalAlterConfigs OpType.APPEND on unset property fails with NullPointerException
[ https://issues.apache.org/jira/browse/KAFKA-9644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-9644. -- Fix Version/s: 2.6.0 Resolution: Fixed Issue resolved by pull request 8216 [https://github.com/apache/kafka/pull/8216] > incrementalAlterConfigs OpType.APPEND on unset property fails with > NullPointerException > --- > > Key: KAFKA-9644 > URL: https://issues.apache.org/jira/browse/KAFKA-9644 > Project: Kafka > Issue Type: Bug > Components: admin >Affects Versions: 2.3.0 >Reporter: Steve Rodrigues >Priority: Minor > Fix For: 2.6.0 > > Attachments: incrementalAlterTest.patch, > kafka.api.PlaintextAdminIntegrationTest.testValidIncrementalAlterConfigs.test.stdout > > > Running incrementalAlterConfigs with an OpType.APPEND when the config > property doesn't already exist fails with a NullPointerException on the > broker. > Attached is a patch to the PlaintextAdminIntegrationTest demonstrating this > failure and the test output showing the NPE. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9670) Benchmark and optimize MetadataResponse preparation
Manikumar created KAFKA-9670: Summary: Benchmark and optimize MetadataResponse preparation Key: KAFKA-9670 URL: https://issues.apache.org/jira/browse/KAFKA-9670 Project: Kafka Issue Type: Task Reporter: Manikumar Assignee: Manikumar Fix For: 2.6.0 This JIRA to remove unnecessary intermediate object conversions in Metadata Response preparation. We will also add Benchmark test to track perf improvements. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-9626) Benchmark and optimize AclAuthorizer.acls()
[ https://issues.apache.org/jira/browse/KAFKA-9626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-9626. -- Resolution: Fixed Issue resolved by pull request 8199 [https://github.com/apache/kafka/pull/8199] > Benchmark and optimize AclAuthorizer.acls() > > > Key: KAFKA-9626 > URL: https://issues.apache.org/jira/browse/KAFKA-9626 > Project: Kafka > Issue Type: Improvement >Affects Versions: 2.4.0 >Reporter: Lucas Bradstreet >Assignee: Manikumar >Priority: Major > Fix For: 2.6.0 > > > AclAuthorizer.acls creates unnecessary intermediate sets which are only > iterated over when another set is created from this set. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-9327) GroupMetadata metrics are not documented
[ https://issues.apache.org/jira/browse/KAFKA-9327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-9327. -- Fix Version/s: 2.6.0 Resolution: Fixed Issue resolved by pull request 7890 [https://github.com/apache/kafka/pull/7890] > GroupMetadata metrics are not documented > > > Key: KAFKA-9327 > URL: https://issues.apache.org/jira/browse/KAFKA-9327 > Project: Kafka > Issue Type: Improvement >Reporter: Gwen Shapira >Assignee: Dongjin Lee >Priority: Major > Fix For: 2.6.0 > > > GroupMetadataManager includes quite a few gauges that look very useful! > NumGroups, NumGroupsPreparingRebalance, NumGroupsStable, NumGroupsDead, > NumGroupsEmpty. > I couldn't find them in our site-docs, but they seem quite useful and will be > nice to include. > LastStableOffsetLag is also missing, but I'm not sure if it is useful enough > to include. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9626) Benchmark and optimize AclAuthorizer.acls()
Manikumar created KAFKA-9626: Summary: Benchmark and optimize AclAuthorizer.acls() Key: KAFKA-9626 URL: https://issues.apache.org/jira/browse/KAFKA-9626 Project: Kafka Issue Type: Improvement Affects Versions: 2.4.0 Reporter: Lucas Bradstreet Assignee: Manikumar Fix For: 2.6.0 AclAuthorizer.acls creates unnecessary intermediate sets which are only iterated over when another set is created from this set. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-9594) speed up the processing of LeaderAndIsrRequest
[ https://issues.apache.org/jira/browse/KAFKA-9594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-9594. -- Resolution: Fixed Issue resolved by pull request 8153 [https://github.com/apache/kafka/pull/8153] > speed up the processing of LeaderAndIsrRequest > -- > > Key: KAFKA-9594 > URL: https://issues.apache.org/jira/browse/KAFKA-9594 > Project: Kafka > Issue Type: Improvement >Reporter: Jun Rao >Assignee: Manikumar >Priority: Minor > Fix For: 2.6.0 > > > Observations from [~junrao] > Currently, Partition.makerFollower() holds a write lock on > leaderIsrUpdateLock. Partition.doAppendRecordsToFollowerOrFutureReplica() > holds a read lock on leaderIsrUpdateLock. So, if there is an ongoing log > append on the follower, the makeFollower() call will be delayed. This path is > a bit different when serving the Partition.makeLeader() call. Before we make > a call on Partition.makerLeader(), we first remove the follower from the > replicaFetcherThread. So, the makerLeader() call won't be delayed because of > log append. This means that when we change one follower to become leader and > another follower to follow the new leader during a controlled shutdown, the > makerLeader() call typically completes faster than the makeFollower() call, > which can delay the follower fetching from the new leader and cause ISR to > shrink. > This only reason that Partition.doAppendRecordsToFollowerOrFutureReplica() > needs to hold a read lock on leaderIsrUpdateLock is for > Partiiton.maybeReplaceCurrentWithFutureReplica() to pause the log append > while checking if the log dir could be replaced. We could potentially add a > separate lock (sth like futureLogLock) that's synced between > maybeReplaceCurrentWithFutureReplica() and > doAppendRecordsToFollowerOrFutureReplica(). Then, > doAppendRecordsToFollowerOrFutureReplica() doesn't need to hold the lock on > leaderIsrUpdateLock. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-9567) Docs and system tests for ZooKeeper 3.5.7 and KIP-515
[ https://issues.apache.org/jira/browse/KAFKA-9567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-9567. -- Fix Version/s: 2.5.0 Resolution: Fixed Issue resolved by pull request 8132 [https://github.com/apache/kafka/pull/8132] > Docs and system tests for ZooKeeper 3.5.7 and KIP-515 > - > > Key: KAFKA-9567 > URL: https://issues.apache.org/jira/browse/KAFKA-9567 > Project: Kafka > Issue Type: Improvement >Affects Versions: 2.5.0 >Reporter: Ron Dagostino >Priority: Blocker > Fix For: 2.5.0 > > > These changes depend on [KIP-515: Enable ZK client to use the new TLS > supported > authentication|https://cwiki.apache.org/confluence/display/KAFKA/KIP-515%3A+Enable+ZK+client+to+use+the+new+TLS+supported+authentication], > which was only added to 2.5.0. The upgrade to ZooKeeper 3.5.7 was merged to > both 2.5.0 and 2.4.1 via https://issues.apache.org/jira/browse/KAFKA-9515, > but this change must only be merged to 2.5.0 (it will break the system tests > if merged to 2.4.1). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9594) speed up the processing of LeaderAndIsrRequest
Manikumar created KAFKA-9594: Summary: speed up the processing of LeaderAndIsrRequest Key: KAFKA-9594 URL: https://issues.apache.org/jira/browse/KAFKA-9594 Project: Kafka Issue Type: Improvement Reporter: Jun Rao Assignee: Manikumar Fix For: 2.6.0 Observations from [~junrao] Currently, Partition.makerFollower() holds a write lock on leaderIsrUpdateLock. Partition.doAppendRecordsToFollowerOrFutureReplica() holds a read lock on leaderIsrUpdateLock. So, if there is an ongoing log append on the follower, the makeFollower() call will be delayed. This path is a bit different when serving the Partition.makeLeader() call. Before we make a call on Partition.makerLeader(), we first remove the follower from the replicaFetcherThread. So, the makerLeader() call won't be delayed because of log append. This means that when we change one follower to become leader and another follower to follow the new leader during a controlled shutdown, the makerLeader() call typically completes faster than the makeFollower() call, which can delay the follower fetching from the new leader and cause ISR to shrink. This only reason that Partition.doAppendRecordsToFollowerOrFutureReplica() needs to hold a read lock on leaderIsrUpdateLock is for Partiiton.maybeReplaceCurrentWithFutureReplica() to pause the log append while checking if the log dir could be replaced. We could potentially add a separate lock (sth like futureLogLock) that's synced between maybeReplaceCurrentWithFutureReplica() and doAppendRecordsToFollowerOrFutureReplica(). Then, doAppendRecordsToFollowerOrFutureReplica() doesn't need to hold the lock on leaderIsrUpdateLock. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-9586) Fix errored json filename in ops documentation
[
https://issues.apache.org/jira/browse/KAFKA-9586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-9586.
--
Resolution: Fixed
Issue resolved by pull request 8149
[https://github.com/apache/kafka/pull/8149]
> Fix errored json filename in ops documentation
> --
>
> Key: KAFKA-9586
> URL: https://issues.apache.org/jira/browse/KAFKA-9586
> Project: Kafka
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 2.5.0
>Reporter: Dongjin Lee
>Assignee: Dongjin Lee
>Priority: Minor
> Fix For: 2.5.0
>
>
> {quote}The --verify option can be used with the tool to check the status of
> the partition reassignment. Note that the same
> +expand-cluster-reassignment.json+ (used with the --execute option) should be
> used with the --verify option:
> bin/kafka-reassign-partitions.sh --zookeeper localhost:2181
> --reassignment-json-file +custom-reassignment.json+ --verify
> {quote}
> {{expand-cluster-reassignment.json}} (underline) should be
> {{custom-reassignment.json}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KAFKA-9575) "Notable changes in 2.5.0" doesn't mention ZooKeeper 3.5.7
[ https://issues.apache.org/jira/browse/KAFKA-9575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-9575. -- Resolution: Fixed Issue resolved by pull request 8139 [https://github.com/apache/kafka/pull/8139] > "Notable changes in 2.5.0" doesn't mention ZooKeeper 3.5.7 > -- > > Key: KAFKA-9575 > URL: https://issues.apache.org/jira/browse/KAFKA-9575 > Project: Kafka > Issue Type: Improvement > Components: docs, documentation >Affects Versions: 2.5.0 >Reporter: Ron Dagostino >Assignee: Ron Dagostino >Priority: Blocker > Fix For: 2.5.0 > > > There is a paragraph in the 2.4.0 upgrade notes talking about ZooKeeper bugs > that make manual intervention recommended while upgrading from ZoKeeper 3.4. > Both of the ZooKeeper bugs that are mentioned in the paragraph are fixed in > 3.5.7, so at a minimum we should mention the ZooKeeper 3.5.7 upgrade in the > AK 2.5.0 upgrade notes section. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-8843) Zookeeper migration tool support for TLS
[ https://issues.apache.org/jira/browse/KAFKA-8843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-8843. -- Fix Version/s: 2.5.0 Resolution: Fixed > Zookeeper migration tool support for TLS > > > Key: KAFKA-8843 > URL: https://issues.apache.org/jira/browse/KAFKA-8843 > Project: Kafka > Issue Type: New Feature >Reporter: Pere Urbon-Bayes >Assignee: Ron Dagostino >Priority: Major > Fix For: 2.5.0 > > > Currently zookeeper-migration tool works based on SASL authentication. What > means only digest and kerberos authentication is supported. > > With the introduction of ZK 3.5, TLS is added, including a new X509 > authentication provider. > > To support this great future and utilise the TLS principals, the > zookeeper-migration-tool script should support the X509 authentication as > well. > > In my newbie view, this should mean adding a new parameter to allow other > ways of authentication around > [https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ZkSecurityMigrator.scala#L65. > > |https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/ZkSecurityMigrator.scala#L65] > > If I understand the process correct, this will require a KIP, right? > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-8146) WARNING: An illegal reflective access operation has occurred
[ https://issues.apache.org/jira/browse/KAFKA-8146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-8146. -- Resolution: Fixed Warnings mentioned in the description are fixed in Kafka 2.4.0 release. > WARNING: An illegal reflective access operation has occurred > > > Key: KAFKA-8146 > URL: https://issues.apache.org/jira/browse/KAFKA-8146 > Project: Kafka > Issue Type: Bug > Components: clients, core >Affects Versions: 2.1.1 > Environment: Java 11 > Kafka v2.1.1 >Reporter: Abhi >Priority: Major > > Hi, > I am running Kafka v2.1.1 and see below warnings at the startup of server and > clients. What is the cause of these warnings and how they can be avoided or > fixed? > *Client side:* > WARNING: Illegal reflective access by > org.apache.kafka.common.network.SaslChannelBuilder > (file:/local/kafka/kafka_installation/kafka_2.12-2.1.1/libs/kafka-clients-2.1.1.jar) > to method sun.security.krb5.Config.getInstance() > WARNING: Please consider reporting this to the maintainers of > org.apache.kafka.common.network.SaslChannelBuilder > WARNING: Use --illegal-access=warn to enable warnings of further illegal > reflective access operations > WARNING: All illegal access operations will be denied in a future release > *Server side:* > WARNING: An illegal reflective access operation has occurred > WARNING: Illegal reflective access by > org.apache.zookeeper.server.util.KerberosUtil > (file:/local/kafka/kafka_installation/kafka_2.12-2.1.1/libs/zookeeper-3.4.13.jar) > to method sun.security.krb5.Config.getInstance() > WARNING: Please consider reporting this to the maintainers of > org.apache.zookeeper.server.util.KerberosUtil > WARNING: Use --illegal-access=warn to enable warnings of further illegal > reflective access operations > WARNING: All illegal access operations will be denied in a future release -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-9316) ConsoleProducer help info not expose default properties
[ https://issues.apache.org/jira/browse/KAFKA-9316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-9316. -- Fix Version/s: 2.5.0 Resolution: Fixed Issue resolved by pull request 7854 [https://github.com/apache/kafka/pull/7854] > ConsoleProducer help info not expose default properties > --- > > Key: KAFKA-9316 > URL: https://issues.apache.org/jira/browse/KAFKA-9316 > Project: Kafka > Issue Type: Improvement > Components: tools >Affects Versions: 2.4.0 >Reporter: huxihx >Assignee: huxihx >Priority: Major > Fix For: 2.5.0 > > > Unlike ConsoleConsumer, ConsoleProducer help info does not expose default > properties. Users cannot know what default properties are supported by > checking the help info. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9305) Add version 2.4 to streams system tests
Manikumar created KAFKA-9305: Summary: Add version 2.4 to streams system tests Key: KAFKA-9305 URL: https://issues.apache.org/jira/browse/KAFKA-9305 Project: Kafka Issue Type: Bug Reporter: Manikumar Fix For: 2.5.0 cc [~mjsax] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-9025) ZkSecurityMigrator not working with zookeeper chroot
[ https://issues.apache.org/jira/browse/KAFKA-9025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-9025. -- Fix Version/s: 2.5.0 Resolution: Fixed Issue resolved by pull request 7618 [https://github.com/apache/kafka/pull/7618] > ZkSecurityMigrator not working with zookeeper chroot > > > Key: KAFKA-9025 > URL: https://issues.apache.org/jira/browse/KAFKA-9025 > Project: Kafka > Issue Type: Bug > Components: security >Affects Versions: 2.3.0 > Environment: Reproduced at least on rhel and macos >Reporter: Laurent Millet >Assignee: huxihx >Priority: Major > Fix For: 2.5.0 > > > The ZkSecurityMigrator tool fails to handle installations where kafka is > configured with a zookeeper chroot (as opposed to using /, the default): > * ACLs on existing nodes are not modified (they are left world-modifiable) > * New nodes created by the tool are created directly under the zookeeper > root instead of under the chroot > The tool does not emit any message, thus the unsuspecting user can only > assume everything went well, when in fact it did not and znodes are still not > secure: > kafka_2.12-2.3.0 $ bin/zookeeper-security-migration.sh --zookeeper.acl=secure > --zookeeper.connect=localhost:2181 > kafka_2.12-2.3.0 $ > For example, with kafka configured to use /kafka as chroot > (zookeeper.connect=localhost:2181/kafka), the following is observed: > * Before running the tool > ** Zookeeper top-level nodes (all kafka nodes are under /kafka): > [zk: localhost:2181(CONNECTED) 1] ls / > [kafka, zookeeper] > ** Example node ACL: > [zk: localhost:2181(CONNECTED) 2] getAcl /kafka/brokers > 'world,'anyone > : cdrwa > * After running the tool: > ** Zookeeper top-level nodes (kafka nodes created by the tool appeared here): > [zk: localhost:2181(CONNECTED) 3] ls / > [admin, brokers, cluster, config, controller, controller_epoch, > delegation_token, isr_change_notification, kafka, kafka-acl, > kafka-acl-changes, kafka-acl-extended, kafka-acl-extended-changes, > latest_producer_id_block, log_dir_event_notification, zookeeper] > ** Example node ACL: > [zk: localhost:2181(CONNECTED) 4] getAcl /kafka/brokers > 'world,'anyone > : cdrwa > ** New node ACL: > [zk: localhost:2181(CONNECTED) 5] getAcl /brokers > 'sasl,'kafka > : cdrwa > 'world,'anyone > : r > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (KAFKA-9212) Keep receiving FENCED_LEADER_EPOCH while sending ListOffsetRequest
[
https://issues.apache.org/jira/browse/KAFKA-9212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar reopened KAFKA-9212:
--
> Keep receiving FENCED_LEADER_EPOCH while sending ListOffsetRequest
> --
>
> Key: KAFKA-9212
> URL: https://issues.apache.org/jira/browse/KAFKA-9212
> Project: Kafka
> Issue Type: Bug
> Components: consumer, offset manager
>Affects Versions: 2.3.0, 2.3.1
> Environment: Linux
>Reporter: Yannick
>Assignee: Jason Gustafson
>Priority: Blocker
> Fix For: 2.4.0, 2.3.2
>
>
> When running Kafka connect s3 sink connector ( confluent 5.3.0), after one
> broker got restarted (leaderEpoch updated at this point), the connect worker
> crashed with the following error :
> [2019-11-19 16:20:30,097] ERROR [Worker clientId=connect-1,
> groupId=connect-ls] Uncaught exception in herder work thread, exiting:
> (org.apache.kafka.connect.runtime.distributed.DistributedHerder:253)
> org.apache.kafka.common.errors.TimeoutException: Failed to get offsets by
> times in 30003ms
>
> After investigation, it seems it's because it got fenced when sending
> ListOffsetRequest in loop and then got timed out , as follows :
> [2019-11-19 16:20:30,020] DEBUG [Consumer clientId=consumer-3,
> groupId=connect-ls] Sending ListOffsetRequest (type=ListOffsetRequest,
> replicaId=-1, partitionTimestamps={connect_ls_config-0={timestamp: -1,
> maxNumOffsets: 1, currentLeaderEpoch: Optional[1]}},
> isolationLevel=READ_UNCOMMITTED) to broker kafka6.fra2.internal:9092 (id: 4
> rack: null) (org.apache.kafka.clients.consumer.internals.Fetcher:905)
> [2019-11-19 16:20:30,044] DEBUG [Consumer clientId=consumer-3,
> groupId=connect-ls] Attempt to fetch offsets for partition
> connect_ls_config-0 failed due to FENCED_LEADER_EPOCH, retrying.
> (org.apache.kafka.clients.consumer.internals.Fetcher:985)
>
> The above happens multiple times until timeout.
>
> According to the debugs, the consumer always get a leaderEpoch of 1 for this
> topic when starting up :
>
> [2019-11-19 13:27:30,802] DEBUG [Consumer clientId=consumer-3,
> groupId=connect-ls] Updating last seen epoch from null to 1 for partition
> connect_ls_config-0 (org.apache.kafka.clients.Metadata:178)
>
>
> But according to our brokers log, the leaderEpoch should be 2, as follows :
>
> [2019-11-18 14:19:28,988] INFO [Partition connect_ls_config-0 broker=4]
> connect_ls_config-0 starts at Leader Epoch 2 from offset 22. Previous Leader
> Epoch was: 1 (kafka.cluster.Partition)
>
>
> This make impossible to restart the worker as it will always get fenced and
> then finally timeout.
>
> It is also impossible to consume with a 2.3 kafka-console-consumer as
> follows :
>
> kafka-console-consumer --bootstrap-server BOOTSTRAPSERVER:9092 --topic
> connect_ls_config --from-beginning
>
> the above will just hang forever ( which is not expected cause there is
> data) and we can see those debug messages :
> [2019-11-19 22:17:59,124] DEBUG [Consumer clientId=consumer-1,
> groupId=console-consumer-3844] Attempt to fetch offsets for partition
> connect_ls_config-0 failed due to FENCED_LEADER_EPOCH, retrying.
> (org.apache.kafka.clients.consumer.internals.Fetcher)
>
>
> Interesting fact, if we do subscribe the same way with kafkacat (1.5.0) we
> can consume without problem ( must be the way kafkacat is consuming ignoring
> FENCED_LEADER_EPOCH):
>
> kafkacat -b BOOTSTRAPSERVER:9092 -t connect_ls_config -o beginning
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Reopened] (KAFKA-9261) NPE when updating client metadata
[
https://issues.apache.org/jira/browse/KAFKA-9261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar reopened KAFKA-9261:
--
> NPE when updating client metadata
> -
>
> Key: KAFKA-9261
> URL: https://issues.apache.org/jira/browse/KAFKA-9261
> Project: Kafka
> Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Major
> Fix For: 2.4.0, 2.3.2
>
>
> We have seen the following exception recently:
> {code}
> java.lang.NullPointerException
> at java.base/java.util.Objects.requireNonNull(Objects.java:221)
> at org.apache.kafka.common.Cluster.(Cluster.java:134)
> at org.apache.kafka.common.Cluster.(Cluster.java:89)
> at
> org.apache.kafka.clients.MetadataCache.computeClusterView(MetadataCache.java:120)
> at org.apache.kafka.clients.MetadataCache.(MetadataCache.java:82)
> at org.apache.kafka.clients.MetadataCache.(MetadataCache.java:58)
> at
> org.apache.kafka.clients.Metadata.handleMetadataResponse(Metadata.java:325)
> at org.apache.kafka.clients.Metadata.update(Metadata.java:252)
> at
> org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.handleCompletedMetadataResponse(NetworkClient.java:1059)
> at
> org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:845)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:548)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:262)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1281)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1225)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1201)
> {code}
> The client assumes that if a leader is included in the response, then node
> information must also be available. There are at least a couple possible
> reasons this assumption can fail:
> 1. The client is able to detect stale partition metadata using leader epoch
> information available. If stale partition metadata is detected, the client
> ignores it and uses the last known metadata. However, it cannot detect stale
> broker information and will always accept the latest update. This means that
> the latest metadata may be a mix of multiple metadata responses and therefore
> the invariant will not generally hold.
> 2. There is no lock which protects both the fetching of partition metadata
> and the live broker when handling a Metadata request. This means an
> UpdateMetadata request can arrive concurrently and break the intended
> invariant.
> It seems case 2 has been possible for a long time, but it should be extremely
> rare. Case 1 was only made possible with KIP-320, which added the leader
> epoch tracking. It should also be rare, but the window for inconsistent
> metadata is probably a bit bigger than the window for a concurrent update.
> To fix this, we should make the client more defensive about metadata updates
> and not assume that the leader is among the live endpoints.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KAFKA-9261) NPE when updating client metadata
[
https://issues.apache.org/jira/browse/KAFKA-9261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-9261.
--
Resolution: Fixed
> NPE when updating client metadata
> -
>
> Key: KAFKA-9261
> URL: https://issues.apache.org/jira/browse/KAFKA-9261
> Project: Kafka
> Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Major
> Fix For: 2.4.0, 2.3.2
>
>
> We have seen the following exception recently:
> {code}
> java.lang.NullPointerException
> at java.base/java.util.Objects.requireNonNull(Objects.java:221)
> at org.apache.kafka.common.Cluster.(Cluster.java:134)
> at org.apache.kafka.common.Cluster.(Cluster.java:89)
> at
> org.apache.kafka.clients.MetadataCache.computeClusterView(MetadataCache.java:120)
> at org.apache.kafka.clients.MetadataCache.(MetadataCache.java:82)
> at org.apache.kafka.clients.MetadataCache.(MetadataCache.java:58)
> at
> org.apache.kafka.clients.Metadata.handleMetadataResponse(Metadata.java:325)
> at org.apache.kafka.clients.Metadata.update(Metadata.java:252)
> at
> org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.handleCompletedMetadataResponse(NetworkClient.java:1059)
> at
> org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:845)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:548)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:262)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1281)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1225)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1201)
> {code}
> The client assumes that if a leader is included in the response, then node
> information must also be available. There are at least a couple possible
> reasons this assumption can fail:
> 1. The client is able to detect stale partition metadata using leader epoch
> information available. If stale partition metadata is detected, the client
> ignores it and uses the last known metadata. However, it cannot detect stale
> broker information and will always accept the latest update. This means that
> the latest metadata may be a mix of multiple metadata responses and therefore
> the invariant will not generally hold.
> 2. There is no lock which protects both the fetching of partition metadata
> and the live broker when handling a Metadata request. This means an
> UpdateMetadata request can arrive concurrently and break the intended
> invariant.
> It seems case 2 has been possible for a long time, but it should be extremely
> rare. Case 1 was only made possible with KIP-320, which added the leader
> epoch tracking. It should also be rare, but the window for inconsistent
> metadata is probably a bit bigger than the window for a concurrent update.
> To fix this, we should make the client more defensive about metadata updates
> and not assume that the leader is among the live endpoints.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KAFKA-9267) ZkSecurityMigrator should not create /controller node
[
https://issues.apache.org/jira/browse/KAFKA-9267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-9267.
--
Fix Version/s: 2.5.0
Resolution: Fixed
Issue resolved by pull request 7778
[https://github.com/apache/kafka/pull/7778]
> ZkSecurityMigrator should not create /controller node
> -
>
> Key: KAFKA-9267
> URL: https://issues.apache.org/jira/browse/KAFKA-9267
> Project: Kafka
> Issue Type: Bug
> Components: admin
>Reporter: NanerLee
>Priority: Major
> Fix For: 2.5.0
>
>
> As we can see in these source codes – [ZkSecurityMigrator.scala#L226|#L226]
> _ZkSecurityMigrator_ checks and sets acl recursively for each path in
> _SecureRootPaths_. And _/controller_ is also in _SecureRootPaths_.
> As we can predicted, _zkClient.makeSurePersistentPathExists()_ will create
> _/controller_ node if _/controller_ is not existed.
> _/controller_ is a *EPHEMERAL* node for controller election, but
> _makeSurePersistentPathExists()_ will create a *PERSISTENT* node with *null*
> data.
> If that happens, null data will cause a *NPE*, and the controller cannot be
> elected, kafka cluster will be unavailable .
> In addition, a *PERSISTENT* node doesn't disappear automatically, we have to
> delete it manually to fix the problem.
>
> *PERSISTENT* _/controller_ node with *null* data in zk:
> {code:java}
> [zk: localhost:2181(CONNECTED) 16] get /kafka/controller
> null
> cZxid = 0x112284
> ctime = Tue Dec 03 18:37:26 CST 2019
> mZxid = 0x112284
> mtime = Tue Dec 03 18:37:26 CST 2019
> pZxid = 0x112284
> cversion = 0
> dataVersion = 0
> aclVersion = 1
> ephemeralOwner = 0x0
> dataLength = 0
> numChildren = 0{code}
> *Normal* /controller node in zk:
> {code:java}
> [zk: localhost:2181(CONNECTED) 21] get /kafka/controller
> {"version":1,"brokerid":1001,"timestamp":"1575370170528"}
> cZxid = 0x1123e1
> ctime = Tue Dec 03 18:49:30 CST 2019
> mZxid = 0x1123e1
> mtime = Tue Dec 03 18:49:30 CST 2019
> pZxid = 0x1123e1
> cversion = 0
> dataVersion = 0
> aclVersion = 0
> ephemeralOwner = 0x16ecb572df50021
> dataLength = 57
> numChildren = 0{code}
> *NPE* in controller.log :
> {code:java}
> [2019-11-21 15:02:41,276] INFO [ControllerEventThread controllerId=1002]
> Starting (kafka.controller.ControllerEventManager$ControllerEventThread)
> [2019-11-21 15:02:41,282] ERROR [ControllerEventThread controllerId=1002]
> Error processing event Startup
> (kafka.controller.ControllerEventManager$ControllerEventThread)
> java.lang.NullPointerException
> at com.fasterxml.jackson.core.JsonFactory.createParser(JsonFactory.java:857)
> at
> com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2572)
> at kafka.utils.Json$.parseBytes(Json.scala:62)
> at kafka.zk.ControllerZNode$.decode(ZkData.scala:56)
> at kafka.zk.KafkaZkClient.getControllerId(KafkaZkClient.scala:902)
> at
> kafka.controller.KafkaController.kafka$controller$KafkaController$$elect(KafkaController.scala:1199)
> at
> kafka.controller.KafkaController$Startup$.process(KafkaController.scala:1148)
> at
> kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply$mcV$sp(ControllerEventManager.scala:86)
> at
> kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply(ControllerEventManager.scala:86)
> at
> kafka.controller.ControllerEventManager$ControllerEventThread$$anonfun$doWork$1.apply(ControllerEventManager.scala:86)
> at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:31)
> at
> kafka.controller.ControllerEventManager$ControllerEventThread.doWork(ControllerEventManager.scala:85)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82){code}
>
> So, I submit a PR that _ZkSecurityMigrator_ will not handle _/controller_
> node when _/controller_ is not existed.
> This bug seems to affect all versions, please review and merge the PR as soon
> as possible.
> Thanks!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Reopened] (KAFKA-9261) NPE when updating client metadata
[
https://issues.apache.org/jira/browse/KAFKA-9261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar reopened KAFKA-9261:
--
> NPE when updating client metadata
> -
>
> Key: KAFKA-9261
> URL: https://issues.apache.org/jira/browse/KAFKA-9261
> Project: Kafka
> Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Major
> Fix For: 2.4.0, 2.3.2
>
>
> We have seen the following exception recently:
> {code}
> java.lang.NullPointerException
> at java.base/java.util.Objects.requireNonNull(Objects.java:221)
> at org.apache.kafka.common.Cluster.(Cluster.java:134)
> at org.apache.kafka.common.Cluster.(Cluster.java:89)
> at
> org.apache.kafka.clients.MetadataCache.computeClusterView(MetadataCache.java:120)
> at org.apache.kafka.clients.MetadataCache.(MetadataCache.java:82)
> at org.apache.kafka.clients.MetadataCache.(MetadataCache.java:58)
> at
> org.apache.kafka.clients.Metadata.handleMetadataResponse(Metadata.java:325)
> at org.apache.kafka.clients.Metadata.update(Metadata.java:252)
> at
> org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.handleCompletedMetadataResponse(NetworkClient.java:1059)
> at
> org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:845)
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:548)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:262)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1281)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1225)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1201)
> {code}
> The client assumes that if a leader is included in the response, then node
> information must also be available. There are at least a couple possible
> reasons this assumption can fail:
> 1. The client is able to detect stale partition metadata using leader epoch
> information available. If stale partition metadata is detected, the client
> ignores it and uses the last known metadata. However, it cannot detect stale
> broker information and will always accept the latest update. This means that
> the latest metadata may be a mix of multiple metadata responses and therefore
> the invariant will not generally hold.
> 2. There is no lock which protects both the fetching of partition metadata
> and the live broker when handling a Metadata request. This means an
> UpdateMetadata request can arrive concurrently and break the intended
> invariant.
> It seems case 2 has been possible for a long time, but it should be extremely
> rare. Case 1 was only made possible with KIP-320, which added the leader
> epoch tracking. It should also be rare, but the window for inconsistent
> metadata is probably a bit bigger than the window for a concurrent update.
> To fix this, we should make the client more defensive about metadata updates
> and not assume that the leader is among the live endpoints.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KAFKA-9123) Add system test with large number of partitions
[ https://issues.apache.org/jira/browse/KAFKA-9123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-9123. -- Resolution: Fixed > Add system test with large number of partitions > --- > > Key: KAFKA-9123 > URL: https://issues.apache.org/jira/browse/KAFKA-9123 > Project: Kafka > Issue Type: Test > Components: system tests >Reporter: David Arthur >Assignee: David Arthur >Priority: Major > Fix For: 2.4.0 > > > Let's add a system test with several thousand replicas and do some validation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9253) Test failure : ReassignPartitionsClusterTest.shouldListMovingPartitionsThroughApi
Manikumar created KAFKA-9253:
Summary: Test failure :
ReassignPartitionsClusterTest.shouldListMovingPartitionsThroughApi
Key: KAFKA-9253
URL: https://issues.apache.org/jira/browse/KAFKA-9253
Project: Kafka
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Manikumar
https://builds.apache.org/blue/organizations/jenkins/kafka-trunk-jdk11/detail/kafka-trunk-jdk11/990/tests
{quote}
java.lang.NullPointerException
Stacktrace
java.lang.NullPointerException
at
kafka.admin.ReassignPartitionsClusterTest.assertIsReassigning(ReassignPartitionsClusterTest.scala:1188)
at
kafka.admin.ReassignPartitionsClusterTest.shouldListMovingPartitionsThroughApi(ReassignPartitionsClusterTest.scala:782)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
at
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:65)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
at org.junit.runners.ParentRunner.run(ParentRunner.java:412)
{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KAFKA-9069) Flaky Test AdminClientIntegrationTest.testCreatePartitions
[
https://issues.apache.org/jira/browse/KAFKA-9069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-9069.
--
Fix Version/s: 2.5.0
Resolution: Fixed
Issue resolved by pull request 7619
[https://github.com/apache/kafka/pull/7619]
> Flaky Test AdminClientIntegrationTest.testCreatePartitions
> --
>
> Key: KAFKA-9069
> URL: https://issues.apache.org/jira/browse/KAFKA-9069
> Project: Kafka
> Issue Type: Bug
> Components: admin, core, unit tests
>Reporter: Matthias J. Sax
>Assignee: huxihx
>Priority: Major
> Labels: flaky-test
> Fix For: 2.5.0
>
>
> [https://builds.apache.org/job/kafka-pr-jdk11-scala2.13/2792/testReport/junit/kafka.api/AdminClientIntegrationTest/testCreatePartitions/]
> {quote}java.lang.AssertionError: validateOnly expected:<3> but was:<1> at
> org.junit.Assert.fail(Assert.java:89) at
> org.junit.Assert.failNotEquals(Assert.java:835) at
> org.junit.Assert.assertEquals(Assert.java:647) at
> kafka.api.AdminClientIntegrationTest.$anonfun$testCreatePartitions$5(AdminClientIntegrationTest.scala:651)
> at
> kafka.api.AdminClientIntegrationTest.$anonfun$testCreatePartitions$5$adapted(AdminClientIntegrationTest.scala:601)
> at scala.collection.immutable.List.foreach(List.scala:305) at
> kafka.api.AdminClientIntegrationTest.testCreatePartitions(AdminClientIntegrationTest.scala:601){quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KAFKA-9046) Connect worker configs require undocumented 'admin.' prefix to configure DLQ for connectors
[
https://issues.apache.org/jira/browse/KAFKA-9046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-9046.
--
Fix Version/s: 2.4.0
2.3.2
Resolution: Fixed
Issue resolved by pull request 7525
[https://github.com/apache/kafka/pull/7525]
> Connect worker configs require undocumented 'admin.' prefix to configure DLQ
> for connectors
> ---
>
> Key: KAFKA-9046
> URL: https://issues.apache.org/jira/browse/KAFKA-9046
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.3.0, 2.4.0, 2.3.1
>Reporter: Chris Egerton
>Assignee: Chris Egerton
>Priority: Blocker
> Fix For: 2.3.2, 2.4.0
>
>
> The changes for KAFKA-8265 involved [adding a prefix of "admin." to Connect
> worker
> configs|https://github.com/apache/kafka/pull/6624/files#diff-316d2c222b623ee65e8065863bf4b9ceR606]
> that would be used to configure the admin client that's used for connector
> DLQs. However, this was never documented in the [corresponding
> KIP|https://cwiki.apache.org/confluence/display/KAFKA/KIP-458%3A+Connector+Client+Config+Override+Policy]
> and has broken backwards compatibility with prior Connect releases since
> workers without the necessary {{"admin."}}-prefixed properties in their
> configuration files will now fail in some circumstances (e.g., when
> interacting with a secured Kafka cluster that requires authentication from
> all admin clients that interact with it).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-9176) Flaky test failure: OptimizedKTableIntegrationTest.shouldApplyUpdatesToStandbyStore
Manikumar created KAFKA-9176: Summary: Flaky test failure: OptimizedKTableIntegrationTest.shouldApplyUpdatesToStandbyStore Key: KAFKA-9176 URL: https://issues.apache.org/jira/browse/KAFKA-9176 Project: Kafka Issue Type: Bug Affects Versions: 2.4.0 Reporter: Manikumar h4. [https://builds.apache.org/blue/organizations/jenkins/kafka-2.4-jdk8/detail/kafka-2.4-jdk8/65/tests] h4. Error org.apache.kafka.streams.errors.InvalidStateStoreException: Cannot get state store source-table because the stream thread is PARTITIONS_ASSIGNED, not RUNNING h4. Stacktrace org.apache.kafka.streams.errors.InvalidStateStoreException: Cannot get state store source-table because the stream thread is PARTITIONS_ASSIGNED, not RUNNING at org.apache.kafka.streams.state.internals.StreamThreadStateStoreProvider.stores(StreamThreadStateStoreProvider.java:51) at org.apache.kafka.streams.state.internals.QueryableStoreProvider.getStore(QueryableStoreProvider.java:59) at org.apache.kafka.streams.KafkaStreams.store(KafkaStreams.java:1129) at org.apache.kafka.streams.integration.OptimizedKTableIntegrationTest.shouldApplyUpdatesToStandbyStore(OptimizedKTableIntegrationTest.java:157) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:65) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305) at org.junit.runners.ParentRunner.run(ParentRunner.java:412) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38) at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62) at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51) at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33) at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94) at com.sun.proxy.$Proxy2.processTestClass(Unknown Source) at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118) at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182) at
[jira] [Resolved] (KAFKA-8984) Improve tagged fields documentation
[ https://issues.apache.org/jira/browse/KAFKA-8984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-8984. -- Fix Version/s: 2.4.0 Resolution: Fixed Issue resolved by pull request 7477 [https://github.com/apache/kafka/pull/7477] > Improve tagged fields documentation > --- > > Key: KAFKA-8984 > URL: https://issues.apache.org/jira/browse/KAFKA-8984 > Project: Kafka > Issue Type: Improvement > Components: protocol >Reporter: Colin McCabe >Assignee: Colin McCabe >Priority: Minor > Fix For: 2.4.0 > > > We should improve the KIP-482 tagged fields documentation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-8677) Flakey test GroupEndToEndAuthorizationTest#testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl
[
https://issues.apache.org/jira/browse/KAFKA-8677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-8677.
--
Assignee: Guozhang Wang
Resolution: Fixed
> Flakey test
> GroupEndToEndAuthorizationTest#testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl
>
>
> Key: KAFKA-8677
> URL: https://issues.apache.org/jira/browse/KAFKA-8677
> Project: Kafka
> Issue Type: Bug
> Components: core, security, unit tests
>Affects Versions: 2.4.0
>Reporter: Boyang Chen
>Assignee: Guozhang Wang
>Priority: Blocker
> Labels: flaky-test
> Fix For: 2.4.0
>
>
> [https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/6325/console]
>
> *18:43:39* kafka.api.GroupEndToEndAuthorizationTest >
> testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl STARTED*18:44:00*
> kafka.api.GroupEndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl
> failed, log available in
> /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk11-scala2.12/core/build/reports/testOutput/kafka.api.GroupEndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl.test.stdout*18:44:00*
> *18:44:00* kafka.api.GroupEndToEndAuthorizationTest >
> testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl FAILED*18:44:00*
> org.scalatest.exceptions.TestFailedException: Consumed 0 records before
> timeout instead of the expected 1 records
> ---
> I found this flaky test is actually exposing a real bug in consumer: within
> {{KafkaConsumer.poll}}, we have an optimization to try to send the next fetch
> request before returning the data in order to pipelining the fetch requests:
> {code}
> if (!records.isEmpty()) {
> // before returning the fetched records, we can send off
> the next round of fetches
> // and avoid block waiting for their responses to enable
> pipelining while the user
> // is handling the fetched records.
> //
> // NOTE: since the consumed position has already been
> updated, we must not allow
> // wakeups or any other errors to be triggered prior to
> returning the fetched records.
> if (fetcher.sendFetches() > 0 ||
> client.hasPendingRequests()) {
> client.pollNoWakeup();
> }
> return this.interceptors.onConsume(new
> ConsumerRecords<>(records));
> }
> {code}
> As the NOTE mentioned, this pollNoWakeup should NOT throw any exceptions,
> since at this point the fetch position has been updated. If an exception is
> thrown here, and the callers decides to capture and continue, those records
> would never be returned again, causing data loss.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KAFKA-9150) DescribeGroup uses member assignment as metadata
[ https://issues.apache.org/jira/browse/KAFKA-9150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-9150. -- Resolution: Fixed > DescribeGroup uses member assignment as metadata > > > Key: KAFKA-9150 > URL: https://issues.apache.org/jira/browse/KAFKA-9150 > Project: Kafka > Issue Type: Bug >Reporter: Jason Gustafson >Assignee: David Jacot >Priority: Blocker > Fix For: 2.4.0 > > > When we converted the DescribeGroup internally to rely on the generated > protocol in KAFKA-7922, we introduced a regression in the response handling. > Basically we serialize the member assignment as both the assignment and > metadata in the response: > [https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaApis.scala#L1326]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-9140) Consumer gets stuck rejoining the group indefinitely
[
https://issues.apache.org/jira/browse/KAFKA-9140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-9140.
--
Assignee: Guozhang Wang
Resolution: Fixed
> Consumer gets stuck rejoining the group indefinitely
>
>
> Key: KAFKA-9140
> URL: https://issues.apache.org/jira/browse/KAFKA-9140
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer
>Affects Versions: 2.4.0
>Reporter: Sophie Blee-Goldman
>Assignee: Guozhang Wang
>Priority: Blocker
> Fix For: 2.4.0
>
> Attachments: debug.tgz, info.tgz, kafka-data-logs-1.tgz,
> kafka-data-logs-2.tgz, server-start-stdout-stderr.log.tgz, streams.log.tgz
>
>
> There seems to be a race condition that is now causing a rejoining member to
> potentially get stuck infinitely initiating a rejoin. The relevant client
> logs are attached (streams.log.tgz; all others attachments are broker logs),
> but basically it repeats this message (and nothing else) continuously until
> killed/shutdown:
>
> {code:java}
> [2019-11-05 01:53:54,699] INFO [Consumer
> clientId=StreamsUpgradeTest-a4c1cff8-7883-49cd-82da-d2cdfc33a2f0-StreamThread-1-consumer,
> groupId=StreamsUpgradeTest] Generation data was cleared by heartbeat thread.
> Initiating rejoin.
> (org.apache.kafka.clients.consumer.internals.AbstractCoordinator)
> {code}
>
> The message that appears was added as part of the bugfix ([PR
> 7460|https://github.com/apache/kafka/pull/7460]) for this related race
> condition: KAFKA-8104.
> This issue was uncovered by the Streams version probing upgrade test, which
> fails with a varying frequency. Here is the rate of failures for different
> system test runs so far:
> trunk (cooperative): 1/1 and 2/10 failures
> 2.4 (cooperative) : 0/10 and 1/15 failures
> trunk (eager): 0/10 failures
> I've kicked off some high-repeat runs to complete overnight and hopefully
> shed more light.
> Note that I have also kicked off runs of both 2.4 and trunk with the PR for
> KAFKA-8104 reverted. Both of them saw 2/10 failures, due to hitting the bug
> that was fixed by [PR 7460|https://github.com/apache/kafka/pull/7460]. It is
> therefore unclear whether [PR 7460|https://github.com/apache/kafka/pull/7460]
> introduced another or a new race condition/bug, or merely uncovered an
> existing one that previously would have first failed due to KAFKA-8104.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KAFKA-9079) System Test Failure: TransactionsTest
[ https://issues.apache.org/jira/browse/KAFKA-9079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-9079. -- Fix Version/s: (was: 2.5.0) 2.4.0 Resolution: Fixed Issue resolved by pull request 7653 [https://github.com/apache/kafka/pull/7653] > System Test Failure: TransactionsTest > - > > Key: KAFKA-9079 > URL: https://issues.apache.org/jira/browse/KAFKA-9079 > Project: Kafka > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Manikumar >Priority: Major > Fix For: 2.4.0 > > > TransactionsTest tests are failing on 2.4 and trunk. > http://confluent-kafka-2-4-system-test-results.s3-us-west-2.amazonaws.com/2019-10-21--001.1571716233--confluentinc--2.4--cb4944f/report.html -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-9080) System Test Failure: MessageFormatChangeTest.testCompatibilty
[
https://issues.apache.org/jira/browse/KAFKA-9080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-9080.
--
Resolution: Fixed
> System Test Failure: MessageFormatChangeTest.testCompatibilty
> -
>
> Key: KAFKA-9080
> URL: https://issues.apache.org/jira/browse/KAFKA-9080
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: Manikumar
>Assignee: Tu Tran
>Priority: Blocker
> Fix For: 2.4.0
>
>
> MessageFormatChangeTest tests are failing on 2.4 and trunk for 0.9.0.1
> version.
> http://confluent-kafka-2-4-system-test-results.s3-us-west-2.amazonaws.com/2019-10-21--001.1571716233--confluentinc--2.4--cb4944f/report.html
> {code}
> Module: kafkatest.tests.client.message_format_change_test
> Class: MessageFormatChangeTest
> Method: test_compatibility
> Arguments:
> {
> "consumer_version": "0.9.0.1",
> "producer_version": "0.9.0.1"
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KAFKA-9078) System Test Failure: ConnectRestApiTest
[ https://issues.apache.org/jira/browse/KAFKA-9078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-9078. -- Resolution: Fixed > System Test Failure: ConnectRestApiTest > --- > > Key: KAFKA-9078 > URL: https://issues.apache.org/jira/browse/KAFKA-9078 > Project: Kafka > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Manikumar >Assignee: Konstantine Karantasis >Priority: Blocker > Fix For: 2.4.0 > > > ConnectRestApiTest tests are failing on 2.4 and trunk. > http://confluent-kafka-2-4-system-test-results.s3-us-west-2.amazonaws.com/2019-10-21--001.1571716233--confluentinc--2.4--cb4944f/report.html -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9080) System Test Failure: MessageFormatChangeTest.testCompatibilty
Manikumar created KAFKA-9080:
Summary: System Test Failure:
MessageFormatChangeTest.testCompatibilty
Key: KAFKA-9080
URL: https://issues.apache.org/jira/browse/KAFKA-9080
Project: Kafka
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Manikumar
Fix For: 2.4.0
MessageFormatChangeTest tests are failing on 2.4 and trunk for 0.9.0.1 version.
http://confluent-kafka-2-4-system-test-results.s3-us-west-2.amazonaws.com/2019-10-21--001.1571716233--confluentinc--2.4--cb4944f/report.html
{code}
Module: kafkatest.tests.client.message_format_change_test
Class: MessageFormatChangeTest
Method: test_compatibility
Arguments:
{
"consumer_version": "0.9.0.1",
"producer_version": "0.9.0.1"
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-9079) System Test Failure: TransactionsTest
Manikumar created KAFKA-9079: Summary: System Test Failure: TransactionsTest Key: KAFKA-9079 URL: https://issues.apache.org/jira/browse/KAFKA-9079 Project: Kafka Issue Type: Bug Affects Versions: 2.4.0 Reporter: Manikumar Fix For: 2.4.0 TransactionsTest tests are failing on 2.4 and trunk. http://confluent-kafka-2-4-system-test-results.s3-us-west-2.amazonaws.com/2019-10-21--001.1571716233--confluentinc--2.4--cb4944f/report.html -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9078) System Test Failure: ConnectRestApiTest
Manikumar created KAFKA-9078: Summary: System Test Failure: ConnectRestApiTest Key: KAFKA-9078 URL: https://issues.apache.org/jira/browse/KAFKA-9078 Project: Kafka Issue Type: Bug Affects Versions: 2.4.0 Reporter: Manikumar Fix For: 2.4.0 ConnectRestApiTest tests are failing on 2.4 and trunk. http://confluent-kafka-2-4-system-test-results.s3-us-west-2.amazonaws.com/2019-10-21--001.1571716233--confluentinc--2.4--cb4944f/report.html -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KAFKA-9077) System Test Failure: StreamsSimpleBenchmarkTest
Manikumar created KAFKA-9077: Summary: System Test Failure: StreamsSimpleBenchmarkTest Key: KAFKA-9077 URL: https://issues.apache.org/jira/browse/KAFKA-9077 Project: Kafka Issue Type: Bug Affects Versions: 2.4.0 Reporter: Manikumar Fix For: 2.4.0 StreamsSimpleBenchmarkTest tests are failing on 2.4 and trunk. http://confluent-kafka-2-4-system-test-results.s3-us-west-2.amazonaws.com/2019-10-21--001.1571716233--confluentinc--2.4--cb4944f/report.html -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-8482) alterReplicaLogDirs should be better documented
[
https://issues.apache.org/jira/browse/KAFKA-8482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-8482.
--
Fix Version/s: (was: 2.4.0)
2.5.0
Resolution: Fixed
> alterReplicaLogDirs should be better documented
> ---
>
> Key: KAFKA-8482
> URL: https://issues.apache.org/jira/browse/KAFKA-8482
> Project: Kafka
> Issue Type: Improvement
> Components: admin
>Reporter: Colin McCabe
>Assignee: Dongjin Lee
>Priority: Major
> Labels: newbie
> Fix For: 2.5.0
>
>
> alterReplicaLogDirs should be better documented. In particular, it should
> document what exceptions it throws in {{AdminClient.java}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KAFKA-6263) Expose metric for group metadata loading duration
[ https://issues.apache.org/jira/browse/KAFKA-6263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-6263. -- Fix Version/s: 2.4.0 Resolution: Fixed > Expose metric for group metadata loading duration > - > > Key: KAFKA-6263 > URL: https://issues.apache.org/jira/browse/KAFKA-6263 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: Jason Gustafson >Assignee: Anastasia Vela >Priority: Major > Fix For: 2.4.0 > > > We have seen in several cases where the log cleaner either wasn't enabled or > had experienced some failure that __consumer_offsets partitions can grow > excessively. When one of these partitions changes leadership, the new > coordinator must load the offset cache from the start of the log, which can > take arbitrarily long depending on how large the partition has grown (we have > seen cases where it took hours). Catching this problem is not always easy > because the condition is rare and the symptom just tends to be a long period > of inactivity in the consumer group which gradually gets worse over time. It > may therefore be useful to have a broker metric for the load time so that it > can be monitored and potentially alerted on. Same thing goes for the > transaction log -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-7471) Multiple Consumer Group Management (Describe, Reset, Delete)
[ https://issues.apache.org/jira/browse/KAFKA-7471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-7471. -- Fix Version/s: 2.4.0 Resolution: Fixed > Multiple Consumer Group Management (Describe, Reset, Delete) > > > Key: KAFKA-7471 > URL: https://issues.apache.org/jira/browse/KAFKA-7471 > Project: Kafka > Issue Type: New Feature > Components: tools >Affects Versions: 1.0.0, 2.0.0 >Reporter: Alex Dunayevsky >Assignee: Alex Dunayevsky >Priority: Major > Fix For: 2.4.0 > > > Functionality needed: > * Describe/Delete/Reset offsets on multiple consumer groups at a time > (including each group by repeating `--group` parameter) > * Describe/Delete/Reset offsets on ALL consumer groups at a time (add new > --groups-all option similar to --topics-all) > * Generate CSV for multiple consumer groups > What are the benifits? > * No need to start a new JVM to perform each query on every single consumer > group > * Abiltity to query groups by their status (for instance, `-v grepping` by > `Stable` to spot problematic/dead/empty groups) > * Ability to export offsets to reset for multiple consumer groups to a CSV > file (needs CSV generation export/import format rework) > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-7800) Extend Admin API to support dynamic application log levels
[
https://issues.apache.org/jira/browse/KAFKA-7800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-7800.
--
Fix Version/s: 2.4.0
Resolution: Fixed
> Extend Admin API to support dynamic application log levels
> --
>
> Key: KAFKA-7800
> URL: https://issues.apache.org/jira/browse/KAFKA-7800
> Project: Kafka
> Issue Type: New Feature
>Reporter: Stanislav Kozlovski
>Assignee: Stanislav Kozlovski
>Priority: Major
> Fix For: 2.4.0
>
>
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-412%3A+Extend+Admin+API+to+support+dynamic+application+log+levels]
> Logging is a critical part of any system's infrastructure. It is the most
> direct way of observing what is happening with a system. In the case of
> issues, it helps us diagnose the problem quickly which in turn helps lower
> the
> [MTTR|http://enterprisedevops.org/article/devops-metric-mean-time-to-recovery-mttr-definition-and-reasoning].
> Kafka supports application logging via the log4j library and outputs messages
> in various log levels (TRACE, DEBUG, INFO, WARN, ERROR). Log4j is a rich
> library that supports fine-grained logging configurations (e.g use INFO-level
> logging in {{kafka.server.ReplicaManager}} and use DEBUG-level in
> {{kafka.server.KafkaApis}}).
> This is statically configurable through the
> [log4j.properties|https://github.com/apache/kafka/blob/trunk/config/log4j.properties]
> file which gets read once at broker start-up.
> A problem with this static configuration is that we cannot alter the log
> levels when a problem arises. It is severely impractical to edit a properties
> file and restart all brokers in order to gain visibility of a problem taking
> place in production.
> It would be very useful if we support dynamically altering the log levels at
> runtime without needing to restart the Kafka process.
> Log4j itself supports dynamically altering the log levels in a programmatic
> way and Kafka exposes a [JMX
> API|https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/utils/Log4jController.scala]
> that lets you alter them. This allows users to change the log levels via a
> GUI (jconsole) or a CLI (jmxterm) that uses JMX.
> There is one problem with changing log levels through JMX that we hope to
> address and that is *Ease of Use*:
> * Establishing a connection - Connecting to a remote process via JMX
> requires configuring and exposing multiple JMX ports to the outside world.
> This is a burden on users, as most production deployments may stand behind
> layers of firewalls and have policies against opening ports. This makes
> opening the ports and connections in the middle of an incident even more
> burdensome
> * Security - JMX and tools around it support authentication and
> authorization but it is an additional hassle to set up credentials for
> another system.
> * Manual process - Changing the whole cluster's log level requires manually
> connecting to each broker. In big deployments, this is severely impractical
> and forces users to build tooling around it.
> h4. Proposition
> Ideally, Kafka would support dynamically changing log levels and address all
> of the aforementioned concerns out of the box.
> We propose extending the IncrementalAlterConfig/DescribeConfig Admin API with
> functionality for dynamically altering the broker's log level.
> This approach would also pave the way for even finer-grained logging logic
> (e.g log DEBUG level only for a certain topic) and would allow us to leverage
> the existing *AlterConfigPolicy* for custom user-defined validation of
> log-level changes.
> These log-level changes will be *temporary* and reverted on broker restart -
> we will not persist them anywhere.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KAFKA-8696) Clean up Sum/Count/Total metrics
[ https://issues.apache.org/jira/browse/KAFKA-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-8696. -- Resolution: Fixed > Clean up Sum/Count/Total metrics > > > Key: KAFKA-8696 > URL: https://issues.apache.org/jira/browse/KAFKA-8696 > Project: Kafka > Issue Type: Improvement >Reporter: John Roesler >Assignee: John Roesler >Priority: Minor > Fix For: 2.4.0 > > > Kafka has a family of metrics consisting of: > org.apache.kafka.common.metrics.stats.Count > org.apache.kafka.common.metrics.stats.Sum > org.apache.kafka.common.metrics.stats.Total > org.apache.kafka.common.metrics.stats.Rate.SampledTotal > org.apache.kafka.streams.processor.internals.metrics.CumulativeCount > These metrics are all related to each other, but their relationship is > obscure (and one is redundant) (and another is internal). > I've recently been involved in a third recapitulation of trying to work out > which metric does what. It seems like it's time to clean up the mess and save > everyone from having to work out the mystery for themselves. > I've proposed https://cwiki.apache.org/confluence/x/kkAyBw to fix it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-8669) Add java security providers in Kafka Security config
[
https://issues.apache.org/jira/browse/KAFKA-8669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-8669.
--
Fix Version/s: 2.4.0
Assignee: Sai Sandeep
Resolution: Fixed
Fixed in https://github.com/apache/kafka/pull/7090
> Add java security providers in Kafka Security config
>
>
> Key: KAFKA-8669
> URL: https://issues.apache.org/jira/browse/KAFKA-8669
> Project: Kafka
> Issue Type: Improvement
>Reporter: Sai Sandeep
>Assignee: Sai Sandeep
>Priority: Minor
> Fix For: 2.4.0
>
>
> Currently kafka supports ssl.keymanager.algorithm and
> ssl.trustmanager.algorithm parameters as part of secure config. These
> parameters can be configured to load the key manager and trust managers which
> provide keys and certificates for ssl handshakes with the clients/server. The
> algorithms configured by parameters need to be registered by Java security
> provider classes. These provider classes are configured as JVM properties
> through java.security file. An example file given below
> {code:java}
> $ cat /usr/lib/jvm/jdk-8-oracle-x64/jre/lib/security/java.security
> ...
> security.provider.1=sun.security.provider.Sun
> security.provider.2=sun.security.rsa.SunRsaSign
> security.provider.3=sun.security.ec.SunEC
> …
> {code}
> Custom keymanager and trustmanager algorithms can be used to supply the kafka
> brokers with keys and certificates, these algorithms can be used to replace
> the traditional, non-scalable static keystore and truststore jks files.
> To take advantage of these custom algorithms, we want to support java
> security provider parameter in security config. This param can be used by
> kafka brokers or kafka clients(when connecting to the kafka brokers). The
> security providers can also be used for configuring security in SASL based
> communication too.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KAFKA-8907) Return topic configs in CreateTopics response
[ https://issues.apache.org/jira/browse/KAFKA-8907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-8907. -- Reviewer: Manikumar Reddy Resolution: Fixed > Return topic configs in CreateTopics response > -- > > Key: KAFKA-8907 > URL: https://issues.apache.org/jira/browse/KAFKA-8907 > Project: Kafka > Issue Type: New Feature > Components: clients >Reporter: Rajini Sivaram >Assignee: Rajini Sivaram >Priority: Major > Fix For: 2.4.0 > > > See > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-525+-+Return+topic+metadata+and+configs+in+CreateTopics+response] > for details -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-6883) KafkaShortnamer should allow to convert Kerberos principal name to upper case user name
[ https://issues.apache.org/jira/browse/KAFKA-6883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-6883. -- Fix Version/s: 2.4.0 Resolution: Fixed Issue resolved by pull request 7375 [https://github.com/apache/kafka/pull/7375] > KafkaShortnamer should allow to convert Kerberos principal name to upper case > user name > --- > > Key: KAFKA-6883 > URL: https://issues.apache.org/jira/browse/KAFKA-6883 > Project: Kafka > Issue Type: Improvement >Reporter: Attila Sasvári >Assignee: Manikumar >Priority: Major > Fix For: 2.4.0 > > > KAFKA-5764 implemented support to convert Kerberos principal name to lower > case Linux user name via auth_to_local rules. > As a follow-up, KafkaShortnamer could be further extended to allow converting > principal names to uppercase by appending /U to the rule. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-8892) Display the sorted configs in Kafka Configs Help Command.
[ https://issues.apache.org/jira/browse/KAFKA-8892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-8892. -- Fix Version/s: 2.4.0 Resolution: Fixed Issue resolved by pull request 7319 [https://github.com/apache/kafka/pull/7319] > Display the sorted configs in Kafka Configs Help Command. > - > > Key: KAFKA-8892 > URL: https://issues.apache.org/jira/browse/KAFKA-8892 > Project: Kafka > Issue Type: Bug >Reporter: Kamal Chandraprakash >Assignee: Kamal Chandraprakash >Priority: Minor > Fix For: 2.4.0 > > > Configs that can be updated dynamically for topics/brokers/users/clients are > shown in the help command. Only the topic configs are sorted alphabetically. > It will be helpful to do quick lookup if the brokers, users and clients > configs are also sorted. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KAFKA-8860) SslPrincipalMapper should handle distinguished names with spaces
[
https://issues.apache.org/jira/browse/KAFKA-8860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-8860.
--
Resolution: Fixed
Issue resolved by pull request 7140
[https://github.com/apache/kafka/pull/7140]
> SslPrincipalMapper should handle distinguished names with spaces
>
>
> Key: KAFKA-8860
> URL: https://issues.apache.org/jira/browse/KAFKA-8860
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Manikumar
>Priority: Major
> Fix For: 2.4.0
>
>
> This Jira is to track the issue reported by
> [t...@teebee.de|mailto:t...@teebee.de] in PR
> [#7140|https://github.com/apache/kafka/pull/7140]
> PR [#6099|https://github.com/apache/kafka/pull/6099] tried to undo the
> splitting of the {{ssl.principal.mapper.rules}}
> [list|https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaConfig.scala#L1054]
> on [comma with
> whitespace|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java#L78]
> by [sophisticated
> rejoining|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/ssl/SslPrincipalMapper.java#L42]
> of the split list using a comma as separator. However, since possibly
> surrounding whitespace is not reconstructed this approach fails in general.
> Consider the following test case:
> {code:java}
> @Test
> public void testCommaWithWhitespace() throws Exception \{
> String value = "RULE:^CN=((, *|\\w)+)(,.*|$)/$1/,DEFAULT";
> @SuppressWarnings("unchecked")
> List rules = (List)
> ConfigDef.parseType("ssl.principal.mapper.rules", value, Type.LIST);
> SslPrincipalMapper mapper = SslPrincipalMapper.fromRules(rules);
> assertEquals("Tkac\\, Adam", mapper.getName("CN=Tkac\\,
> Adam,OU=ITZ,DC=geodis,DC=cz"));
> }
> {code}
> The space after the escaped comma is
> [essential|https://sogo.nu/bugs/view.php?id=2152]. Unfortunately, it has
> disappeared after splitting and rejoining.
> Moreover, in
> [{{joinSplitRules}}|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/ssl/SslPrincipalMapper.java#L42]
> the decision to rejoin list elements is based on local information only
> which might not be sufficient. It works for
> {quote}"RULE:^CN=([^,ADEFLTU,]+)(,.*|$)/$1/"{quote} but fails for the
> _equivalent_ regular expression
> {quote}RULE:^CN=([^,DEFAULT,]+)(,.*|$)/$1/"{quote}
> The approach of the current PR is to change the type of the
> {{ssl.principal.mapper.rules}} attribute from
> [LIST|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java#L781]
> to
> [STRING|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java#L781]
> and to delegate the splitting of the rules to the
> [SslPrincipalMapper|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/ssl/SslPrincipalMapper.java].
> It knows about the structure of the rules and can perform the splitting
> context-based.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (KAFKA-8860) SslPrincipalMapper should handle distinguished names with spaces
Manikumar created KAFKA-8860:
Summary: SslPrincipalMapper should handle distinguished names with
spaces
Key: KAFKA-8860
URL: https://issues.apache.org/jira/browse/KAFKA-8860
Project: Kafka
Issue Type: Bug
Reporter: Manikumar
This Jira is to track the issue reported by
[t...@teebee.de|mailto:t...@teebee.de] in PR
[#7140|https://github.com/apache/kafka/pull/7140]
PR [#6099|https://github.com/apache/kafka/pull/6099] tried to undo the
splitting of the {{ssl.principal.mapper.rules}}
[list|https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaConfig.scala#L1054]
on [comma with
whitespace|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java#L78]
by [sophisticated
rejoining|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/ssl/SslPrincipalMapper.java#L42]
of the split list using a comma as separator. However, since possibly
surrounding whitespace is not reconstructed this approach fails in general.
Consider the following test case:
{code:java}
@Test
public void testCommaWithWhitespace() throws Exception \{
String value = "RULE:^CN=((, *|\\w)+)(,.*|$)/$1/,DEFAULT";
@SuppressWarnings("unchecked")
List rules = (List)
ConfigDef.parseType("ssl.principal.mapper.rules", value, Type.LIST);
SslPrincipalMapper mapper = SslPrincipalMapper.fromRules(rules);
assertEquals("Tkac\\, Adam", mapper.getName("CN=Tkac\\,
Adam,OU=ITZ,DC=geodis,DC=cz"));
}
{code}
The space after the escaped comma is
[essential|https://sogo.nu/bugs/view.php?id=2152]. Unfortunately, it has
disappeared after splitting and rejoining.
Moreover, in
[{{joinSplitRules}}|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/ssl/SslPrincipalMapper.java#L42]
the decision to rejoin list elements is based on local information only which
might not be sufficient. It works for
{{"RULE:^CN=([^,ADEFLTU,]+)(,.*|$)/$1/"*+}} *but fails for the _equivalent_
regular expression {{"RULE:^CN=([^,DEFAULT,])(,.}}*{{|$)/$1/"}}.
The approach of the current PR is to change the type of the
{{ssl.principal.mapper.rules}} attribute from
[LIST|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java#L781]
to
[STRING|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java#L781]
and to delegate the splitting of the rules to the
[SslPrincipalMapper|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/security/ssl/SslPrincipalMapper.java].
It knows about the structure of the rules and can perform the splitting
context-based.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Resolved] (KAFKA-8698) ListOffsets Response protocol documentation
[
https://issues.apache.org/jira/browse/KAFKA-8698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-8698.
--
Fix Version/s: 2.4.0
Resolution: Fixed
> ListOffsets Response protocol documentation
> ---
>
> Key: KAFKA-8698
> URL: https://issues.apache.org/jira/browse/KAFKA-8698
> Project: Kafka
> Issue Type: Bug
> Components: documentation
>Reporter: Fábio Silva
>Assignee: Asutosh Pandya
>Priority: Minor
> Labels: documentation, protocol-documentation
> Fix For: 2.4.0
>
>
> The documentation of ListOffsets Response (Version: 0) appears to have an
> typo on offsets field name, suffixed with `'`.
> {code:java}
> [offsets']{code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Resolved] (KAFKA-8598) Replace RenewDelegationToken request/response with automated protocol
[ https://issues.apache.org/jira/browse/KAFKA-8598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-8598. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 7038 [https://github.com/apache/kafka/pull/7038] > Replace RenewDelegationToken request/response with automated protocol > - > > Key: KAFKA-8598 > URL: https://issues.apache.org/jira/browse/KAFKA-8598 > Project: Kafka > Issue Type: Sub-task >Reporter: Mickael Maison >Assignee: Mickael Maison >Priority: Major > Fix For: 2.4.0 > > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (KAFKA-8599) Replace ExpireDelegationToken request/response with automated protocol
[ https://issues.apache.org/jira/browse/KAFKA-8599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-8599. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 7098 [https://github.com/apache/kafka/pull/7098] > Replace ExpireDelegationToken request/response with automated protocol > -- > > Key: KAFKA-8599 > URL: https://issues.apache.org/jira/browse/KAFKA-8599 > Project: Kafka > Issue Type: Sub-task >Reporter: Mickael Maison >Assignee: Mickael Maison >Priority: Major > Fix For: 2.4.0 > > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Resolved] (KAFKA-8390) Replace CreateDelegationToken request/response with automated protocol
[ https://issues.apache.org/jira/browse/KAFKA-8390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-8390. -- Resolution: Fixed Fix Version/s: 2.4.0 > Replace CreateDelegationToken request/response with automated protocol > -- > > Key: KAFKA-8390 > URL: https://issues.apache.org/jira/browse/KAFKA-8390 > Project: Kafka > Issue Type: Sub-task >Reporter: Mickael Maison >Assignee: Mickael Maison >Priority: Major > Fix For: 2.4.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-8536) Error creating ACL Alter Topic in 2.2
[ https://issues.apache.org/jira/browse/KAFKA-8536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-8536. -- Resolution: Fixed Assignee: Manikumar Fix Version/s: 2.1.2 2.0.2 Sorry, I forgot to backport the fix. backported the PR [https://github.com/apache/kafka/pull/6263] to 2.0, 2.1, 2.2 branches. > Error creating ACL Alter Topic in 2.2 > - > > Key: KAFKA-8536 > URL: https://issues.apache.org/jira/browse/KAFKA-8536 > Project: Kafka > Issue Type: Bug > Components: security >Affects Versions: 2.2.1 >Reporter: Alvaro Peris >Assignee: Manikumar >Priority: Critical > Fix For: 2.0.2, 2.1.2, 2.2.2 > > > When we try to execute the statement to create an Alter Topic ACL in version > 2.2 of > Kafka through the kafka-acls. > """ > kafka-acls --authorizer-properties > zookeeper.connect=fastdata-zk-discovery:2181 \ > --add \ > --allow-principal User:MyUser \ > --operation Alter \ > --topic topic \ > """ > We get the following error: > > ResourceType TOPIC only supports operations > > """ > Read,All,AlterConfigs,DescribeConfigs,Delete,Write,Create,Describe > """ > It should be possible to create an Alter Topic ACL, according to the > documentation. > Thanks -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-8461) Flakey test UncleanLeaderElectionTest#testUncleanLeaderElectionDisabledByTopicOverride
[ https://issues.apache.org/jira/browse/KAFKA-8461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-8461. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 6887 [https://github.com/apache/kafka/pull/6887] > Flakey test > UncleanLeaderElectionTest#testUncleanLeaderElectionDisabledByTopicOverride > -- > > Key: KAFKA-8461 > URL: https://issues.apache.org/jira/browse/KAFKA-8461 > Project: Kafka > Issue Type: Bug > Components: core >Reporter: Boyang Chen >Priority: Major > Fix For: 2.4.0 > > > [https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/5168/consoleFull] > *15:47:56* kafka.integration.UncleanLeaderElectionTest > > testUncleanLeaderElectionDisabledByTopicOverride FAILED*15:47:56* > org.scalatest.exceptions.TestFailedException: Timing out after 3 ms since > expected new leader 1 was not elected for partition > topic-9147891452427084986-0, leader is Some(-1)*15:47:56* at > org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:530)*15:47:56* > at > org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:529)*15:47:56* > at > org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1389)*15:47:56* > at org.scalatest.Assertions.fail(Assertions.scala:1091)*15:47:56* > at org.scalatest.Assertions.fail$(Assertions.scala:1087)*15:47:56* > at org.scalatest.Assertions$.fail(Assertions.scala:1389)*15:47:56* > at > kafka.utils.TestUtils$.$anonfun$waitUntilLeaderIsElectedOrChanged$8(TestUtils.scala:722)*15:47:56* > at scala.Option.getOrElse(Option.scala:138)*15:47:56* at > kafka.utils.TestUtils$.waitUntilLeaderIsElectedOrChanged(TestUtils.scala:712)*15:47:56* > at > kafka.integration.UncleanLeaderElectionTest.verifyUncleanLeaderElectionDisabled(UncleanLeaderElectionTest.scala:258)*15:47:56* > at > kafka.integration.UncleanLeaderElectionTest.testUncleanLeaderElectionDisabledByTopicOverride(UncleanLeaderElectionTest.scala:153)*15:47:56* -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-8152) Offline partition state not propagated by controller
[
https://issues.apache.org/jira/browse/KAFKA-8152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-8152.
--
Resolution: Duplicate
> Offline partition state not propagated by controller
>
>
> Key: KAFKA-8152
> URL: https://issues.apache.org/jira/browse/KAFKA-8152
> Project: Kafka
> Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Major
>
> Currently when the controller starts up, only the state of online partitions
> will be sent to other brokers. Any broker which is started or restarted after
> the controller will see only a subset of the partitions of any topic which
> has offline partitions. If all the partitions for a topic are offline, then
> the broker will not know of the topic at all. As far as I can tell, the bug
> is the fact that `ReplicaStateMachine.startup` only does an initial state
> change for replicas which are online.
> This can be reproduced with the following steps:
> # Startup two brokers
> # Create a single partition topic with rf=1
> # Shutdown the broker where the replica landed
> # Shutdown the other broker
> # Restart the broker without the replica
> # Run `kafka-topics --describe --bootstrap-server \{server ip}`
> Note that the metadata inconsistency will only be apparent when using
> `bootstrap-server` in `kafka-topics.sh`. Using zookeeper, everything will
> seem normal.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KAFKA-3143) inconsistent state in ZK when all replicas are dead
[ https://issues.apache.org/jira/browse/KAFKA-3143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-3143. -- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 5041 [https://github.com/apache/kafka/pull/5041] > inconsistent state in ZK when all replicas are dead > --- > > Key: KAFKA-3143 > URL: https://issues.apache.org/jira/browse/KAFKA-3143 > Project: Kafka > Issue Type: Bug >Reporter: Jun Rao >Assignee: Ismael Juma >Priority: Major > Labels: reliability > Fix For: 2.3.0 > > > This issue can be recreated in the following steps. > 1. Start 3 brokers, 1, 2 and 3. > 2. Create a topic with a single partition and 2 replicas, say on broker 1 and > 2. > If we stop both replicas 1 and 2, depending on where the controller is, the > leader and isr stored in ZK in the end are different. > If the controller is on broker 3, what's stored in ZK will be -1 for leader > and an empty set for ISR. > On the other hand, if the controller is on broker 2 and we stop broker 1 > followed by broker 2, what's stored in ZK will be 2 for leader and 2 for ISR. > The issue is that in the first case, the controller will call > ReplicaStateMachine to transition to OfflineReplica, which will change the > leader and isr. However, in the second case, the controller fails over, but > we don't transition ReplicaStateMachine to OfflineReplica during controller > initialization. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-8376) Flaky test ClientAuthenticationFailureTest.testTransactionalProducerWithInvalidCredentials test.
Manikumar created KAFKA-8376: Summary: Flaky test ClientAuthenticationFailureTest.testTransactionalProducerWithInvalidCredentials test. Key: KAFKA-8376 URL: https://issues.apache.org/jira/browse/KAFKA-8376 Project: Kafka Issue Type: Bug Reporter: Manikumar Fix For: 2.3.0 Attachments: t1.txt The test is going into hang state and test run was not completing. I think the flakiness is due to timing issues and https://github.com/apache/kafka/pull/5971 Attaching the thread dump. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-7455) JmxTool cannot connect to an SSL-enabled JMX RMI port
[
https://issues.apache.org/jira/browse/KAFKA-7455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-7455.
--
Resolution: Fixed
Fix Version/s: 2.3.0
> JmxTool cannot connect to an SSL-enabled JMX RMI port
> -
>
> Key: KAFKA-7455
> URL: https://issues.apache.org/jira/browse/KAFKA-7455
> Project: Kafka
> Issue Type: Bug
> Components: tools
>Reporter: Attila Sasvari
>Priority: Major
> Fix For: 2.3.0
>
>
> When JmxTool tries to connect to an SSL-enabled JMX RMI port with
> JMXConnectorFactory'connect(), the connection attempt results in a
> "java.rmi.ConnectIOException: non-JRMP server at remote endpoint":
> {code}
> $ export
> KAFKA_OPTS="-Djavax.net.ssl.trustStore=/tmp/kafka.server.truststore.jks
> -Djavax.net.ssl.trustStorePassword=test"
> $ bin/kafka-run-class.sh kafka.tools.JmxTool --object-name
> "kafka.server:type=kafka-metrics-count" --jmx-url
> service:jmx:rmi:///jndi/rmi://localhost:9393/jmxrmi
> ConnectIOException: non-JRMP server at remote endpoint].
> java.io.IOException: Failed to retrieve RMIServer stub:
> javax.naming.CommunicationException [Root exception is
> java.rmi.ConnectIOException: non-JRMP server at remote endpoint]
> at
> javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369)
> at
> javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
> at kafka.tools.JmxTool$.main(JmxTool.scala:120)
> at kafka.tools.JmxTool.main(JmxTool.scala)
> {code}
> The problem is that {{JmxTool}} does not specify
> {{SslRMIClientSocketFactory}} when it tries to connect
> https://github.com/apache/kafka/blob/70d90c371833b09cf934c8c2358171433892a085/core/src/main/scala/kafka/tools/JmxTool.scala#L120
> {code}
> jmxc = JMXConnectorFactory.connect(url, null)
> {code}
> To connect to a secured RMI port, it should pass an envionrment map that
> contains a {{("com.sun.jndi.rmi.factory.socket", new
> SslRMIClientSocketFactory)}} entry.
> More info:
> - https://docs.oracle.com/cd/E19698-01/816-7609/security-35/index.html
> - https://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KAFKA-7466) Implement KIP-339: Create a new IncrementalAlterConfigs API
[ https://issues.apache.org/jira/browse/KAFKA-7466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-7466. -- Resolution: Fixed Reviewer: Colin P. McCabe Fix Version/s: 2.3.0 > Implement KIP-339: Create a new IncrementalAlterConfigs API > --- > > Key: KAFKA-7466 > URL: https://issues.apache.org/jira/browse/KAFKA-7466 > Project: Kafka > Issue Type: Improvement >Reporter: Colin P. McCabe >Assignee: Manikumar >Priority: Major > Fix For: 2.3.0 > > > Implement KIP-339: Create a new IncrementalAlterConfigs API -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-8090) Replace ControlledShutdown request/response with automated protocol
[ https://issues.apache.org/jira/browse/KAFKA-8090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-8090. -- Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 6423 [https://github.com/apache/kafka/pull/6423] > Replace ControlledShutdown request/response with automated protocol > --- > > Key: KAFKA-8090 > URL: https://issues.apache.org/jira/browse/KAFKA-8090 > Project: Kafka > Issue Type: Sub-task >Reporter: Mickael Maison >Assignee: Mickael Maison >Priority: Major > Fix For: 2.3.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-7978) Flaky Test SaslSslAdminClientIntegrationTest#testConsumerGroups
[
https://issues.apache.org/jira/browse/KAFKA-7978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-7978.
--
Resolution: Duplicate
Resolving as duplicate of KAFKA-8098. Feel free to reopen the issue, If occurs
again.
> Flaky Test SaslSslAdminClientIntegrationTest#testConsumerGroups
> ---
>
> Key: KAFKA-7978
> URL: https://issues.apache.org/jira/browse/KAFKA-7978
> Project: Kafka
> Issue Type: Bug
> Components: core, unit tests
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Matthias J. Sax
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0, 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/25/]
> {quote}java.lang.AssertionError: expected:<2> but was:<0> at
> org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.failNotEquals(Assert.java:834) at
> org.junit.Assert.assertEquals(Assert.java:645) at
> org.junit.Assert.assertEquals(Assert.java:631) at
> kafka.api.AdminClientIntegrationTest.testConsumerGroups(AdminClientIntegrationTest.scala:1157)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
