date:20210823

Conditional Produce in Kafka (KAFKA-2260)

2021-08-23 Thread Niket Goel

Hi all,

I am new to the Kafka project and while having a discussion with a colleague, I
was pointed to the JIRA KAFKA-2260
[1] which talks about adding
the ability to perform "conditional produce" requests to Kafka. This idea
sounded very exciting to me and I can see it draw parallels with Optimistic
Concurrency Control in Traditional Databases wherein Kafka could be used as a
"system of record”, or as an arbiter for disparate systems working on the same
data. The natural scale-out architecture of Kafka makes this ability even more
interesting than the more monolithic databases that generally offer this. I
found a system which is doing something like this - Waltz
altz
[2] today already.

Following the discussion in the JIRA [1] itself and then on KIP-27

[3], it seems like there was a lot of interest in this proposal, but it never
materialized. Also IIUC, with the introduction of KIP-98

[4], some of the use cases that were motivating the discussion seem to have
been addressed (e.g. idempotent producer).

In theory, introducing this ability to Kafka seems pretty great to me, but
being new to Kafka I am not able to articulate real-life use cases where users
of Kafka would utilize such an ability. This is where I would like help.

My questions to the group here today are:
1. Is there a use case that you are using/or want to use Kafka for, where
having the ability to "conditionally produce” or "optimistically lock a key"
would simplify your life?
2. If there are such scenarios today, how are people making it work? Is it
through other Kafka features, or through application side logic.

Please feel to also add your thoughts on this feature’s usefulness, or even
point me towards more suggested reading if I am missing things. :)

Thanks
Niket Goel

[1] https://issues.apache.org/jira/browse/KAFKA-2260

[2] https://wepay.github.io/waltz/docs/introduction

[3]
https://cwiki.apache.org/confluence/display/KAFKA/KIP-27+-+Conditional+Publish

[4]
https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #420

2021-08-23 Thread Apache Jenkins Server

See 


Changes:


--
[...truncated 486467 lines...]
[2021-08-24T00:10:35.937Z] > Task :raft:testClasses UP-TO-DATE
[2021-08-24T00:10:35.937Z] > Task :connect:json:testJar
[2021-08-24T00:10:35.937Z] > Task :connect:json:testSrcJar
[2021-08-24T00:10:35.937Z] > Task :metadata:compileTestJava UP-TO-DATE
[2021-08-24T00:10:35.937Z] > Task :metadata:testClasses UP-TO-DATE
[2021-08-24T00:10:35.937Z] > Task 
:clients:generateMetadataFileForMavenJavaPublication
[2021-08-24T00:10:35.937Z] > Task 
:clients:generatePomFileForMavenJavaPublication
[2021-08-24T00:10:35.937Z] 
[2021-08-24T00:10:35.937Z] > Task :streams:processMessages
[2021-08-24T00:10:35.937Z] Execution optimizations have been disabled for task 
':streams:processMessages' to ensure correctness due to the following reasons:
[2021-08-24T00:10:35.937Z]   - Gradle detected a problem with the following 
location: 
'/home/jenkins/workspace/Kafka_kafka_trunk/streams/src/generated/java/org/apache/kafka/streams/internals/generated'.
 Reason: Task ':streams:srcJar' uses this output of task 
':streams:processMessages' without declaring an explicit or implicit 
dependency. This can lead to incorrect results being produced, depending on 
what order the tasks are executed. Please refer to 
https://docs.gradle.org/7.1.1/userguide/validation_problems.html#implicit_dependency
 for more details about this problem.
[2021-08-24T00:10:35.937Z] MessageGenerator: processed 1 Kafka message JSON 
files(s).
[2021-08-24T00:10:35.938Z] 
[2021-08-24T00:10:35.938Z] > Task :streams:compileJava UP-TO-DATE
[2021-08-24T00:10:35.938Z] > Task :streams:classes UP-TO-DATE
[2021-08-24T00:10:35.938Z] > Task :streams:test-utils:compileJava UP-TO-DATE
[2021-08-24T00:10:36.942Z] > Task :streams:copyDependantLibs
[2021-08-24T00:10:36.942Z] > Task :streams:jar UP-TO-DATE
[2021-08-24T00:10:36.942Z] > Task 
:streams:generateMetadataFileForMavenJavaPublication
[2021-08-24T00:10:39.967Z] > Task :connect:api:javadoc
[2021-08-24T00:10:39.967Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2021-08-24T00:10:39.967Z] > Task :connect:api:jar UP-TO-DATE
[2021-08-24T00:10:39.967Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2021-08-24T00:10:39.967Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2021-08-24T00:10:39.967Z] > Task :connect:json:jar UP-TO-DATE
[2021-08-24T00:10:39.967Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2021-08-24T00:10:39.967Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2021-08-24T00:10:39.967Z] > Task :connect:json:publishToMavenLocal
[2021-08-24T00:10:39.967Z] > Task :connect:api:javadocJar
[2021-08-24T00:10:39.967Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2021-08-24T00:10:39.967Z] > Task :connect:api:testClasses UP-TO-DATE
[2021-08-24T00:10:39.967Z] > Task :connect:api:testJar
[2021-08-24T00:10:39.967Z] > Task :connect:api:testSrcJar
[2021-08-24T00:10:39.967Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2021-08-24T00:10:39.967Z] > Task :connect:api:publishToMavenLocal
[2021-08-24T00:10:44.182Z] > Task :streams:javadoc
[2021-08-24T00:10:44.182Z] > Task :streams:javadocJar
[2021-08-24T00:10:45.187Z] > Task :clients:javadoc
[2021-08-24T00:10:46.193Z] > Task :clients:javadocJar
[2021-08-24T00:10:46.193Z] 
[2021-08-24T00:10:46.193Z] > Task :clients:srcJar
[2021-08-24T00:10:46.193Z] Execution optimizations have been disabled for task 
':clients:srcJar' to ensure correctness due to the following reasons:
[2021-08-24T00:10:46.193Z]   - Gradle detected a problem with the following 
location: 
'/home/jenkins/workspace/Kafka_kafka_trunk/clients/src/generated/java'. Reason: 
Task ':clients:srcJar' uses this output of task ':clients:processMessages' 
without declaring an explicit or implicit dependency. This can lead to 
incorrect results being produced, depending on what order the tasks are 
executed. Please refer to 
https://docs.gradle.org/7.1.1/userguide/validation_problems.html#implicit_dependency
 for more details about this problem.
[2021-08-24T00:10:47.198Z] 
[2021-08-24T00:10:47.198Z] > Task :clients:testJar
[2021-08-24T00:10:47.198Z] > Task :clients:testSrcJar
[2021-08-24T00:10:48.369Z] > Task 
:clients:publishMavenJavaPublicationToMavenLocal
[2021-08-24T00:10:48.369Z] > Task :clients:publishToMavenLocal
[2021-08-24T00:11:07.117Z] > Task :core:compileScala
[2021-08-24T00:12:16.534Z] > Task :core:classes
[2021-08-24T00:12:16.534Z] > Task :core:compileTestJava NO-SOURCE
[2021-08-24T00:12:40.411Z] > Task :core:compileTestScala
[2021-08-24T00:13:29.849Z] > Task :core:testClasses
[2021-08-24T00:13:41.637Z] > Task :streams:compileTestJava
[2021-08-24T00:13:41.637Z] > Task :streams:testClasses
[2021-08-24T00:13:41.637Z] > Task :streams:testJar
[2021-08-24T00:13:41.637Z] > Task :streams:testSrcJar
[2021-08-24T00:13:41.637Z] > Task

[jira] [Created] (KAFKA-13224) broker.id does not appear in config's originals map when setting just node.id

2021-08-23 Thread Ron Dagostino (Jira)

Ron Dagostino created KAFKA-13224:
-

 Summary: broker.id does not appear in config's originals map when 
setting just node.id
 Key: KAFKA-13224
 URL: https://issues.apache.org/jira/browse/KAFKA-13224
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Ron Dagostino
Assignee: Ron Dagostino


Plugins may expect broker.id to exist as a key in the config's various 
originals()-related maps, but with KRaft we rely solely on node.id for the 
broker's ID, and with the Zk-based brokers we provide the option to specify 
node.id in addition to (or as a full replacement for) broker.id. 
 There are multiple problems related to this switch to node.id:

# We do not enforce consistency between explicitly-specified broker.id and 
node.id properties in the config -- it is entirely possible right now that we 
could set broker.id=0 and also set node.id=1, and the broker will use 1 for 
it's ID. This is confusing at best; the broker should detect this inconsistency 
and fail to start with a ConfigException.
# When node.id is set, both that value and any explicitly-set broker.id value 
will exist in the config's *originals()-related maps*. Downstream components 
are often configured based on these maps, and they may ask for the broker.id, 
so downstream components may be misconfigured if the values differ, or they may 
fail during configuration if no broker.id key exists in the map at all.
# The config's *values()-related maps* will contain either the 
explicitly-specified broker.id value or the default value of -1.  When node.id 
is set, both that value (which cannot be negative) and the (potentially -1) 
broker.id value will exist in the config's values()-related maps. Downstream 
components are often configured based on these maps, and they may ask for the 
broker.id, so downstream components may be misconfigured if the broker.id value 
differs from the broker's true ID.

The broker should detect inconsistency between explicitly-specified broker.id 
and node.id values and fail startup accordingly. It should also ensures that 
the config's originals()- and values()-related maps contain the same mapped 
values for both broker.id and node.id keys.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #419

2021-08-23 Thread Apache Jenkins Server

See 


Changes:


--
[...truncated 488967 lines...]
[Pipeline] {
[Pipeline] sh
[2021-08-23T22:06:07.621Z] 
[2021-08-23T22:06:07.621Z] LogCleanerParameterizedIntegrationTest > 
testCleanerWithMessageFormatV0(CompressionType) > 
kafka.log.LogCleanerParameterizedIntegrationTest.testCleanerWithMessageFormatV0(CompressionType)[1]
 PASSED
[2021-08-23T22:06:07.621Z] 
[2021-08-23T22:06:07.621Z] LogCleanerParameterizedIntegrationTest > 
testCleanerWithMessageFormatV0(CompressionType) > 
kafka.log.LogCleanerParameterizedIntegrationTest.testCleanerWithMessageFormatV0(CompressionType)[2]
 STARTED
[2021-08-23T22:06:07.804Z] + mvn compile
[2021-08-23T22:06:07.873Z] 
[2021-08-23T22:06:07.873Z] FetchRequestBetweenDifferentIbpTest > 
testControllerNewToOldIBP() PASSED
[2021-08-23T22:06:09.701Z] [INFO] Scanning for projects...
[2021-08-23T22:06:09.701Z] [INFO] 
[2021-08-23T22:06:09.701Z] [INFO] -< 
streams.examples:streams.examples >--
[2021-08-23T22:06:09.701Z] [INFO] Building Kafka Streams Quickstart :: Java 0.1
[2021-08-23T22:06:09.701Z] [INFO] [ jar 
]-
[2021-08-23T22:06:09.701Z] [INFO] 
[2021-08-23T22:06:09.701Z] [INFO] --- maven-resources-plugin:2.6:resources 
(default-resources) @ streams.examples ---
[2021-08-23T22:06:09.930Z] 
[2021-08-23T22:06:09.930Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 8.0.
[2021-08-23T22:06:09.930Z] 
[2021-08-23T22:06:09.930Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2021-08-23T22:06:09.930Z] 
[2021-08-23T22:06:09.930Z] See 
https://docs.gradle.org/7.1.1/userguide/command_line_interface.html#sec:command_line_warnings
[2021-08-23T22:06:09.930Z] 
[2021-08-23T22:06:09.930Z] BUILD SUCCESSFUL in 1h 35m 58s
[2021-08-23T22:06:09.930Z] 202 actionable tasks: 109 executed, 93 up-to-date
[2021-08-23T22:06:09.930Z] 
[2021-08-23T22:06:09.930Z] See the profiling report at: 
file:///home/jenkins/workspace/Kafka_kafka_trunk/build/reports/profile/profile-2021-08-23-20-30-17.html
[2021-08-23T22:06:09.930Z] A fine-grained performance profile is available: use 
the --scan option.
[Pipeline] junit
[2021-08-23T22:06:10.627Z] [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
[2021-08-23T22:06:10.627Z] [INFO] Copying 1 resource
[2021-08-23T22:06:10.627Z] [INFO] 
[2021-08-23T22:06:10.627Z] [INFO] --- maven-compiler-plugin:3.1:compile 
(default-compile) @ streams.examples ---
[2021-08-23T22:06:10.627Z] [INFO] Changes detected - recompiling the module!
[2021-08-23T22:06:10.627Z] [INFO] Compiling 3 source files to 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/quickstart/test-streams-archetype/streams.examples/target/classes
[2021-08-23T22:06:10.959Z] Recording test results
[2021-08-23T22:06:12.359Z] [INFO] 

[2021-08-23T22:06:12.359Z] [INFO] BUILD SUCCESS
[2021-08-23T22:06:12.359Z] [INFO] 

[2021-08-23T22:06:12.359Z] [INFO] Total time:  2.710 s
[2021-08-23T22:06:12.359Z] [INFO] Finished at: 2021-08-23T22:06:11Z
[2021-08-23T22:06:12.359Z] [INFO] 

[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[2021-08-23T22:06:26.563Z] 
[2021-08-23T22:06:26.563Z] LogCleanerParameterizedIntegrationTest > 
testCleanerWithMessageFormatV0(CompressionType) > 
kafka.log.LogCleanerParameterizedIntegrationTest.testCleanerWithMessageFormatV0(CompressionType)[2]
 PASSED
[2021-08-23T22:06:26.563Z] 
[2021-08-23T22:06:26.563Z] LogCleanerParameterizedIntegrationTest > 
testCleanerWithMessageFormatV0(CompressionType) > 
kafka.log.LogCleanerParameterizedIntegrationTest.testCleanerWithMessageFormatV0(CompressionType)[3]
 STARTED
[2021-08-23T22:06:27.692Z] 
[2021-08-23T22:06:27.692Z] ReplicationQuotasTest > 
shouldBootstrapTwoBrokersWithFollowerThrottle() PASSED
[2021-08-23T22:06:27.692Z] 
[2021-08-23T22:06:27.692Z] ControllerFailoverTest > 
testHandleIllegalStateException() STARTED
[2021-08-23T22:06:28.625Z] 
[2021-08-23T22:06:28.625Z] ControllerFailoverTest > 
testHandleIllegalStateException() PASSED
[2021-08-23T22:06:28.625Z] 
[2021-08-23T22:06:28.625Z] MetricsTest > testUpdateJMXFilter() STARTED
[2021-08-23T22:06:31.245Z] 
[2021-08-23T22:06:31.245Z] MetricsTest > testUpdateJMXFilter() PASSED
[2021-08-23T22:06:31.245Z] 
[2021-08-23T22:06:31.245Z] MetricsTest >

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.0 #115

2021-08-23 Thread Apache Jenkins Server

See

Build failed in Jenkins: Kafka » Kafka Branch Builder » 2.8 #68

2021-08-23 Thread Apache Jenkins Server

See 


Changes:


--
[...truncated 3691 lines...]
[2021-08-23T21:06:25.662Z] > Task :spotlessScalaCheck
[2021-08-23T21:06:32.733Z] 
[2021-08-23T21:06:32.733Z] > Task :rat
[2021-08-23T21:06:32.733Z] Rat report: 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_2.8/build/rat/rat-report.html
[2021-08-23T21:06:36.395Z] 
[2021-08-23T21:06:36.395Z] > Task :generator:spotbugsMain
[2021-08-23T21:06:39.935Z] > Task :clients:compileJava
[2021-08-23T21:06:39.935Z] > Task :clients:classes
[2021-08-23T21:06:41.846Z] > Task :clients:jar
[2021-08-23T21:06:43.756Z] > Task :connect:mirror-client:compileJava
[2021-08-23T21:06:43.756Z] > Task :connect:mirror-client:classes
[2021-08-23T21:06:43.756Z] > Task :log4j-appender:compileJava
[2021-08-23T21:06:43.756Z] > Task :log4j-appender:classes
[2021-08-23T21:06:43.756Z] > Task :log4j-appender:jar
[2021-08-23T21:06:43.756Z] > Task :connect:api:compileJava
[2021-08-23T21:06:43.756Z] > Task :connect:api:classes
[2021-08-23T21:06:43.756Z] > Task :metadata:compileJava
[2021-08-23T21:06:43.756Z] > Task :metadata:classes
[2021-08-23T21:06:43.756Z] > Task :metadata:jar
[2021-08-23T21:06:43.756Z] > Task :log4j-appender:checkstyleMain
[2021-08-23T21:06:45.669Z] > Task :connect:mirror-client:checkstyleMain
[2021-08-23T21:06:45.669Z] > Task :raft:compileJava
[2021-08-23T21:06:45.669Z] > Task :raft:classes
[2021-08-23T21:06:45.669Z] > Task :raft:jar
[2021-08-23T21:06:45.669Z] > Task :log4j-appender:spotbugsMain
[2021-08-23T21:06:45.669Z] > Task :core:compileJava
[2021-08-23T21:06:45.669Z] > Task :tools:compileJava
[2021-08-23T21:06:45.669Z] > Task :tools:classes
[2021-08-23T21:06:45.669Z] > Task :connect:api:checkstyleMain
[2021-08-23T21:06:47.586Z] > Task :raft:checkstyleMain
[2021-08-23T21:06:51.144Z] > Task :metadata:checkstyleMain
[2021-08-23T21:06:51.144Z] > Task :connect:api:spotbugsMain
[2021-08-23T21:06:51.144Z] > Task :tools:checkstyleMain
[2021-08-23T21:06:56.372Z] > Task :connect:mirror-client:spotbugsMain
[2021-08-23T21:07:09.293Z] > Task :tools:spotbugsMain
[2021-08-23T21:07:11.202Z] > Task :clients:compileTestJava
[2021-08-23T21:07:11.202Z] > Task :clients:testClasses
[2021-08-23T21:07:13.117Z] > Task :connect:mirror-client:copyDependantLibs
[2021-08-23T21:07:13.117Z] > Task :connect:api:copyDependantLibs
[2021-08-23T21:07:13.117Z] > Task :connect:mirror-client:jar
[2021-08-23T21:07:13.117Z] > Task :connect:api:jar
[2021-08-23T21:07:13.117Z] > Task :tools:copyDependantLibs
[2021-08-23T21:07:13.117Z] > Task :connect:basic-auth-extension:compileJava
[2021-08-23T21:07:13.117Z] > Task :connect:json:compileJava
[2021-08-23T21:07:13.117Z] > Task :connect:json:classes
[2021-08-23T21:07:13.117Z] > Task :connect:file:compileJava
[2021-08-23T21:07:13.117Z] > Task :log4j-appender:compileTestJava
[2021-08-23T21:07:13.117Z] > Task :connect:json:copyDependantLibs
[2021-08-23T21:07:13.117Z] > Task :connect:json:jar
[2021-08-23T21:07:13.117Z] > Task :tools:jar
[2021-08-23T21:07:13.117Z] > Task :raft:spotbugsMain
[2021-08-23T21:07:13.117Z] > Task :connect:transforms:compileJava
[2021-08-23T21:07:13.117Z] > Task :connect:transforms:classes
[2021-08-23T21:07:15.028Z] > Task :connect:transforms:copyDependantLibs
[2021-08-23T21:07:15.028Z] > Task :connect:transforms:jar
[2021-08-23T21:07:15.028Z] > Task :tools:compileTestJava
[2021-08-23T21:07:15.028Z] > Task :connect:basic-auth-extension:classes
[2021-08-23T21:07:15.028Z] > Task :connect:basic-auth-extension:compileTestJava
[2021-08-23T21:07:15.028Z] > Task :connect:file:classes
[2021-08-23T21:07:16.938Z] > Task :connect:api:compileTestJava
[2021-08-23T21:07:16.938Z] > Task :core:compileScala
[2021-08-23T21:07:16.938Z] > Task :connect:file:compileTestJava
[2021-08-23T21:07:16.938Z] > Task :connect:mirror-client:compileTestJava
[2021-08-23T21:07:16.938Z] > Task :raft:compileTestJava
[2021-08-23T21:07:16.938Z] > Task :metadata:compileTestJava
[2021-08-23T21:07:16.938Z] > Task :connect:json:compileTestJava
[2021-08-23T21:07:16.938Z] > Task :connect:transforms:compileTestJava
[2021-08-23T21:07:16.938Z] > Task :connect:basic-auth-extension:checkstyleMain
[2021-08-23T21:07:16.938Z] > Task :connect:file:checkstyleMain
[2021-08-23T21:07:16.938Z] > Task :log4j-appender:testClasses
[2021-08-23T21:07:18.848Z] > Task :log4j-appender:checkstyleTest
[2021-08-23T21:07:18.848Z] > Task :metadata:testClasses
[2021-08-23T21:07:18.848Z] > Task :connect:json:checkstyleMain
[2021-08-23T21:07:18.848Z] > Task :raft:testClasses
[2021-08-23T21:07:18.848Z] > Task :connect:runtime:compileJava
[2021-08-23T21:07:18.848Z] > Task :connect:runtime:classes
[2021-08-23T21:07:18.848Z] > Task :connect:transforms:checkstyleMain
[2021-08-23T21:07:18.848Z] > Task :tools:testClasses
[2021-08-23T21:07:20.758Z] > Task :tools:checkstyleTest
[2021-08-23T21:07:20.758Z] > Task :connect:api:testClasses
[2021-08-23T21:07:20.758Z] > Task :metadata:spotbugsMain

Re: [VOTE] KIP-748: Add Broker Count Metrics (restarted)

2021-08-23 Thread David Arthur

Thanks, Colin. Looks good to me!

+1 binding

-David

On Mon, Aug 16, 2021 at 3:37 PM David Jacot  wrote:

> Hi Colin,
>
> Thanks for restarting this KIP.
>
> +1 (binding)
>
> Best,
> David
>
> Le lun. 16 août 2021 à 21:32, Colin McCabe  a écrit :
>
> > Hi all,
> >
> > Two months ago we tried to vote on this minor KIP adding two new broker
> > metrics. However, things got busy and we weren't able to complete the
> vote.
> > Now that things have quieted down a bit, let's restart the vote!
> >
> > Note that I made some minor changes to the KIP after some offline
> > discussion with Ryan Dielhenn and David Jacot. Specifically, I added a
> > metric for the number of fenced brokers.
> >
> > Here's the KIP:
> > https://cwiki.apache.org/confluence/x/N4rOCg
> >
> > Here's the discussion thread:
> >
> >
> https://lists.apache.org/thread.html/r308364dfbb3020e6151cef47237c28a4a540e187b8af84ddafec83af%40%3Cdev.kafka.apache.org%3E
> >
> > Please take a look if you have a chance!
> >
> > best,
> > Colin
> >
>


-- 
David Arthur

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #418

2021-08-23 Thread Apache Jenkins Server

See 


Changes:


--
[...truncated 489110 lines...]
[2021-08-23T20:16:18.829Z] > Task :raft:testClasses UP-TO-DATE
[2021-08-23T20:16:18.829Z] > Task :connect:json:testJar
[2021-08-23T20:16:18.829Z] > Task :connect:json:testSrcJar
[2021-08-23T20:16:18.829Z] > Task :metadata:compileTestJava UP-TO-DATE
[2021-08-23T20:16:18.829Z] > Task :metadata:testClasses UP-TO-DATE
[2021-08-23T20:16:18.829Z] > Task 
:clients:generateMetadataFileForMavenJavaPublication
[2021-08-23T20:16:18.829Z] > Task 
:clients:generatePomFileForMavenJavaPublication
[2021-08-23T20:16:18.829Z] > Task :core:compileScala UP-TO-DATE
[2021-08-23T20:16:18.829Z] > Task :core:classes UP-TO-DATE
[2021-08-23T20:16:18.829Z] > Task :core:compileTestJava NO-SOURCE
[2021-08-23T20:16:18.829Z] 
[2021-08-23T20:16:18.829Z] > Task :streams:processMessages
[2021-08-23T20:16:18.829Z] Execution optimizations have been disabled for task 
':streams:processMessages' to ensure correctness due to the following reasons:
[2021-08-23T20:16:18.829Z]   - Gradle detected a problem with the following 
location: 
'/home/jenkins/workspace/Kafka_kafka_trunk/streams/src/generated/java/org/apache/kafka/streams/internals/generated'.
 Reason: Task ':streams:srcJar' uses this output of task 
':streams:processMessages' without declaring an explicit or implicit 
dependency. This can lead to incorrect results being produced, depending on 
what order the tasks are executed. Please refer to 
https://docs.gradle.org/7.1.1/userguide/validation_problems.html#implicit_dependency
 for more details about this problem.
[2021-08-23T20:16:18.829Z] MessageGenerator: processed 1 Kafka message JSON 
files(s).
[2021-08-23T20:16:18.829Z] 
[2021-08-23T20:16:18.829Z] > Task :streams:compileJava UP-TO-DATE
[2021-08-23T20:16:18.829Z] > Task :streams:classes UP-TO-DATE
[2021-08-23T20:16:18.829Z] > Task :streams:copyDependantLibs UP-TO-DATE
[2021-08-23T20:16:18.829Z] > Task :streams:jar UP-TO-DATE
[2021-08-23T20:16:18.829Z] > Task :streams:test-utils:compileJava UP-TO-DATE
[2021-08-23T20:16:18.829Z] > Task 
:streams:generateMetadataFileForMavenJavaPublication
[2021-08-23T20:16:18.829Z] > Task :core:compileTestScala UP-TO-DATE
[2021-08-23T20:16:18.829Z] > Task :core:testClasses UP-TO-DATE
[2021-08-23T20:16:23.023Z] > Task :connect:api:javadoc
[2021-08-23T20:16:23.023Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2021-08-23T20:16:23.023Z] > Task :connect:api:jar UP-TO-DATE
[2021-08-23T20:16:23.023Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2021-08-23T20:16:23.023Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2021-08-23T20:16:23.023Z] > Task :connect:json:jar UP-TO-DATE
[2021-08-23T20:16:23.023Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2021-08-23T20:16:23.023Z] > Task :connect:api:javadocJar
[2021-08-23T20:16:23.023Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2021-08-23T20:16:23.023Z] > Task :connect:api:testClasses UP-TO-DATE
[2021-08-23T20:16:23.023Z] > Task :connect:api:testJar
[2021-08-23T20:16:23.023Z] > Task :connect:api:testSrcJar
[2021-08-23T20:16:23.023Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2021-08-23T20:16:23.023Z] > Task :connect:json:publishToMavenLocal
[2021-08-23T20:16:23.023Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2021-08-23T20:16:23.023Z] > Task :connect:api:publishToMavenLocal
[2021-08-23T20:16:25.076Z] > Task :streams:javadoc
[2021-08-23T20:16:25.076Z] > Task :streams:javadocJar
[2021-08-23T20:16:26.097Z] > Task :streams:compileTestJava UP-TO-DATE
[2021-08-23T20:16:26.097Z] > Task :streams:testClasses UP-TO-DATE
[2021-08-23T20:16:26.097Z] > Task :streams:testJar
[2021-08-23T20:16:26.097Z] > Task :streams:testSrcJar
[2021-08-23T20:16:26.097Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2021-08-23T20:16:26.097Z] > Task :streams:publishToMavenLocal
[2021-08-23T20:16:27.121Z] > Task :clients:javadoc
[2021-08-23T20:16:27.121Z] > Task :clients:javadocJar
[2021-08-23T20:16:28.143Z] 
[2021-08-23T20:16:28.143Z] > Task :clients:srcJar
[2021-08-23T20:16:28.143Z] Execution optimizations have been disabled for task 
':clients:srcJar' to ensure correctness due to the following reasons:
[2021-08-23T20:16:28.143Z]   - Gradle detected a problem with the following 
location: 
'/home/jenkins/workspace/Kafka_kafka_trunk/clients/src/generated/java'. Reason: 
Task ':clients:srcJar' uses this output of task ':clients:processMessages' 
without declaring an explicit or implicit dependency. This can lead to 
incorrect results being produced, depending on what order the tasks are 
executed. Please refer to 
https://docs.gradle.org/7.1.1/userguide/validation_problems.html#implicit_dependency
 for more details about this problem.
[2021-08-23T20:16:29.165Z] 
[2021-08-23T20:16:29.165Z] > Task :clients:testJar
[2021-08-23T20:16:29.165Z] > Task :clients:testSrcJar

Re: [DISCUSS] KIP-768: Extend SASL/OAUTHBEARER with Support for OIDC

2021-08-23 Thread Manikumar

Hi Kirk,

Thanks for the KIP!

1. Do we want to support validating issuers using the issuer uri?
2. Can the access token be reused for multiple connections from the same
client?
3. Do we support configuring separate ssl configs for connecting
authorization server/jwks endpoint?
4. Do we want support any readable username as principal if it is present
in the token claims
5. I agree with Ron, We should move the public classes to
"o.a.k.c.s.oauthbearer.secured" package.

Thanks,
Manikumar

On Thu, Aug 19, 2021 at 11:46 PM Kirk True  wrote:

> Hi Ron,
>
> On Sat, Aug 14, 2021, at 11:27 AM, Ron Dagostino wrote:
> > Hi Kirk -- thanks for the KIP!  Having concrete implementations
> > out-of-the-box will be very helpful.
> >
> > > As seen in this diagram, the login callback is executed on the client
> and
> > the validate callback is executed on the broker.
> >
> > There was no diagram when I looked.  Maybe there is a broken link or
> > something?
>
> I double-checked and it's showing for me on the published version of the
> wiki, even after I've logged out.
>
> Would you mind checking again when you get the chance?
>
> > > The name of the implementation class will be
> >
> org.apache.kafka.common.security.oauthbearer.internals.secured.OAuthBearerLoginCallbackHandler
> >
> > I think the internals package was meant for non-public stuff  Most of it
> > seems that way, although the "unsecured" implementation is in there --
> but
> > that's maybe a grey area since it isn't meant to be used in production
> > scenarios and is mostly leveraged in unit tests.  Perhaps move the
> proposed
> > class into a "o.a.k.c.s.oauthbearer.secured" package?  Then any
> > implementation details beyond the public stuff can live under the
> > "...internals.secured" package that you mentioned?  The same comment
> > applies to the validator callback handler class.
>
> In a draft I had the secured package directly under the oauthbearer
> package as you describe but I received some out-of-band feedback to aim for
> parity with the unsecured package layout.
>
> I don't have a preference for either. I do agree that it seems weird for a
> package named internals to be used in configuration since its name implies
> that things could change.
>
> > I'm confused by loginRetryMaxWaitMs and loginRetryWaitMs.  The former has
> > "Max" in the name, but only the description of the latter mentions it
> being
> > a max amount of time?  Are the descriptions incorrect or perhaps
> reversed?
>
> Yes. Thanks for catching that. I've added more description in a separate
> paragraph above the enumerated configurations.
>
> > >  Ensure the encoding algorithm isn't none and matches what the expected
> > algorithm expecting
> >
> > "expected algorithm expecting" some kind of grammar issue?
>
> Haha! Yes - thanks for catching that too!
>
> It now reads:
>
> > Ensure the encoding algorithm (`alg` from the header) isn't `none` and
> matches the expected algorithm for the JWK ID
>
> > Thanks again -- very exciting!
>
> Thanks for the feedback!!!
>
> Kirk
>
> >
> > Ron
> >
> >
> >
> >
> >
> > On Fri, Aug 13, 2021 at 3:53 PM Kirk True  wrote:
> >
> > > Hi all!
> > >
> > > I have created a new KIP for a new OAuth/OIDC related authentication
> > > feature.
> > >
> > > This task is to provide a concrete implementation of the interfaces
> > > defined in KIP-255 to allow Kafka to connect to an OAuth / OIDC
> identity
> > > provider for authentication and token retrieval. While KIP-255
> provides an
> > > unsecured JWT example for development purposes, this will fill in the
> gap
> > > and provide a production-grade implementation.
> > >
> > > Here's the KIP:
> > >
> > >
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=186877575
> > >
> > > Thanks!
> > > Kirk
> >
>

[jira] [Resolved] (KAFKA-13223) Idempotent producer error with Kraft

2021-08-23 Thread Laurynas Butkus (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-13223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laurynas Butkus resolved KAFKA-13223.
-
Resolution: Information Provided

> Idempotent producer error with Kraft 
> -
>
> Key: KAFKA-13223
> URL: https://issues.apache.org/jira/browse/KAFKA-13223
> Project: Kafka
>  Issue Type: Bug
>  Components: kraft, producer 
>Reporter: Laurynas Butkus
>Priority: Major
> Fix For: 3.0.0
>
>
> I get an error *"The broker does not support INIT_PRODUCER_ID"* if I try to 
> produce a message idempotence enabled.
> Result:
> {code:java}
> ➜  ./bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic 
> test --request-required-acks -1 --producer-property enable.idempotence=true
> >test
> >[2021-08-23 19:40:33,356] ERROR [Producer clientId=console-producer] 
> >Aborting producer batches due to fatal error 
> >(org.apache.kafka.clients.producer.internals.Sender)
> org.apache.kafka.common.errors.UnsupportedVersionException: The broker does 
> not support INIT_PRODUCER_ID
> [2021-08-23 19:40:33,358] ERROR Error when sending message to topic test with 
> key: null, value: 4 bytes with error: 
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> org.apache.kafka.common.errors.UnsupportedVersionException: The broker does 
> not support INIT_PRODUCER_ID
> {code}
>  
> It works fine with idempotence disabled. Also it works fine if using 
> zookeeper.
> Tested with altered docker image: 
> {code:java}
> FROM confluentinc/cp-kafka:6.2.0
> RUN sed -i '/KAFKA_ZOOKEEPER_CONNECT/d' /etc/confluent/docker/configure && \
> # Docker workaround: Ignore cub zk-ready
> sed -i 's/cub zk-ready/echo ignore zk-ready/' /etc/confluent/docker/ensure && 
> \
> # KRaft required step: Format the storage directory with a new cluster ID
> echo "kafka-storage format --ignore-formatted -t $(kafka-storage random-uuid) 
> -c /etc/kafka/kafka.properties" >> /etc/confluent/docker/ensure
> {code}
> docker-compose.yml
> {code:java}
> version: '3.4'
> services:
>   kafka:
> build: kafka
> restart: unless-stopped
> environment:
>   ALLOW_PLAINTEXT_LISTENER: "yes"
>   KAFKA_HEAP_OPTS: -Xms256m -Xmx256m
>   LOG4J_LOGGER_KAFKA: "WARN"
>   KAFKA_BROKER_ID: 1
>   KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 
> 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
>   KAFKA_ADVERTISED_LISTENERS: 
> 'PLAINTEXT://kafka:29092,PLAINTEXT_HOST://127.0.0.1:9092'
>   KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
>   KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
>   KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
>   KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
>   KAFKA_PROCESS_ROLES: 'broker,controller'
>   KAFKA_NODE_ID: 1
>   KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka:29093'
>   KAFKA_LISTENERS: 
> 'PLAINTEXT://kafka:29092,CONTROLLER://kafka:29093,PLAINTEXT_HOST://:9092'
>   KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
>   KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
>   KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
> ports:
>   - "127.0.0.1:9092:9092/tcp"
> command: "/etc/confluent/docker/run"
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Reopened] (KAFKA-13223) Idempotent producer error with Kraft

2021-08-23 Thread Laurynas Butkus (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-13223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laurynas Butkus reopened KAFKA-13223:
-

> Idempotent producer error with Kraft 
> -
>
> Key: KAFKA-13223
> URL: https://issues.apache.org/jira/browse/KAFKA-13223
> Project: Kafka
>  Issue Type: Bug
>  Components: kraft, producer 
>Reporter: Laurynas Butkus
>Priority: Major
> Fix For: 3.0.0
>
>
> I get an error *"The broker does not support INIT_PRODUCER_ID"* if I try to 
> produce a message idempotence enabled.
> Result:
> {code:java}
> ➜  ./bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic 
> test --request-required-acks -1 --producer-property enable.idempotence=true
> >test
> >[2021-08-23 19:40:33,356] ERROR [Producer clientId=console-producer] 
> >Aborting producer batches due to fatal error 
> >(org.apache.kafka.clients.producer.internals.Sender)
> org.apache.kafka.common.errors.UnsupportedVersionException: The broker does 
> not support INIT_PRODUCER_ID
> [2021-08-23 19:40:33,358] ERROR Error when sending message to topic test with 
> key: null, value: 4 bytes with error: 
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> org.apache.kafka.common.errors.UnsupportedVersionException: The broker does 
> not support INIT_PRODUCER_ID
> {code}
>  
> It works fine with idempotence disabled. Also it works fine if using 
> zookeeper.
> Tested with altered docker image: 
> {code:java}
> FROM confluentinc/cp-kafka:6.2.0
> RUN sed -i '/KAFKA_ZOOKEEPER_CONNECT/d' /etc/confluent/docker/configure && \
> # Docker workaround: Ignore cub zk-ready
> sed -i 's/cub zk-ready/echo ignore zk-ready/' /etc/confluent/docker/ensure && 
> \
> # KRaft required step: Format the storage directory with a new cluster ID
> echo "kafka-storage format --ignore-formatted -t $(kafka-storage random-uuid) 
> -c /etc/kafka/kafka.properties" >> /etc/confluent/docker/ensure
> {code}
> docker-compose.yml
> {code:java}
> version: '3.4'
> services:
>   kafka:
> build: kafka
> restart: unless-stopped
> environment:
>   ALLOW_PLAINTEXT_LISTENER: "yes"
>   KAFKA_HEAP_OPTS: -Xms256m -Xmx256m
>   LOG4J_LOGGER_KAFKA: "WARN"
>   KAFKA_BROKER_ID: 1
>   KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 
> 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
>   KAFKA_ADVERTISED_LISTENERS: 
> 'PLAINTEXT://kafka:29092,PLAINTEXT_HOST://127.0.0.1:9092'
>   KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
>   KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
>   KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
>   KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
>   KAFKA_PROCESS_ROLES: 'broker,controller'
>   KAFKA_NODE_ID: 1
>   KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka:29093'
>   KAFKA_LISTENERS: 
> 'PLAINTEXT://kafka:29092,CONTROLLER://kafka:29093,PLAINTEXT_HOST://:9092'
>   KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
>   KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
>   KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
> ports:
>   - "127.0.0.1:9092:9092/tcp"
> command: "/etc/confluent/docker/run"
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-13223) Idempotent producer error with Kraft

2021-08-23 Thread Laurynas Butkus (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-13223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Laurynas Butkus resolved KAFKA-13223.
-
Fix Version/s: 3.0.0
   Resolution: Fixed

> Idempotent producer error with Kraft 
> -
>
> Key: KAFKA-13223
> URL: https://issues.apache.org/jira/browse/KAFKA-13223
> Project: Kafka
>  Issue Type: Bug
>  Components: kraft, producer 
>Reporter: Laurynas Butkus
>Priority: Major
> Fix For: 3.0.0
>
>
> I get an error *"The broker does not support INIT_PRODUCER_ID"* if I try to 
> produce a message idempotence enabled.
> Result:
> {code:java}
> ➜  ./bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic 
> test --request-required-acks -1 --producer-property enable.idempotence=true
> >test
> >[2021-08-23 19:40:33,356] ERROR [Producer clientId=console-producer] 
> >Aborting producer batches due to fatal error 
> >(org.apache.kafka.clients.producer.internals.Sender)
> org.apache.kafka.common.errors.UnsupportedVersionException: The broker does 
> not support INIT_PRODUCER_ID
> [2021-08-23 19:40:33,358] ERROR Error when sending message to topic test with 
> key: null, value: 4 bytes with error: 
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> org.apache.kafka.common.errors.UnsupportedVersionException: The broker does 
> not support INIT_PRODUCER_ID
> {code}
>  
> It works fine with idempotence disabled. Also it works fine if using 
> zookeeper.
> Tested with altered docker image: 
> {code:java}
> FROM confluentinc/cp-kafka:6.2.0
> RUN sed -i '/KAFKA_ZOOKEEPER_CONNECT/d' /etc/confluent/docker/configure && \
> # Docker workaround: Ignore cub zk-ready
> sed -i 's/cub zk-ready/echo ignore zk-ready/' /etc/confluent/docker/ensure && 
> \
> # KRaft required step: Format the storage directory with a new cluster ID
> echo "kafka-storage format --ignore-formatted -t $(kafka-storage random-uuid) 
> -c /etc/kafka/kafka.properties" >> /etc/confluent/docker/ensure
> {code}
> docker-compose.yml
> {code:java}
> version: '3.4'
> services:
>   kafka:
> build: kafka
> restart: unless-stopped
> environment:
>   ALLOW_PLAINTEXT_LISTENER: "yes"
>   KAFKA_HEAP_OPTS: -Xms256m -Xmx256m
>   LOG4J_LOGGER_KAFKA: "WARN"
>   KAFKA_BROKER_ID: 1
>   KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 
> 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
>   KAFKA_ADVERTISED_LISTENERS: 
> 'PLAINTEXT://kafka:29092,PLAINTEXT_HOST://127.0.0.1:9092'
>   KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
>   KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
>   KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
>   KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
>   KAFKA_PROCESS_ROLES: 'broker,controller'
>   KAFKA_NODE_ID: 1
>   KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka:29093'
>   KAFKA_LISTENERS: 
> 'PLAINTEXT://kafka:29092,CONTROLLER://kafka:29093,PLAINTEXT_HOST://:9092'
>   KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
>   KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
>   KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
> ports:
>   - "127.0.0.1:9092:9092/tcp"
> command: "/etc/confluent/docker/run"
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build became unstable: Kafka » Kafka Branch Builder » 3.0 #114

2021-08-23 Thread Apache Jenkins Server

See

[jira] [Resolved] (KAFKA-13091) Increment HW after shrinking ISR through AlterIsr

2021-08-23 Thread Jason Gustafson (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-13091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-13091.
-
Fix Version/s: 3.0.1
   Resolution: Fixed

> Increment HW after shrinking ISR through AlterIsr
> -
>
> Key: KAFKA-13091
> URL: https://issues.apache.org/jira/browse/KAFKA-13091
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Major
> Fix For: 3.0.1
>
>
> After we have shrunk the ISR, we have an opportunity to advance the high 
> watermark. We do this currently in `maybeShrinkIsr` after the synchronous 
> update through ZK. For the AlterIsr path, however, we cannot rely on this 
> call since the request is sent asynchronously. Instead we should attempt to 
> advance the high watermark in the callback when the AlterIsr response returns 
> successfully.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-13223) Idempotent producer error with Kraft

2021-08-23 Thread Laurynas Butkus (Jira)

Laurynas Butkus created KAFKA-13223:
---

 Summary: Idempotent producer error with Kraft 
 Key: KAFKA-13223
 URL: https://issues.apache.org/jira/browse/KAFKA-13223
 Project: Kafka
  Issue Type: Bug
  Components: kraft, producer 
Reporter: Laurynas Butkus


I get an error *"The broker does not support INIT_PRODUCER_ID"* if I try to 
produce a message idempotence enabled.

Result:
{code:java}
➜  ./bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic 
test --request-required-acks -1 --producer-property enable.idempotence=true
>test
>[2021-08-23 19:40:33,356] ERROR [Producer clientId=console-producer] Aborting 
>producer batches due to fatal error 
>(org.apache.kafka.clients.producer.internals.Sender)
org.apache.kafka.common.errors.UnsupportedVersionException: The broker does not 
support INIT_PRODUCER_ID
[2021-08-23 19:40:33,358] ERROR Error when sending message to topic test with 
key: null, value: 4 bytes with error: 
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.UnsupportedVersionException: The broker does not 
support INIT_PRODUCER_ID
{code}
 

It works fine with idempotence disabled. Also it works fine if using zookeeper.

Tested with altered docker image: 
{code:java}
FROM confluentinc/cp-kafka:6.2.0

RUN sed -i '/KAFKA_ZOOKEEPER_CONNECT/d' /etc/confluent/docker/configure && \
# Docker workaround: Ignore cub zk-ready
sed -i 's/cub zk-ready/echo ignore zk-ready/' /etc/confluent/docker/ensure && \
# KRaft required step: Format the storage directory with a new cluster ID
echo "kafka-storage format --ignore-formatted -t $(kafka-storage random-uuid) 
-c /etc/kafka/kafka.properties" >> /etc/confluent/docker/ensure
{code}
docker-compose.yml
{code:java}
version: '3.4'

services:
  kafka:
build: kafka
restart: unless-stopped
environment:
  ALLOW_PLAINTEXT_LISTENER: "yes"
  KAFKA_HEAP_OPTS: -Xms256m -Xmx256m
  LOG4J_LOGGER_KAFKA: "WARN"
  KAFKA_BROKER_ID: 1
  KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 
'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
  KAFKA_ADVERTISED_LISTENERS: 
'PLAINTEXT://kafka:29092,PLAINTEXT_HOST://127.0.0.1:9092'
  KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
  KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
  KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
  KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
  KAFKA_PROCESS_ROLES: 'broker,controller'
  KAFKA_NODE_ID: 1
  KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka:29093'
  KAFKA_LISTENERS: 
'PLAINTEXT://kafka:29092,CONTROLLER://kafka:29093,PLAINTEXT_HOST://:9092'
  KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
  KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
  KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
ports:
  - "127.0.0.1:9092:9092/tcp"
command: "/etc/confluent/docker/run"
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-770: Replace "buffered.records.per.partition" with "input.buffer.max.bytes"

2021-08-23 Thread Sagar

Thanks Guozhang.

1) Just for my confirmation, when you say we should proceed with the even
distribution of bytes, are you referring to the Proposed Solution in the
KIP or the option you had considered in the JIRA?
2) Default value for the config is something that I missed. I agree we
can't have really large values as it might be detrimental to the
performance. Maybe, as a starting point, we assume that only 1 Stream Task
is running so what could be the ideal value in such a scenario? Somewhere
around 10MB similar to the caching config?
3) When you say,  *a task level metric indicating the current totally
aggregated metrics, * you mean the bytes aggregated at a task level?
4) I am ok with the name change, but would like to know others' thoughts.

Thanks!
Sagar.

On Sun, Aug 22, 2021 at 11:54 PM Guozhang Wang  wrote:

> Thanks Sagar for writing this PR.
>
> I think twice about the options that have been proposed in
> https://issues.apache.org/jira/browse/KAFKA-13152, and feel that at the
> moment it's simpler to just do the even distribution of the configured
> total bytes. My rationale is that right now we have a static tasks ->
> threads mapping, and hence each partition would only be fetched by a single
> thread / consumer at a given time. If in the future we break that static
> mapping into dynamic mapping, then we would not be able to do this even
> distribution. Instead we would have other threads polling from consumer
> only, and those threads would be responsible for checking the config and
> pause non-empty partitions if it goes beyond the threshold. But since at
> that time we would not change the config but just how it would be
> implemented behind the scenes we would not need another KIP to change it.
>
> Some more comments:
>
> 1. We need to discuss a bit about the default value of this new config.
> Personally I think we need to be a bit conservative with large values so
> that it would not have any perf regression compared with old configs
> especially with large topology and large number of partitions.
> 2. I looked at the existing metrics, and do not have corresponding sensors.
> How about also adding a task level metric indicating the current totally
> aggregated metrics. The reason I do not suggest this metric on the
> per-thread level is that in the future we may break the static mapping of
> tasks -> threads.
>
> [optional] As an orthogonal thought, I'm thinking maybe we can rename the
> other "*cache.max.bytes.buffering*" as "statestore.cache.max.bytes" (via
> deprecation of course), piggy-backed in this KIP? Would like to hear
> others' thoughts.
>
>
> Guozhang
>
>
>
> On Sun, Aug 22, 2021 at 9:29 AM Sagar  wrote:
>
> > Hi All,
> >
> > I would like to start a discussion on the following KIP:
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=186878390
> >
> > Thanks!
> > Sagar.
> >
>
>
> --
> -- Guozhang
>

Re: [VOTE] KIP-690: Add additional configuration to control MirrorMaker 2 internal topics naming convention

2021-08-23 Thread Omnia Ibrahim

Thanks, everyone for the feedback and votes. I'm closing the vote with:

   - 3 +1 (binding) votes from Mickael, Gwen and David


   - 4 +1 (non-binding) votes from Ryanne, Igor, Lopo and Josep


On Mon, Aug 23, 2021 at 11:31 AM David Jacot 
wrote:

> +1 (binding). Thanks for the KIP, Omnia.
>
> On Mon, Aug 23, 2021 at 11:42 AM Omnia Ibrahim 
> wrote:
>
> > Can I get one last +1 binding for this KIP, please?
> >
> > On Mon, Aug 16, 2021 at 5:16 PM Omnia Ibrahim 
> > wrote:
> >
> > > Thanks Gwen.
> > > Can I get one more +1 binding for the KIP?
> > >
> > > On Mon, Aug 2, 2021 at 2:51 AM Gwen Shapira  >
> > > wrote:
> > >
> > >> +1 (binding). Thank you for your patience and clear explanations,
> Omnia.
> > >>
> > >> On Mon, Jul 26, 2021 at 3:39 PM Omnia Ibrahim <
> o.g.h.ibra...@gmail.com>
> > >> wrote:
> > >>
> > >> > Bumping up this voting thread.
> > >> >
> > >> > On Fri, Jul 16, 2021 at 1:57 PM Omnia Ibrahim <
> > o.g.h.ibra...@gmail.com>
> > >> > wrote:
> > >> >
> > >> > > Hi,
> > >> > > Can I get 2 more +1 binding for this KIP?
> > >> > > Thanks
> > >> > >
> > >> > > On Fri, Jul 2, 2021 at 5:14 PM Omnia Ibrahim <
> > o.g.h.ibra...@gmail.com
> > >> >
> > >> > > wrote:
> > >> > >
> > >> > >> Hi All,
> > >> > >>
> > >> > >> Just thought of bumping this voting thread again to see if we can
> > >> form a
> > >> > >> consensus around this.
> > >> > >>
> > >> > >> Thanks
> > >> > >>
> > >> > >> On Thu, Jun 24, 2021 at 5:55 PM Mickael Maison <
> > >> > mickael.mai...@gmail.com>
> > >> > >> wrote:
> > >> > >>
> > >> > >>> +1 (binding)
> > >> > >>> Thanks for the KIP!
> > >> > >>>
> > >> > >>> On Tue, May 4, 2021 at 3:23 PM Igor Soarez
> >  > >> >
> > >> > >>> wrote:
> > >> > >>> >
> > >> > >>> > Another +1 here, also non-binding.
> > >> > >>> >
> > >> > >>> > Thank you Omnia!
> > >> > >>> >
> > >> > >>> > --
> > >> > >>> > Igor
> > >> > >>> >
> > >> > >>> >
> > >> > >>> > On Fri, Apr 30, 2021, at 3:15 PM, Ryanne Dolan wrote:
> > >> > >>> > > +1 (non-binding), thanks!
> > >> > >>> > >
> > >> > >>> > > On Thu, Jan 21, 2021, 4:31 AM Omnia Ibrahim <
> > >> > o.g.h.ibra...@gmail.com>
> > >> > >>> wrote:
> > >> > >>> > >
> > >> > >>> > >> Hi
> > >> > >>> > >> Can I get a vote on this, please?
> > >> > >>> > >>
> > >> > >>> > >> Best
> > >> > >>> > >> Omnia
> > >> > >>> > >>
> > >> > >>> > >> On Tue, Dec 15, 2020 at 12:16 PM Omnia Ibrahim <
> > >> > >>> o.g.h.ibra...@gmail.com>
> > >> > >>> > >> wrote:
> > >> > >>> > >>
> > >> > >>> > >>> If anyone interested in reading the discussions you can
> find
> > >> it
> > >> > >>> here
> > >> > >>> > >>>
> > >> https://www.mail-archive.com/dev@kafka.apache.org/msg113373.html
> > >> > >>> > >>>
> > >> > >>> > >>> On Tue, Dec 8, 2020 at 4:01 PM Omnia Ibrahim <
> > >> > >>> o.g.h.ibra...@gmail.com>
> > >> > >>> > >>> wrote:
> > >> > >>> > >>>
> > >> > >>> >  Hi everyone,
> > >> > >>> >  I’m proposing a new KIP for MirrorMaker 2 to add the
> > ability
> > >> to
> > >> > >>> control
> > >> > >>> >  internal topics naming convention. The proposal details
> are
> > >> here
> > >> > >>> > 
> > >> > >>>
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention
> > >> > >>> > 
> > >> > >>> >  Please vote in this thread.
> > >> > >>> >  Thanks
> > >> > >>> >  Omnia
> > >> > >>> > 
> > >> > >>> > >>>
> > >> > >>> > >
> > >> > >>>
> > >> > >>
> > >> >
> > >>
> > >>
> > >> --
> > >> Gwen Shapira
> > >> Engineering Manager | Confluent
> > >> 650.450.2760 | @gwenshap
> > >> Follow us: Twitter | blog
> > >>
> > >
> >
>

Re: [VOTE] KIP-690: Add additional configuration to control MirrorMaker 2 internal topics naming convention

2021-08-23 Thread David Jacot

+1 (binding). Thanks for the KIP, Omnia.

On Mon, Aug 23, 2021 at 11:42 AM Omnia Ibrahim 
wrote:

> Can I get one last +1 binding for this KIP, please?
>
> On Mon, Aug 16, 2021 at 5:16 PM Omnia Ibrahim 
> wrote:
>
> > Thanks Gwen.
> > Can I get one more +1 binding for the KIP?
> >
> > On Mon, Aug 2, 2021 at 2:51 AM Gwen Shapira 
> > wrote:
> >
> >> +1 (binding). Thank you for your patience and clear explanations, Omnia.
> >>
> >> On Mon, Jul 26, 2021 at 3:39 PM Omnia Ibrahim 
> >> wrote:
> >>
> >> > Bumping up this voting thread.
> >> >
> >> > On Fri, Jul 16, 2021 at 1:57 PM Omnia Ibrahim <
> o.g.h.ibra...@gmail.com>
> >> > wrote:
> >> >
> >> > > Hi,
> >> > > Can I get 2 more +1 binding for this KIP?
> >> > > Thanks
> >> > >
> >> > > On Fri, Jul 2, 2021 at 5:14 PM Omnia Ibrahim <
> o.g.h.ibra...@gmail.com
> >> >
> >> > > wrote:
> >> > >
> >> > >> Hi All,
> >> > >>
> >> > >> Just thought of bumping this voting thread again to see if we can
> >> form a
> >> > >> consensus around this.
> >> > >>
> >> > >> Thanks
> >> > >>
> >> > >> On Thu, Jun 24, 2021 at 5:55 PM Mickael Maison <
> >> > mickael.mai...@gmail.com>
> >> > >> wrote:
> >> > >>
> >> > >>> +1 (binding)
> >> > >>> Thanks for the KIP!
> >> > >>>
> >> > >>> On Tue, May 4, 2021 at 3:23 PM Igor Soarez
>  >> >
> >> > >>> wrote:
> >> > >>> >
> >> > >>> > Another +1 here, also non-binding.
> >> > >>> >
> >> > >>> > Thank you Omnia!
> >> > >>> >
> >> > >>> > --
> >> > >>> > Igor
> >> > >>> >
> >> > >>> >
> >> > >>> > On Fri, Apr 30, 2021, at 3:15 PM, Ryanne Dolan wrote:
> >> > >>> > > +1 (non-binding), thanks!
> >> > >>> > >
> >> > >>> > > On Thu, Jan 21, 2021, 4:31 AM Omnia Ibrahim <
> >> > o.g.h.ibra...@gmail.com>
> >> > >>> wrote:
> >> > >>> > >
> >> > >>> > >> Hi
> >> > >>> > >> Can I get a vote on this, please?
> >> > >>> > >>
> >> > >>> > >> Best
> >> > >>> > >> Omnia
> >> > >>> > >>
> >> > >>> > >> On Tue, Dec 15, 2020 at 12:16 PM Omnia Ibrahim <
> >> > >>> o.g.h.ibra...@gmail.com>
> >> > >>> > >> wrote:
> >> > >>> > >>
> >> > >>> > >>> If anyone interested in reading the discussions you can find
> >> it
> >> > >>> here
> >> > >>> > >>>
> >> https://www.mail-archive.com/dev@kafka.apache.org/msg113373.html
> >> > >>> > >>>
> >> > >>> > >>> On Tue, Dec 8, 2020 at 4:01 PM Omnia Ibrahim <
> >> > >>> o.g.h.ibra...@gmail.com>
> >> > >>> > >>> wrote:
> >> > >>> > >>>
> >> > >>> >  Hi everyone,
> >> > >>> >  I’m proposing a new KIP for MirrorMaker 2 to add the
> ability
> >> to
> >> > >>> control
> >> > >>> >  internal topics naming convention. The proposal details are
> >> here
> >> > >>> > 
> >> > >>>
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention
> >> > >>> > 
> >> > >>> >  Please vote in this thread.
> >> > >>> >  Thanks
> >> > >>> >  Omnia
> >> > >>> > 
> >> > >>> > >>>
> >> > >>> > >
> >> > >>>
> >> > >>
> >> >
> >>
> >>
> >> --
> >> Gwen Shapira
> >> Engineering Manager | Confluent
> >> 650.450.2760 | @gwenshap
> >> Follow us: Twitter | blog
> >>
> >
>

Re: [VOTE] KIP-690: Add additional configuration to control MirrorMaker 2 internal topics naming convention

2021-08-23 Thread Omnia Ibrahim

Can I get one last +1 binding for this KIP, please?

On Mon, Aug 16, 2021 at 5:16 PM Omnia Ibrahim 
wrote:

> Thanks Gwen.
> Can I get one more +1 binding for the KIP?
>
> On Mon, Aug 2, 2021 at 2:51 AM Gwen Shapira 
> wrote:
>
>> +1 (binding). Thank you for your patience and clear explanations, Omnia.
>>
>> On Mon, Jul 26, 2021 at 3:39 PM Omnia Ibrahim 
>> wrote:
>>
>> > Bumping up this voting thread.
>> >
>> > On Fri, Jul 16, 2021 at 1:57 PM Omnia Ibrahim 
>> > wrote:
>> >
>> > > Hi,
>> > > Can I get 2 more +1 binding for this KIP?
>> > > Thanks
>> > >
>> > > On Fri, Jul 2, 2021 at 5:14 PM Omnia Ibrahim > >
>> > > wrote:
>> > >
>> > >> Hi All,
>> > >>
>> > >> Just thought of bumping this voting thread again to see if we can
>> form a
>> > >> consensus around this.
>> > >>
>> > >> Thanks
>> > >>
>> > >> On Thu, Jun 24, 2021 at 5:55 PM Mickael Maison <
>> > mickael.mai...@gmail.com>
>> > >> wrote:
>> > >>
>> > >>> +1 (binding)
>> > >>> Thanks for the KIP!
>> > >>>
>> > >>> On Tue, May 4, 2021 at 3:23 PM Igor Soarez > >
>> > >>> wrote:
>> > >>> >
>> > >>> > Another +1 here, also non-binding.
>> > >>> >
>> > >>> > Thank you Omnia!
>> > >>> >
>> > >>> > --
>> > >>> > Igor
>> > >>> >
>> > >>> >
>> > >>> > On Fri, Apr 30, 2021, at 3:15 PM, Ryanne Dolan wrote:
>> > >>> > > +1 (non-binding), thanks!
>> > >>> > >
>> > >>> > > On Thu, Jan 21, 2021, 4:31 AM Omnia Ibrahim <
>> > o.g.h.ibra...@gmail.com>
>> > >>> wrote:
>> > >>> > >
>> > >>> > >> Hi
>> > >>> > >> Can I get a vote on this, please?
>> > >>> > >>
>> > >>> > >> Best
>> > >>> > >> Omnia
>> > >>> > >>
>> > >>> > >> On Tue, Dec 15, 2020 at 12:16 PM Omnia Ibrahim <
>> > >>> o.g.h.ibra...@gmail.com>
>> > >>> > >> wrote:
>> > >>> > >>
>> > >>> > >>> If anyone interested in reading the discussions you can find
>> it
>> > >>> here
>> > >>> > >>>
>> https://www.mail-archive.com/dev@kafka.apache.org/msg113373.html
>> > >>> > >>>
>> > >>> > >>> On Tue, Dec 8, 2020 at 4:01 PM Omnia Ibrahim <
>> > >>> o.g.h.ibra...@gmail.com>
>> > >>> > >>> wrote:
>> > >>> > >>>
>> > >>> >  Hi everyone,
>> > >>> >  I’m proposing a new KIP for MirrorMaker 2 to add the ability
>> to
>> > >>> control
>> > >>> >  internal topics naming convention. The proposal details are
>> here
>> > >>> > 
>> > >>>
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention
>> > >>> > 
>> > >>> >  Please vote in this thread.
>> > >>> >  Thanks
>> > >>> >  Omnia
>> > >>> > 
>> > >>> > >>>
>> > >>> > >
>> > >>>
>> > >>
>> >
>>
>>
>> --
>> Gwen Shapira
>> Engineering Manager | Confluent
>> 650.450.2760 | @gwenshap
>> Follow us: Twitter | blog
>>
>

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #417

2021-08-23 Thread Apache Jenkins Server

See

Conditional Produce in Kafka (KAFKA-2260)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #420

[jira] [Created] (KAFKA-13224) broker.id does not appear in config's originals map when setting just node.id

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #419

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.0 #115

Build failed in Jenkins: Kafka » Kafka Branch Builder » 2.8 #68

Re: [VOTE] KIP-748: Add Broker Count Metrics (restarted)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #418

Re: [DISCUSS] KIP-768: Extend SASL/OAUTHBEARER with Support for OIDC

[jira] [Resolved] (KAFKA-13223) Idempotent producer error with Kraft

[jira] [Reopened] (KAFKA-13223) Idempotent producer error with Kraft

[jira] [Resolved] (KAFKA-13223) Idempotent producer error with Kraft

Jenkins build became unstable: Kafka » Kafka Branch Builder » 3.0 #114

[jira] [Resolved] (KAFKA-13091) Increment HW after shrinking ISR through AlterIsr

[jira] [Created] (KAFKA-13223) Idempotent producer error with Kraft

Re: [DISCUSS] KIP-770: Replace "buffered.records.per.partition" with "input.buffer.max.bytes"

Re: [VOTE] KIP-690: Add additional configuration to control MirrorMaker 2 internal topics naming convention

Re: [VOTE] KIP-690: Add additional configuration to control MirrorMaker 2 internal topics naming convention

Re: [VOTE] KIP-690: Add additional configuration to control MirrorMaker 2 internal topics naming convention

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #417

20 matches

Site Navigation

Mail list logo

Footer information