[jira] [Resolved] (KAFKA-15061) CoordinatorPartitionWriter should reuse buffer

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-15061.
-
Fix Version/s: 3.7.0
   Resolution: Fixed

> CoordinatorPartitionWriter should reuse buffer
> --
>
> Key: KAFKA-15061
> URL: https://issues.apache.org/jira/browse/KAFKA-15061
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: David Jacot
>Assignee: David Jacot
>Priority: Major
>  Labels: kip-848-preview
> Fix For: 3.7.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15972) Add support to exclude labels in client telemetry

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-15972:
-

 Summary: Add support to exclude labels in client telemetry
 Key: KAFKA-15972
 URL: https://issues.apache.org/jira/browse/KAFKA-15972
 Project: Kafka
  Issue Type: Sub-task
Reporter: Apoorv Mittal
Assignee: Apoorv Mittal


Some of the labels/tags which are present in metric should be skipped while 
collecting telemetry as data might already be known to broker hence, we should 
minimize the data transfer. One of such labels is client_id which is already 
present in RequestContext hence broker can append that label prior emitting 
metrics to telemetry backend. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] 3.6.1 RC0

[Justine Olshan]

Not sure if all folks can see this link but the results are here:
https://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/system-test-kafka-branch-builder--1701739886--apache--3.6--ec5faaa90d/2023-12-04--001./2023-12-04--001./report.html


Looks like there were a few failures but most of them were due to known
versioning issues with tests (ie the dev version string causes issues when
trying to assert on it or versions 2.6.3, 2.7.2, and 3.3.2 are not in the
tests -- see PR here where I tried to fix this
https://github.com/apache/kafka/pull/14641) I believe the quota tests have
also been broken for a while and I've talked to folks about fixing them in
the past.

Taking in the dev (3.6.1-SNAPSHOT), 2.6.3, 2.7.2, and 3.3.2, and the quota
tests I think the failures look reasonable.

+1 binding from me given I did the previous validations.

Justine

P.S. I will get to that versioning fix again soon!

On Mon, Dec 4, 2023 at 2:10 PM Justine Olshan  wrote:

> Just wanted to give an update on the system tests since I said I started
> mine last week. I accidentally misconfigured my build, so I restarted them
> today. I should have results and if things look good, I should be able to
> make a vote early tomorrow.
>
> Justine
>
> On Mon, Dec 4, 2023 at 1:55 PM David Arthur
>  wrote:
>
>> I have a fix for KAFKA-15968
>>  here
>> https://github.com/apache/kafka/pull/14919/. After a bit of digging, I
>> found that this behavior has existed in the KRaft controller since the
>> beginning, so it is not a regression.
>>
>> Another thing I observed while investigating this is that MetadataLoader
>> *does* treat CorruptRecordExceptions as fatal, which leads to the crash we
>> want. RaftClient calls handleCommit serially for all its listeners, so if
>> QuorumController#handleCommit is called first and does not crash, the call
>> to MetadataLoader#handleCommit will crash.
>>
>> Considering these two factors, I don't strongly feel like we need to block
>> the release for this fix.
>>
>> -David
>>
>>
>> On Mon, Dec 4, 2023 at 10:49 AM David Arthur 
>> wrote:
>>
>> > Mickael,
>> >
>> > I just filed https://issues.apache.org/jira/browse/KAFKA-15968 while
>> > investigating a log corruption issue on the controller. I'm still
>> > investigating the issue to see how far back this goes, but I think this
>> > could be a blocker.
>> >
>> > Essentially, the bug is that the controller does not treat a
>> > CorruptRecordException as fatal, so the process will continue running.
>> If
>> > this happens on an active controller, it could corrupt the cluster's
>> > metadata in general (since missing a single metadata record can cause
>> lots
>> > of downstream problems).
>> >
>> > I'll update this thread by the end of day with a stronger
>> > blocker/non-blocker opinion.
>> >
>> > Thanks,
>> > David
>> >
>> >
>> > On Mon, Dec 4, 2023 at 6:48 AM Luke Chen  wrote:
>> >
>> >> Hi Mickael:
>> >>
>> >> I did:
>> >>1. Validated all checksums, signatures, and hashes
>> >>2. Ran quick start for KRaft using scala 2.12 artifacts
>> >>3. Spot checked the documentation and Javadoc
>> >>4. Validated the licence file
>> >>
>> >> When running the validation to scala 2.12 package, I found these
>> libraries
>> >> are missing: (We only include scala 2.13 libraries in licence file)
>> >> scala-java8-compat_2.12-1.0.2 is missing in license file
>> >> scala-library-2.12.18 is missing in license file
>> >> scala-logging_2.12-3.9.4 is missing in license file
>> >> scala-reflect-2.12.18 is missing in license file
>> >>
>> >> It looks like this issue has been there for a long time, so it won't
>> be a
>> >> block issue for v3.6.1.
>> >>
>> >> +1 (binding) from me.
>> >>
>> >> Thank you.
>> >> Luke
>> >>
>> >> On Sat, Dec 2, 2023 at 5:46 AM Bill Bejeck  wrote:
>> >>
>> >> > Hi Mickael,
>> >> >
>> >> > I did the following:
>> >> >
>> >> >1. Validated all checksums, signatures, and hashes
>> >> >2. Built from source
>> >> >3. Ran all the unit tests
>> >> >4. Spot checked the documentation and Javadoc
>> >> >5. Ran the ZK, Kraft, and Kafka Streams quickstart guides
>> >> >
>> >> > I did notice that the `fillDotVersion` in `js/templateData.js` needs
>> >> > updating to `3.6.1`, but this is minor and should not block the
>> release.
>> >> >
>> >> > It's a +1(binding) for me, pending the successful system test run
>> >> >
>> >> > Thanks,
>> >> > Bill
>> >> >
>> >> > On Fri, Dec 1, 2023 at 1:49 PM Justine Olshan
>> >> > >> > >
>> >> > wrote:
>> >> >
>> >> > > I've started a system test run on my end.
>> >> > >
>> >> > > Justine
>> >> > >
>> >> > > On Wed, Nov 29, 2023 at 1:55 PM Justine Olshan <
>> jols...@confluent.io>
>> >> > > wrote:
>> >> > >
>> >> > > > I built from source and ran a simple transactional produce
>> bench. I
>> >> > ran a
>> >> > > > handful of unit tests as well.
>> >> > > > I scanned the docs and everything looked reasonable.
>> >> > > >
>> 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2447

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 334458 lines...]
> Task :connect:json:publishMavenJavaPublicationToMavenLocal
> Task :connect:json:publishToMavenLocal
> Task :server:compileTestJava
> Task :server:testClasses
> Task :server-common:compileTestJava
> Task :server-common:testClasses
> Task :raft:compileTestJava
> Task :raft:testClasses
> Task :core:compileScala
> Task :group-coordinator:compileTestJava
> Task :group-coordinator:testClasses

> Task :clients:javadoc
/home/jenkins/workspace/Kafka_kafka_trunk/clients/src/main/java/org/apache/kafka/clients/admin/ScramMechanism.java:32:
 warning - Tag @see: missing final '>': "https://cwiki.apache.org/confluence/display/KAFKA/KIP-554%3A+Add+Broker-side+SCRAM+Config+API;>KIP-554:
 Add Broker-side SCRAM Config API

 This code is duplicated in 
org.apache.kafka.common.security.scram.internals.ScramMechanism.
 The type field in both files must match and must not change. The type field
 is used both for passing ScramCredentialUpsertion and for the internal
 UserScramCredentialRecord. Do not change the type field."
/home/jenkins/workspace/Kafka_kafka_trunk/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/package-info.java:21:
 warning - Tag @link: reference not found: 
org.apache.kafka.common.security.oauthbearer

> Task :streams:generateMetadataFileForMavenJavaPublication

> Task :clients:javadoc
2 warnings

> Task :metadata:compileTestJava
> Task :metadata:testClasses
> Task :clients:javadocJar
> Task :clients:srcJar
> Task :clients:testJar
> Task :clients:testSrcJar
> Task :clients:publishMavenJavaPublicationToMavenLocal
> Task :clients:publishToMavenLocal
> Task :connect:api:generateMetadataFileForMavenJavaPublication
> Task :connect:api:compileTestJava UP-TO-DATE
> Task :connect:api:testClasses UP-TO-DATE
> Task :connect:api:testJar
> Task :connect:api:testSrcJar
> Task :connect:api:publishMavenJavaPublicationToMavenLocal
> Task :connect:api:publishToMavenLocal
> Task :streams:javadoc
> Task :streams:javadocJar
> Task :streams:srcJar
> Task :streams:processTestResources UP-TO-DATE
> Task :core:classes
> Task :core:compileTestJava NO-SOURCE
> Task :core:compileTestScala
> Task :core:testClasses
> Task :streams:compileTestJava
> Task :streams:testClasses
> Task :streams:testJar
> Task :streams:testSrcJar
> Task :streams:publishMavenJavaPublicationToMavenLocal
> Task :streams:publishToMavenLocal

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings 
and determine if they come from your own scripts or plugins.

For more on this, please refer to 
https://docs.gradle.org/8.5/userguide/command_line_interface.html#sec:command_line_warnings
 in the Gradle documentation.

BUILD SUCCESSFUL in 5m 20s
95 actionable tasks: 41 executed, 54 up-to-date

Publishing build scan...
https://ge.apache.org/s/ds35tfblh3nhe

[Pipeline] sh
+ grep ^version= gradle.properties
+ cut -d= -f 2
[Pipeline] dir
Running in /home/jenkins/workspace/Kafka_kafka_trunk/streams/quickstart
[Pipeline] {
[Pipeline] sh
+ mvn clean install -Dgpg.skip
[INFO] Scanning for projects...
[INFO] 
[INFO] Reactor Build Order:
[INFO] 
[INFO] Kafka Streams :: Quickstart[pom]
[INFO] streams-quickstart-java[maven-archetype]
[INFO] 
[INFO] < org.apache.kafka:streams-quickstart >-
[INFO] Building Kafka Streams :: Quickstart 3.7.0-SNAPSHOT[1/2]
[INFO]   from pom.xml
[INFO] [ pom ]-
[INFO] 
[INFO] --- clean:3.0.0:clean (default-clean) @ streams-quickstart ---
[INFO] 
[INFO] --- remote-resources:1.5:process (process-resource-bundles) @ 
streams-quickstart ---
[INFO] 
[INFO] --- site:3.5.1:attach-descriptor (attach-descriptor) @ 
streams-quickstart ---
[INFO] 
[INFO] --- gpg:1.6:sign (sign-artifacts) @ streams-quickstart ---
[INFO] 
[INFO] --- install:2.5.2:install (default-install) @ streams-quickstart ---
[INFO] Installing 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/quickstart/pom.xml to 
/home/jenkins/.m2/repository/org/apache/kafka/streams-quickstart/3.7.0-SNAPSHOT/streams-quickstart-3.7.0-SNAPSHOT.pom
[INFO] 
[INFO] --< org.apache.kafka:streams-quickstart-java >--
[INFO] Building streams-quickstart-java 3.7.0-SNAPSHOT[2/2]
[INFO]   from java/pom.xml
[INFO] --[ maven-archetype ]---
[INFO] 
[INFO] --- clean:3.0.0:clean (default-clean) @ streams-quickstart-java ---
[INFO] 
[INFO] --- remote-resources:1.5:process (process-resource-bundles) @ 
streams-quickstart-java ---
[INFO] 
[INFO] --- resources:2.7:resources 

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.6 #124

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2446

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 129942 lines...]
> Task :server:compileTestJava
> Task :server:testClasses
> Task :server-common:compileTestJava
> Task :server-common:testClasses
> Task :raft:compileTestJava
> Task :raft:testClasses

> Task :clients:javadoc
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/clients/src/main/java/org/apache/kafka/clients/admin/ScramMechanism.java:32:
 warning - Tag @see: missing final '>': "https://cwiki.apache.org/confluence/display/KAFKA/KIP-554%3A+Add+Broker-side+SCRAM+Config+API;>KIP-554:
 Add Broker-side SCRAM Config API

 This code is duplicated in 
org.apache.kafka.common.security.scram.internals.ScramMechanism.
 The type field in both files must match and must not change. The type field
 is used both for passing ScramCredentialUpsertion and for the internal
 UserScramCredentialRecord. Do not change the type field."

> Task :core:compileScala
> Task :group-coordinator:compileTestJava
> Task :group-coordinator:testClasses
> Task :streams:generateMetadataFileForMavenJavaPublication

> Task :clients:javadoc
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/package-info.java:21:
 warning - Tag @link: reference not found: 
org.apache.kafka.common.security.oauthbearer
2 warnings

> Task :clients:javadocJar
> Task :metadata:compileTestJava
> Task :metadata:testClasses
> Task :clients:srcJar
> Task :clients:testJar
> Task :clients:testSrcJar
> Task :clients:publishMavenJavaPublicationToMavenLocal
> Task :clients:publishToMavenLocal
> Task :connect:api:generateMetadataFileForMavenJavaPublication
> Task :connect:api:compileTestJava UP-TO-DATE
> Task :connect:api:testClasses UP-TO-DATE
> Task :connect:api:testJar
> Task :connect:api:testSrcJar
> Task :connect:api:publishMavenJavaPublicationToMavenLocal
> Task :connect:api:publishToMavenLocal
> Task :streams:javadoc
> Task :streams:javadocJar
> Task :streams:srcJar
> Task :streams:processTestResources UP-TO-DATE
> Task :core:classes
> Task :core:compileTestJava NO-SOURCE
> Task :core:compileTestScala
> Task :core:testClasses
> Task :streams:compileTestJava
> Task :streams:testClasses
> Task :streams:testJar
> Task :streams:testSrcJar
> Task :streams:publishMavenJavaPublicationToMavenLocal
> Task :streams:publishToMavenLocal

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings 
and determine if they come from your own scripts or plugins.

For more on this, please refer to 
https://docs.gradle.org/8.5/userguide/command_line_interface.html#sec:command_line_warnings
 in the Gradle documentation.

BUILD SUCCESSFUL in 5m 19s
95 actionable tasks: 41 executed, 54 up-to-date

Publishing build scan...
https://ge.apache.org/s/tcfhj2fy3tiew

[Pipeline] sh
+ grep ^version= gradle.properties
+ cut -d= -f 2
[Pipeline] dir
Running in 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/quickstart
[Pipeline] {
[Pipeline] sh
+ mvn clean install -Dgpg.skip
[INFO] Scanning for projects...
[INFO] 
[INFO] Reactor Build Order:
[INFO] 
[INFO] Kafka Streams :: Quickstart[pom]
[INFO] streams-quickstart-java[maven-archetype]
[INFO] 
[INFO] < org.apache.kafka:streams-quickstart >-
[INFO] Building Kafka Streams :: Quickstart 3.7.0-SNAPSHOT[1/2]
[INFO]   from pom.xml
[INFO] [ pom ]-
[INFO] 
[INFO] --- clean:3.0.0:clean (default-clean) @ streams-quickstart ---
[INFO] 
[INFO] --- remote-resources:1.5:process (process-resource-bundles) @ 
streams-quickstart ---
[INFO] 
[INFO] --- site:3.5.1:attach-descriptor (attach-descriptor) @ 
streams-quickstart ---
[INFO] 
[INFO] --- gpg:1.6:sign (sign-artifacts) @ streams-quickstart ---
[INFO] 
[INFO] --- install:2.5.2:install (default-install) @ streams-quickstart ---
[INFO] Installing 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/quickstart/pom.xml
 to 
/home/jenkins/.m2/repository/org/apache/kafka/streams-quickstart/3.7.0-SNAPSHOT/streams-quickstart-3.7.0-SNAPSHOT.pom
[INFO] 
[INFO] --< org.apache.kafka:streams-quickstart-java >--
[INFO] Building streams-quickstart-java 3.7.0-SNAPSHOT[2/2]
[INFO]   from java/pom.xml
[INFO] --[ maven-archetype ]---
[INFO] 
[INFO] --- clean:3.0.0:clean (default-clean) @ streams-quickstart-java ---
[INFO] 
[INFO] --- remote-resources:1.5:process (process-resource-bundles) @ 
streams-quickstart-java ---
[INFO] 
[INFO] --- resources:2.7:resources (default-resources) @ 
streams-quickstart-java 

Re: [DISCUSS] KIP-967: Support custom SSL configuration for Kafka Connect RestServer

[Greg Harris]

Hi Chris,

Thank you for your comments above. I disagree with your recommendation
for a new SslEngineFactory variant/hierarchy.

1. A superinterface could be more confusing to users. Since this is an
interface in `clients`, the connect-specific interface would also need
to be in clients, despite being superfluous for clients users and
broker developers.
2. A user could implement the reduced interface, and then have issues
loading their implementation in a broker, and need to find
documentation/javadocs to explain to them why.
3. This interface has been in use since 2020, so I don't believe that
the burden of implementing these methods has been excessive. I assume
there's at least one "static" implementation out there that would have
benefitted from the proposed super-interface, but they managed to
adapt to the standardized interface.
4. Implementations that don't want to provide basic implementations
can throw UnsupportedOperationException from the extra methods as a
last resort.

On the other hand, how much would it take to support the full suite of
SslEngineFactory operations in Connect? Could part of this KIP be
improving Connect to make use of these methods? This would help unify
the experience for users that implement the interface specifically for
the dynamic reconfiguration support, and rely on it on the broker
side.

Taras, are you interested in dynamic SSL reconfiguration in Connect?
Would you be willing to investigate the details of that for the KIP?

Thanks,
Greg

On Mon, Dec 4, 2023 at 1:17 PM Chris Egerton  wrote:
>
> Hi Taras,
>
> Regarding slimming down the interface: IMO, we should do this right the
> first time, and that includes not requiring unnecessary methods from users.
> I think BaseSslEngineFactory is good enough as a superinterface.
>
>
> Regarding the parsing logic: I think the KIP needs to be more explicit. We
> should add something like this to the proposed changes section:
>
> "If any properties are present in the worker config with a prefix of
> "listeners.https.", then only properties with that prefix will be passed to
> the SSL engine factory. Otherwise, all top-level worker properties will be
> passed to the SSL engine factory. Note that this differs slightly from
> existing logic in that the set of properties (prefixed or otherwise) will
> not be filtered based on a predefined set of keys; this will enable custom
> SSL engine factories to define and accept custom properties."
>
> I also took a quick look at the prototype (I usually try not to do this
> since we vote on KIP documents, not PRs). I don't think we should populate
> default values for SSL-related properties before sending properties to the
> SSL engine factory, since it may confuse users who have written custom SSL
> engine factories to see that properties not specified in their Connect
> worker config are being passed to their factory. Instead, the default SSL
> engine factory used by Connect can perform this logic, and we can let other
> custom factories be responsible for their own default values.
>
>
> Cheers,
>
> Chris
>
> On Wed, Nov 29, 2023 at 8:36 AM Taras Ledkov  wrote:
>
> > Hi team,
> >
> > Ping for review / vote for KIP-967 [1].
> > Voting thread is here [2]
> >
> > [1].
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-967%3A+Support+custom+SSL+configuration+for+Kafka+Connect+RestServer
> > [2]. https://github.com/apache/kafka/pull/14203
> > [2]. https://lists.apache.org/thread/wc4v5v3pynl15g1q547m2croqsqgmzpw
> >
> > --
> > With best regards,
> > Taras Ledkov
> >

Re: [VOTE] 3.6.1 RC0

[Justine Olshan]

Just wanted to give an update on the system tests since I said I started
mine last week. I accidentally misconfigured my build, so I restarted them
today. I should have results and if things look good, I should be able to
make a vote early tomorrow.

Justine

On Mon, Dec 4, 2023 at 1:55 PM David Arthur
 wrote:

> I have a fix for KAFKA-15968
>  here
> https://github.com/apache/kafka/pull/14919/. After a bit of digging, I
> found that this behavior has existed in the KRaft controller since the
> beginning, so it is not a regression.
>
> Another thing I observed while investigating this is that MetadataLoader
> *does* treat CorruptRecordExceptions as fatal, which leads to the crash we
> want. RaftClient calls handleCommit serially for all its listeners, so if
> QuorumController#handleCommit is called first and does not crash, the call
> to MetadataLoader#handleCommit will crash.
>
> Considering these two factors, I don't strongly feel like we need to block
> the release for this fix.
>
> -David
>
>
> On Mon, Dec 4, 2023 at 10:49 AM David Arthur 
> wrote:
>
> > Mickael,
> >
> > I just filed https://issues.apache.org/jira/browse/KAFKA-15968 while
> > investigating a log corruption issue on the controller. I'm still
> > investigating the issue to see how far back this goes, but I think this
> > could be a blocker.
> >
> > Essentially, the bug is that the controller does not treat a
> > CorruptRecordException as fatal, so the process will continue running. If
> > this happens on an active controller, it could corrupt the cluster's
> > metadata in general (since missing a single metadata record can cause
> lots
> > of downstream problems).
> >
> > I'll update this thread by the end of day with a stronger
> > blocker/non-blocker opinion.
> >
> > Thanks,
> > David
> >
> >
> > On Mon, Dec 4, 2023 at 6:48 AM Luke Chen  wrote:
> >
> >> Hi Mickael:
> >>
> >> I did:
> >>1. Validated all checksums, signatures, and hashes
> >>2. Ran quick start for KRaft using scala 2.12 artifacts
> >>3. Spot checked the documentation and Javadoc
> >>4. Validated the licence file
> >>
> >> When running the validation to scala 2.12 package, I found these
> libraries
> >> are missing: (We only include scala 2.13 libraries in licence file)
> >> scala-java8-compat_2.12-1.0.2 is missing in license file
> >> scala-library-2.12.18 is missing in license file
> >> scala-logging_2.12-3.9.4 is missing in license file
> >> scala-reflect-2.12.18 is missing in license file
> >>
> >> It looks like this issue has been there for a long time, so it won't be
> a
> >> block issue for v3.6.1.
> >>
> >> +1 (binding) from me.
> >>
> >> Thank you.
> >> Luke
> >>
> >> On Sat, Dec 2, 2023 at 5:46 AM Bill Bejeck  wrote:
> >>
> >> > Hi Mickael,
> >> >
> >> > I did the following:
> >> >
> >> >1. Validated all checksums, signatures, and hashes
> >> >2. Built from source
> >> >3. Ran all the unit tests
> >> >4. Spot checked the documentation and Javadoc
> >> >5. Ran the ZK, Kraft, and Kafka Streams quickstart guides
> >> >
> >> > I did notice that the `fillDotVersion` in `js/templateData.js` needs
> >> > updating to `3.6.1`, but this is minor and should not block the
> release.
> >> >
> >> > It's a +1(binding) for me, pending the successful system test run
> >> >
> >> > Thanks,
> >> > Bill
> >> >
> >> > On Fri, Dec 1, 2023 at 1:49 PM Justine Olshan
> >>  >> > >
> >> > wrote:
> >> >
> >> > > I've started a system test run on my end.
> >> > >
> >> > > Justine
> >> > >
> >> > > On Wed, Nov 29, 2023 at 1:55 PM Justine Olshan <
> jols...@confluent.io>
> >> > > wrote:
> >> > >
> >> > > > I built from source and ran a simple transactional produce bench.
> I
> >> > ran a
> >> > > > handful of unit tests as well.
> >> > > > I scanned the docs and everything looked reasonable.
> >> > > >
> >> > > > I was wondering if we got the system test results mentioned >
> System
> >> > > > tests: Still running I'll post an update once they complete.
> >> > > >
> >> > > > Justine
> >> > > >
> >> > > > On Wed, Nov 29, 2023 at 6:33 AM Mickael Maison <
> >> > mickael.mai...@gmail.com
> >> > > >
> >> > > > wrote:
> >> > > >
> >> > > >> Hi Josep,
> >> > > >>
> >> > > >> Good catch!
> >> > > >> If it's the only issue we find, I don't think we should block the
> >> > > >> release just to fix that.
> >> > > >>
> >> > > >> If we find another issue, I'll backport it before running another
> >> RC,
> >> > > >> otherwise I'll backport it once 3.6.1 is released.
> >> > > >>
> >> > > >> Thanks,
> >> > > >> Mickael
> >> > > >>
> >> > > >> On Wed, Nov 29, 2023 at 11:55 AM Josep Prat
> >> >  >> > > >
> >> > > >> wrote:
> >> > > >> >
> >> > > >> > Hi Mickael,
> >> > > >> > This PR[1] made me realize NOTICE-binary is missing the notice
> >> for
> >> > > >> > commons-io. I don't know if it's a blocker or not. I can cherry
> >> pick
> >> > > the
> >> > > >> > commit to the 3.6 branch if you want.
> >> > > >> >
> >> > > >> > 

Re: [VOTE] 3.6.1 RC0

[David Arthur]

I have a fix for KAFKA-15968
 here
https://github.com/apache/kafka/pull/14919/. After a bit of digging, I
found that this behavior has existed in the KRaft controller since the
beginning, so it is not a regression.

Another thing I observed while investigating this is that MetadataLoader
*does* treat CorruptRecordExceptions as fatal, which leads to the crash we
want. RaftClient calls handleCommit serially for all its listeners, so if
QuorumController#handleCommit is called first and does not crash, the call
to MetadataLoader#handleCommit will crash.

Considering these two factors, I don't strongly feel like we need to block
the release for this fix.

-David


On Mon, Dec 4, 2023 at 10:49 AM David Arthur 
wrote:

> Mickael,
>
> I just filed https://issues.apache.org/jira/browse/KAFKA-15968 while
> investigating a log corruption issue on the controller. I'm still
> investigating the issue to see how far back this goes, but I think this
> could be a blocker.
>
> Essentially, the bug is that the controller does not treat a
> CorruptRecordException as fatal, so the process will continue running. If
> this happens on an active controller, it could corrupt the cluster's
> metadata in general (since missing a single metadata record can cause lots
> of downstream problems).
>
> I'll update this thread by the end of day with a stronger
> blocker/non-blocker opinion.
>
> Thanks,
> David
>
>
> On Mon, Dec 4, 2023 at 6:48 AM Luke Chen  wrote:
>
>> Hi Mickael:
>>
>> I did:
>>1. Validated all checksums, signatures, and hashes
>>2. Ran quick start for KRaft using scala 2.12 artifacts
>>3. Spot checked the documentation and Javadoc
>>4. Validated the licence file
>>
>> When running the validation to scala 2.12 package, I found these libraries
>> are missing: (We only include scala 2.13 libraries in licence file)
>> scala-java8-compat_2.12-1.0.2 is missing in license file
>> scala-library-2.12.18 is missing in license file
>> scala-logging_2.12-3.9.4 is missing in license file
>> scala-reflect-2.12.18 is missing in license file
>>
>> It looks like this issue has been there for a long time, so it won't be a
>> block issue for v3.6.1.
>>
>> +1 (binding) from me.
>>
>> Thank you.
>> Luke
>>
>> On Sat, Dec 2, 2023 at 5:46 AM Bill Bejeck  wrote:
>>
>> > Hi Mickael,
>> >
>> > I did the following:
>> >
>> >1. Validated all checksums, signatures, and hashes
>> >2. Built from source
>> >3. Ran all the unit tests
>> >4. Spot checked the documentation and Javadoc
>> >5. Ran the ZK, Kraft, and Kafka Streams quickstart guides
>> >
>> > I did notice that the `fillDotVersion` in `js/templateData.js` needs
>> > updating to `3.6.1`, but this is minor and should not block the release.
>> >
>> > It's a +1(binding) for me, pending the successful system test run
>> >
>> > Thanks,
>> > Bill
>> >
>> > On Fri, Dec 1, 2023 at 1:49 PM Justine Olshan
>> > > >
>> > wrote:
>> >
>> > > I've started a system test run on my end.
>> > >
>> > > Justine
>> > >
>> > > On Wed, Nov 29, 2023 at 1:55 PM Justine Olshan 
>> > > wrote:
>> > >
>> > > > I built from source and ran a simple transactional produce bench. I
>> > ran a
>> > > > handful of unit tests as well.
>> > > > I scanned the docs and everything looked reasonable.
>> > > >
>> > > > I was wondering if we got the system test results mentioned > System
>> > > > tests: Still running I'll post an update once they complete.
>> > > >
>> > > > Justine
>> > > >
>> > > > On Wed, Nov 29, 2023 at 6:33 AM Mickael Maison <
>> > mickael.mai...@gmail.com
>> > > >
>> > > > wrote:
>> > > >
>> > > >> Hi Josep,
>> > > >>
>> > > >> Good catch!
>> > > >> If it's the only issue we find, I don't think we should block the
>> > > >> release just to fix that.
>> > > >>
>> > > >> If we find another issue, I'll backport it before running another
>> RC,
>> > > >> otherwise I'll backport it once 3.6.1 is released.
>> > > >>
>> > > >> Thanks,
>> > > >> Mickael
>> > > >>
>> > > >> On Wed, Nov 29, 2023 at 11:55 AM Josep Prat
>> > > > > >
>> > > >> wrote:
>> > > >> >
>> > > >> > Hi Mickael,
>> > > >> > This PR[1] made me realize NOTICE-binary is missing the notice
>> for
>> > > >> > commons-io. I don't know if it's a blocker or not. I can cherry
>> pick
>> > > the
>> > > >> > commit to the 3.6 branch if you want.
>> > > >> >
>> > > >> > Best,
>> > > >> >
>> > > >> >
>> > > >> > [1]: https://github.com/apache/kafka/pull/14865
>> > > >> >
>> > > >> > On Tue, Nov 28, 2023 at 10:25 AM Josep Prat > >
>> > > >> wrote:
>> > > >> >
>> > > >> > > Hi Mickael,
>> > > >> > > Thanks for running the release. It's a +1 for me (non-binding).
>> > > >> > > I did the following:
>> > > >> > > - Verified artifact's signatures and hashes
>> > > >> > > - Checked JavaDoc (with navigation to Oracle JavaDoc)
>> > > >> > > - Compiled source code
>> > > >> > > - Run unit tests and integration tests
>> > > >> > > - Run getting started with ZK and KRaft
>> > > >> > >
>> > > >> 

Re: [DISCUSS] KIP-967: Support custom SSL configuration for Kafka Connect RestServer

[Chris Egerton]

Hi Taras,

Regarding slimming down the interface: IMO, we should do this right the
first time, and that includes not requiring unnecessary methods from users.
I think BaseSslEngineFactory is good enough as a superinterface.


Regarding the parsing logic: I think the KIP needs to be more explicit. We
should add something like this to the proposed changes section:

"If any properties are present in the worker config with a prefix of
"listeners.https.", then only properties with that prefix will be passed to
the SSL engine factory. Otherwise, all top-level worker properties will be
passed to the SSL engine factory. Note that this differs slightly from
existing logic in that the set of properties (prefixed or otherwise) will
not be filtered based on a predefined set of keys; this will enable custom
SSL engine factories to define and accept custom properties."

I also took a quick look at the prototype (I usually try not to do this
since we vote on KIP documents, not PRs). I don't think we should populate
default values for SSL-related properties before sending properties to the
SSL engine factory, since it may confuse users who have written custom SSL
engine factories to see that properties not specified in their Connect
worker config are being passed to their factory. Instead, the default SSL
engine factory used by Connect can perform this logic, and we can let other
custom factories be responsible for their own default values.


Cheers,

Chris

On Wed, Nov 29, 2023 at 8:36 AM Taras Ledkov  wrote:

> Hi team,
>
> Ping for review / vote for KIP-967 [1].
> Voting thread is here [2]
>
> [1].
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-967%3A+Support+custom+SSL+configuration+for+Kafka+Connect+RestServer
> [2]. https://github.com/apache/kafka/pull/14203
> [2]. https://lists.apache.org/thread/wc4v5v3pynl15g1q547m2croqsqgmzpw
>
> --
> With best regards,
> Taras Ledkov
>

[jira] [Resolved] (KAFKA-15533) Ensure HeartbeatRequestManager only send out some fields once

[Andrew Schofield (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Schofield resolved KAFKA-15533.
--
Resolution: Resolved

Already resolved

> Ensure HeartbeatRequestManager only send out some fields once
> -
>
> Key: KAFKA-15533
> URL: https://issues.apache.org/jira/browse/KAFKA-15533
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, consumer
>Reporter: Philip Nee
>Assignee: Philip Nee
>Priority: Minor
>  Labels: kip-848, kip-848-client-support
>
> We want to ensure ConsumerGroupHeartbeatRequest is as lightweight as 
> possible, so a lot of fields in it don't need to be resend. An example would 
> be the rebalanceTimeoutMs, currently we have the following code:
>  
>  
> {code:java}
> ConsumerGroupHeartbeatRequestData data = new 
> ConsumerGroupHeartbeatRequestData()
> .setGroupId(membershipManager.groupId())
> .setMemberEpoch(membershipManager.memberEpoch())
> .setMemberId(membershipManager.memberId())
> .setRebalanceTimeoutMs(rebalanceTimeoutMs); {code}
>  
>  
> We should encapsulate these once-used fields into a class such as 
> HeartbeatMetdataBuilder, and it should maintain a state of whether a certain 
> field needs to be sent or not.
>  
> Note that, currently only 3 fields are mandatory in the request:
>  * groupId
>  * memberEpoch
>  * memberId
> Note that on retriable errors and network errors (ex. timeout) a full request 
> should be sent to the broker.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15971) Re-enable consumer integration tests for new consumer

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-15971:


 Summary: Re-enable consumer integration tests for new consumer
 Key: KAFKA-15971
 URL: https://issues.apache.org/jira/browse/KAFKA-15971
 Project: Kafka
  Issue Type: Improvement
  Components: clients
Affects Versions: 3.7.0
Reporter: Andrew Schofield
Assignee: Andrew Schofield
 Fix For: 3.7.0


Re-enable the consumer integration tests for the new consumer making sure that 
build stability is not impacted.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2445

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-15970) KIP-951, port newly added tests in FetcherTest.java to FetchRequestManagerTest.ajva

[Mayank Shekhar Narula (Jira)]

Mayank Shekhar Narula created KAFKA-15970:
-

 Summary: KIP-951, port newly added tests in FetcherTest.java to 
FetchRequestManagerTest.ajva
 Key: KAFKA-15970
 URL: https://issues.apache.org/jira/browse/KAFKA-15970
 Project: Kafka
  Issue Type: Improvement
  Components: clients
Affects Versions: 3.7.0
Reporter: Mayank Shekhar Narula
Assignee: Mayank Shekhar Narula


Java client changes for 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-951%3A+Leader+discovery+optimisations+for+the+client



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15965) Test failure: org.apache.kafka.common.requests.BrokerRegistrationRequestTest

[Jun Rao (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-15965.
-
Fix Version/s: 3.7.0
 Assignee: Colin McCabe
   Resolution: Fixed

This is fixed by https://github.com/apache/kafka/pull/14887.

> Test failure: org.apache.kafka.common.requests.BrokerRegistrationRequestTest
> 
>
> Key: KAFKA-15965
> URL: https://issues.apache.org/jira/browse/KAFKA-15965
> Project: Kafka
>  Issue Type: Bug
>Reporter: Apoorv Mittal
>Assignee: Colin McCabe
>Priority: Major
> Fix For: 3.7.0
>
>
> 2 tests for versions 0 and 1 fails consistently.
> Build: 
> https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-14767/15/tests/
>  
> {code:java}
> org.opentest4j.AssertionFailedError: 
> BrokerRegistrationRequestData(brokerId=0, clusterId='test', 
> incarnationId=Xil73H5bSZ2vYTWUVlf07Q, listeners=[], features=[], rack='a', 
> isMigratingZkBroker=false, logDirs=[], previousBrokerEpoch=1) ==> expected: 
> <-1> but was: <1>Stacktraceorg.opentest4j.AssertionFailedError: 
> BrokerRegistrationRequestData(brokerId=0, clusterId='test', 
> incarnationId=Xil73H5bSZ2vYTWUVlf07Q, listeners=[], features=[], rack='a', 
> isMigratingZkBroker=false, logDirs=[], previousBrokerEpoch=1) ==> expected: 
> <-1> but was: <1>  at 
> app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
>at 
> app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
>at 
> app//org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)  
> at 
> app//org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:166)  
> at app//org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:660)
>   at 
> app//org.apache.kafka.common.requests.BrokerRegistrationRequestTest.testBasicBuild(BrokerRegistrationRequestTest.java:57)
> at 
> java.base@17.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native 
> Method)at 
> java.base@17.0.7/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
>   at 
> java.base@17.0.7/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.base@17.0.7/java.lang.reflect.Method.invoke(Method.java:568)
> at 
> app//org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:728)
>   at 
> app//org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
>at 
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
>  at 
> app//org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
> at 
> app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
>   at 
> app//org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestTemplateMethod(TimeoutExtension.java:94)
>at 
> app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
> at 
> app//org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
>  at 
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
> at 
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
>at 
> app//org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
>  {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-858: Handle JBOD broker disk failure in KRaft

[Igor Soarez]

Hi Viktor,

Thanks for pointing this out.

I forgot to make this clear in the KIP. I'll update it.

ClusterAction on Cluster resource is exactly right,
see `ControllerApis.handleAssignReplicasToDirs`. [1]

--
Igor

[1]: 
https://github.com/apache/kafka/pull/14863/files#diff-91060c918c99d25342f625c146f14425716eda9d8fcfe1126b2c45feff388362R1073

On Mon, Dec 4, 2023, at 12:28 PM, Viktor Somogyi-Vass wrote:
> Hi Igor,
> 
> I'm just reading through your KIP and noticed that the new protocol you
> created doesn't say anything about ACLs of the new AssignReplicasToDirs
> API. Would it make sense to authorize these requests as other inter-broker
> protocol calls are usually authorized, that is ClusterAction on Cluster
> resource?
> 
> Thanks,
> Viktor
> 
> On Tue, Nov 28, 2023 at 4:18 PM Igor Soarez  wrote:
> 
> > Hi everyone,
> >
> > There have been a number of further changes.
> >
> > I have updated the KIP to reflect them, but for reference,
> > I'd also like to update this thread with a summary.
> >
> > 1. The reserved Uuids and their names for directories have changed.
> > The first 100 Uuids are reserved for future use.
> >
> > 2. During the ZooKeeper to KRaft migration, if a broker still
> > configured in ZK mode has any log directory offline, it will
> > shutdown and refuse to startup. The expectation is that this
> > escalation from a log directory's unavailability to the entire
> > broker's unavailability will be temporary, limited to the migration
> > period. And that the simplification will help develop, test and
> > support this feature.
> >
> > 3. The representation of replica directories in metadata records is
> > no longer tightly coupled with the respective broker IDs. Instead of
> > replacing the int[] replicas field in PartitionRecord and
> > PartitionChangeRecord, we are instead introducing a new field Uuid[]
> > named directories, that should be kept in the same size and order as
> > the existing replicas field. See
> > https://github.com/apache/kafka/pull/14290 for further details.
> >
> > 4. Assignments that are respective to failed log directories are no
> > longer prioritized. Previously the KIP proposed prioritizing
> > assignments that related to failed log directories, aiming to
> > synchronize the necessary replica to directory mapping on the
> > controller before handling the directory failure. Recently, we have
> > decided to remove any prioritization of these assignments, as
> > delaying the reporting of directory failures is considered
> > detrimental for any reason
> >
> > 5. Uuids for log directories that failed after startup are always
> > included in every broker heartbeat request. Previously the KIP
> > proposed sending Uuids for failed directories in the broker
> > heartbeat until a successful reply is received. However, due to the
> > overload mode handling of broker heartbeats, because broker
> > heartbeat requests may receive a successful response without being
> > fully processed, it is preferable to always send the cumulative list
> > of directory IDs that have failed since startup. In the future, this
> > list can be trimmed to remove directory IDs that are seen to be
> > removed from the broker registration, as the broker catches up with
> > metadata.
> >
> > 6. The proposal to shutdown the broker log.dir.failure.timeout.ms
> > after not being able to communicate that some log directory is
> > offline is now more of an open question. It's unclear if this will
> > actually be necessary.
> >
> > Please share if you have any thoughts.
> >
> > Best,
> >
> > --
> > Igor
> >
> >
> > On Tue, Oct 10, 2023, at 5:28 AM, Igor Soarez wrote:
> > > Hi Colin,
> > >
> > > Thanks for the renaming suggestions. UNASSIGNED is better then
> > > UNKNOWN, MIGRATING is also better than SELECTED and I don't
> > > expect it to be used outside of the migration phase.
> > > LOST can also work instead of OFFLINE, but I think there will
> > > be other uses for this value outside of the migration, like
> > > in the intra-broker replica movement edge cases described in the KIP.
> > > I've updated the KIP and also filed a tiny PR with your suggestion,
> > > except I'm keeping the description of LOST more broad than just
> > > scoped to the migration.
> > >
> > >   https://github.com/apache/kafka/pull/14517
> > >
> > >
> > > The KIP already proposes that the broker does not want to unfence
> > > until it has confirmed all the assignments are communicated
> > > with the controller. And you're right about the interaction
> > > with ReplicaManager, we definitely don't want RPCs coming
> > > out of there. My intention is to introduce a new manager, as you
> > > suggest, with its own event loop, that batches and prioritizes
> > > assignment and dir failure events, called DirectoryEventManager.
> > > There's already an open PR, perhaps you could have a look?
> > >
> > >   KAFKA-15357: Aggregate and propagate assignments and logdir failures
> > >   https://github.com/apache/kafka/pull/14369
> > >
> > >
> > > 

Re: [VOTE] KIP-858: Handle JBOD broker disk failure in KRaft

[Colin McCabe]

Yes, this should be CLUSTER_ACTION on CLUSTER, to be consistent with our other 
internal RPCs.

best,
Colin


On Mon, Dec 4, 2023, at 04:28, Viktor Somogyi-Vass wrote:
> Hi Igor,
>
> I'm just reading through your KIP and noticed that the new protocol you
> created doesn't say anything about ACLs of the new AssignReplicasToDirs
> API. Would it make sense to authorize these requests as other inter-broker
> protocol calls are usually authorized, that is ClusterAction on Cluster
> resource?
>
> Thanks,
> Viktor
>
> On Tue, Nov 28, 2023 at 4:18 PM Igor Soarez  wrote:
>
>> Hi everyone,
>>
>> There have been a number of further changes.
>>
>> I have updated the KIP to reflect them, but for reference,
>> I'd also like to update this thread with a summary.
>>
>> 1. The reserved Uuids and their names for directories have changed.
>> The first 100 Uuids are reserved for future use.
>>
>> 2. During the ZooKeeper to KRaft migration, if a broker still
>> configured in ZK mode has any log directory offline, it will
>> shutdown and refuse to startup. The expectation is that this
>> escalation from a log directory's unavailability to the entire
>> broker's unavailability will be temporary, limited to the migration
>> period. And that the simplification will help develop, test and
>> support this feature.
>>
>> 3. The representation of replica directories in metadata records is
>> no longer tightly coupled with the respective broker IDs. Instead of
>> replacing the int[] replicas field in PartitionRecord and
>> PartitionChangeRecord, we are instead introducing a new field Uuid[]
>> named directories, that should be kept in the same size and order as
>> the existing replicas field. See
>> https://github.com/apache/kafka/pull/14290 for further details.
>>
>> 4. Assignments that are respective to failed log directories are no
>> longer prioritized. Previously the KIP proposed prioritizing
>> assignments that related to failed log directories, aiming to
>> synchronize the necessary replica to directory mapping on the
>> controller before handling the directory failure. Recently, we have
>> decided to remove any prioritization of these assignments, as
>> delaying the reporting of directory failures is considered
>> detrimental for any reason
>>
>> 5. Uuids for log directories that failed after startup are always
>> included in every broker heartbeat request. Previously the KIP
>> proposed sending Uuids for failed directories in the broker
>> heartbeat until a successful reply is received. However, due to the
>> overload mode handling of broker heartbeats, because broker
>> heartbeat requests may receive a successful response without being
>> fully processed, it is preferable to always send the cumulative list
>> of directory IDs that have failed since startup. In the future, this
>> list can be trimmed to remove directory IDs that are seen to be
>> removed from the broker registration, as the broker catches up with
>> metadata.
>>
>> 6. The proposal to shutdown the broker log.dir.failure.timeout.ms
>> after not being able to communicate that some log directory is
>> offline is now more of an open question. It's unclear if this will
>> actually be necessary.
>>
>> Please share if you have any thoughts.
>>
>> Best,
>>
>> --
>> Igor
>>
>>
>> On Tue, Oct 10, 2023, at 5:28 AM, Igor Soarez wrote:
>> > Hi Colin,
>> >
>> > Thanks for the renaming suggestions. UNASSIGNED is better then
>> > UNKNOWN, MIGRATING is also better than SELECTED and I don't
>> > expect it to be used outside of the migration phase.
>> > LOST can also work instead of OFFLINE, but I think there will
>> > be other uses for this value outside of the migration, like
>> > in the intra-broker replica movement edge cases described in the KIP.
>> > I've updated the KIP and also filed a tiny PR with your suggestion,
>> > except I'm keeping the description of LOST more broad than just
>> > scoped to the migration.
>> >
>> >   https://github.com/apache/kafka/pull/14517
>> >
>> >
>> > The KIP already proposes that the broker does not want to unfence
>> > until it has confirmed all the assignments are communicated
>> > with the controller. And you're right about the interaction
>> > with ReplicaManager, we definitely don't want RPCs coming
>> > out of there. My intention is to introduce a new manager, as you
>> > suggest, with its own event loop, that batches and prioritizes
>> > assignment and dir failure events, called DirectoryEventManager.
>> > There's already an open PR, perhaps you could have a look?
>> >
>> >   KAFKA-15357: Aggregate and propagate assignments and logdir failures
>> >   https://github.com/apache/kafka/pull/14369
>> >
>> >
>> > > With regard to the failure detection "gap" during hybrid mode: the
>> > > kraft controller sends a full LeaderAndIsrRequest to the brokers
>> > > that are in hybrid mode, right? And there is a per-partition
>> > > response as well. Right now, we don't pay attention to the error
>> > > codes sent back in the response. But we 

[jira] [Created] (KAFKA-15969) Align RemoteStorageThreadPool metrics name with KIP-405

[Lixin Yao (Jira)]

Lixin Yao created KAFKA-15969:
-

 Summary: Align RemoteStorageThreadPool metrics name with KIP-405
 Key: KAFKA-15969
 URL: https://issues.apache.org/jira/browse/KAFKA-15969
 Project: Kafka
  Issue Type: Bug
  Components: metrics
Affects Versions: 3.6.0
Reporter: Lixin Yao
 Fix For: 3.7.0


In KIP-405, there are 2 metrics defined below:
^kafka.log.remote:type=RemoteStorageThreadPool, 
name=RemoteLogReaderTaskQueueSize^
and
^kafka.log.remote:type=RemoteStorageThreadPool, 
name=RemoteLogReaderAvgIdlePercent^

However, in Kafka 3.6 release, the actual metrics exposes are:
^org.apache.kafka.storage.internals.log:name=RemoteLogReaderAvgIdlePercent,type=RemoteStorageThreadPool^
^org.apache.kafka.storage.internals.log:name=RemoteLogReaderTaskQueueSize,type=RemoteStorageThreadPool^

The problem is the bean domain name is changed from ^{{kafka.log.remote}}^ to 
{{{}^org.apache.kafka.storage.internals.log^{}}}. And the type name is also 
changed.

We should either update the metrics path in KIP, or fix the path in the code.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-997 Support fetch(fromKey, toKey, from, to) to WindowRangeQuery and unify WindowKeyQuery and WindowRangeQuery

[Hanyu (Peter) Zheng]

Thank you Alieh,

After discussion with Matthias, we decide use oldTimeFrom and
oldTimeTo(these two will Deprecated soom) to implement
withWindowStartRange() and use timeFrom and timeTo to implement
withAllKey() and withKeyRange(), I will update the KIP.

Sincerely,
Hanyu

On Sun, Dec 3, 2023 at 3:11 PM Alieh Saeedi 
wrote:

> Thanks, Hanyu, for the KIP and all the updates.
> I just do not understand the purpose of defining new time ranges
> (`newTimeFrom`, `newTimeTo`). Why don't we simply re-use the existing time
> range variables?
>
> Bests,
> Alieh
>
> On Thu, Nov 30, 2023 at 8:34 PM Hanyu (Peter) Zheng
>  wrote:
>
> > new KIP link:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-997%3A++update+WindowRangeQuery+and+unify+WindowKeyQuery+and+WindowRangeQuery
> >
> > On Wed, Nov 29, 2023 at 10:12 PM Hanyu (Peter) Zheng <
> pzh...@confluent.io>
> > wrote:
> >
> > > Thank you Bruno,
> > > 1. Thank you for the notification. I have updated the ticket link
> > > accordingly.
> > > 2. Certainly, I'll update the KIP name. Should I initiate a new
> > discussion
> > > for it, because if I change the name, the link will change.
> > > 3. Understood, I will add that to the KIP.
> > > 4. I propose we accept both
> > > `WindowRangeQuery.withAllKeys().fromTime(time1).toTime(time2)` and
> > > `WindowRangeQuery.withKeyRange(key1,
> > key2).fromTime(time1).toTime(time2)`,
> > > while also reusing the existing `withKey` method.
> > > 5. Following a discussion with Matthias, we've decided to defer the
> > > implementation of order guarantees to a future KIP.
> > >
> > > Sincerely,
> > > Hanyu
> > >
> > > On Wed, Nov 29, 2023 at 6:22 AM Bruno Cadonna 
> > wrote:
> > >
> > >> Hi,
> > >>
> > >> Thanks for the updates!
> > >>
> > >>
> > >> 1.
> > >> Could you please link the correct ticket in the KIP?
> > >>
> > >> 2.
> > >> Could you please adapt the motivation section and the title to the
> > >> updated goal of the KIP? There is no fetch() or fetchAll() method in
> the
> > >> query class.
> > >>
> > >> 3.
> > >> Could you please add the "// newly added" comment to all parts that
> were
> > >> newly added? That is methods lowerKeyBound() and upperKeyBound().
> > >>
> > >> 4.
> > >> We should use a more fluent API as I proposed in my last e-mail:
> > >>
> > >> Here again
> > >>
> > >> WindowRangeQuery.withAllKeys().fromTime(time1).toTime(time2);
> > >> WindowRangeQuery.withKey(key1).fromTime(time1).toTime(time2);
> > >> WindowRangeQuery.withKeyRange(key1,
> key2).fromTime(time1).toTime(time2);
> > >>
> > >> 5.
> > >> We should also consider the order of the results similar as we did in
> > >> KIP-968. Alternatively, we do not guarantee any order and postpone the
> > >> order guarantees to a future KIP.
> > >>
> > >>
> > >> Best,
> > >> Bruno
> > >>
> > >>
> > >>
> > >> On 11/17/23 3:11 AM, Matthias J. Sax wrote:
> > >> > Thanks for the KIP.
> > >> >
> > >> > Given how `WindowRangeQuery` works right now, it's really time to
> > >> > improve it.
> > >> >
> > >> >
> > >> > 1) Agree. It's not clear what will be added right now. I think we
> > >> should
> > >> > deprecate existing `getKey()` w/o an actually replacement? For
> > >> > `getFromKey` and `getToKey` we should actually be `lowerKeyBound()`
> > and
> > >> > `upperKeyBound()` to align to KIP-969?
> > >> >
> > >> > Also wondering if we should deprecate existing `withKey()` and
> > >> > `withWindowStartRange`? `withKey` only works for SessionStores and
> > >> > implements a single-key full-time-range query. Similarly,
> > >> > `withWindowStartRange` only works for WindowedStore and implements
> an
> > >> > all-key time-range query. Thus, both are rather special and it seems
> > >> the
> > >> > aim of this KIP is to generalize `WindowRangeQuery` to arbitrary
> > >> > key-range/time-range queries?
> > >> >
> > >> > What raises one question about time-range semantics, given that we
> > >> query
> > >> > windows with different semantics.
> > >> >
> > >> >   - The current `WindowStore` semantics used for
> > >> > `WindowRangeQuery#withWindowStartRange` is considering only the
> window
> > >> > start time, ie, the window-start time must fall into the query
> > >> > time-range to be returned.
> > >> >
> > >> >   - In contrast, `SessionStore` time ranges base on `findSession`
> use
> > >> > earliest-session-end-time and latest-session-end-time and thus
> > >> implement
> > >> > an "window-bounds / search-time-range overlap query".
> > >> >
> > >> > Is there any concern about semantic differences? I would also be
> > >> > possible to use the same semantics for both query types, and maybe
> > even
> > >> > let the user pick with semantics they want (let users chose might
> > >> > actually be the best thing to do)? -- We can also do this
> > >> incrementally,
> > >> > and limit the scope of this KIP (or keep the full KIP scope but
> > >> > implement it incrementally only)
> > >> >
> > >> > Btw: I think we should not add any ordering at this point, and
> > >> > explicitly 

Re: [VOTE] 3.6.1 RC0

[David Arthur]

Mickael,

I just filed https://issues.apache.org/jira/browse/KAFKA-15968 while
investigating a log corruption issue on the controller. I'm still
investigating the issue to see how far back this goes, but I think this
could be a blocker.

Essentially, the bug is that the controller does not treat a
CorruptRecordException as fatal, so the process will continue running. If
this happens on an active controller, it could corrupt the cluster's
metadata in general (since missing a single metadata record can cause lots
of downstream problems).

I'll update this thread by the end of day with a stronger
blocker/non-blocker opinion.

Thanks,
David


On Mon, Dec 4, 2023 at 6:48 AM Luke Chen  wrote:

> Hi Mickael:
>
> I did:
>1. Validated all checksums, signatures, and hashes
>2. Ran quick start for KRaft using scala 2.12 artifacts
>3. Spot checked the documentation and Javadoc
>4. Validated the licence file
>
> When running the validation to scala 2.12 package, I found these libraries
> are missing: (We only include scala 2.13 libraries in licence file)
> scala-java8-compat_2.12-1.0.2 is missing in license file
> scala-library-2.12.18 is missing in license file
> scala-logging_2.12-3.9.4 is missing in license file
> scala-reflect-2.12.18 is missing in license file
>
> It looks like this issue has been there for a long time, so it won't be a
> block issue for v3.6.1.
>
> +1 (binding) from me.
>
> Thank you.
> Luke
>
> On Sat, Dec 2, 2023 at 5:46 AM Bill Bejeck  wrote:
>
> > Hi Mickael,
> >
> > I did the following:
> >
> >1. Validated all checksums, signatures, and hashes
> >2. Built from source
> >3. Ran all the unit tests
> >4. Spot checked the documentation and Javadoc
> >5. Ran the ZK, Kraft, and Kafka Streams quickstart guides
> >
> > I did notice that the `fillDotVersion` in `js/templateData.js` needs
> > updating to `3.6.1`, but this is minor and should not block the release.
> >
> > It's a +1(binding) for me, pending the successful system test run
> >
> > Thanks,
> > Bill
> >
> > On Fri, Dec 1, 2023 at 1:49 PM Justine Olshan
>  > >
> > wrote:
> >
> > > I've started a system test run on my end.
> > >
> > > Justine
> > >
> > > On Wed, Nov 29, 2023 at 1:55 PM Justine Olshan 
> > > wrote:
> > >
> > > > I built from source and ran a simple transactional produce bench. I
> > ran a
> > > > handful of unit tests as well.
> > > > I scanned the docs and everything looked reasonable.
> > > >
> > > > I was wondering if we got the system test results mentioned > System
> > > > tests: Still running I'll post an update once they complete.
> > > >
> > > > Justine
> > > >
> > > > On Wed, Nov 29, 2023 at 6:33 AM Mickael Maison <
> > mickael.mai...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > >> Hi Josep,
> > > >>
> > > >> Good catch!
> > > >> If it's the only issue we find, I don't think we should block the
> > > >> release just to fix that.
> > > >>
> > > >> If we find another issue, I'll backport it before running another
> RC,
> > > >> otherwise I'll backport it once 3.6.1 is released.
> > > >>
> > > >> Thanks,
> > > >> Mickael
> > > >>
> > > >> On Wed, Nov 29, 2023 at 11:55 AM Josep Prat
> >  > > >
> > > >> wrote:
> > > >> >
> > > >> > Hi Mickael,
> > > >> > This PR[1] made me realize NOTICE-binary is missing the notice for
> > > >> > commons-io. I don't know if it's a blocker or not. I can cherry
> pick
> > > the
> > > >> > commit to the 3.6 branch if you want.
> > > >> >
> > > >> > Best,
> > > >> >
> > > >> >
> > > >> > [1]: https://github.com/apache/kafka/pull/14865
> > > >> >
> > > >> > On Tue, Nov 28, 2023 at 10:25 AM Josep Prat 
> > > >> wrote:
> > > >> >
> > > >> > > Hi Mickael,
> > > >> > > Thanks for running the release. It's a +1 for me (non-binding).
> > > >> > > I did the following:
> > > >> > > - Verified artifact's signatures and hashes
> > > >> > > - Checked JavaDoc (with navigation to Oracle JavaDoc)
> > > >> > > - Compiled source code
> > > >> > > - Run unit tests and integration tests
> > > >> > > - Run getting started with ZK and KRaft
> > > >> > >
> > > >> > > Best,
> > > >> > >
> > > >> > > On Tue, Nov 28, 2023 at 8:51 AM Kamal Chandraprakash <
> > > >> > > kamal.chandraprak...@gmail.com> wrote:
> > > >> > >
> > > >> > >> +1 (non-binding)
> > > >> > >>
> > > >> > >> 1. Built the source from 3.6.1-rc0 tag in scala 2.12 and 2.13
> > > >> > >> 2. Ran all the unit and integration tests.
> > > >> > >> 3. Ran quickstart and verified the produce-consume on a 3 node
> > > >> cluster.
> > > >> > >> 4. Verified the tiered storage functionality with local-tiered
> > > >> storage.
> > > >> > >>
> > > >> > >> On Tue, Nov 28, 2023 at 12:55 AM Federico Valeri <
> > > >> fedeval...@gmail.com>
> > > >> > >> wrote:
> > > >> > >>
> > > >> > >> > Hi Mickael,
> > > >> > >> >
> > > >> > >> > - Build from source (Java 17, Scala 2.13)
> > > >> > >> > - Run unit and integration tests
> > > >> > >> > - Run custom client apps using staging artifacts
> > > >> > >> >
> > > >> > >> > +1 (non 

Re: [DISCUSS] KIP-1004: Enforce tasks.max property in Kafka Connect

[Chris Egerton]

Hi all,

It seems like there are no objections to this KIP, so I've kicked off a
vote thread:
https://lists.apache.org/thread/dgq332o5j25rwddbvfydf7ttrclldw17

Cheers,

Chris

On Fri, Nov 24, 2023 at 10:39 PM Chris Egerton 
wrote:

> Hi Yash,
>
> Thanks for taking a look! Yeah, it looks like we'll be forced to hold onto
> the property until 5.0 if we can't introduce it at least one minor release
> before 4.0. I don't think this is the worst thing. Although it'd be nice to
> have the property completely removed when designing features like KIP-987,
> if necessary, we can also declare any new features incompatible with
> connectors that have opted out of enforcement of the tasks.max property
> (and likely enforce this restraint programmatically via preflight
> validation, failing connectors/tasks, log messages, etc.).
>
> Cheers,
>
> Chris
>
> On Wed, Nov 22, 2023 at 10:04 PM Yash Mayya  wrote:
>
> > Hi Chris,
> >
> > Thanks for the well written and comprehensive KIP! Given that we're
> already
> > past the KIP freeze deadline for 3.7.0 (
> > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.7.0)
> and
> > there may not be a 3.8.0 release before the 4.0.0 release, would we then
> be
> > forced to punt the removal of "tasks.max.enforce" to a future 5.0.0
> > release? I don't have any other comments, and the proposed changes make
> > sense to me.
> >
> > Thanks,
> > Yash
> >
> > On Mon, Nov 20, 2023 at 10:50 PM Chris Egerton 
> > wrote:
> >
> > > Hi Hector,
> > >
> > > Thanks for taking a look! I think the key difference between the
> proposed
> > > behavior and the rejected alternative is that the set of tasks that
> will
> > be
> > > running with the former is still a complete set of tasks, whereas the
> set
> > > of tasks in the latter is a subset of tasks. Also noteworthy but
> slightly
> > > less important: the problem will be more visible to users with the
> former
> > > (the connector will still be marked FAILED) than with the latter.
> > >
> > > Cheers,
> > >
> > > Chris
> > >
> > > On Tue, Nov 21, 2023, 00:53 Hector Geraldino (BLOOMBERG/ 919 3RD A) <
> > > hgerald...@bloomberg.net> wrote:
> > >
> > > > Thanks for the KIP Chris, adding this check makes total sense.
> > > >
> > > > I do have one question. The second paragraph in the Public Interfaces
> > > > section states:
> > > >
> > > > "If the connector generated excessive tasks after being reconfigured,
> > > then
> > > > any existing tasks for the connector will be allowed to continue
> > running,
> > > > unless that existing set of tasks also exceeds the tasks.max
> property."
> > > >
> > > > Would not failing the connector land us in the second scenario of
> > > > 'Rejected Alternatives'?
> > > >
> > > > From: dev@kafka.apache.org At: 11/11/23 00:27:44 UTC-5:00To:
> > > > dev@kafka.apache.org
> > > > Subject: [DISCUSS] KIP-1004: Enforce tasks.max property in Kafka
> > Connect
> > > >
> > > > Hi all,
> > > >
> > > > I'd like to open up KIP-1004 for discussion:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1004%3A+Enforce+tasks.max+
> > > > property+in+Kafka+Connect
> > > >
> > > > As a brief summary: this KIP proposes that the Kafka Connect runtime
> > > start
> > > > failing connectors that generate a greater number of tasks than the
> > > > tasks.max property, with an optional emergency override that can be
> > used
> > > to
> > > > continue running these (probably-buggy) connectors if absolutely
> > > necessary.
> > > >
> > > > I'll be taking time off most of the next three weeks, so response
> > latency
> > > > may be a bit higher than usual, but I wanted to kick off the
> discussion
> > > in
> > > > case we can land this in time for the upcoming 3.7.0 release.
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > >
> > > >
> > >
> >
>

[VOTE] KIP-1004: Enforce tasks.max property in Kafka Connect

[Chris Egerton]

Hi all,

I'd like to call for a vote on KIP-1004, which adds enforcement for the
tasks.max connector property in Kafka Connect.

The KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1004%3A+Enforce+tasks.max+property+in+Kafka+Connect

The discussion thread:
https://lists.apache.org/thread/scx75cjwm19jyt19wxky41q9smf5nx6d

Cheers,

Chris

[jira] [Created] (KAFKA-15968) QuorumController does not treat CorruptRecordException as fatal

[David Arthur (Jira)]

David Arthur created KAFKA-15968:


 Summary: QuorumController does not treat CorruptRecordException as 
fatal
 Key: KAFKA-15968
 URL: https://issues.apache.org/jira/browse/KAFKA-15968
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.6.0, 3.7.0
Reporter: David Arthur


When QuorumController encounters a CorruptRecordException, it does not include 
the exception in the log message. Since CorruptRecordException extends 
ApiException, it gets caught by the first condition in 
EventHandlerExceptionInfo#fromInternal.

The controller treats ApiException as an excepted case (for things like authz 
errors of creating a topic that already exists) so it does not cause a 
failover. If the active controller sees a corrupt record, it should be a fatal 
error.

While we are fixing this, we should audit the subclasses of ApiException and 
make sure we are handling the fatal ones correctly.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-995: Allow users to specify initial offsets while creating connectors

[Chris Egerton]

Oh, one more thing--can we add the KIP to the list of KIPs?
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#KafkaImprovementProposals-KIPsunderdiscussion

On Mon, Dec 4, 2023 at 10:33 AM Chris Egerton  wrote:

> Hi Ashwin,
>
> Thanks for the KIP! This would be a nice simplification to the process for
> migrating connectors enabled by KIP-980, and would also add global support
> for a feature I've seen implemented by hand in at least a couple connectors
> over the years.
>
>
> High-level thoughts:
>
> 1. If there are leftover offsets from a previous connector, what will the
> impact on those offsets (if any) be if a new connector is created with the
> same name with initial offsets specified? I can think of at least two
> options: we leave those offsets as-are but allow any of the initial offsets
> in the new connector request to overwrite them, or we automatically wipe
> all existing offsets for the connector first before writing its initial
> offsets and then creating it. I have a slight preference for the first
> because it's simpler to implement and aligns with existing precedent for
> offset handling where we never wipe them unless explicitly requested by the
> user or connector, but it could be argued that the second is less likely to
> generate footguns for users. Interested in your thoughts!
>
> 2. IMO preflight validation (i.e., the "Validate initial_offset before
> creating the connector in STOPPED state rejected alternative) is a
> must-have for this kind of feature. I acknowledge the possible race
> condition (and I don't think it's worth it to track in-flight connector
> creation requests in the herder in order to prevent this race, since
> ever-blocking Connector operations would be cumbersome to deal with), and
> the extra implementation effort. But I don't think either of those tip the
> scales far enough to override the benefit of ensuring the submitted offsets
> are valid before creating the connector.
>
> 3. On the topic of preflight validation--I also think it's important that
> we validate both the connector config and the initial offsets before either
> creating the connector or storing the initial offsets. I don't think this
> point requires any changes to the KIP that aren't already proposed with
> point 2. above, but wanted to see if we could adopt this as a primary goal
> for the design of the feature and keep it in mind with future changes.
>
> 4. In the proposed changes section, the first step is "Create a connector
> in STOPPED state", and the second step is "Validate the offset...". What
> exactly is entailed by "Create a connector"? Submit the config to the
> config topic (presumably after a preflight validation of the config)?
> Participate in the ensuing rebalance? I'm a little hesitant to start
> performing rebalances inside REST requests, wondering if we can find a
> lighter-weight way to implement this.
>
>
> Nits:
>
> 5. You can remove the italicized "This page is meant as a template for
> writing a KIP" section after the table of contents.
>
> 6. Can you file a JIRA ticket for this change and add a link to it in the
> KIP under the status section?
>
> 7. What do you think about changing the name for the new field from
> "initial_offset" (singular) to "initial_offsets" (plural)? This is
> especially relevant for connectors that read from multiple source
> partitions, like MM2 and the various JDBC source connectors out there.
>
> 8. IMO it's not necessary to include this section: "Please note that sink
> and source connectors have different schemas for offset." While it's
> technically true that the fields of the objects inside the "partition" and
> "offset" fields will likely differ between sink and source connectors,
> they'll also likely differ in the exact same way across different sink
> connectors. I think it's enough to link to the relevant section in KIP-875
> on the details for the format.
>
> 9. Instead of "Connector-defined source partition" and "Connector-defined
> source offset" in the comments for the sample connector creation body
> (which aren't strictly accurate for sink connectors), can we say something
> like "Source partition" and "Desired initial offset"?
>
> 10. In the compatibility section the KIP states that "This new feature
> will use the current OffsetStorageWriter." IMO we should refrain from
> referring to internal API class names in KIPs when possible, since those
> class names may change and users may also mistakenly assume that they're
> part of the public API. Can we say something like "This new feature will
> reuse existing logic for storing offsets" instead?
>
>
> Thanks again for the KIP!
>
> Cheers,
>
> Chris
>
> On Thu, Nov 30, 2023 at 2:26 AM Ashwin 
> wrote:
>
>> Hi all,
>>
>> I'd like to begin discussion on KIP-995 which proposes to allow users
>> to specify initial offset as part of the request to create a connector
>>
>>
>> 

Re: [DISCUSS] KIP-995: Allow users to specify initial offsets while creating connectors

[Chris Egerton]

Hi Ashwin,

Thanks for the KIP! This would be a nice simplification to the process for
migrating connectors enabled by KIP-980, and would also add global support
for a feature I've seen implemented by hand in at least a couple connectors
over the years.


High-level thoughts:

1. If there are leftover offsets from a previous connector, what will the
impact on those offsets (if any) be if a new connector is created with the
same name with initial offsets specified? I can think of at least two
options: we leave those offsets as-are but allow any of the initial offsets
in the new connector request to overwrite them, or we automatically wipe
all existing offsets for the connector first before writing its initial
offsets and then creating it. I have a slight preference for the first
because it's simpler to implement and aligns with existing precedent for
offset handling where we never wipe them unless explicitly requested by the
user or connector, but it could be argued that the second is less likely to
generate footguns for users. Interested in your thoughts!

2. IMO preflight validation (i.e., the "Validate initial_offset before
creating the connector in STOPPED state rejected alternative) is a
must-have for this kind of feature. I acknowledge the possible race
condition (and I don't think it's worth it to track in-flight connector
creation requests in the herder in order to prevent this race, since
ever-blocking Connector operations would be cumbersome to deal with), and
the extra implementation effort. But I don't think either of those tip the
scales far enough to override the benefit of ensuring the submitted offsets
are valid before creating the connector.

3. On the topic of preflight validation--I also think it's important that
we validate both the connector config and the initial offsets before either
creating the connector or storing the initial offsets. I don't think this
point requires any changes to the KIP that aren't already proposed with
point 2. above, but wanted to see if we could adopt this as a primary goal
for the design of the feature and keep it in mind with future changes.

4. In the proposed changes section, the first step is "Create a connector
in STOPPED state", and the second step is "Validate the offset...". What
exactly is entailed by "Create a connector"? Submit the config to the
config topic (presumably after a preflight validation of the config)?
Participate in the ensuing rebalance? I'm a little hesitant to start
performing rebalances inside REST requests, wondering if we can find a
lighter-weight way to implement this.


Nits:

5. You can remove the italicized "This page is meant as a template for
writing a KIP" section after the table of contents.

6. Can you file a JIRA ticket for this change and add a link to it in the
KIP under the status section?

7. What do you think about changing the name for the new field from
"initial_offset" (singular) to "initial_offsets" (plural)? This is
especially relevant for connectors that read from multiple source
partitions, like MM2 and the various JDBC source connectors out there.

8. IMO it's not necessary to include this section: "Please note that sink
and source connectors have different schemas for offset." While it's
technically true that the fields of the objects inside the "partition" and
"offset" fields will likely differ between sink and source connectors,
they'll also likely differ in the exact same way across different sink
connectors. I think it's enough to link to the relevant section in KIP-875
on the details for the format.

9. Instead of "Connector-defined source partition" and "Connector-defined
source offset" in the comments for the sample connector creation body
(which aren't strictly accurate for sink connectors), can we say something
like "Source partition" and "Desired initial offset"?

10. In the compatibility section the KIP states that "This new feature will
use the current OffsetStorageWriter." IMO we should refrain from referring
to internal API class names in KIPs when possible, since those class names
may change and users may also mistakenly assume that they're part of the
public API. Can we say something like "This new feature will reuse existing
logic for storing offsets" instead?


Thanks again for the KIP!

Cheers,

Chris

On Thu, Nov 30, 2023 at 2:26 AM Ashwin  wrote:

> Hi all,
>
> I'd like to begin discussion on KIP-995 which proposes to allow users
> to specify initial offset as part of the request to create a connector
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-995%3A+Allow+users+to+specify+initial+offsets+while+creating+connectors
>
> During the discussion for KIP-980
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-980%3A+Allow+creating+connectors+in+a+stopped+state
> >,
> which proposed the creation of connectors in STOPPED state, there was
> a suggestion to also allow setting the initial offset for a connector
> in the connector creation API. The proposal was deemed 

Re: [DISCUSS] KIP-971 Expose replication-offset-lag MirrorMaker2 metric

[Viktor Somogyi-Vass]

Elkhan, do you want to propose a vote for this KIP or do you have any other
ideas to include?

On Tue, Oct 17, 2023 at 2:47 PM Viktor Somogyi-Vass <
viktor.somo...@cloudera.com> wrote:

> Hi hudeqi,
>
> Good thinking about the OOM and resource leaks.
> The "update.replication.lag.interval.time" I think is almost good but we
> should include that it is about a metric (like
> "replication.lag.interval.metric.update.time") so it's obvious without the
> docs too.
>
> Thanks,
> Viktor
>
> On Sat, Oct 7, 2023 at 8:53 AM hudeqi <16120...@bjtu.edu.cn> wrote:
>
>> Hi, Elkhan, Viktor.
>>
>> I took a look at the updated KIP. I think Viktor mentioned that he did
>> not see the relevant configuration, which refers to "(Optional) -
>> MirrorConnectorConfig - a configuration to control the poll interval for
>> the Consumer.endOffsets() call at LEO acquisition mentioned below". I think
>> we can introduce the name of this configuration here, such as
>> "update.replication.lag.interval.time", which means that in a separate
>> periodic scheduling thread, the lag is calculated by this interval time
>> through "consumer.endOffsets - LRO". In addition, for the LRO cache, you
>> can add an expired time attribute for each partition. If this expired
>> interval time is exceeded before next updated, the LRO of this partition
>> can be removed from the cache to avoid possible leaks and OOM.
>>
>> best,
>> hudeqi
>
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2444

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-858: Handle JBOD broker disk failure in KRaft

[Viktor Somogyi-Vass]

Hi Igor,

I'm just reading through your KIP and noticed that the new protocol you
created doesn't say anything about ACLs of the new AssignReplicasToDirs
API. Would it make sense to authorize these requests as other inter-broker
protocol calls are usually authorized, that is ClusterAction on Cluster
resource?

Thanks,
Viktor

On Tue, Nov 28, 2023 at 4:18 PM Igor Soarez  wrote:

> Hi everyone,
>
> There have been a number of further changes.
>
> I have updated the KIP to reflect them, but for reference,
> I'd also like to update this thread with a summary.
>
> 1. The reserved Uuids and their names for directories have changed.
> The first 100 Uuids are reserved for future use.
>
> 2. During the ZooKeeper to KRaft migration, if a broker still
> configured in ZK mode has any log directory offline, it will
> shutdown and refuse to startup. The expectation is that this
> escalation from a log directory's unavailability to the entire
> broker's unavailability will be temporary, limited to the migration
> period. And that the simplification will help develop, test and
> support this feature.
>
> 3. The representation of replica directories in metadata records is
> no longer tightly coupled with the respective broker IDs. Instead of
> replacing the int[] replicas field in PartitionRecord and
> PartitionChangeRecord, we are instead introducing a new field Uuid[]
> named directories, that should be kept in the same size and order as
> the existing replicas field. See
> https://github.com/apache/kafka/pull/14290 for further details.
>
> 4. Assignments that are respective to failed log directories are no
> longer prioritized. Previously the KIP proposed prioritizing
> assignments that related to failed log directories, aiming to
> synchronize the necessary replica to directory mapping on the
> controller before handling the directory failure. Recently, we have
> decided to remove any prioritization of these assignments, as
> delaying the reporting of directory failures is considered
> detrimental for any reason
>
> 5. Uuids for log directories that failed after startup are always
> included in every broker heartbeat request. Previously the KIP
> proposed sending Uuids for failed directories in the broker
> heartbeat until a successful reply is received. However, due to the
> overload mode handling of broker heartbeats, because broker
> heartbeat requests may receive a successful response without being
> fully processed, it is preferable to always send the cumulative list
> of directory IDs that have failed since startup. In the future, this
> list can be trimmed to remove directory IDs that are seen to be
> removed from the broker registration, as the broker catches up with
> metadata.
>
> 6. The proposal to shutdown the broker log.dir.failure.timeout.ms
> after not being able to communicate that some log directory is
> offline is now more of an open question. It's unclear if this will
> actually be necessary.
>
> Please share if you have any thoughts.
>
> Best,
>
> --
> Igor
>
>
> On Tue, Oct 10, 2023, at 5:28 AM, Igor Soarez wrote:
> > Hi Colin,
> >
> > Thanks for the renaming suggestions. UNASSIGNED is better then
> > UNKNOWN, MIGRATING is also better than SELECTED and I don't
> > expect it to be used outside of the migration phase.
> > LOST can also work instead of OFFLINE, but I think there will
> > be other uses for this value outside of the migration, like
> > in the intra-broker replica movement edge cases described in the KIP.
> > I've updated the KIP and also filed a tiny PR with your suggestion,
> > except I'm keeping the description of LOST more broad than just
> > scoped to the migration.
> >
> >   https://github.com/apache/kafka/pull/14517
> >
> >
> > The KIP already proposes that the broker does not want to unfence
> > until it has confirmed all the assignments are communicated
> > with the controller. And you're right about the interaction
> > with ReplicaManager, we definitely don't want RPCs coming
> > out of there. My intention is to introduce a new manager, as you
> > suggest, with its own event loop, that batches and prioritizes
> > assignment and dir failure events, called DirectoryEventManager.
> > There's already an open PR, perhaps you could have a look?
> >
> >   KAFKA-15357: Aggregate and propagate assignments and logdir failures
> >   https://github.com/apache/kafka/pull/14369
> >
> >
> > > With regard to the failure detection "gap" during hybrid mode: the
> > > kraft controller sends a full LeaderAndIsrRequest to the brokers
> > > that are in hybrid mode, right? And there is a per-partition
> > > response as well. Right now, we don't pay attention to the error
> > > codes sent back in the response. But we could. Any replica with an
> > > error could be transitioned from MIGRATING -> LOST, right? That
> > > would close the failure detection gap.
> >
> > Almost. The missing bit is that the controller would also need to
> > watch the /log_dir_event_notification znode, and on any change
> > 

Re: [VOTE] 3.6.1 RC0

[Luke Chen]

Hi Mickael:

I did:
   1. Validated all checksums, signatures, and hashes
   2. Ran quick start for KRaft using scala 2.12 artifacts
   3. Spot checked the documentation and Javadoc
   4. Validated the licence file

When running the validation to scala 2.12 package, I found these libraries
are missing: (We only include scala 2.13 libraries in licence file)
scala-java8-compat_2.12-1.0.2 is missing in license file
scala-library-2.12.18 is missing in license file
scala-logging_2.12-3.9.4 is missing in license file
scala-reflect-2.12.18 is missing in license file

It looks like this issue has been there for a long time, so it won't be a
block issue for v3.6.1.

+1 (binding) from me.

Thank you.
Luke

On Sat, Dec 2, 2023 at 5:46 AM Bill Bejeck  wrote:

> Hi Mickael,
>
> I did the following:
>
>1. Validated all checksums, signatures, and hashes
>2. Built from source
>3. Ran all the unit tests
>4. Spot checked the documentation and Javadoc
>5. Ran the ZK, Kraft, and Kafka Streams quickstart guides
>
> I did notice that the `fillDotVersion` in `js/templateData.js` needs
> updating to `3.6.1`, but this is minor and should not block the release.
>
> It's a +1(binding) for me, pending the successful system test run
>
> Thanks,
> Bill
>
> On Fri, Dec 1, 2023 at 1:49 PM Justine Olshan  >
> wrote:
>
> > I've started a system test run on my end.
> >
> > Justine
> >
> > On Wed, Nov 29, 2023 at 1:55 PM Justine Olshan 
> > wrote:
> >
> > > I built from source and ran a simple transactional produce bench. I
> ran a
> > > handful of unit tests as well.
> > > I scanned the docs and everything looked reasonable.
> > >
> > > I was wondering if we got the system test results mentioned > System
> > > tests: Still running I'll post an update once they complete.
> > >
> > > Justine
> > >
> > > On Wed, Nov 29, 2023 at 6:33 AM Mickael Maison <
> mickael.mai...@gmail.com
> > >
> > > wrote:
> > >
> > >> Hi Josep,
> > >>
> > >> Good catch!
> > >> If it's the only issue we find, I don't think we should block the
> > >> release just to fix that.
> > >>
> > >> If we find another issue, I'll backport it before running another RC,
> > >> otherwise I'll backport it once 3.6.1 is released.
> > >>
> > >> Thanks,
> > >> Mickael
> > >>
> > >> On Wed, Nov 29, 2023 at 11:55 AM Josep Prat
>  > >
> > >> wrote:
> > >> >
> > >> > Hi Mickael,
> > >> > This PR[1] made me realize NOTICE-binary is missing the notice for
> > >> > commons-io. I don't know if it's a blocker or not. I can cherry pick
> > the
> > >> > commit to the 3.6 branch if you want.
> > >> >
> > >> > Best,
> > >> >
> > >> >
> > >> > [1]: https://github.com/apache/kafka/pull/14865
> > >> >
> > >> > On Tue, Nov 28, 2023 at 10:25 AM Josep Prat 
> > >> wrote:
> > >> >
> > >> > > Hi Mickael,
> > >> > > Thanks for running the release. It's a +1 for me (non-binding).
> > >> > > I did the following:
> > >> > > - Verified artifact's signatures and hashes
> > >> > > - Checked JavaDoc (with navigation to Oracle JavaDoc)
> > >> > > - Compiled source code
> > >> > > - Run unit tests and integration tests
> > >> > > - Run getting started with ZK and KRaft
> > >> > >
> > >> > > Best,
> > >> > >
> > >> > > On Tue, Nov 28, 2023 at 8:51 AM Kamal Chandraprakash <
> > >> > > kamal.chandraprak...@gmail.com> wrote:
> > >> > >
> > >> > >> +1 (non-binding)
> > >> > >>
> > >> > >> 1. Built the source from 3.6.1-rc0 tag in scala 2.12 and 2.13
> > >> > >> 2. Ran all the unit and integration tests.
> > >> > >> 3. Ran quickstart and verified the produce-consume on a 3 node
> > >> cluster.
> > >> > >> 4. Verified the tiered storage functionality with local-tiered
> > >> storage.
> > >> > >>
> > >> > >> On Tue, Nov 28, 2023 at 12:55 AM Federico Valeri <
> > >> fedeval...@gmail.com>
> > >> > >> wrote:
> > >> > >>
> > >> > >> > Hi Mickael,
> > >> > >> >
> > >> > >> > - Build from source (Java 17, Scala 2.13)
> > >> > >> > - Run unit and integration tests
> > >> > >> > - Run custom client apps using staging artifacts
> > >> > >> >
> > >> > >> > +1 (non binding)
> > >> > >> >
> > >> > >> > Thanks
> > >> > >> > Fede
> > >> > >> >
> > >> > >> >
> > >> > >> >
> > >> > >> > On Sun, Nov 26, 2023 at 11:34 AM Jakub Scholz  >
> > >> wrote:
> > >> > >> > >
> > >> > >> > > +1 non-binding. I used the staged Scala 2.13 artifacts and
> the
> > >> staged
> > >> > >> > Maven
> > >> > >> > > repo for my tests. All seems to work fine.
> > >> > >> > >
> > >> > >> > > Thanks
> > >> > >> > > Jakub
> > >> > >> > >
> > >> > >> > > On Fri, Nov 24, 2023 at 4:37 PM Mickael Maison <
> > >> mimai...@apache.org>
> > >> > >> > wrote:
> > >> > >> > >
> > >> > >> > > > Hello Kafka users, developers and client-developers,
> > >> > >> > > >
> > >> > >> > > > This is the first candidate for release of Apache Kafka
> > 3.6.1.
> > >> > >> > > >
> > >> > >> > > > This is a bugfix release with several fixes, including
> > >> dependency
> > >> > >> > > > version bumps for CVEs.
> > >> > >> > > >
> > >> > >> > > > Release notes for the 3.6.1 

[jira] [Created] (KAFKA-15967) Fix revocation in reconcilation logic

[Lucas Brutschy (Jira)]

Lucas Brutschy created KAFKA-15967:
--

 Summary: Fix revocation in reconcilation logic
 Key: KAFKA-15967
 URL: https://issues.apache.org/jira/browse/KAFKA-15967
 Project: Kafka
  Issue Type: Sub-task
Reporter: Lucas Brutschy


Looks like there is a problem in the reconciliation logic.

We are getting 6 partitions from an HB, we add them to 
{{{}assignmentReadyToReconcile{}}}. Next HB we get only 4 partitions (2 are 
revoked), we also add them to {{{}assignmentReadyToReconcile{}}}, but the 2 
partitions that were supposed to be removed from the assignment are never 
removed because they are still in {{{}assignmentReadyToReconcile{}}}.

This was discovered during integration testing of 
[https://github.com/apache/kafka/pull/14878] - part of the test 
testRemoteAssignorRange was disabled and should be re-enabled once this is 
fixed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15690) EosIntegrationTest is flaky.

[Lucas Brutschy (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lucas Brutschy resolved KAFKA-15690.

Resolution: Fixed

> EosIntegrationTest is flaky.
> 
>
> Key: KAFKA-15690
> URL: https://issues.apache.org/jira/browse/KAFKA-15690
> Project: Kafka
>  Issue Type: Bug
>  Components: streams, unit tests
>Reporter: Calvin Liu
>Assignee: Lucas Brutschy
>Priority: Major
>  Labels: flaky-test
>
> EosIntegrationTest
> shouldNotViolateEosIfOneTaskGetsFencedUsingIsolatedAppInstances[exactly_once_v2,
>  processing threads = false]
> {code:java}
> org.junit.runners.model.TestTimedOutException: test timed out after 600 
> seconds   at 
> org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.handleServerDisconnect(NetworkClient.java:)
> at 
> org.apache.kafka.clients.NetworkClient.processDisconnection(NetworkClient.java:821)
>   at 
> org.apache.kafka.clients.NetworkClient.processTimeoutDisconnection(NetworkClient.java:779)
>at 
> org.apache.kafka.clients.NetworkClient.handleTimedOutRequests(NetworkClient.java:837)
> org.apache.kafka.streams.errors.StreamsException: Exception caught in 
> process. taskId=0_1, processor=KSTREAM-SOURCE-00, 
> topic=multiPartitionInputTopic, partition=1, offset=15, 
> stacktrace=java.lang.RuntimeException: Detected we've been interrupted.   
> at 
> org.apache.kafka.streams.integration.EosIntegrationTest$1$1.transform(EosIntegrationTest.java:892)
>at 
> org.apache.kafka.streams.integration.EosIntegrationTest$1$1.transform(EosIntegrationTest.java:867)
>at 
> org.apache.kafka.streams.kstream.internals.TransformerSupplierAdapter$1.transform(TransformerSupplierAdapter.java:49)
> at 
> org.apache.kafka.streams.kstream.internals.TransformerSupplierAdapter$1.transform(TransformerSupplierAdapter.java:38)
> at 
> org.apache.kafka.streams.kstream.internals.KStreamFlatTransform$KStreamFlatTransformProcessor.process(KStreamFlatTransform.java:66)
>  {code}
>   shouldWriteLatestOffsetsToCheckpointOnShutdown[exactly_once_v2, processing 
> threads = false] 
> {code:java}
> java.lang.RuntimeException: java.util.concurrent.ExecutionException: 
> org.apache.kafka.common.errors.UnknownServerException: The server experienced 
> an unexpected error when processing the request.   at 
> org.apache.kafka.streams.integration.utils.KafkaEmbedded.deleteTopic(KafkaEmbedded.java:204)
>  at 
> org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster.deleteTopicsAndWait(EmbeddedKafkaCluster.java:286)
>at 
> org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster.deleteTopicsAndWait(EmbeddedKafkaCluster.java:274)
>at 
> org.apache.kafka.streams.integration.EosIntegrationTest.createTopics(EosIntegrationTest.java:174)
> at 
> java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
> org.apache.kafka.streams.errors.StreamsException: Exception caught in 
> process. taskId=0_1, processor=KSTREAM-SOURCE-00, 
> topic=multiPartitionInputTopic, partition=1, offset=15, 
> stacktrace=java.lang.RuntimeException: Detected we've been interrupted.   
> at 
> org.apache.kafka.streams.integration.EosIntegrationTest$1$1.transform(EosIntegrationTest.java:892)
>at 
> org.apache.kafka.streams.integration.EosIntegrationTest$1$1.transform(EosIntegrationTest.java:867)
>at 
> org.apache.kafka.streams.kstream.internals.TransformerSupplierAdapter$1.transform(TransformerSupplierAdapter.java:49)
> at 
> org.apache.kafka.streams.kstream.internals.TransformerSupplierAdapter$1.transform(TransformerSupplierAdapter.java:38)
> at 
> org.apache.kafka.streams.kstream.internals.KStreamFlatTransform$KStreamFlatTransformProcessor.process(KStreamFlatTransform.java:66)
>  {code}
> shouldWriteLatestOffsetsToCheckpointOnShutdown[exactly_once, processing 
> threads = false] 
> {code:java}
> org.opentest4j.AssertionFailedError: Condition not met within timeout 6. 
> StreamsTasks did not request commit. ==> expected:  but was: 
>at 
> app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
>at 
> app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
>at app//org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)   
>  at app//org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36) 
> at app//org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:210)
> java.lang.IllegalStateException: Replica 
> [Topic=__transaction_state,Partition=2,Replica=1] should be in the 
> OfflineReplica,ReplicaDeletionStarted states before moving to 
> ReplicaDeletionIneligible state. Instead it is in OnlineReplica state   
> at 
> 

Re: [REVIEW REQUEST] ConsumerGroupCommand move to tools

[Николай Ижиков]

Hello.

Dear Kafka committers.
As far as I know - we moving Kafka from scala to java.

I prepared a series of PRs to move `ConsumerGroupCommand` from scala to java.
From my experience, right now, Mickael Maison and Justine Olshan are 
overwhelmed with other activities and can review my patches from time to time 
only.

So, if any Kafka committer has any chance to review simple patches and keep 
pace of moving scala code to java, please, do it.
Help me to move these command to tools.

Looks like all code are prepared and quality review is all we need :).

> 30 нояб. 2023 г., в 13:28, Николай Ижиков  написал(а):
> 
> Hello.
> 
> I prepared a PR to move `ConsumerGroupCommand` to `tools` module.
> Full changes are pretty huge [1] so I split them into several PR’s.
> 
> Please, take a look at first PR in the series - 
> https://github.com/apache/kafka/pull/14856
> 
> PR adds `ConsumerGroupCommandOptions` class and other command case classes to 
> `tools` module.
> 
> [1] https://github.com/apache/kafka/pull/14471
> 

[jira] [Resolved] (KAFKA-14438) Throw error when consumer configured with empty/whitespace-only group.id for AsyncKafkaConsumer

[Bruno Cadonna (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bruno Cadonna resolved KAFKA-14438.
---
Resolution: Fixed

> Throw error when consumer configured with empty/whitespace-only group.id for 
> AsyncKafkaConsumer
> ---
>
> Key: KAFKA-14438
> URL: https://issues.apache.org/jira/browse/KAFKA-14438
> Project: Kafka
>  Issue Type: Task
>  Components: consumer
>Reporter: Philip Nee
>Assignee: Bruno Cadonna
>Priority: Blocker
>  Labels: kip-848-client-support, kip-848-e2e, kip-848-preview
> Fix For: 3.7.0
>
>
> Currently, a warning message is logged upon using an empty consumer groupId. 
> In the next major release, we should drop the support of empty ("") consumer 
> groupId.
> cc [~hachikuji]
> See 
> [KIP-289|https://cwiki.apache.org/confluence/display/KAFKA/KIP-289%3A+Improve+the+default+group+id+behavior+in+KafkaConsumer]
>  for more detail.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)