[jira] [Created] (KAFKA-14866) When the controller changes, the old controller needs to clean up some related resources when resigning

[hudeqi (Jira)]

hudeqi created KAFKA-14866:
--

 Summary: When the controller changes, the old controller needs to 
clean up some related resources when resigning
 Key: KAFKA-14866
 URL: https://issues.apache.org/jira/browse/KAFKA-14866
 Project: Kafka
  Issue Type: Improvement
  Components: controller, core
Affects Versions: 3.3.2
Reporter: hudeqi






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14865) Consumer::poll returns early on with an empty result for highly compacted KTables

[Jira]

Balázs Németh created KAFKA-14865:
-

 Summary: Consumer::poll returns early on with an empty result for 
highly compacted KTables
 Key: KAFKA-14865
 URL: https://issues.apache.org/jira/browse/KAFKA-14865
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.2.0
Reporter: Balázs Németh


This behaviour/regression was introduced by 
https://issues.apache.org/jira/browse/KAFKA-12980 / 
[https://github.com/apache/kafka/pull/11046] 

The issue happens when processing a topic that has huge offset ranges that has 
been "compacted away". It's the scenario that triggers 
[https://github.com/apache/kafka/blob/5afedd9ac37c4d740f47867cfd31eaed15dc542f/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L1266-L1269C21]
 

Before that change this pseudocode worked:

 
{code:java}
for (;;) {
  ConsumerRecords result = consumer.poll(..);
  if (result.isEmpty()) {
scheduleLater(..);
break;
  }
  process(result);
}{code}
It worked because the poll waited until the first fetch that had a record, or 
the timeout happened. 

 

Now it only waits for the very first fetch. In many cases the returned `Fetch` 
object return false for `isEmpty()` yet it contains no records. 
(`positionAdvanced` is true, `records` is empty) 
[https://github.com/apache/kafka/blob/5afedd9ac37c4d740f47867cfd31eaed15dc542f/clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetch.java#L102-L104]

So now it calls the `scheduleLater` after every fetch in those ranges. That 
might throttle the ingestion a lot.

Before the change:

- we have record(s) -> return immediately with a nonempty result
- we encountered a gap -> iterates over it until a record is found (and return 
with a nonempty result) or the timeout expires (which with sufficiently big 
timeout effectively never happens)
- we encountered the end -> tries to read for the timeout, and returns with an 
empty result

After the change:

- we have record(s) -> return immediately with a nonempty result
- we encountered a gap -> return the result of the very first fetch, if the gap 
is bigger than that it returns an empty result otherwise nonempty result
- we encountered the end -> tries to read for the timeout, and returns with an 
empty result

As you can see before the change an empty result most likely meant that we have 
reached the end of the stream. After the change it has an additional meaning, 
and there is no way to make a distinction to figure out which scenario we 
encountered.

`org.apache.kafka.clients.consumer.internals.Fetch` has that info, but when it 
gets converted to `org.apache.kafka.clients.consumer.ConsumerRecords` that 
context is lost.

So a `Fetch` with isEmpty() == false produces a `ConsumerRecords` with 
isEmpty() == true.

 

ps.: For example another project where it causes issues: 
[https://github.com/apache/beam/blob/3efd3c38b9917ae1f51f466a237a79d9fe57b2a6/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/ReadFromKafkaDoFn.java#L383C36-L388]
 (apache/beam allows the usage of any version of `kafka-clients` you prefer, so 
if you pick the 3.2.0 or any newer this issue might happen).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Kafka Cluster WITHOUT Zookeeper

[Brebner, Paul]

David, hi, interesting thanks! Paul

From: David Arthur 
Date: Wednesday, 29 March 2023 at 2:37 am
To: paul.breb...@netapp.com.invalid 
Cc: us...@kafka.apache.org , dev@kafka.apache.org 
, mmcfarl...@cavulus.com , Israel 
Ekpo , ranlupov...@gmail.com , 
scante...@gmail.com , show...@gmail.com 
, sunilmchaudhar...@gmail.com 
Subject: Re: Kafka Cluster WITHOUT Zookeeper
NetApp Security WARNING: This is an external email. Do not click links or open 
attachments unless you recognize the sender and know the content is safe.




Paul, thanks for the articles. It's great to see someone digging into
KRaft! It would be interesting to see your experiments against a newer
version of KRaft, such as 3.4.0. Also, we hope to improve on the maximum
number of partitions with the addition of
https://cwiki.apache.org/confluence/display/KAFKA/KIP-868+Metadata+Transactions
which is likely to land in Kafka 3.5.0.

Cheers,
David

On Mon, Mar 27, 2023 at 6:32 PM Brebner, Paul
 wrote:

> I have a recent 3 part blog series on Kraft (expanded version of ApacheCon
> 2022 talk):
>
>
>
>
> https://www.instaclustr.com/blog/apache-kafka-kraft-abandons-the-zookeeper-part-1-partitions-and-data-performance/
>
>
> https://www.instaclustr.com/blog/apache-kafka-kraft-abandons-the-zookeeper-part-2-partitions-and-meta-data-performance/
>
>
> https://www.instaclustr.com/blog/apache-kafka-kraft-abandons-the-zookeeper-part-3-maximum-partitions-and-conclusions/
>
>
>
> Regards, Paul
>
>
>
> *From: *Chia-Ping Tsai 
> *Date: *Monday, 27 March 2023 at 5:37 pm
> *To: *dev@kafka.apache.org 
> *Cc: *us...@kafka.apache.org ,
> mmcfarl...@cavulus.com , Israel Ekpo <
> israele...@gmail.com>, ranlupov...@gmail.com ,
> scante...@gmail.com , show...@gmail.com <
> show...@gmail.com>, sunilmchaudhar...@gmail.com <
> sunilmchaudhar...@gmail.com>
> *Subject: *Re: Kafka Cluster WITHOUT Zookeeper
>
> *NetApp Security WARNING*: This is an external email. Do not click links
> or open attachments unless you recognize the sender and know the content is
> safe.
>
>
>
> hi
>
>
>
> You can use the keyword “kraft” to get the answer by google or chatgpt.
> For example:
>
>
>
> Introduction:
>
> KRaft - Apache Kafka Without ZooKeeper
> 
>
> developer.confluent.io 
>
>
>
>
>
> QuickStart:
>
> Apache Kafka 
>
> kafka.apache.org 
>
>
>
>
>
> —
>
> Chia-Ping
>
>
>
>
>
>
>
> Kafka Life  於 2023年3月27日 下午1:33 寫道：
>
> Hello  Kafka experts
>
> Is there a way where we can have Kafka Cluster be functional serving
> producers and consumers without having Zookeeper cluster manage the
> instance .
>
> Any particular version of kafka for this or how can we achieve this please
>
>

--
-David

[jira] [Created] (KAFKA-14864) Memory leak in KStreamWindowAggregate with ON_WINDOW_CLOSE emit strategy

[Victoria Xia (Jira)]

Victoria Xia created KAFKA-14864:


 Summary: Memory leak in KStreamWindowAggregate with 
ON_WINDOW_CLOSE emit strategy
 Key: KAFKA-14864
 URL: https://issues.apache.org/jira/browse/KAFKA-14864
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: Victoria Xia
Assignee: Victoria Xia
 Fix For: 3.5.0


The Streams DSL processor implementation for the ON_WINDOW_CLOSE emit strategy 
during KStream windowed aggregations opens a key-value iterator but does not 
call `close()` on it 
([link|https://github.com/apache/kafka/blob/5afedd9ac37c4d740f47867cfd31eaed15dc542f/streams/src/main/java/org/apache/kafka/streams/kstream/internals/AbstractKStreamTimeWindowAggregateProcessor.java#L203]),
 despite the Javadocs for the iterator making clear that users must do so in 
order to release resources 
([link|https://github.com/apache/kafka/blob/5afedd9ac37c4d740f47867cfd31eaed15dc542f/streams/src/main/java/org/apache/kafka/streams/state/KeyValueIterator.java#L27]).
  

I discovered this bug while running load testing benchmarks and noticed that 
some runs were sporadically hitting OOMs, so it is definitely possible to hit 
this in practice.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

[Jeff Kim]

Hi Alexandre,

Thanks for the feedback.

100. You're correct. I updated the KIP to explicitly state that once we
introduce a new tagged field in a later version, that version can never be
downgraded to the listed versions.

101. I don't think there is a way to differentiate the two but also not
sure if there will be a need to.

Best,
Jeff

On Tue, Mar 28, 2023 at 2:13 PM Alexandre Dupriez <
alexandre.dupr...@gmail.com> wrote:

> Hi Jeff,
>
> Thank you for the fast answer!
>
> 100. Got it, I think I am starting to understand based on your example
> exposing three Kafka versions. Please let me know if the following
> corresponds to the correct interpretation.
>
> Taking OffsetCommitValue as an example, the idea is to bump the schema
> to version 4 to make the record flexible. The schema version 4 will be
> back-ported to versions 3.0.3, 3.1.3, 3.2.4, 3.3.3, 3.4.1 and
> potentially 3.5. These versions will continue to serialise the record
> with version 3 so that downgrades from there to earlier minor and/or
> major versions are possible. And, these versions will deserialize the
> records with version 4 so that they can support flexible records,
> although they do not make use of tagged fields (if any is present).
>
> Then, in a future version, 3.x, x >= 6, a tag field will be added to
> the record - say, topic id. Downgrade from 3.x to 3.0.3, 3.1.3, 3.2.4,
> 3.3.3, 3.4.1 and 3.5 will be possible because these versions will
> perform deserialisation with the record schema version 4, that is,
> supporting tagged fields.
>
> Hence, this approach is both backward and forward looking and allows
> to extend the scope of compatible versions for downgrades.
>
> [N] 3.x, x >= 6 --> downgradable to [I] 3.0.3, 3.1.3, 3.2.4, 3.3.3,
> 3.4.1, 3.5 --> downgradable to e.g. [O] 3.0.0.
>
> One note though is that a transitive downgrade from 3.x, x >= 6, to
> 3.0.0 via the version domain [I], will not be supported. Should it be
> explicitly mentioned in the KIP that downgrades from 3.0.3, 3.1.3,
> 3.2.4, 3.3.3, 3.4.1, 3.5 to earlier versions may not be possible (if
> offsets or group metadata records were generated by a higher version
> 3.x, x >= 6)? Or am I still misunderstanding?
>
> 101. I will try to provide an example to illustrate what I mean by
> version. Consider the GroupMetadataValue schema version 4 and the
> tagged field topicId introduced in 3.6. Let's say a new optional field
> needs to be added in 3.7. We will have two records version 4, one with
> topic id, the other with topic id and the new field. The new field is
> semantically optional (structurally it is always optional since it is
> a tagged field) but we want to make the distinction between a record
> generated by 3.6 and one generated by 3.7. How do we resolve the
> ambiguity?
>
> Thanks!
> Alexandre
>
> Le mar. 28 mars 2023 à 16:13, Jeff Kim  a
> écrit :
> >
> > Hi Alexandre,
> >
> > Thank you for taking a look!
> >
> > 100. I am not sure I fully understand what you mean by forcefully adding
> > tagged fields. Let's say VX does not have a flexible version,
> > VY allows deserialization but serializes with a non-flexible version, and
> > VZ introduces a new tagged field.
> > VX upgrade to VY then downgrade back to VX works because even if group
> > metadata changes VY will serialize with
> > the highest non-flexible version. VZ to VY back to VZ also works because
> > even if VY serializes with a non-flexible field
> > VZ will be able to deserialize it as it is a supported version. Does this
> > answer your question?
> >
> > 101. The future versioning scheme needs to be backward compatible with
> > older coordinators. Wouldn't segregating into 2 versions
> > be incompatible?
> >
> > Thanks,
> > Jeff
> >
> > On Tue, Mar 28, 2023 at 5:47 AM Alexandre Dupriez <
> > alexandre.dupr...@gmail.com> wrote:
> >
> > > Hi Jeff, Team,
> > >
> > > Thank you for the KIP. This is a very interesting approach. I feel it
> > > is simpler than the described alternative although it comes with
> > > tradeoffs, thanks for highlighting those. If I may, I would like to
> > > share two naive questions.
> > >
> > > 100. The KIP mentions that records will be serialised with the highest
> > > non-flexible version (e.g. 3 for GroupMetadataValue and
> > > OffsetCommitValue) so that records can be deserialized with earlier
> > > versions of Kafka. I am not sure I understand correctly: is the idea
> > > to forcefully add tagged fields at the end of the records while
> > > maintaining the existing version (3 for the two record types just
> > > mentioned) so that they can be deserialized by existing Kafka versions
> > > for which the version of these record types is not known as flexible,
> > > while at the same time preserving the new tagged fields to new Kafka
> > > versions abreast of the addition of a new flexible version for these
> > > record types? If so, is it "bypassing" the protocol convention which
> > > prescribes the use of a flexible version to allow the use of tagged
> > > 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.3 #165

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1714

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 274159 lines...]
[2023-03-28T21:17:37.157Z] > Task :clients:compileTestJava UP-TO-DATE
[2023-03-28T21:17:37.157Z] > Task :clients:testClasses UP-TO-DATE
[2023-03-28T21:17:37.157Z] > Task 
:streams:generateMetadataFileForMavenJavaPublication
[2023-03-28T21:17:37.157Z] > Task :storage:api:compileTestJava UP-TO-DATE
[2023-03-28T21:17:37.157Z] > Task :storage:api:testClasses UP-TO-DATE
[2023-03-28T21:17:37.157Z] > Task :server-common:compileTestJava UP-TO-DATE
[2023-03-28T21:17:37.157Z] > Task :server-common:testClasses UP-TO-DATE
[2023-03-28T21:17:37.157Z] > Task :connect:json:compileTestJava UP-TO-DATE
[2023-03-28T21:17:37.157Z] > Task :raft:compileTestJava UP-TO-DATE
[2023-03-28T21:17:37.157Z] > Task :raft:testClasses UP-TO-DATE
[2023-03-28T21:17:37.157Z] > Task :connect:json:testClasses UP-TO-DATE
[2023-03-28T21:17:37.157Z] > Task :group-coordinator:compileTestJava UP-TO-DATE
[2023-03-28T21:17:37.157Z] > Task :group-coordinator:testClasses UP-TO-DATE
[2023-03-28T21:17:37.157Z] > Task :connect:json:testJar
[2023-03-28T21:17:37.157Z] > Task :metadata:compileTestJava UP-TO-DATE
[2023-03-28T21:17:37.157Z] > Task :metadata:testClasses UP-TO-DATE
[2023-03-28T21:17:37.157Z] > Task :connect:json:testSrcJar
[2023-03-28T21:17:37.157Z] > Task 
:clients:generateMetadataFileForMavenJavaPublication
[2023-03-28T21:17:39.618Z] 
[2023-03-28T21:17:39.618Z] > Task :connect:api:javadoc
[2023-03-28T21:17:39.618Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/connect/api/src/main/java/org/apache/kafka/connect/source/SourceRecord.java:44:
 warning - Tag @link: reference not found: org.apache.kafka.connect.data
[2023-03-28T21:17:41.710Z] 1 warning
[2023-03-28T21:17:41.710Z] 
[2023-03-28T21:17:41.710Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2023-03-28T21:17:41.710Z] > Task :connect:api:jar UP-TO-DATE
[2023-03-28T21:17:42.752Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2023-03-28T21:17:42.752Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2023-03-28T21:17:42.752Z] > Task :connect:json:jar UP-TO-DATE
[2023-03-28T21:17:42.752Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2023-03-28T21:17:42.752Z] > Task :connect:api:javadocJar
[2023-03-28T21:17:42.752Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2023-03-28T21:17:42.752Z] > Task :connect:api:testClasses UP-TO-DATE
[2023-03-28T21:17:42.752Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2023-03-28T21:17:42.752Z] > Task :connect:json:publishToMavenLocal
[2023-03-28T21:17:42.752Z] > Task :connect:api:testJar
[2023-03-28T21:17:42.752Z] > Task :connect:api:testSrcJar
[2023-03-28T21:17:42.752Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2023-03-28T21:17:42.752Z] > Task :connect:api:publishToMavenLocal
[2023-03-28T21:17:45.877Z] > Task :streams:javadoc
[2023-03-28T21:17:45.877Z] > Task :streams:javadocJar
[2023-03-28T21:17:47.962Z] 
[2023-03-28T21:17:47.962Z] > Task :clients:javadoc
[2023-03-28T21:17:47.962Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/package-info.java:21:
 warning - Tag @link: reference not found: 
org.apache.kafka.common.security.oauthbearer
[2023-03-28T21:17:49.004Z] 1 warning
[2023-03-28T21:17:50.045Z] 
[2023-03-28T21:17:50.045Z] > Task :clients:javadocJar
[2023-03-28T21:17:51.086Z] > Task :clients:srcJar
[2023-03-28T21:17:52.128Z] > Task :clients:testJar
[2023-03-28T21:17:52.128Z] > Task :clients:testSrcJar
[2023-03-28T21:17:52.128Z] > Task 
:clients:publishMavenJavaPublicationToMavenLocal
[2023-03-28T21:17:52.128Z] > Task :clients:publishToMavenLocal
[2023-03-28T21:18:06.215Z] > Task :core:compileScala
[2023-03-28T21:19:45.865Z] > Task :core:classes
[2023-03-28T21:19:45.865Z] > Task :core:compileTestJava NO-SOURCE
[2023-03-28T21:20:15.970Z] > Task :core:compileTestScala
[2023-03-28T21:21:53.601Z] > Task :core:testClasses
[2023-03-28T21:21:53.601Z] > Task :streams:compileTestJava UP-TO-DATE
[2023-03-28T21:21:53.601Z] > Task :streams:testClasses UP-TO-DATE
[2023-03-28T21:21:53.601Z] > Task :streams:testJar
[2023-03-28T21:21:53.601Z] > Task :streams:testSrcJar
[2023-03-28T21:21:53.601Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2023-03-28T21:21:53.601Z] > Task :streams:publishToMavenLocal
[2023-03-28T21:21:53.601Z] 
[2023-03-28T21:21:53.601Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 9.0.
[2023-03-28T21:21:53.601Z] 
[2023-03-28T21:21:53.601Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2023-03-28T21:21:53.601Z] 
[2023-03-28T21:21:53.601Z] See 
https://docs.gradle.org/8.0.2/userguide/command_line_interface.html#sec:command_line_warnings
[2023-03-28T21:21:53.601Z

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.4 #101

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 447577 lines...]
[2023-03-28T21:20:31.542Z] 
[2023-03-28T21:20:31.542Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testGetDataAndStat() PASSED
[2023-03-28T21:20:31.542Z] 
[2023-03-28T21:20:31.542Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testReassignPartitionsInProgress() STARTED
[2023-03-28T21:20:31.542Z] 
[2023-03-28T21:20:31.542Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testReassignPartitionsInProgress() PASSED
[2023-03-28T21:20:31.542Z] 
[2023-03-28T21:20:31.542Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testChrootExistsAndRootIsLocked() STARTED
[2023-03-28T21:20:32.586Z] 
[2023-03-28T21:20:32.586Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testChrootExistsAndRootIsLocked() PASSED
[2023-03-28T21:20:32.586Z] 
[2023-03-28T21:20:32.586Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testCreateTopLevelPaths() STARTED
[2023-03-28T21:20:32.586Z] 
[2023-03-28T21:20:32.586Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testCreateTopLevelPaths() PASSED
[2023-03-28T21:20:32.586Z] 
[2023-03-28T21:20:32.586Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > 
testGetAllTopicsInClusterDoesNotTriggerWatch() STARTED
[2023-03-28T21:20:33.631Z] 
[2023-03-28T21:20:33.631Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > 
testGetAllTopicsInClusterDoesNotTriggerWatch() PASSED
[2023-03-28T21:20:33.631Z] 
[2023-03-28T21:20:33.631Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testIsrChangeNotificationGetters() STARTED
[2023-03-28T21:20:33.631Z] 
[2023-03-28T21:20:33.631Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testIsrChangeNotificationGetters() PASSED
[2023-03-28T21:20:33.631Z] 
[2023-03-28T21:20:33.631Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testLogDirEventNotificationsDeletion() 
STARTED
[2023-03-28T21:20:33.631Z] 
[2023-03-28T21:20:33.631Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testLogDirEventNotificationsDeletion() PASSED
[2023-03-28T21:20:33.631Z] 
[2023-03-28T21:20:33.631Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testGetLogConfigs() STARTED
[2023-03-28T21:20:34.676Z] 
[2023-03-28T21:20:34.676Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testGetLogConfigs() PASSED
[2023-03-28T21:20:34.676Z] 
[2023-03-28T21:20:34.676Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testBrokerSequenceIdMethods() STARTED
[2023-03-28T21:20:34.676Z] 
[2023-03-28T21:20:34.676Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testBrokerSequenceIdMethods() PASSED
[2023-03-28T21:20:34.676Z] 
[2023-03-28T21:20:34.676Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testAclMethods() STARTED
[2023-03-28T21:20:34.676Z] 
[2023-03-28T21:20:34.676Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testAclMethods() PASSED
[2023-03-28T21:20:34.676Z] 
[2023-03-28T21:20:34.676Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testCreateSequentialPersistentPath() STARTED
[2023-03-28T21:20:35.721Z] 
[2023-03-28T21:20:35.721Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testCreateSequentialPersistentPath() PASSED
[2023-03-28T21:20:35.721Z] 
[2023-03-28T21:20:35.721Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testConditionalUpdatePath() STARTED
[2023-03-28T21:20:35.721Z] 
[2023-03-28T21:20:35.721Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testConditionalUpdatePath() PASSED
[2023-03-28T21:20:35.721Z] 
[2023-03-28T21:20:35.721Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testGetAllTopicsInClusterTriggersWatch() 
STARTED
[2023-03-28T21:20:35.721Z] 
[2023-03-28T21:20:35.721Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testGetAllTopicsInClusterTriggersWatch() 
PASSED
[2023-03-28T21:20:35.721Z] 
[2023-03-28T21:20:35.721Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 172 > KafkaZkClientTest > testDeleteTopicZNode() STARTED
[2023-03-28T21:20:36.766Z] 
[2023-03-28T21:20:36.766Z] Gradle Test Run :core:integration

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.2 #101

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 379200 lines...]
[2023-03-28T21:07:50.454Z] 
[2023-03-28T21:07:50.454Z] 
org.apache.kafka.streams.integration.StreamTableJoinTopologyOptimizationIntegrationTest
 > shouldDoStreamTableJoinWithDifferentNumberOfPartitions[Optimization = all] 
STARTED
[2023-03-28T21:07:52.808Z] 
[2023-03-28T21:07:52.808Z] 
org.apache.kafka.streams.integration.StreamTableJoinTopologyOptimizationIntegrationTest
 > shouldDoStreamTableJoinWithDifferentNumberOfPartitions[Optimization = all] 
PASSED
[2023-03-28T21:07:52.808Z] 
[2023-03-28T21:07:52.808Z] 
org.apache.kafka.streams.integration.StreamTableJoinTopologyOptimizationIntegrationTest
 > shouldDoStreamTableJoinWithDifferentNumberOfPartitions[Optimization = none] 
STARTED
[2023-03-28T21:07:56.410Z] 
[2023-03-28T21:07:56.410Z] 
org.apache.kafka.streams.integration.StreamTableJoinTopologyOptimizationIntegrationTest
 > shouldDoStreamTableJoinWithDifferentNumberOfPartitions[Optimization = none] 
PASSED
[2023-03-28T21:07:58.711Z] 
[2023-03-28T21:07:58.711Z] 
org.apache.kafka.streams.integration.StreamsUpgradeTestIntegrationTest > 
testVersionProbingUpgrade STARTED
[2023-03-28T21:08:46.686Z] 
[2023-03-28T21:08:46.686Z] 
org.apache.kafka.streams.integration.StreamsUpgradeTestIntegrationTest > 
testVersionProbingUpgrade PASSED
[2023-03-28T21:08:46.686Z] 
[2023-03-28T21:08:46.686Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldInheritSerdes STARTED
[2023-03-28T21:08:46.686Z] 
[2023-03-28T21:08:46.686Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldInheritSerdes PASSED
[2023-03-28T21:08:46.686Z] 
[2023-03-28T21:08:46.686Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldShutdownWhenRecordConstraintIsViolated STARTED
[2023-03-28T21:08:47.701Z] 
[2023-03-28T21:08:47.701Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldShutdownWhenRecordConstraintIsViolated PASSED
[2023-03-28T21:08:47.701Z] 
[2023-03-28T21:08:47.701Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldUseDefaultSerdes STARTED
[2023-03-28T21:08:50.533Z] 
[2023-03-28T21:08:50.533Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldUseDefaultSerdes PASSED
[2023-03-28T21:08:50.533Z] 
[2023-03-28T21:08:50.533Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldAllowDisablingChangelog STARTED
[2023-03-28T21:08:54.461Z] 
[2023-03-28T21:08:54.461Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldAllowDisablingChangelog PASSED
[2023-03-28T21:08:54.461Z] 
[2023-03-28T21:08:54.461Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldAllowOverridingChangelogConfig STARTED
[2023-03-28T21:08:57.069Z] 
[2023-03-28T21:08:57.069Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldAllowOverridingChangelogConfig PASSED
[2023-03-28T21:08:57.069Z] 
[2023-03-28T21:08:57.069Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldShutdownWhenBytesConstraintIsViolated STARTED
[2023-03-28T21:09:02.057Z] 
[2023-03-28T21:09:02.057Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldShutdownWhenBytesConstraintIsViolated PASSED
[2023-03-28T21:09:02.057Z] 
[2023-03-28T21:09:02.057Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldCreateChangelogByDefault STARTED
[2023-03-28T21:09:07.052Z] 
[2023-03-28T21:09:07.052Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldCreateChangelogByDefault PASSED
[2023-03-28T21:09:07.979Z] 
[2023-03-28T21:09:07.979Z] 
org.apache.kafka.streams.integration.TaskAssignorIntegrationTest > 
shouldProperlyConfigureTheAssignor STARTED
[2023-03-28T21:09:08.990Z] 
[2023-03-28T21:09:08.990Z] 
org.apache.kafka.streams.integration.TaskAssignorIntegrationTest > 
shouldProperlyConfigureTheAssignor PASSED
[2023-03-28T21:09:12.227Z] 
[2023-03-28T21:09:12.227Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorLargeNumConsumers STARTED
[2023-03-28T21:09:28.263Z] 
[2023-03-28T21:09:28.263Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorLargeNumConsumers PASSED
[2023-03-28T21:09:28.263Z] 
[2023-03-28T21:09:28.263Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorLargePartitionCount STARTED
[2023-03-28T21:09:28.263Z] 
[2023-03-28T21:09:28.263Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorLargePartitionCount PASSED
[2023-03-28T21:09:28.263Z] 
[2023-03-28T21:09:28.263Z] 
org.apache.kafka.streams.processor.internals.StreamsAssignmentScaleTest > 
testHighAvailabilityTaskAssignorManyThreadsPerClient STARTED
[2023-0

[jira] [Created] (KAFKA-14863) Plugins which do not have a valid no-args constructor are visible in the REST API

[Greg Harris (Jira)]

Greg Harris created KAFKA-14863:
---

 Summary: Plugins which do not have a valid no-args constructor are 
visible in the REST API
 Key: KAFKA-14863
 URL: https://issues.apache.org/jira/browse/KAFKA-14863
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Reporter: Greg Harris
Assignee: Greg Harris


Currently, the Connect plugin discovery mechanisms only assert that a no-args 
constructor is present when necessary. In particular, this assertion happens 
for Connectors when the framework needs to evaluate the connector's version 
method.
It also happens for ConnectorConfigOverridePolicy, ConnectRestExtension, and 
ConfigProvider plugins, which are loaded via the ServiceLoader. The 
ServiceLoader constructs instances of plugins with their no-args constructor 
during discovery, so these plugins are discovered even if they are not 
Versioned.

This has the effect that these unusable plugins which are missing a default 
constructor appear in the REST API, but are not able to be instantiated or 
used. To make the ServiceLoader and Reflections discovery mechanisms behave 
more similar, this assertion should be applied to all plugins, and a log 
message emitted when plugins do not follow the constructor requirements.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

[Alexandre Dupriez]

Hi Jeff,

Thank you for the fast answer!

100. Got it, I think I am starting to understand based on your example
exposing three Kafka versions. Please let me know if the following
corresponds to the correct interpretation.

Taking OffsetCommitValue as an example, the idea is to bump the schema
to version 4 to make the record flexible. The schema version 4 will be
back-ported to versions 3.0.3, 3.1.3, 3.2.4, 3.3.3, 3.4.1 and
potentially 3.5. These versions will continue to serialise the record
with version 3 so that downgrades from there to earlier minor and/or
major versions are possible. And, these versions will deserialize the
records with version 4 so that they can support flexible records,
although they do not make use of tagged fields (if any is present).

Then, in a future version, 3.x, x >= 6, a tag field will be added to
the record - say, topic id. Downgrade from 3.x to 3.0.3, 3.1.3, 3.2.4,
3.3.3, 3.4.1 and 3.5 will be possible because these versions will
perform deserialisation with the record schema version 4, that is,
supporting tagged fields.

Hence, this approach is both backward and forward looking and allows
to extend the scope of compatible versions for downgrades.

[N] 3.x, x >= 6 --> downgradable to [I] 3.0.3, 3.1.3, 3.2.4, 3.3.3,
3.4.1, 3.5 --> downgradable to e.g. [O] 3.0.0.

One note though is that a transitive downgrade from 3.x, x >= 6, to
3.0.0 via the version domain [I], will not be supported. Should it be
explicitly mentioned in the KIP that downgrades from 3.0.3, 3.1.3,
3.2.4, 3.3.3, 3.4.1, 3.5 to earlier versions may not be possible (if
offsets or group metadata records were generated by a higher version
3.x, x >= 6)? Or am I still misunderstanding?

101. I will try to provide an example to illustrate what I mean by
version. Consider the GroupMetadataValue schema version 4 and the
tagged field topicId introduced in 3.6. Let's say a new optional field
needs to be added in 3.7. We will have two records version 4, one with
topic id, the other with topic id and the new field. The new field is
semantically optional (structurally it is always optional since it is
a tagged field) but we want to make the distinction between a record
generated by 3.6 and one generated by 3.7. How do we resolve the
ambiguity?

Thanks!
Alexandre

Le mar. 28 mars 2023 à 16:13, Jeff Kim  a écrit :
>
> Hi Alexandre,
>
> Thank you for taking a look!
>
> 100. I am not sure I fully understand what you mean by forcefully adding
> tagged fields. Let's say VX does not have a flexible version,
> VY allows deserialization but serializes with a non-flexible version, and
> VZ introduces a new tagged field.
> VX upgrade to VY then downgrade back to VX works because even if group
> metadata changes VY will serialize with
> the highest non-flexible version. VZ to VY back to VZ also works because
> even if VY serializes with a non-flexible field
> VZ will be able to deserialize it as it is a supported version. Does this
> answer your question?
>
> 101. The future versioning scheme needs to be backward compatible with
> older coordinators. Wouldn't segregating into 2 versions
> be incompatible?
>
> Thanks,
> Jeff
>
> On Tue, Mar 28, 2023 at 5:47 AM Alexandre Dupriez <
> alexandre.dupr...@gmail.com> wrote:
>
> > Hi Jeff, Team,
> >
> > Thank you for the KIP. This is a very interesting approach. I feel it
> > is simpler than the described alternative although it comes with
> > tradeoffs, thanks for highlighting those. If I may, I would like to
> > share two naive questions.
> >
> > 100. The KIP mentions that records will be serialised with the highest
> > non-flexible version (e.g. 3 for GroupMetadataValue and
> > OffsetCommitValue) so that records can be deserialized with earlier
> > versions of Kafka. I am not sure I understand correctly: is the idea
> > to forcefully add tagged fields at the end of the records while
> > maintaining the existing version (3 for the two record types just
> > mentioned) so that they can be deserialized by existing Kafka versions
> > for which the version of these record types is not known as flexible,
> > while at the same time preserving the new tagged fields to new Kafka
> > versions abreast of the addition of a new flexible version for these
> > record types? If so, is it "bypassing" the protocol convention which
> > prescribes the use of a flexible version to allow the use of tagged
> > fields?
> >
> > 101. After the bump of the records to a new version indicated as
> > flexible, the record version is expected to never change while the
> > underlying tagged fields could potentially still evolve over time. One
> > potential downside is that we lose the benefits of the versioning
> > scheme enforced by the serde protocol. Could this become a problem in
> > the future if there is ever a need to segregate two distinct
> > "versions" of the appended record structure held by the tagged fields?
> >
> > Thanks,
> > Alexandre
> >
> > Le jeu. 23 mars 2023 à 18:15, Jeff Kim  a
> > écrit :
> > >
> > > 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1713

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 275286 lines...]
[2023-03-28T16:47:24.719Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 175 > KafkaZkClientTest > testIsrChangeNotificationGetters() STARTED
[2023-03-28T16:47:24.719Z] 
[2023-03-28T16:47:24.719Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 175 > KafkaZkClientTest > testIsrChangeNotificationGetters() PASSED
[2023-03-28T16:47:24.719Z] 
[2023-03-28T16:47:24.719Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 175 > KafkaZkClientTest > testLogDirEventNotificationsDeletion() 
STARTED
[2023-03-28T16:47:24.719Z] 
[2023-03-28T16:47:24.719Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 175 > KafkaZkClientTest > testLogDirEventNotificationsDeletion() PASSED
[2023-03-28T16:47:24.719Z] 
[2023-03-28T16:47:24.719Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 175 > KafkaZkClientTest > testGetLogConfigs() STARTED
[2023-03-28T16:47:24.719Z] 
[2023-03-28T16:47:24.719Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 175 > KafkaZkClientTest > testGetLogConfigs() PASSED
[2023-03-28T16:47:24.719Z] 
[2023-03-28T16:47:24.719Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 175 > KafkaZkClientTest > testBrokerSequenceIdMethods() STARTED
[2023-03-28T16:47:24.719Z] 
[2023-03-28T16:47:24.719Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 175 > KafkaZkClientTest > testBrokerSequenceIdMethods() PASSED
[2023-03-28T16:47:24.719Z] 
[2023-03-28T16:47:24.719Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 175 > KafkaZkClientTest > testAclMethods() STARTED
[2023-03-28T16:47:24.719Z] 
[2023-03-28T16:47:24.719Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 175 > KafkaZkClientTest > testAclMethods() PASSED
[2023-03-28T16:47:24.719Z] 
[2023-03-28T16:47:24.719Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 175 > KafkaZkClientTest > testCreateSequentialPersistentPath() STARTED
[2023-03-28T16:47:24.719Z] 
[2023-03-28T16:47:24.719Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 175 > KafkaZkClientTest > testCreateSequentialPersistentPath() PASSED
[2023-03-28T16:47:24.719Z] 
[2023-03-28T16:47:24.719Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 175 > KafkaZkClientTest > testConditionalUpdatePath() STARTED
[2023-03-28T16:47:25.685Z] 
[2023-03-28T16:47:25.685Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 175 > KafkaZkClientTest > testConditionalUpdatePath() PASSED
[2023-03-28T16:47:25.685Z] 
[2023-03-28T16:47:25.685Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 175 > KafkaZkClientTest > testGetAllTopicsInClusterTriggersWatch() 
STARTED
[2023-03-28T16:47:25.685Z] 
[2023-03-28T16:47:25.685Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 175 > KafkaZkClientTest > testGetAllTopicsInClusterTriggersWatch() 
PASSED
[2023-03-28T16:47:25.685Z] 
[2023-03-28T16:47:25.685Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 175 > KafkaZkClientTest > testDeleteTopicZNode() STARTED
[2023-03-28T16:47:27.643Z] 
[2023-03-28T16:47:27.643Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 175 > KafkaZkClientTest > testDeleteTopicZNode() PASSED
[2023-03-28T16:47:27.643Z] 
[2023-03-28T16:47:27.643Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 175 > KafkaZkClientTest > testDeletePath() STARTED
[2023-03-28T16:47:27.643Z] 
[2023-03-28T16:47:27.643Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 175 > KafkaZkClientTest > testDeletePath() PASSED
[2023-03-28T16:47:27.643Z] 
[2023-03-28T16:47:27.643Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 175 > KafkaZkClientTest > testGetBrokerMethods() STARTED
[2023-03-28T16:47:27.643Z] 
[2023-03-28T16:47:27.643Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 175 > KafkaZkClientTest > testGetBrokerMethods() PASSED
[2023-03-28T16:47:27.643Z] 
[2023-03-28T16:47:27.643Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 175 > KafkaZkClientTest > testJuteMaxBufffer() STARTED
[2023-03-28T16:47:27.643Z] 
[2023-03-28T16:47:27.643Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 175 > KafkaZkClientTest > testJuteMaxBufffer() PASSED
[2023-03-28T16:47:27.643Z] 
[2023-03-28T16:47:27.643Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 175 > KafkaZkClientTest > testCreateTokenChangeNotification() STARTED
[2023-03-28T16:47:27.643Z] 
[2023-03-28T16:47:27.643Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 175 > KafkaZkClientTest > testCreateTokenChangeNotification() PASSED
[2023-03-28T16:47:27.643Z] 
[2023-03-28T16:47:27.643Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 175 > KafkaZkClientTest > testGetTopicsAndPartitions() STARTED
[2023-03-28T16:47:30.22

Re: Kafka Cluster WITHOUT Zookeeper

[David Arthur]

Paul, thanks for the articles. It's great to see someone digging into
KRaft! It would be interesting to see your experiments against a newer
version of KRaft, such as 3.4.0. Also, we hope to improve on the maximum
number of partitions with the addition of
https://cwiki.apache.org/confluence/display/KAFKA/KIP-868+Metadata+Transactions
which is likely to land in Kafka 3.5.0.

Cheers,
David

On Mon, Mar 27, 2023 at 6:32 PM Brebner, Paul
 wrote:

> I have a recent 3 part blog series on Kraft (expanded version of ApacheCon
> 2022 talk):
>
>
>
>
> https://www.instaclustr.com/blog/apache-kafka-kraft-abandons-the-zookeeper-part-1-partitions-and-data-performance/
>
>
> https://www.instaclustr.com/blog/apache-kafka-kraft-abandons-the-zookeeper-part-2-partitions-and-meta-data-performance/
>
>
> https://www.instaclustr.com/blog/apache-kafka-kraft-abandons-the-zookeeper-part-3-maximum-partitions-and-conclusions/
>
>
>
> Regards, Paul
>
>
>
> *From: *Chia-Ping Tsai 
> *Date: *Monday, 27 March 2023 at 5:37 pm
> *To: *dev@kafka.apache.org 
> *Cc: *us...@kafka.apache.org ,
> mmcfarl...@cavulus.com , Israel Ekpo <
> israele...@gmail.com>, ranlupov...@gmail.com ,
> scante...@gmail.com , show...@gmail.com <
> show...@gmail.com>, sunilmchaudhar...@gmail.com <
> sunilmchaudhar...@gmail.com>
> *Subject: *Re: Kafka Cluster WITHOUT Zookeeper
>
> *NetApp Security WARNING*: This is an external email. Do not click links
> or open attachments unless you recognize the sender and know the content is
> safe.
>
>
>
> hi
>
>
>
> You can use the keyword “kraft” to get the answer by google or chatgpt.
> For example:
>
>
>
> Introduction:
>
> KRaft - Apache Kafka Without ZooKeeper
> 
>
> developer.confluent.io 
>
>
>
>
>
> QuickStart:
>
> Apache Kafka 
>
> kafka.apache.org 
>
>
>
>
>
> —
>
> Chia-Ping
>
>
>
>
>
>
>
> Kafka Life  於 2023年3月27日 下午1:33 寫道：
>
> Hello  Kafka experts
>
> Is there a way where we can have Kafka Cluster be functional serving
> producers and consumers without having Zookeeper cluster manage the
> instance .
>
> Any particular version of kafka for this or how can we achieve this please
>
>

-- 
-David

[jira] [Resolved] (KAFKA-14843) Connector plugins config endpoint does not include Common configs

[Chris Egerton (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Egerton resolved KAFKA-14843.
---
Fix Version/s: 3.5.0
   Resolution: Fixed

> Connector plugins config endpoint does not include Common configs
> -
>
> Key: KAFKA-14843
> URL: https://issues.apache.org/jira/browse/KAFKA-14843
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 3.2.0, 3.3.0, 3.2.1, 3.4.0, 3.2.2, 3.2.3, 3.3.1, 3.3.2
>Reporter: Jorge Esteban Quilcate Otoya
>Assignee: Jorge Esteban Quilcate Otoya
>Priority: Major
> Fix For: 3.5.0
>
>
> Connector plugins GET config endpoint introduced in 
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+APIs+to+list+all+connector+plugins+and+retrieve+their+configuration+definitions]
>   allows to get plugin configuration from the rest endpoint.
> This configuration only includes the plugin configuration, but not the base 
> configuration of the Sink/Source Connector.
> For instance, when validating the configuration of a plugin, _all_ configs 
> are returned:
> ```
> curl -s 
> $CONNECT_URL/connector-plugins/io.aiven.kafka.connect.http.HttpSinkConnector/config
>  | jq -r '.[].name' | sort -u | wc -l     
> 21
> curl -s 
> $CONNECT_URL/connector-plugins/io.aiven.kafka.connect.http.HttpSinkConnector/config/validate
>  -XPUT -H 'Content-type: application/json' --data "\{\"connector.class\": 
> \"io.aiven.kafka.connect.http.HttpSinkConnector\", \"topics\": 
> \"example-topic-name\"}" | jq -r '.configs[].definition.name' | sort -u | wc 
> -l
> 39
> ```
> and the missing configs are all from base config:
> ```
> diff validate.txt config.txt                                                  
>                                                   
> 6,14d5
> < config.action.reload
> < connector.class
> < errors.deadletterqueue.context.headers.enable
> < errors.deadletterqueue.topic.name
> < errors.deadletterqueue.topic.replication.factor
> < errors.log.enable
> < errors.log.include.messages
> < errors.retry.delay.max.ms
> < errors.retry.timeout
> 16d6
> < header.converter
> 24d13
> < key.converter
> 26d14
> < name
> 33d20
> < predicates
> 35,39d21
> < tasks.max
> < topics
> < topics.regex
> < transforms
> < value.converter
> ```
> Would be great to get the base configs from the same endpoint as well, so we 
> could rely on it instead of using the validate endpoint to get all configs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

[Jeff Kim]

Hi Alexandre,

Thank you for taking a look!

100. I am not sure I fully understand what you mean by forcefully adding
tagged fields. Let's say VX does not have a flexible version,
VY allows deserialization but serializes with a non-flexible version, and
VZ introduces a new tagged field.
VX upgrade to VY then downgrade back to VX works because even if group
metadata changes VY will serialize with
the highest non-flexible version. VZ to VY back to VZ also works because
even if VY serializes with a non-flexible field
VZ will be able to deserialize it as it is a supported version. Does this
answer your question?

101. The future versioning scheme needs to be backward compatible with
older coordinators. Wouldn't segregating into 2 versions
be incompatible?

Thanks,
Jeff

On Tue, Mar 28, 2023 at 5:47 AM Alexandre Dupriez <
alexandre.dupr...@gmail.com> wrote:

> Hi Jeff, Team,
>
> Thank you for the KIP. This is a very interesting approach. I feel it
> is simpler than the described alternative although it comes with
> tradeoffs, thanks for highlighting those. If I may, I would like to
> share two naive questions.
>
> 100. The KIP mentions that records will be serialised with the highest
> non-flexible version (e.g. 3 for GroupMetadataValue and
> OffsetCommitValue) so that records can be deserialized with earlier
> versions of Kafka. I am not sure I understand correctly: is the idea
> to forcefully add tagged fields at the end of the records while
> maintaining the existing version (3 for the two record types just
> mentioned) so that they can be deserialized by existing Kafka versions
> for which the version of these record types is not known as flexible,
> while at the same time preserving the new tagged fields to new Kafka
> versions abreast of the addition of a new flexible version for these
> record types? If so, is it "bypassing" the protocol convention which
> prescribes the use of a flexible version to allow the use of tagged
> fields?
>
> 101. After the bump of the records to a new version indicated as
> flexible, the record version is expected to never change while the
> underlying tagged fields could potentially still evolve over time. One
> potential downside is that we lose the benefits of the versioning
> scheme enforced by the serde protocol. Could this become a problem in
> the future if there is ever a need to segregate two distinct
> "versions" of the appended record structure held by the tagged fields?
>
> Thanks,
> Alexandre
>
> Le jeu. 23 mars 2023 à 18:15, Jeff Kim  a
> écrit :
> >
> > Hi Yi,
> >
> > > Does it mean with a flexible version, the future
> > version of these value types will stay at the version where the
> flexibility
> > is first introduced? Will there be any need to bump the version again in
> > the future?
> >
> > Yes, there will be no need to bump the version since we will only be
> adding
> > tagged fields but in the chance the version is bumped, we will
> deserialize
> > to the highest known (flexible) version which will ignore unknown tagged
> > fields.
> >
> > > To enforce the version not bumping, is it possible to have a unit test
> to
> > gate?
> >
> > Do you have some tests in mind? I don't think we can tell whether a
> version
> > was bumped or not.
> >
> > Best,
> > Jeff
> >
> > On Thu, Mar 23, 2023 at 12:07 PM Yi Ding 
> wrote:
> >
> > > Hi Jeff,
> > >
> > > Thanks for the update. Does it mean with a flexible version, the future
> > > version of these value types will stay at the version where the
> flexibility
> > > is first introduced? Will there be any need to bump the version again
> in
> > > the future?
> > > To enforce the version not bumping, is it possible to have a unit test
> to
> > > gate?
> > >
> > >
> > > On Wed, Mar 22, 2023 at 3:19 PM Jeff Kim  >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > After discussing with my colleagues, I have repurposed the KIP for a
> > > > general downgrade solution for both transaction and group
> coordinators.
> > > The
> > > > KIP no longer discusses the downgrade path but instead lays out the
> > > > foundation for future downgrade solutions.
> > > >
> > > > Link:
> > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-915%3A+Txn+and+Group+Coordinator+Downgrade+Foundation
> > > >
> > > > Thanks,
> > > > Jeff
> > > >
> > > > On Mon, Mar 20, 2023 at 7:37 PM Jeff Kim 
> wrote:
> > > >
> > > > > Hi David and Justine,
> > > > >
> > > > > Thank you both for the detailed feedback.
> > > > >
> > > > > David,
> > > > >
> > > > > 1. That makes sense. I revised the "Reading new fields" section
> with
> > > how
> > > > > we can downgrade to the highest known version and that this was
> > > confirmed
> > > > > via unit testing. I also attempted to dive deeper into using tagged
> > > > fields
> > > > > and the rejected alternative. Please let me know what you think.
> > > > >
> > > > > 2. Under "Restrictions and Guidelines" I updated the paragraph to
> > > clearly
> > > > > state that we want to use tagged f

Re: [DISCUSS] KIP-910: Update Source offsets for Source Connectors without producing records

[Chris Egerton]

Hi Sagar,

Thanks for the KIP! I have some thoughts.

Nits:

1. Shouldn't KAFKA-3821 [1] be linked as the Jira ticket on the KIP? Or is
there a different ticket that should be associated with it?
2. The current state is listed as "Draft". Considering it's been brought up
for discussion, maybe the KIP should be updated to "Discussion"?
3. Can you add a link for the discussion thread to the KIP?
4. The KIP states that "In this process, offsets are written at regular
intervals(driven by `offset.flush.interval.ms`)". This isn't strictly
accurate since, when exactly-once support is enabled, offset commits can
also be performed for each record batch (which is the default) or when
explicitly requested by the task instance (if the connector implements the
API to define its own transactions and the user has configured it to do
so). Maybe better to just say "Offsets are written periodically"?
5. The description for the (per-connector) "heartbeat.records.topic "
property states that it is "Only applicable in distributed mode; in
standalone mode, setting this property will have no effect". Is this
correct?

Non-nits:

6. It seems (based on both the KIP and discussion on KAFKA-3821) that the
only use case for being able to emit offsets without also emitting source
records that's been identified so far is for CDC source connectors like
Debezium. But Debezium already has support for this exact feature (emitting
heartbeat records that include offsets that cannot be associated with
other, "regular" source records). Why should we add this feature to Kafka
Connect when the problem it addresses is already solved in the set
connectors that (it seems) would have any need for it, and the size of that
set is extremely small? If there are other practical use cases for
connectors that would benefit from this feature, please let me know.

7. If a task produces heartbeat records and source records that use the
same source partition, which offset will ultimately be committed?

8. The SourceTask::produceHeartbeatRecords method returns a
List, and users can control the heartbeat topic for a
connector via the (connector- or worker-level) "heartbeat.records.topic"
property. Since every constructor for the SourceRecord class [2] requires a
topic to be supplied, what will happen to that topic? Will it be ignored?
If so, I think we should look for a cleaner solution.

9. A large concern raised in the discussion for KAFKA-3821 was the allowing
connectors to control the ordering of these special "offsets-only"
emissions and the regular source records returned from SourceTask::poll.
Are we choosing to ignore that concern? If so, can you add this to the
rejected alternatives section along with a rationale?

10. If, sometime in the future, we wanted to add framework-level support
for sending heartbeat records that doesn't require connectors to implement
any new APIs (e.g., SourceTask::produceHeartbeatRecords), a lot of this
would paint us into a corner design-wise. We'd have to think carefully
about which property names would be used, how to account for connectors
that have already implemented the SourceTask::produceHeartbeatRecords
method, etc. In general, it seems like we're trying to solve two completely
different problems with this single KIP: adding framework-level support for
emitting heartbeat records for source connectors, and allowing source
connectors to emit offsets without also emitting source records. I don't
mind addressing the two at the same time if the result is elegant and
doesn't compromise on the solution for either problem, but that doesn't
seem to be the case here. Of the two problems, could we describe one as the
primary and one as the secondary? If so, we might consider dropping the
secondary problm from this KIP and addressing it separately.

[1] - https://issues.apache.org/jira/browse/KAFKA-3821
[2] -
https://kafka.apache.org/34/javadoc/org/apache/kafka/connect/source/SourceRecord.html

Cheers,

Chris

On Sat, Mar 25, 2023 at 11:18 PM Sagar  wrote:

> Hi John,
>
> Thanks for taking. look at the KIP!
>
> The point about stream time not advancing in case of infrequent updates is
> an interesting one. I can imagine if the upstream producer to a Kafka
> Streams application is a Source Connector which isn't sending records
> frequently(due to the nature of the data ingestion for example), then the
> downstream stream processing can land into the issues you described above.
>
> Which also brings me to the second point you made about how this would be
> used by downstream consumers. IIUC, you are referring to the consumers of
> the newly added topic i.e the heartbeat topic. In my mind, the heartbeat
> topic is an internal topic (similar to offsets/config/status topic in
> connect), the main purpose of which is to trick the framework to produce
> records to the offsets topic and advance the offsets. Since every connector
> could have a different definition of offsets(LSN, BinLogID etc for
> example), that logic to determine what

[jira] [Created] (KAFKA-14862) Outer stream-stream join does not output all results with multiple input partitions

[Bruno Cadonna (Jira)]

Bruno Cadonna created KAFKA-14862:
-

 Summary: Outer stream-stream join does not output all results with 
multiple input partitions
 Key: KAFKA-14862
 URL: https://issues.apache.org/jira/browse/KAFKA-14862
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: Bruno Cadonna


If I execute the following Streams app once with two input topics each with 1 
partition and then with input topics each with two partitions, I get different 
results.
  
{code:java}
final KStream leftSide = builder.stream(leftSideTopic);
final KStream rightSide = 
builder.stream(rightSideTopic);

final KStream leftAndRight = leftSide.outerJoin(
rightSide,
(leftValue, rightValue) ->
(rightValue == null) ? leftValue + "/NOTPRESENT": leftValue + 
"/" + rightValue,
JoinWindows.ofTimeDifferenceAndGrace(Duration.ofSeconds(20), 
Duration.ofSeconds(10)),
StreamJoined.with(
Serdes.String(), /* key */
Serdes.String(), /* left value */
Serdes.String()  /* right value */
)
);
leftAndRight.print(Printed.toSysOut());
{code}

To reproduce, produce twice the following batch of records with an interval 
greater than window + grace period (i.e. > 30 seconds) in between the two 
batches:
{code}
(0, 0)
(1, 1)
(2, 2)
(3, 3)
(4, 4)
(5, 5)
(6, 6)
(7, 7)
(8, 8)
(9, 9)
{code}

With input topics with 1 partition I get:
{code}
[KSTREAM-PROCESSVALUES-08]: 0, 0/NOTPRESENT
[KSTREAM-PROCESSVALUES-08]: 1, 1/NOTPRESENT
[KSTREAM-PROCESSVALUES-08]: 2, 2/NOTPRESENT
[KSTREAM-PROCESSVALUES-08]: 3, 3/NOTPRESENT
[KSTREAM-PROCESSVALUES-08]: 4, 4/NOTPRESENT
[KSTREAM-PROCESSVALUES-08]: 5, 5/NOTPRESENT
[KSTREAM-PROCESSVALUES-08]: 6, 6/NOTPRESENT
[KSTREAM-PROCESSVALUES-08]: 7, 7/NOTPRESENT
[KSTREAM-PROCESSVALUES-08]: 8, 8/NOTPRESENT
[KSTREAM-PROCESSVALUES-08]: 9, 9/NOTPRESENT
{code}

With input topics with 2 partitions I get:
{code}
[KSTREAM-PROCESSVALUES-08]: 1, 1/NOTPRESENT
[KSTREAM-PROCESSVALUES-08]: 3, 3/NOTPRESENT
[KSTREAM-PROCESSVALUES-08]: 4, 4/NOTPRESENT
[KSTREAM-PROCESSVALUES-08]: 7, 7/NOTPRESENT
[KSTREAM-PROCESSVALUES-08]: 8, 8/NOTPRESENT
[KSTREAM-PROCESSVALUES-08]: 9, 9/NOTPRESENT
{code}

I would expect to get the same set of records, maybe in a different order due 
to the partitioning.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14861) Migration Documentation should state the need for connectivity from controller to brokers when provisioning the KRaft controller quorum

[Proven Provenzano (Jira)]

Proven Provenzano created KAFKA-14861:
-

 Summary: Migration Documentation should state the need for 
connectivity from controller to brokers when provisioning the KRaft controller 
quorum
 Key: KAFKA-14861
 URL: https://issues.apache.org/jira/browse/KAFKA-14861
 Project: Kafka
  Issue Type: Improvement
  Components: documentation, kraft
Affects Versions: 3.4.0
Reporter: Proven Provenzano


Normal KRaft cluster operations with no nodes configured with {{process.roles}} 
as both controller and broker, there is no need to configure connectivity from 
the controllers to the brokers. Only brokers need to connect to controllers.

The ZK to KRaft migration does need controller to broker connectivity during 
the migration and this should be made explicit in the documentation in the 
*Provisioning the KRaft quorum controller* section.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14860) New controllers created for migration should validate connectivity to brokers before starting migration

[Proven Provenzano (Jira)]

Proven Provenzano created KAFKA-14860:
-

 Summary: New controllers created for migration should validate 
connectivity to brokers before starting migration
 Key: KAFKA-14860
 URL: https://issues.apache.org/jira/browse/KAFKA-14860
 Project: Kafka
  Issue Type: Bug
  Components: kraft
Reporter: Proven Provenzano
Assignee: Proven Provenzano


The process of migration starts with creating controllers and adding them to 
the Kafka cluster at which point they wait for the brokers to register in a 
state ready for migrations. Brokers are then updated in a rolling manner to 
enable migrations. When all brokers have enabled migration, then migration 
starts.

After the metadata is migrated, the controller tries to connect back to the 
brokers to signal that metadata migration is complete. If the controller cannot 
create this connection (because of authentication issues) it will continuously 
retry (no timeout). All metadata operations initiated by the brokers that have 
not been contacted will fail. Rollback from this state back to ZK with no 
controllers though is a manual step.

I want the controllers to create the connection back to the brokers after all 
brokers have registered but before metadata migration is started. This will 
prevent metadata operations from failing if the controller cannot connect to 
any broker and allow for simple rollback of the cluster by disabling migration 
in the controllers and restarting them in a rolling manner.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14859) Support SCRAM ZK to KRaft Migration

[Proven Provenzano (Jira)]

Proven Provenzano created KAFKA-14859:
-

 Summary: Support SCRAM ZK to KRaft Migration
 Key: KAFKA-14859
 URL: https://issues.apache.org/jira/browse/KAFKA-14859
 Project: Kafka
  Issue Type: Improvement
Reporter: Proven Provenzano
Assignee: Proven Provenzano


I want to allow existing ZK installations to be able to migrate to KRaft and 
support their existing SCRAM credentials.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-14842) MirrorCheckpointTask can reduce the rpc calls of "listConsumerGroupOffsets(group)" of irrelevant groups at each poll

[Chris Egerton (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Egerton resolved KAFKA-14842.
---
Fix Version/s: 3.5.0
   Resolution: Done

> MirrorCheckpointTask can reduce the rpc calls of 
> "listConsumerGroupOffsets(group)" of irrelevant groups at each poll
> 
>
> Key: KAFKA-14842
> URL: https://issues.apache.org/jira/browse/KAFKA-14842
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect, mirrormaker
>Affects Versions: 3.3.2
>Reporter: hudeqi
>Assignee: hudeqi
>Priority: Major
> Fix For: 3.5.0
>
>
> sorry, wrong related.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14858) Standalone herder does not handle exceptions thrown from connector taskConfigs method

[Chris Egerton (Jira)]

Chris Egerton created KAFKA-14858:
-

 Summary: Standalone herder does not handle exceptions thrown from 
connector taskConfigs method
 Key: KAFKA-14858
 URL: https://issues.apache.org/jira/browse/KAFKA-14858
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Reporter: Chris Egerton


In distributed mode, if a connector throws an exception from its 
{{taskConfigs}} method (invoked by the herder, through the {{Worker}} class, 
[here|https://github.com/apache/kafka/blob/f3e4dd922933bf28b2c091e846cbc4e5255dd1d5/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1960]),
 we wait for an exponential backoff period (see KAFKA-14732) and then [retry 
the 
operation|https://github.com/apache/kafka/blob/f3e4dd922933bf28b2c091e846cbc4e5255dd1d5/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1907-L1911].

However, in standalone mode, not only do we not retry the operation, we do not 
even log the exception. In addition, when REST calls are made that require 
generating new task configs for a connector (which include creating and 
reconfiguring a connector), if the connector's {{taskConfigs}} method throws an 
exception, those requests will time out since the 
[callback|https://github.com/apache/kafka/blob/f3e4dd922933bf28b2c091e846cbc4e5255dd1d5/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/standalone/StandaloneHerder.java#L183]
 we use to respond to those requests never gets invoked.

At a bare minimum, we should:
 * Log any exceptions thrown from the {{taskConfigs}} method at {{ERROR}} level
 * Invoke any callbacks passed in to the relevant {{StandaloneHerder}} methods 
with any exceptions thrown by the {{taskConfigs}} method

We might also consider introducing the same kind of exponential backoff retry 
logic used by distributed mode, but this can be addressed separately since it 
would be a much larger change in behavior and may break existing user's 
deployments.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Kafka Cluster WITHOUT Zookeeper

[Kafka Life]

This is really great information Paul . Thank you .

On Tue, Mar 28, 2023 at 4:01 AM Brebner, Paul
 wrote:

> I have a recent 3 part blog series on Kraft (expanded version of ApacheCon
> 2022 talk):
>
>
>
>
> https://www.instaclustr.com/blog/apache-kafka-kraft-abandons-the-zookeeper-part-1-partitions-and-data-performance/
>
>
> https://www.instaclustr.com/blog/apache-kafka-kraft-abandons-the-zookeeper-part-2-partitions-and-meta-data-performance/
>
>
> https://www.instaclustr.com/blog/apache-kafka-kraft-abandons-the-zookeeper-part-3-maximum-partitions-and-conclusions/
>
>
>
> Regards, Paul
>
>
>
> *From: *Chia-Ping Tsai 
> *Date: *Monday, 27 March 2023 at 5:37 pm
> *To: *dev@kafka.apache.org 
> *Cc: *us...@kafka.apache.org ,
> mmcfarl...@cavulus.com , Israel Ekpo <
> israele...@gmail.com>, ranlupov...@gmail.com ,
> scante...@gmail.com , show...@gmail.com <
> show...@gmail.com>, sunilmchaudhar...@gmail.com <
> sunilmchaudhar...@gmail.com>
> *Subject: *Re: Kafka Cluster WITHOUT Zookeeper
>
> *NetApp Security WARNING*: This is an external email. Do not click links
> or open attachments unless you recognize the sender and know the content is
> safe.
>
>
>
> hi
>
>
>
> You can use the keyword “kraft” to get the answer by google or chatgpt.
> For example:
>
>
>
> Introduction:
>
> KRaft - Apache Kafka Without ZooKeeper
> 
>
> developer.confluent.io 
>
>
>
>
>
> QuickStart:
>
> Apache Kafka 
>
> kafka.apache.org 
>
>
>
>
>
> —
>
> Chia-Ping
>
>
>
>
>
>
>
> Kafka Life  於 2023年3月27日 下午1:33 寫道：
>
> Hello  Kafka experts
>
> Is there a way where we can have Kafka Cluster be functional serving
> producers and consumers without having Zookeeper cluster manage the
> instance .
>
> Any particular version of kafka for this or how can we achieve this please
>
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1712

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 274092 lines...]
[2023-03-28T11:16:51.401Z] > Task :storage:api:compileTestJava UP-TO-DATE
[2023-03-28T11:16:51.401Z] > Task :storage:api:testClasses UP-TO-DATE
[2023-03-28T11:16:51.401Z] > Task :server-common:compileTestJava UP-TO-DATE
[2023-03-28T11:16:51.401Z] > Task :server-common:testClasses UP-TO-DATE
[2023-03-28T11:16:51.401Z] > Task :connect:json:compileTestJava UP-TO-DATE
[2023-03-28T11:16:51.401Z] > Task :connect:json:testClasses UP-TO-DATE
[2023-03-28T11:16:51.401Z] > Task :raft:compileTestJava UP-TO-DATE
[2023-03-28T11:16:51.401Z] > Task :raft:testClasses UP-TO-DATE
[2023-03-28T11:16:51.401Z] > Task :group-coordinator:compileTestJava UP-TO-DATE
[2023-03-28T11:16:51.401Z] > Task :group-coordinator:testClasses UP-TO-DATE
[2023-03-28T11:16:51.401Z] > Task :connect:json:testJar
[2023-03-28T11:16:51.401Z] > Task :connect:json:testSrcJar
[2023-03-28T11:16:51.401Z] > Task 
:streams:generateMetadataFileForMavenJavaPublication
[2023-03-28T11:16:53.620Z] > Task :metadata:compileTestJava UP-TO-DATE
[2023-03-28T11:16:53.620Z] > Task :metadata:testClasses UP-TO-DATE
[2023-03-28T11:16:53.620Z] > Task 
:clients:generateMetadataFileForMavenJavaPublication
[2023-03-28T11:16:55.102Z] 
[2023-03-28T11:16:55.102Z] > Task :connect:api:javadoc
[2023-03-28T11:16:55.102Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk_2/connect/api/src/main/java/org/apache/kafka/connect/source/SourceRecord.java:44:
 warning - Tag @link: reference not found: org.apache.kafka.connect.data
[2023-03-28T11:16:58.412Z] 1 warning
[2023-03-28T11:16:58.412Z] 
[2023-03-28T11:16:58.412Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2023-03-28T11:16:58.412Z] > Task :connect:api:jar UP-TO-DATE
[2023-03-28T11:16:58.412Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2023-03-28T11:16:58.412Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2023-03-28T11:16:58.412Z] > Task :connect:json:jar UP-TO-DATE
[2023-03-28T11:16:58.412Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2023-03-28T11:16:58.412Z] > Task :connect:api:javadocJar
[2023-03-28T11:16:58.412Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2023-03-28T11:16:58.412Z] > Task :connect:json:publishToMavenLocal
[2023-03-28T11:16:58.412Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2023-03-28T11:16:58.412Z] > Task :connect:api:testClasses UP-TO-DATE
[2023-03-28T11:16:58.412Z] > Task :connect:api:testJar
[2023-03-28T11:16:58.412Z] > Task :connect:api:testSrcJar
[2023-03-28T11:16:58.412Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2023-03-28T11:16:58.412Z] > Task :connect:api:publishToMavenLocal
[2023-03-28T11:17:01.464Z] > Task :streams:javadoc
[2023-03-28T11:17:01.464Z] > Task :streams:javadocJar
[2023-03-28T11:17:04.193Z] 
[2023-03-28T11:17:04.193Z] > Task :clients:javadoc
[2023-03-28T11:17:04.193Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk_2/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/package-info.java:21:
 warning - Tag @link: reference not found: 
org.apache.kafka.common.security.oauthbearer
[2023-03-28T11:17:06.663Z] 1 warning
[2023-03-28T11:17:06.663Z] 
[2023-03-28T11:17:06.663Z] > Task :clients:javadocJar
[2023-03-28T11:17:08.462Z] > Task :clients:srcJar
[2023-03-28T11:17:08.462Z] > Task :clients:testJar
[2023-03-28T11:17:08.462Z] > Task :clients:testSrcJar
[2023-03-28T11:17:08.462Z] > Task 
:clients:publishMavenJavaPublicationToMavenLocal
[2023-03-28T11:17:08.462Z] > Task :clients:publishToMavenLocal
[2023-03-28T11:17:21.109Z] > Task :core:compileScala
[2023-03-28T11:19:15.997Z] > Task :core:classes
[2023-03-28T11:19:15.997Z] > Task :core:compileTestJava NO-SOURCE
[2023-03-28T11:19:36.652Z] > Task :core:compileTestScala
[2023-03-28T11:21:06.012Z] > Task :core:testClasses
[2023-03-28T11:21:07.358Z] > Task :streams:compileTestJava UP-TO-DATE
[2023-03-28T11:21:07.358Z] > Task :streams:testClasses UP-TO-DATE
[2023-03-28T11:21:07.358Z] > Task :streams:testJar
[2023-03-28T11:21:09.974Z] > Task :streams:testSrcJar
[2023-03-28T11:21:09.974Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2023-03-28T11:21:09.974Z] > Task :streams:publishToMavenLocal
[2023-03-28T11:21:09.974Z] 
[2023-03-28T11:21:09.974Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 9.0.
[2023-03-28T11:21:09.974Z] 
[2023-03-28T11:21:09.974Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2023-03-28T11:21:09.974Z] 
[2023-03-28T11:21:09.974Z] See 
https://docs.gradle.org/8.0.2/userguide/command_line_interface.html#sec:command_line_warnings
[2023-03-28T11:21:09.974Z] 
[2023-03-28T11:21:09.974Z] BUILD SUCCESSFUL in 4m 44s
[2023-03-28T11:21:09.974Z] 86 actionable tasks: 33 executed, 53 up-to-date

Re: [DISCUSS] KIP-915: Next Gen Group Coordinator Downgrade Path

[Alexandre Dupriez]

Hi Jeff, Team,

Thank you for the KIP. This is a very interesting approach. I feel it
is simpler than the described alternative although it comes with
tradeoffs, thanks for highlighting those. If I may, I would like to
share two naive questions.

100. The KIP mentions that records will be serialised with the highest
non-flexible version (e.g. 3 for GroupMetadataValue and
OffsetCommitValue) so that records can be deserialized with earlier
versions of Kafka. I am not sure I understand correctly: is the idea
to forcefully add tagged fields at the end of the records while
maintaining the existing version (3 for the two record types just
mentioned) so that they can be deserialized by existing Kafka versions
for which the version of these record types is not known as flexible,
while at the same time preserving the new tagged fields to new Kafka
versions abreast of the addition of a new flexible version for these
record types? If so, is it "bypassing" the protocol convention which
prescribes the use of a flexible version to allow the use of tagged
fields?

101. After the bump of the records to a new version indicated as
flexible, the record version is expected to never change while the
underlying tagged fields could potentially still evolve over time. One
potential downside is that we lose the benefits of the versioning
scheme enforced by the serde protocol. Could this become a problem in
the future if there is ever a need to segregate two distinct
"versions" of the appended record structure held by the tagged fields?

Thanks,
Alexandre

Le jeu. 23 mars 2023 à 18:15, Jeff Kim  a écrit :
>
> Hi Yi,
>
> > Does it mean with a flexible version, the future
> version of these value types will stay at the version where the flexibility
> is first introduced? Will there be any need to bump the version again in
> the future?
>
> Yes, there will be no need to bump the version since we will only be adding
> tagged fields but in the chance the version is bumped, we will deserialize
> to the highest known (flexible) version which will ignore unknown tagged
> fields.
>
> > To enforce the version not bumping, is it possible to have a unit test to
> gate?
>
> Do you have some tests in mind? I don't think we can tell whether a version
> was bumped or not.
>
> Best,
> Jeff
>
> On Thu, Mar 23, 2023 at 12:07 PM Yi Ding  wrote:
>
> > Hi Jeff,
> >
> > Thanks for the update. Does it mean with a flexible version, the future
> > version of these value types will stay at the version where the flexibility
> > is first introduced? Will there be any need to bump the version again in
> > the future?
> > To enforce the version not bumping, is it possible to have a unit test to
> > gate?
> >
> >
> > On Wed, Mar 22, 2023 at 3:19 PM Jeff Kim 
> > wrote:
> >
> > > Hi all,
> > >
> > > After discussing with my colleagues, I have repurposed the KIP for a
> > > general downgrade solution for both transaction and group coordinators.
> > The
> > > KIP no longer discusses the downgrade path but instead lays out the
> > > foundation for future downgrade solutions.
> > >
> > > Link:
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-915%3A+Txn+and+Group+Coordinator+Downgrade+Foundation
> > >
> > > Thanks,
> > > Jeff
> > >
> > > On Mon, Mar 20, 2023 at 7:37 PM Jeff Kim  wrote:
> > >
> > > > Hi David and Justine,
> > > >
> > > > Thank you both for the detailed feedback.
> > > >
> > > > David,
> > > >
> > > > 1. That makes sense. I revised the "Reading new fields" section with
> > how
> > > > we can downgrade to the highest known version and that this was
> > confirmed
> > > > via unit testing. I also attempted to dive deeper into using tagged
> > > fields
> > > > and the rejected alternative. Please let me know what you think.
> > > >
> > > > 2. Under "Restrictions and Guidelines" I updated the paragraph to
> > clearly
> > > > state that we want to use tagged fields across all record types
> > > introduced
> > > > in KIP-848 including OffsetCommitValue.
> > > >
> > > > 3. Would it be possible to bump the OffsetCommitValue record version to
> > > > make it flexible along with the changes to parse with the highest known
> > > > version? I'm not sure I understand why we cannot make both changes
> > > together.
> > > >
> > > > 4. I completely missed this. Added some notes at the end of
> > "Restrictions
> > > > and Guidelines". Unfortunately I can't think of a solution at the
> > moment.
> > > > Will get back to you.
> > > >
> > > > 5. I have a section under "Identifying New Record Types" that discusses
> > > > this:
> > > >  > We can automate the cleanup by writing tombstones when the
> > coordinator
> > > > reads unrecognizable records. This may add duplicate work if tombstones
> > > > were already added but not yet pruned by the log cleaner.
> > > > This is a sure way to delete any unknown record types even if the
> > > operator
> > > > does not follow the steps.
> > > >
> > > > 6. Thanks, I have expanded on the section on transactional offs

Re: [DISCUSS] KIP-881: Rack-aware Partition Assignment for Kafka Consumers

[Rajini Sivaram]

Hi all,

With KIP-881, consumer partition assignors attempt to improve locality by
matching consumer racks with partition replica racks. At the moment, we
don't rebalance when replicas change, so changes in replica racks are not
taken into account for partition assignment until the next rebalance, which
could be a very long time. To overcome this limitation, we would like to
also trigger rebalance when metadata indicates that the racks on which a
partition has replicas has changed. The rebalance will be triggered by the
leader if `client.rack` is configured and the set of racks of any of the
partitions changes. Rebalances are not triggered for transient changes like
a replica being in ISR or being temporarily offline. But for less frequent,
but long-lived changes due to reassignment, rebalance will be triggered to
improve locality when rack-awareness is enabled for consumers.

Please let me know if there are any concerns with this change. Otherwise I
will update the KIP and submit a PR.

Thank you,

Rajini


On Thu, Dec 1, 2022 at 9:12 AM Rajini Sivaram 
wrote:

> Hi Artem,
>
> Yes, clients can reset their `client.rack` config if the broker doesn't
> use a rack-aware selector. This config is only useful for improving
> locality with a rack-aware selector.
>
> On Wed, Nov 30, 2022 at 10:05 PM Artem Livshits
>  wrote:
>
>> I think it's reasonable for practical scenarios.  Is it possible to turn
>> off rack awareness in the clients in case the broker selector plugin is
>> not
>> compatible with rack-aware logic in the client?
>>
>> -Artem
>>
>> On Wed, Nov 30, 2022 at 2:46 AM Rajini Sivaram 
>> wrote:
>>
>> > Hi Artem,
>> >
>> > Understood your concern - brokers could use a replica selector that uses
>> > some other client metadata other than rack id to decide the preferred
>> read
>> > replica, or just use the leader. In that case, consumers wouldn't really
>> > benefit from aligning partition assignment on rack ids. So the question
>> is
>> > whether we should make the default consumer assignors use rack ids for
>> > partition assignment or whether we should have different rack-aware
>> > assignors that can be configured when brokers use rack-aware replica
>> > selector. We had a similar discussion earlier in the thread (the KIP had
>> > originally proposed new rack-aware assignors). We decided to update the
>> > default assignors because:
>> > 1) Consumers using fetch-from-follower automatically benefit from
>> improved
>> > locality, without having to update another config.
>> > 2) Consumers configure rack id for improved locality, so aligning on
>> > replica rack ids generally makes sense.
>> > 3) We prioritize balanced assignment over locality in the consumer, so
>> the
>> > default assignors should work effectively regardless of broker's replica
>> > selector.
>> >
>> > Does that make sense?
>> >
>> >
>> > Thank you,
>> >
>> > Rajini
>> >
>> >
>> >
>> > On Tue, Nov 29, 2022 at 1:05 AM Artem Livshits
>> >  wrote:
>> >
>> > > I'm thinking of a case where the broker uses a plugin that does not
>> use
>> > > rack ids to determine client locality, in that case the consumer might
>> > > arrive at the wrong decision based on rack ids.
>> > >
>> > > -Artem
>> > >
>> > > On Wed, Nov 23, 2022 at 3:54 AM Rajini Sivaram <
>> rajinisiva...@gmail.com>
>> > > wrote:
>> > >
>> > > > Hi Artem,
>> > > >
>> > > > Thanks for reviewing the KIP. The client doesn't rely on a
>> particular
>> > > > replica assignment logic in the broker. Instead, it matches the
>> actual
>> > > rack
>> > > > assignment for partitions from the metadata with consumer racks. So
>> > there
>> > > > is no assumption in the client implementation about the use of
>> > > > RackAwareReplicaSelector in the broker. Did I misunderstand your
>> > concern?
>> > > >
>> > > > Regards,
>> > > >
>> > > > Rajini
>> > > >
>> > > >
>> > > > On Tue, Nov 22, 2022 at 11:03 PM Artem Livshits
>> > > >  wrote:
>> > > >
>> > > > > Hi Rajini,
>> > > > >
>> > > > > Thank you for the KIP, the KIP looks good to match
>> > > > RackAwareReplicaSelector
>> > > > > behavior that is available out-of-box.  Which should probably be
>> good
>> > > > > enough in practice.
>> > > > >
>> > > > > From the design perspective, though, RackAwareReplicaSelector is
>> just
>> > > one
>> > > > > possible plugin, in theory the broker could use a plugin that
>> > leverages
>> > > > > networking information to get client locality or some other way,
>> so
>> > it
>> > > > > seems like we're making an assumption about broker replica
>> selection
>> > in
>> > > > the
>> > > > > default assignment implementation.  So I wonder if we should use a
>> > > > separate
>> > > > > plugin that would be set when RackAwareReplicaSelector is set,
>> rather
>> > > > than
>> > > > > assume broker behavior in the client implementation.
>> > > > >
>> > > > > -Artem
>> > > > >
>> > > > > On Wed, Nov 16, 2022 at 8:08 AM Jun Rao > >
>> > > > wrote:
>> > > > >
>> > > > > > Hi, David and Rajini,
>> > > > > >
>> > > > >