Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2678

[Apache Jenkins Server]

See 

Re: DISCUSS KIP-984 Add pluggable compression interface to Kafka

[Luke Chen]

Hi Assane,

I also share the same concern as Greg has, which is that the KIP is not
kafka ecosystem friendly.
And this will make the kafka client and broker have high dependencies that
once you use the pluggable compression interface, the producer must be java
client.
This seems to go against the original Kafka's design.

If the proposal can support all kinds of clients, that would be great.

Thanks.
Luke

On Tue, Feb 27, 2024 at 7:44 AM Diop, Assane  wrote:

> Hi Greg,
>
> Thanks for taking the time to give some feedback. It was very insightful.
>
> I have some answers:
>
> 1. The current proposal is Java centric. We want to figure out with Java
> first and then later incorporate other languages. We will get there.
>
> 2. The question of where the plugins would live is an important one. I
> would like to get the community engagement on where a plugin would live.
>Officially supported plugins could be part of Kafka and others could
> live in a plugin repository. Is there currently a way to store plugins in
> Kafka and load them into the classpath? If such a space could be allowed
> then it would provide an standard way of installing officially supported
> plugins.
>In OpenSearch for example, there is a plugin utility that takes the jar
> and installs it across the cluster, privileges can be granted by an admin.
> Such utility could be implemented in Kafka.
>
> 3. There is many way to look at this, we could change the message format
> that use the pluggable interface to be for example v3 and synchronize
> against that.
>In order to use the pluggable codec, you will have to be at message
> version 3 for example.
>
> 4. Passing the class name as metadata is one way to have the producer talk
> to the broker about which plugin to use. However there could be other
> implementation
>where you could set every thing to know about the topic using topic
> level compression. In this case for example a rule could be that in order
> to use the
>pluggable interface, you should use topic level compression.
>
>  I would like to have your valuable inputs on this!!
>
> Thanks before end,
> Assane
>
> -Original Message-
> From: Greg Harris 
> Sent: Wednesday, February 14, 2024 2:36 PM
> To: dev@kafka.apache.org
> Subject: Re: DISCUSS KIP-984 Add pluggable compression interface to Kafka
>
> Hi Assane,
>
> Thanks for the KIP!
> Looking back, it appears that the project has only ever added compression
> types twice: lz4 in 2014 and zstd in 2018, and perhaps Kafka has fallen
> behind the state-of-the-art compression algorithms.
> Thanks for working to fix that!
>
> I do have some concerns:
>
> 1. I think this is a very "java centric" proposal, and doesn't take
> non-java clients into enough consideration. librdkafka [1] is a great
> example of an implementation of the Kafka protocol which doesn't have the
> same classloading and plugin infrastructure that Java has, which would make
> implementing this feature much more difficult.
>
> 2. By making the interface pluggable, it puts the burden of maintaining
> individual compression codecs onto external developers, which may not be
> willing to maintain a codec for the service-lifetime of such a codec.
> An individual developer can easily implement a plugin to allow them to use
> a cutting-edge compression algorithm without consulting the Kafka project,
> but as soon as data is compressed using that algorithm, they are on the
> hook to support that plugin going forward by the organizations using their
> implementation.
> Part of the collective benefits of the Kafka project is to ensure the
> ongoing maintenance of such codecs, and provide a long deprecation window
> when a codec reaches EOL. I think the Kafka project is well-equipped to
> evaluate the maturity and properties of compression codecs and then
> maintain them going forward.
>
> 3. Also by making the interface pluggable, it reduces the scope of
> individual compression codecs. No longer is there a single lineage of Kafka
> protocols, where vN+1 of a protocol supports a codec that vN does not. Now
> there will be "flavors" of the protocol, and operators will need to ensure
> that their servers and their clients support the same "flavors" or else
> encounter errors.
> This is the sort of protocol forking which can be dangerous to the Kafka
> community going forward. If there is a single lineage of codecs such that
> the upstream Kafka vX.Y supports codec Z, it is much simpler for other
> implementations to check and specify "Kafka vX.Y compatible", than it is to
> check & specify "Kafka vX.Y & Z compatible".
>
> 4. The Java class namespace is distributed, as anyone can name their class
> anything. It achieves this by being very verbose, with long fully-qualified
> names for classes. This is in conflict with a binary protocol, where it is
> desirable for the overhead to be as small as possible.
> This may incentivise developers to keep their class names short, which
> also makes conflict more 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2677

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-853: KRaft Controller Membership Changes

[Jun Rao]

Hi, Jose,

Thanks for the reply.

15.1 It would be useful to document the process of changing the controller
listener. For example, what configs need to be changed in what order and
how a voter/observer selects the listener when multiple listeners are
returned.

15.2 Thinking a bit more. The primary usage of multiple listeners is for
changing the listeners online. So, we may not need to support multiple
listeners in the initial setup phase.

30. Well, VotersRecords and LeaderChangeRecord depend on RPCs like
AddVoter, UpdateVoter and Vote, etc. Are those RPCs controlled by
metadata.version? What happens if metadata.version doesn't enable those
RPCs, but kraft.version requires them.

Jun


On Mon, Feb 26, 2024 at 8:39 AM José Armando García Sancio
 wrote:

> Hi Jun, see my comments below.
>
> I need to make some changes to the KIP to reflect our conversation.
>
> On Fri, Feb 23, 2024 at 1:39 PM Jun Rao  wrote:
> > 15.1 "In this configuration, the local replica needs to use
> > CONTROLLER_PLAINTEXT because that is what is supported by all of the
> > voters."
> >Hmm, my understanding is that what listener to use can be made
> pairwise
> > between the sender and the receiver. There is no need to pick the same
> > listener across all voters, right?
>
> Yes, this is true. Thinking about it some more, there are a couple of
> issues.
>
> In the current implementation, the Selector we use for the
> NetworkClient only supports one ChannelBuilder and that channel
> builder can only use one security protocol. Until we fix that in the
> implementation the preferred (first) listener needs to be a listener
> that is supported by all of the voters. I'll make sure to document
> that in the Kafka documentation until we fix the implementation.
>
> The other issue is that when the leader sends its endpoint in the
> BeginQuorumEpoch it needs to send all of its endpoints. The leader
> needs to do that because replicas may connect to each other using any
> of the configured listeners. For example, let's assume that the voters
> (controllers) have been upgraded to support the listeners
> CONTROLLER_SSL and CONTROLLER_PLAINTEXT with CONTROLLER_SSL being the
> preferred listener. It is possible that the observers (brokers) have
> yet to be reconfigured to use CONTROLLER_SSL and are instead using
> CONTROLLER_PLAINTEXT. That means that the voters need to know all of
> the leader's listeners because observers discover the leader by
> sending a Fetch request to a random endpoint in
> controller.quorum.bootstrap.servers using the security protocol of the
> first controller listener. I'll make that change to the KIP.
>
> > 15.2  When bootstrapping with just one controller, we could register a
> > single voter with multiple listeners from controller.listener.names and
> > listeners. However, when bootstrapping with multiple controllers,
> > kafka-storage only has the option to specify one listener
> > (--controller-quorum-voters -@:)
> per
> > voter. Should we make them consistent?
>
> This is true. I was trying to avoid this but this configuration can
> get quite complicated. The user will have a better experience if they
> can express it as a JSON file. The configuration should just be a
> VotersRecord but in JSON. Let me update the KIP.
>
> > 18.3 : "It will not use the metadata layer (QuorumController) to update
> and
> > persist the finalized kraft.version."
> > So, we depend on KRaftVersionRecord to propagate finalized kraft.version
> to
> > brokers/controllers? It would be useful to document that.
>
> Yes. I'll make that clear in the KIP.
>
> > 27. "My plan is to rely on KIP-996: Pre-vote:".
> > Hmm, Diego's paper says "Unfortunately, the Pre-Vote phase does not solve
> > the problem of disruptive servers". Is that a real issue?
>
> An active leader gets disrupted if a replica increases their epoch
> past the active leader's epoch.
>
> Diego's Raft allows the replica to increase their epoch and become a
> candidate once it wins a pre-vote. Diego defines pre-vote as follows:
> "A candidate would first ask other servers whether its log was
> up-to-date enough to get their vote. Only if the candidate believed it
> could get votes from a majority of the cluster would it increment its
> term and start a normal election.
>
> Unfortunately, the Pre-Vote phase does not solve the problem of
> disruptive servers: there are situations where the disruptive server’s
> log is sufficiently up-to-date, but starting an election would still
> be disruptive."
>
> In other words, Diego's Raft only compares log lengths during the
> pre-vote phase. That is why Diego then mentions:
> "We modify the RequestVote RPC to achieve this: if a server receives a
> RequestVote request within the minimum election timeout of hearing
> from a current leader, it does not update its term or grant its vote."
>
> Diego can do this for his definition of Raft because he is only
> concerned with voters. There is no support for observers. Diego's Raft
> nodes are either 

[jira] [Resolved] (KAFKA-16277) CooperativeStickyAssignor does not spread topics evenly among consumer group

[A. Sophie Blee-Goldman (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

A. Sophie Blee-Goldman resolved KAFKA-16277.

Resolution: Fixed

> CooperativeStickyAssignor does not spread topics evenly among consumer group
> 
>
> Key: KAFKA-16277
> URL: https://issues.apache.org/jira/browse/KAFKA-16277
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, consumer
>Reporter: Cameron Redpath
>Priority: Major
> Fix For: 3.8.0
>
> Attachments: image-2024-02-19-13-00-28-306.png
>
>
> Consider the following scenario:
> `topic-1`: 12 partitions
> `topic-2`: 12 partitions
>  
> Of note, `topic-1` gets approximately 10 times more messages through it than 
> `topic-2`. 
>  
> Both of these topics are consumed by a single application, single consumer 
> group, which scales under load. Each member of the consumer group subscribes 
> to both topics. The `partition.assignment.strategy` being used is 
> `org.apache.kafka.clients.consumer.CooperativeStickyAssignor`. The 
> application may start with one consumer. It consumes all partitions from both 
> topics.
>  
> The problem begins when the application scales up to two consumers. What is 
> seen is that all partitions from `topic-1` go to one consumer, and all 
> partitions from `topic-2` go to the other consumer. In the case with one 
> topic receiving more messages than the other, this results in a very 
> imbalanced group where one consumer is receiving 10x the traffic of the other 
> due to partition assignment.
>  
> This is the issue being seen in our cluster at the moment. See this graph of 
> the number of messages being processed by each consumer as the group scales 
> from one to four consumers:
> !image-2024-02-19-13-00-28-306.png|width=537,height=612!
> Things to note from this graphic:
>  * With two consumers, the partitions for a topic all go to a single consumer 
> each
>  * With three consumers, the partitions for a topic are split between two 
> consumers each
>  * With four consumers, the partitions for a topic are split between three 
> consumers each
>  * The total number of messages being processed by each consumer in the group 
> is very imbalanced throughout the entire period
>  
> With regard to the number of _partitions_ being assigned to each consumer, 
> the group is balanced. However, the assignment appears to be biased so that 
> partitions from the same topic go to the same consumer. In our scenario, this 
> leads to very undesirable partition assignment.
>  
> I question if the behaviour of the assignor should be revised, so that each 
> topic has its partitions maximally spread across all available members of the 
> consumer group. In the above scenario, this would result in much more even 
> distribution of load. The behaviour would then be:
>  * With two consumers, 6 partitions from each topic go to each consumer
>  * With three consumers, 4 partitions from each topic go to each consumer
>  * With four consumers, 3 partitions from each topic go to each consumer
>  
> Of note, we only saw this behaviour after migrating to the 
> `CooperativeStickyAssignor`. It was not an issue with the default partition 
> assignment strategy.
>  
> It is possible this may be intended behaviour. In which case, what is the 
> preferred workaround for our scenario? Our current workaround if we decide to 
> go ahead with the update to `CooperativeStickyAssignor` may be to limit our 
> consumers so they only subscribe to one topic, and have two consumer threads 
> per instance of the application.  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] KIP-1022 Formatting and Updating Features

[Justine Olshan]

Hello folks,

I'm proposing a KIP that allows for setting and upgrading new features
(other than metadata version) via the kafka storage format and feature
tools. This KIP extends on the feature versioning changes introduced by
KIP-584 by allowing for the features to be set and upgraded.

Please take a look:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1023%3A+Formatting+and+Updating+Features

Thanks,

Justine

[jira] [Created] (KAFKA-16308) Formatting and Updating Kafka Features

[Justine Olshan (Jira)]

Justine Olshan created KAFKA-16308:
--

 Summary: Formatting and Updating Kafka Features
 Key: KAFKA-16308
 URL: https://issues.apache.org/jira/browse/KAFKA-16308
 Project: Kafka
  Issue Type: Task
Reporter: Justine Olshan
Assignee: Justine Olshan


As part of KIP-1022, we need to extend the storage and upgrade tools to support 
features other than metadata version. 

See 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1023%3A+Formatting+and+Updating+Features



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-956: Tiered Storage Quotas

[Jun Rao]

Hi, Abhijeet,

Sorry for the late reply. It seems that you haven't updated the KIP based
on the discussion? One more comment.

11. Currently, we already have a quota system for both the producers and
consumers. I can understand why we need an additional
remote.log.manager.write.quota.default quota. For example, when tier
storage is enabled for the first time, there could be a lot of segments
that need to be written to the remote storage, even though there is no
increase in the produced data. However, I am not sure about an
additional remote.log.manager.read.quota.default. The KIP says that the
reason is "This may happen when the majority of the consumers start reading
from the earliest offset of their respective Kafka topics.". However, this
can happen with or without tier storage and the current quota system for
consumers is designed for solving this exact problem. Could you explain the
usage of this additional quota?

Thanks,

Jun

On Mon, Feb 12, 2024 at 11:08 AM Abhijeet Kumar 
wrote:

> Comments inline
>
> On Wed, Dec 6, 2023 at 1:12 AM Jun Rao  wrote:
>
> > Hi, Abhijeet,
> >
> > Thanks for the KIP. A few comments.
> >
> > 10. remote.log.manager.write.quota.default:
> > 10.1 For other configs, we
> > use replica.alter.log.dirs.io.max.bytes.per.second. To be consistent,
> > perhaps this can be sth like
> remote.log.manager.write.max.bytes.per.second.
> >
>
> This makes sense, we can rename the following configs to be consistent.
>
> Remote.log.manager.write.quota.default ->
> remote.log.manager.write.max.bytes.per.second
>
> Remote.log.manager.read.quota.default ->
> remote.log.manager.read.max.bytes.per.second.
>
>
>
> > 10.2 Could we list the new metrics associated with the new quota.
> >
>
> We will add the following metrics as mentioned in the other response.
> *RemoteFetchThrottleTime* - The amount of time needed to bring the observed
> remote fetch rate within the read quota
> *RemoteCopyThrottleTime *- The amount of time needed to bring the observed
> remote copy rate with the copy quota.
>
> 10.3 Is this dynamically configurable? If so, could we document the impact
> > to tools like kafka-configs.sh and AdminClient?
> >
>
> Yes, the quotas are dynamically configurable. We will add them as Dynamic
> Broker Configs. Users will be able to change
> the following configs using either kafka-configs.sh or AdminClient by
> specifying the config name and new value. For eg.
>
> Using kafka-configs.sh
>
> bin/kafka-configs.sh --bootstrap-server  --entity-type
> brokers --entity-default --alter --add-config
> remote.log.manager.write.max.bytes.per.second=52428800
>
> Using AdminClient
>
> ConfigEntry configEntry = new
> ConfigEntry("remote.log.manager.write.max.bytes.per.second", "5242800");
> AlterConfigOp alterConfigOp = new AlterConfigOp(configEntry,
> AlterConfigOp.OpType.SET);
> List alterConfigOps =
> Collections.singletonList(alterConfigOp);
>
> ConfigResource resource = new ConfigResource(ConfigResource.Type.BROKER,
> "");
> Map> updateConfig =
> ImmutableMap.of(resource, alterConfigOps);
> adminClient.incrementalAlterConfigs(updateConfig);
>
>
> >
> > Jun
> >
> > On Tue, Nov 28, 2023 at 2:19 AM Luke Chen  wrote:
> >
> > > Hi Abhijeet,
> > >
> > > Thanks for the KIP!
> > > This is an important feature for tiered storage.
> > >
> > > Some comments:
> > > 1. Will we introduce new metrics for this tiered storage quotas?
> > > This is important because the admin can know the throttling status by
> > > checking the metrics while the remote write/read are slow, like the
> rate
> > of
> > > uploading/reading byte rate, the throttled time for upload/read... etc.
> > >
> > > 2. Could you give some examples for the throttling algorithm in the KIP
> > to
> > > explain it? That will make it much clearer.
> > >
> > > 3. To solve this problem, we can break down the RLMTask into two
> smaller
> > > tasks - one for segment upload and the other for handling expired
> > segments.
> > > How do we handle the situation when a segment is still waiting for
> > > offloading while this segment is expired and eligible to be deleted?
> > > Maybe it'll be easier to not block the RLMTask when quota exceeded, and
> > > just check it each time the RLMTask runs?
> > >
> > > Thank you.
> > > Luke
> > >
> > > On Wed, Nov 22, 2023 at 6:27 PM Abhijeet Kumar <
> > abhijeet.cse@gmail.com
> > > >
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I have created KIP-956 for defining read and write quota for tiered
> > > > storage.
> > > >
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-956+Tiered+Storage+Quotas
> > > >
> > > > Feedback and suggestions are welcome.
> > > >
> > > > Regards,
> > > > Abhijeet.
> > > >
> > >
> >
>
>
> --
> Abhijeet.
>

Re: [DISCUSS] Apache Kafka 3.8.0 release

[Chris Egerton]

Thanks Josep, I'm +1 as well.

On Mon, Feb 26, 2024 at 12:32 PM Justine Olshan
 wrote:

> Thanks Joesp. +1 from me.
>
> On Mon, Feb 26, 2024 at 3:37 AM Josep Prat 
> wrote:
>
> > Hi all,
> >
> > I'd like to volunteer as release manager for the Apache Kafka 3.8.0
> > release.
> > If there are no objections, I'll start building a release plan (or
> adapting
> > the one Colin made some weeks ago) in the wiki in the next days.
> >
> > Thank you.
> >
> > --
> > [image: Aiven] 
> >
> > *Josep Prat*
> > Open Source Engineering Director, *Aiven*
> > josep.p...@aiven.io   |   +491715557497
> > aiven.io    |   <
> https://www.facebook.com/aivencloud
> > >
> >      <
> > https://twitter.com/aiven_io>
> > *Aiven Deutschland GmbH*
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
>

RE: DISCUSS KIP-984 Add pluggable compression interface to Kafka

[Diop, Assane]

Hi Greg, 

Thanks for taking the time to give some feedback. It was very insightful.

I have some answers: 

1. The current proposal is Java centric. We want to figure out with Java first 
and then later incorporate other languages. We will get there. 

2. The question of where the plugins would live is an important one. I would 
like to get the community engagement on where a plugin would live. 
   Officially supported plugins could be part of Kafka and others could live in 
a plugin repository. Is there currently a way to store plugins in Kafka and 
load them into the classpath? If such a space could be allowed then it would 
provide an standard way of installing officially supported plugins.
   In OpenSearch for example, there is a plugin utility that takes the jar and 
installs it across the cluster, privileges can be granted by an admin. Such 
utility could be implemented in Kafka. 
   
3. There is many way to look at this, we could change the message format that 
use the pluggable interface to be for example v3 and synchronize against that.
   In order to use the pluggable codec, you will have to be at message version 
3 for example. 

4. Passing the class name as metadata is one way to have the producer talk to 
the broker about which plugin to use. However there could be other 
implementation
   where you could set every thing to know about the topic using topic level 
compression. In this case for example a rule could be that in order to use the 
   pluggable interface, you should use topic level compression.

 I would like to have your valuable inputs on this!!

Thanks before end,
Assane

-Original Message-
From: Greg Harris  
Sent: Wednesday, February 14, 2024 2:36 PM
To: dev@kafka.apache.org
Subject: Re: DISCUSS KIP-984 Add pluggable compression interface to Kafka

Hi Assane,

Thanks for the KIP!
Looking back, it appears that the project has only ever added compression types 
twice: lz4 in 2014 and zstd in 2018, and perhaps Kafka has fallen behind the 
state-of-the-art compression algorithms.
Thanks for working to fix that!

I do have some concerns:

1. I think this is a very "java centric" proposal, and doesn't take non-java 
clients into enough consideration. librdkafka [1] is a great example of an 
implementation of the Kafka protocol which doesn't have the same classloading 
and plugin infrastructure that Java has, which would make implementing this 
feature much more difficult.

2. By making the interface pluggable, it puts the burden of maintaining 
individual compression codecs onto external developers, which may not be 
willing to maintain a codec for the service-lifetime of such a codec.
An individual developer can easily implement a plugin to allow them to use a 
cutting-edge compression algorithm without consulting the Kafka project, but as 
soon as data is compressed using that algorithm, they are on the hook to 
support that plugin going forward by the organizations using their 
implementation.
Part of the collective benefits of the Kafka project is to ensure the ongoing 
maintenance of such codecs, and provide a long deprecation window when a codec 
reaches EOL. I think the Kafka project is well-equipped to evaluate the 
maturity and properties of compression codecs and then maintain them going 
forward.

3. Also by making the interface pluggable, it reduces the scope of individual 
compression codecs. No longer is there a single lineage of Kafka protocols, 
where vN+1 of a protocol supports a codec that vN does not. Now there will be 
"flavors" of the protocol, and operators will need to ensure that their servers 
and their clients support the same "flavors" or else encounter errors.
This is the sort of protocol forking which can be dangerous to the Kafka 
community going forward. If there is a single lineage of codecs such that the 
upstream Kafka vX.Y supports codec Z, it is much simpler for other 
implementations to check and specify "Kafka vX.Y compatible", than it is to 
check & specify "Kafka vX.Y & Z compatible".

4. The Java class namespace is distributed, as anyone can name their class 
anything. It achieves this by being very verbose, with long fully-qualified 
names for classes. This is in conflict with a binary protocol, where it is 
desirable for the overhead to be as small as possible.
This may incentivise developers to keep their class names short, which also 
makes conflict more likely. If you have the option of naming your class "B" 
instead of "org.example.blah.BrotlCompressionCodecVersionOne", and meaningfully 
save a flat 47 bytes on every request, somebody/everybody is going to do that.
This now increases the likelihood for conflict, as perhaps two developers want 
the same short name. Yes there are 52 one-letter class names, but to ensure 
that no two codecs ever conflict requires global coordination that a pluggable 
interface tries to avoid.
Operators then take on the burden of ensuring that the "B" codec on the other 
machine is indeed the "B" 

Re: [DISCUSS] Apache Kafka 3.8.0 release

[Justine Olshan]

Thanks Joesp. +1 from me.

On Mon, Feb 26, 2024 at 3:37 AM Josep Prat 
wrote:

> Hi all,
>
> I'd like to volunteer as release manager for the Apache Kafka 3.8.0
> release.
> If there are no objections, I'll start building a release plan (or adapting
> the one Colin made some weeks ago) in the wiki in the next days.
>
> Thank you.
>
> --
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |    >
>      <
> https://twitter.com/aiven_io>
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B
>

Re: [DISCUSS]

[Justine Olshan]

Hey there,

Thanks for bringing this discussion to the mailing list. I think this is an
interesting one.

I am little concerned about the ability to add any topic ID to the create
topic request. It means that we can no longer rely on topic IDs being
globally unique. I worry folks may misunderstand and think they should be
creating their own IDs for topics and getting into sticky situations.

I wonder if there is a way to encapsulate what you want to do into an API
(ie some sort of formalized migration of a topic). This would make it
clearer that this is meant to be the "same" topic.
I also wonder if this is something that would be on the tiered storage
side. Is the use case combining two topics or recreating the topic and
using the same tiered storage? I'm curious about this use case and if the
topic is intended to be the "same" topic on the recreated cluster.

Thanks,
Justine

On Mon, Feb 26, 2024 at 5:39 AM Anton Agestam
 wrote:

> Hello everyone,
>
> I'm hoping to start a discussion to get some feedback on a topic, I assume
> this is something that would require a KIP, but wanted to start out with a
> discussion to gather some early feedback.
>
> We have a use case internally for creating topics with a given topic ID.
> The need for this arises because we're working on the capability to delete
> a cluster and later recreate its tiered storage topics in an entirely new
> cluster. For that to work, since the organization of the tiered data is
> dependent on the topic ID, the topic created in the fresh cluster must have
> the same ID as the original one in the deleted cluster.
>
> We have a small patch only used for internal testing so far, showing that
> adding TopicID as a tagged field on CreateTopicsRequest and implementing
> the capability is a small and uncomplicated change:
>
> https://github.com/aiven/kafka/pull/48/commits/fa5ebbca642391af630f1c7f7cc013fe56856e13
> .
>
> We also foresee that the same need will arise from tools that replicate
> topics between clusters, such as mirrormaker2.
>
> Are there any thoughts about this proposal, or any pointers for moving
> forward formalizing this in a KIP?
>
> Best regards,
> Anton
>
> --
> [image: Aiven] 
> *Anton Agestam* (he/him or they/them)
> Software Engineer, *Aiven*
> anton.ages...@aiven.io   |   +46 704 486 289
> aiven.io    |
> 
>  >
>

Re: [DISCUSS] KIP-853: KRaft Controller Membership Changes

[José Armando García Sancio]

Hi Jun, see my comments below.

I need to make some changes to the KIP to reflect our conversation.

On Fri, Feb 23, 2024 at 1:39 PM Jun Rao  wrote:
> 15.1 "In this configuration, the local replica needs to use
> CONTROLLER_PLAINTEXT because that is what is supported by all of the
> voters."
>Hmm, my understanding is that what listener to use can be made pairwise
> between the sender and the receiver. There is no need to pick the same
> listener across all voters, right?

Yes, this is true. Thinking about it some more, there are a couple of issues.

In the current implementation, the Selector we use for the
NetworkClient only supports one ChannelBuilder and that channel
builder can only use one security protocol. Until we fix that in the
implementation the preferred (first) listener needs to be a listener
that is supported by all of the voters. I'll make sure to document
that in the Kafka documentation until we fix the implementation.

The other issue is that when the leader sends its endpoint in the
BeginQuorumEpoch it needs to send all of its endpoints. The leader
needs to do that because replicas may connect to each other using any
of the configured listeners. For example, let's assume that the voters
(controllers) have been upgraded to support the listeners
CONTROLLER_SSL and CONTROLLER_PLAINTEXT with CONTROLLER_SSL being the
preferred listener. It is possible that the observers (brokers) have
yet to be reconfigured to use CONTROLLER_SSL and are instead using
CONTROLLER_PLAINTEXT. That means that the voters need to know all of
the leader's listeners because observers discover the leader by
sending a Fetch request to a random endpoint in
controller.quorum.bootstrap.servers using the security protocol of the
first controller listener. I'll make that change to the KIP.

> 15.2  When bootstrapping with just one controller, we could register a
> single voter with multiple listeners from controller.listener.names and
> listeners. However, when bootstrapping with multiple controllers,
> kafka-storage only has the option to specify one listener
> (--controller-quorum-voters -@:) per
> voter. Should we make them consistent?

This is true. I was trying to avoid this but this configuration can
get quite complicated. The user will have a better experience if they
can express it as a JSON file. The configuration should just be a
VotersRecord but in JSON. Let me update the KIP.

> 18.3 : "It will not use the metadata layer (QuorumController) to update and
> persist the finalized kraft.version."
> So, we depend on KRaftVersionRecord to propagate finalized kraft.version to
> brokers/controllers? It would be useful to document that.

Yes. I'll make that clear in the KIP.

> 27. "My plan is to rely on KIP-996: Pre-vote:".
> Hmm, Diego's paper says "Unfortunately, the Pre-Vote phase does not solve
> the problem of disruptive servers". Is that a real issue?

An active leader gets disrupted if a replica increases their epoch
past the active leader's epoch.

Diego's Raft allows the replica to increase their epoch and become a
candidate once it wins a pre-vote. Diego defines pre-vote as follows:
"A candidate would first ask other servers whether its log was
up-to-date enough to get their vote. Only if the candidate believed it
could get votes from a majority of the cluster would it increment its
term and start a normal election.

Unfortunately, the Pre-Vote phase does not solve the problem of
disruptive servers: there are situations where the disruptive server’s
log is sufficiently up-to-date, but starting an election would still
be disruptive."

In other words, Diego's Raft only compares log lengths during the
pre-vote phase. That is why Diego then mentions:
"We modify the RequestVote RPC to achieve this: if a server receives a
RequestVote request within the minimum election timeout of hearing
from a current leader, it does not update its term or grant its vote."

Diego can do this for his definition of Raft because he is only
concerned with voters. There is no support for observers. Diego's Raft
nodes are either voters or they are not part of the cluster.

We cannot do that for KRaft because we have observers. When a replica
is removed from the voter set it becomes an observer. I want that
replica to be able to continue participating in the cluster. That
means that we need to stop observers from increasing their epoch even
if they think they are in the voter set.

For KRaft our definition of pre-vote is different. Voters are going to
perform both checks during the pre-vote phase. A voter rejects a Vote
request if either:
1. The sender's log is shorter than the receiver's log
2. The receiver has contacted the leader either through Fetch,
FetchSnapshot or BeginQuorumEpoch during the Fetch timeout.

Check 2 should stop an observer that thinks they are a voter from
increasing their epoch and disrupting the leader. Let me give an
example.

1. Assume a voter set of 1, 2, 3, 4 and it has been replicated to all
of the voters 

[jira] [Created] (KAFKA-16307) investigate EventAccumulator thread idle ratio metric

[Jeff Kim (Jira)]

Jeff Kim created KAFKA-16307:


 Summary: investigate EventAccumulator thread idle ratio metric
 Key: KAFKA-16307
 URL: https://issues.apache.org/jira/browse/KAFKA-16307
 Project: Kafka
  Issue Type: Sub-task
Reporter: Jeff Kim






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16306) GroupCoordinatorService logger is not configured

[Jeff Kim (Jira)]

Jeff Kim created KAFKA-16306:


 Summary: GroupCoordinatorService logger is not configured
 Key: KAFKA-16306
 URL: https://issues.apache.org/jira/browse/KAFKA-16306
 Project: Kafka
  Issue Type: Sub-task
Reporter: Jeff Kim


The GroupCoordinatorService constructor initializes with the wrong logger class:

```

GroupCoordinatorService(
LogContext logContext,
GroupCoordinatorConfig config,
CoordinatorRuntime runtime,
GroupCoordinatorMetrics groupCoordinatorMetrics
) {
    this.log = logContext.logger(CoordinatorLoader.class);
    this.config = config;
    this.runtime = runtime;
    this.groupCoordinatorMetrics = groupCoordinatorMetrics;
}

```

change this to GroupCoordinatorService.class



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2676

[Apache Jenkins Server]

See 

[DISCUSS]

[Anton Agestam]

Hello everyone,

I'm hoping to start a discussion to get some feedback on a topic, I assume
this is something that would require a KIP, but wanted to start out with a
discussion to gather some early feedback.

We have a use case internally for creating topics with a given topic ID.
The need for this arises because we're working on the capability to delete
a cluster and later recreate its tiered storage topics in an entirely new
cluster. For that to work, since the organization of the tiered data is
dependent on the topic ID, the topic created in the fresh cluster must have
the same ID as the original one in the deleted cluster.

We have a small patch only used for internal testing so far, showing that
adding TopicID as a tagged field on CreateTopicsRequest and implementing
the capability is a small and uncomplicated change:
https://github.com/aiven/kafka/pull/48/commits/fa5ebbca642391af630f1c7f7cc013fe56856e13
.

We also foresee that the same need will arise from tools that replicate
topics between clusters, such as mirrormaker2.

Are there any thoughts about this proposal, or any pointers for moving
forward formalizing this in a KIP?

Best regards,
Anton

-- 
[image: Aiven] 
*Anton Agestam* (he/him or they/them)
Software Engineer, *Aiven*
anton.ages...@aiven.io   |   +46 704 486 289
aiven.io    |

[jira] [Created] (KAFKA-16305) # Optimisation in SslTransportLayer:handshakeUnwrap stalls TLS handshake

[Gaurav Narula (Jira)]

Gaurav Narula created KAFKA-16305:
-

 Summary: # Optimisation in SslTransportLayer:handshakeUnwrap 
stalls TLS handshake
 Key: KAFKA-16305
 URL: https://issues.apache.org/jira/browse/KAFKA-16305
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.6.1, 3.7.0
Reporter: Gaurav Narula


Kafka allows users to configure custom {{SSLEngine}} via the 
{{SslEngineFactory}} interface. There have been attempts to use an OpenSSL 
based {{SSLEngine}} using {{netty-handler}} over the JDK based implementation 
for performance reasons.

While trying to use a Netty/Openssl based SSLEngine, we observe that the server 
hangs while performing the TLS handshake.  We observe the following logs


{code}
2024-02-26 01:40:00,117 
data-plane-kafka-network-thread-101-ListenerName(TEST)-SASL_SSL-0 [TRACE] 
SslTransportLayer- [SslTransportLayer 
channelId=127.0.0.1:60045-127.0.0.1:60046-0 
key=channel=java.nio.channels.SocketChannel[connected local=/127.0.0.1:60045 
remote=/127.0.0.1:60046], selector=sun.nio.ch.KQueueSelectorImpl@18d201be, 
interestOps=1, readyOps=0] XX-Gaurav ready isReady false channelId 
127.0.0.1:60045-127.0.0.1:60046-0 state NOT_INITIALIZED
2024-02-26 01:40:00,117 
data-plane-kafka-network-thread-101-ListenerName(TEST)-SASL_SSL-0 [INFO ] 
Selector - [SocketServer listenerType=ZK_BROKER, nodeId=101] 
XX-Gaurav: calling prepare channelId 127.0.0.1:60045-127.0.0.1:60046-0
2024-02-26 01:40:00,117 
data-plane-kafka-network-thread-101-ListenerName(TEST)-SASL_SSL-0 [TRACE] 
SslTransportLayer- [SslTransportLayer 
channelId=127.0.0.1:60045-127.0.0.1:60046-0 
key=channel=java.nio.channels.SocketChannel[connected local=/127.0.0.1:60045 
remote=/127.0.0.1:60046], selector=sun.nio.ch.KQueueSelectorImpl@18d201be, 
interestOps=1, readyOps=0] XX-Gaurav ready isReady false channelId 
127.0.0.1:60045-127.0.0.1:60046-0 state NOT_INITIALIZED
2024-02-26 01:40:00,117 
data-plane-kafka-network-thread-101-ListenerName(TEST)-SASL_SSL-0 [TRACE] 
SslTransportLayer- [SslTransportLayer 
channelId=127.0.0.1:60045-127.0.0.1:60046-0 
key=channel=java.nio.channels.SocketChannel[connected local=/127.0.0.1:60045 
remote=/127.0.0.1:60046], selector=sun.nio.ch.KQueueSelectorImpl@18d201be, 
interestOps=1, readyOps=0] XX-Gaurav ready isReady false channelId 
127.0.0.1:60045-127.0.0.1:60046-0 state HANDSHAKE
2024-02-26 01:40:00,117 
data-plane-kafka-network-thread-101-ListenerName(TEST)-SASL_SSL-0 [TRACE] 
SslTransportLayer- [SslTransportLayer 
channelId=127.0.0.1:60045-127.0.0.1:60046-0 
key=channel=java.nio.channels.SocketChannel[connected local=/127.0.0.1:60045 
remote=/127.0.0.1:60046], selector=sun.nio.ch.KQueueSelectorImpl@18d201be, 
interestOps=1, readyOps=0] SSLHandshake NEED_UNWRAP channelId 
127.0.0.1:60045-127.0.0.1:60046-0, appReadBuffer pos 0, netReadBuffer pos 0, 
netWriteBuffer pos 0
2024-02-26 01:40:00,117 
data-plane-kafka-network-thread-101-ListenerName(TEST)-SASL_SSL-0 [TRACE] 
SslTransportLayer- [SslTransportLayer 
channelId=127.0.0.1:60045-127.0.0.1:60046-0 
key=channel=java.nio.channels.SocketChannel[connected local=/127.0.0.1:60045 
remote=/127.0.0.1:60046], selector=sun.nio.ch.KQueueSelectorImpl@18d201be, 
interestOps=1, readyOps=0] SSLHandshake handshakeUnwrap channelId 
127.0.0.1:60045-127.0.0.1:60046-0 doRead true
2024-02-26 01:40:00,118 
data-plane-kafka-network-thread-101-ListenerName(TEST)-SASL_SSL-0 [TRACE] 
SslTransportLayer- [SslTransportLayer 
channelId=127.0.0.1:60045-127.0.0.1:60046-0 
key=channel=java.nio.channels.SocketChannel[connected local=/127.0.0.1:60045 
remote=/127.0.0.1:60046], selector=sun.nio.ch.KQueueSelectorImpl@18d201be, 
interestOps=1, readyOps=0] SSLHandshake post handshakeUnwrap: channelId 
127.0.0.1:60045-127.0.0.1:60046-0 handshakeStatus NEED_UNWRAP status 
BUFFER_UNDERFLOW read 0
2024-02-26 01:40:00,118 
data-plane-kafka-network-thread-101-ListenerName(TEST)-SASL_SSL-0 [TRACE] 
SslTransportLayer- [SslTransportLayer 
channelId=127.0.0.1:60045-127.0.0.1:60046-0 
key=channel=java.nio.channels.SocketChannel[connected local=/127.0.0.1:60045 
remote=/127.0.0.1:60046], selector=sun.nio.ch.KQueueSelectorImpl@18d201be, 
interestOps=1, readyOps=0] SSLHandshake post unwrap NEED_UNWRAP channelId 
127.0.0.1:60045-127.0.0.1:60046-0, handshakeResult Status = BUFFER_UNDERFLOW 
HandshakeStatus = NEED_UNWRAP bytesConsumed = 0 bytesProduced = 0, 
appReadBuffer pos 0, netReadBuffer pos 0, netWriteBuffer pos 0 handshakeStatus 
NEED_UNWRAP
2024-02-26 01:40:00,118 
data-plane-kafka-network-thread-101-ListenerName(TEST)-SASL_SSL-0 [TRACE] 
SslTransportLayer- [SslTransportLayer 
channelId=127.0.0.1:60045-127.0.0.1:60046-0 
key=channel=java.nio.channels.SocketChannel[connected local=/127.0.0.1:60045 
remote=/127.0.0.1:60046], selector=sun.nio.ch.KQueueSelectorImpl@18d201be, 
interestOps=1, readyOps=0] XX-Gaurav ready isReady false channelId 

[DISCUSS] Apache Kafka 3.8.0 release

[Josep Prat]

Hi all,

I'd like to volunteer as release manager for the Apache Kafka 3.8.0 release.
If there are no objections, I'll start building a release plan (or adapting
the one Colin made some weeks ago) in the wiki in the next days.

Thank you.

-- 
[image: Aiven] 

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io   |   +491715557497
aiven.io    |   
     
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2675

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-16008) Fix PlaintextConsumerTest.testMaxPollIntervalMs

[Lucas Brutschy (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lucas Brutschy resolved KAFKA-16008.

Resolution: Duplicate

> Fix PlaintextConsumerTest.testMaxPollIntervalMs
> ---
>
> Key: KAFKA-16008
> URL: https://issues.apache.org/jira/browse/KAFKA-16008
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, consumer
>Affects Versions: 3.7.0
>Reporter: Kirk True
>Assignee: Lucas Brutschy
>Priority: Critical
>  Labels: consumer-threading-refactor, integration-tests, timeout
> Fix For: 3.8.0
>
>
> The integration test {{PlaintextConsumerTest.testMaxPollIntervalMs}} is 
> failing when using the {{AsyncKafkaConsumer}}.
> The error is:
> {code}
> org.opentest4j.AssertionFailedError: Timed out before expected rebalance 
> completed
>     at org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38)
> at org.junit.jupiter.api.Assertions.fail(Assertions.java:134)
> at 
> kafka.api.AbstractConsumerTest.awaitRebalance(AbstractConsumerTest.scala:317)
> at 
> kafka.api.PlaintextConsumerTest.testMaxPollIntervalMs(PlaintextConsumerTest.scala:194)
> {code}
> The logs include this line:
>  
> {code}
> [2023-12-13 15:11:16,134] WARN [Consumer clientId=ConsumerTestConsumer, 
> groupId=my-test] consumer poll timeout has expired. This means the time 
> between subsequent calls to poll() was longer than the configured 
> max.poll.interval.ms, which typically implies that the poll loop is spending 
> too much time processing messages. You can address this either by increasing 
> max.poll.interval.ms or by reducing the maximum size of batches returned in 
> poll() with max.poll.records. 
> (org.apache.kafka.clients.consumer.internals.HeartbeatRequestManager:188)
> {code} 
> I don't know if that's related or not.
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] 3.7.0 RC4

[Stanislav Kozlovski]

*(double-sending this to kafka-clients & users, as I forgot initially)*

This vote passes with *10 +1 votes* (3 bindings) and no 0 or -1 votes.

+1 votes

PMC Members (binding):
* Mickael Maison
* Justine Olshan
* Divij Vaidya

Community (non-binding):
* Proven Provenzano
* Federico Valeri
* Vedarth Sharma
* Andrew Schofield
* Paolo Patierno
* Jakub Scholz
* Josep Prat



0 votes

* No votes



-1 votes

* No votes



Vote thread:
https://lists.apache.org/thread/71djwz292y2lzgwzm7n6n8o7x56zbgh9

I'll continue with the release process and the release announcement will
follow ASAP.

On Mon, Feb 26, 2024 at 10:42 AM Stanislav Kozlovski 
wrote:

>
> This vote passes with *10 +1 votes* (3 bindings) and no 0 or -1 votes.
>
> +1 votes
>
> PMC Members (binding):
> * Mickael Maison
> * Justine Olshan
> * Divij Vaidya
>
> Community (non-binding):
> * Proven Provenzano
> * Federico Valeri
> * Vedarth Sharma
> * Andrew Schofield
> * Paolo Patierno
> * Jakub Scholz
> * Josep Prat
>
> 
>
> 0 votes
>
> * No votes
>
> 
>
> -1 votes
>
> * No votes
>
> 
>
> Vote thread:
> https://lists.apache.org/thread/71djwz292y2lzgwzm7n6n8o7x56zbgh9
>
> I'll continue with the release process and the release announcement will
> follow ASAP.
>
> Best,
> Stanislav
>
> On Sun, Feb 25, 2024 at 7:08 PM Mickael Maison 
> wrote:
>
>> Hi,
>>
>> Thanks for sorting out the docs issues.
>> +1 (binding)
>>
>> Mickael
>>
>> On Fri, Feb 23, 2024 at 11:50 AM Stanislav Kozlovski
>>  wrote:
>> >
>> > Some quick updates:
>> >
>> > There were some inconsistencies between the documentation in the
>> > apache/kafka repo and the one in kafka-site. The process is such that
>> the
>> > apache/kafka docs are the source of truth, but we had a few divergences
>> in
>> > the other repo. I have worked on correcting those with:
>> > - MINOR: Reconcile upgrade.html with kafka-site/36's version
>> >  and cherry-picked it into
>> the
>> > 3.6 and 3.7 branches too
>> >
>> > Additionally, the 3.7 upgrade notes have been merged in apache/kafka -
>> MINOR:
>> > Add 3.7 upgrade notes > >.
>> >
>> > With that, I have opened a PR to move them to the kafka-site repository
>> -
>> > https://github.com/apache/kafka-site/pull/587. That is awaiting review.
>> >
>> > Similarly, the 3.7 blog post is ready for review again
>> >  and awaiting a review
>> on 37:
>> > Update default docs to point to the 3.7.0 release docs
>> > 
>> >
>> > I also have a WIP for fixing the 3.6 docs in the kafka-site repo
>> > . This isn't really
>> related
>> > to the release, but it's good to do.
>> >
>> > On Wed, Feb 21, 2024 at 4:55 AM Luke Chen  wrote:
>> >
>> > > Hi all,
>> > >
>> > > I found there is a bug (KAFKA-16283
>> > > ) in the built-in
>> > > `RoundRobinPartitioner`, and it will cause only half of the partitions
>> > > receiving the records. (I'm surprised our tests didn't catch it!?)
>> > > After further testing, I found this issue already existed in v3.0.0.
>> (I
>> > > didn't test 2.x versions)
>> > > I think this should not be a blocker to v3.7.0 since it's been there
>> for
>> > > years.
>> > > But I think we should, at least, document it to notify users about
>> this
>> > > known issue.
>> > > I've created 2 PRs to document it:
>> > > https://github.com/apache/kafka/pull/15400
>> > > https://github.com/apache/kafka-site/pull/585
>> > >
>> > > Let me know what you think.
>> > >
>> > > Thanks.
>> > > Luke
>> > >
>> > > On Wed, Feb 21, 2024 at 10:52 AM Proven Provenzano
>> > >  wrote:
>> > >
>> > > > HI,
>> > > >
>> > > > I've downloaded, built from source and then validated JBOD with
>> KRaft
>> > > works
>> > > > along with migrating a cluster with JBOD from ZK to KRaft works.
>> > > >
>> > > > +1 (nonbinding) from me.
>> > > >
>> > > > --Proven
>> > > >
>> > > > On Tue, Feb 20, 2024 at 2:13 PM Justine Olshan
>> > > > 
>> > > > wrote:
>> > > >
>> > > > > Hey folks,
>> > > > >
>> > > > > I've done the following to validate the release:
>> > > > >
>> > > > > -- validated the keys for all the artifacts
>> > > > > -- built from source and started a ZK cluster -- ran a few
>> workloads on
>> > > > it.
>> > > > > -- ran 2.12 Kraft cluster and ran a few workloads on it
>> > > > >
>> > > > > I see there is a lot of ongoing discussion about the upgrade
>> notes. +1
>> > > > > (binding) from me given Mickael is voting +1 as well.
>> > > > >
>> > > > > Justine
>> > > > >
>> > > > > On Tue, Feb 20, 2024 at 6:18 AM Divij Vaidya <
>> divijvaidy...@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > > I am a bit unclear on the precise process regarding what
>> parts of
>> > > > this
>> > > > > > get merged at what time, and whether the release first needs to
>> be
>> > > done
>> > > > > or
>> > > 

Re: [VOTE] 3.7.0 RC4

[Stanislav Kozlovski]

This vote passes with *10 +1 votes* (3 bindings) and no 0 or -1 votes.

+1 votes

PMC Members (binding):
* Mickael Maison
* Justine Olshan
* Divij Vaidya

Community (non-binding):
* Proven Provenzano
* Federico Valeri
* Vedarth Sharma
* Andrew Schofield
* Paolo Patierno
* Jakub Scholz
* Josep Prat



0 votes

* No votes



-1 votes

* No votes



Vote thread:
https://lists.apache.org/thread/71djwz292y2lzgwzm7n6n8o7x56zbgh9

I'll continue with the release process and the release announcement will
follow ASAP.

Best,
Stanislav

On Sun, Feb 25, 2024 at 7:08 PM Mickael Maison 
wrote:

> Hi,
>
> Thanks for sorting out the docs issues.
> +1 (binding)
>
> Mickael
>
> On Fri, Feb 23, 2024 at 11:50 AM Stanislav Kozlovski
>  wrote:
> >
> > Some quick updates:
> >
> > There were some inconsistencies between the documentation in the
> > apache/kafka repo and the one in kafka-site. The process is such that the
> > apache/kafka docs are the source of truth, but we had a few divergences
> in
> > the other repo. I have worked on correcting those with:
> > - MINOR: Reconcile upgrade.html with kafka-site/36's version
> >  and cherry-picked it into
> the
> > 3.6 and 3.7 branches too
> >
> > Additionally, the 3.7 upgrade notes have been merged in apache/kafka -
> MINOR:
> > Add 3.7 upgrade notes  >.
> >
> > With that, I have opened a PR to move them to the kafka-site repository -
> > https://github.com/apache/kafka-site/pull/587. That is awaiting review.
> >
> > Similarly, the 3.7 blog post is ready for review again
> >  and awaiting a review
> on 37:
> > Update default docs to point to the 3.7.0 release docs
> > 
> >
> > I also have a WIP for fixing the 3.6 docs in the kafka-site repo
> > . This isn't really
> related
> > to the release, but it's good to do.
> >
> > On Wed, Feb 21, 2024 at 4:55 AM Luke Chen  wrote:
> >
> > > Hi all,
> > >
> > > I found there is a bug (KAFKA-16283
> > > ) in the built-in
> > > `RoundRobinPartitioner`, and it will cause only half of the partitions
> > > receiving the records. (I'm surprised our tests didn't catch it!?)
> > > After further testing, I found this issue already existed in v3.0.0. (I
> > > didn't test 2.x versions)
> > > I think this should not be a blocker to v3.7.0 since it's been there
> for
> > > years.
> > > But I think we should, at least, document it to notify users about this
> > > known issue.
> > > I've created 2 PRs to document it:
> > > https://github.com/apache/kafka/pull/15400
> > > https://github.com/apache/kafka-site/pull/585
> > >
> > > Let me know what you think.
> > >
> > > Thanks.
> > > Luke
> > >
> > > On Wed, Feb 21, 2024 at 10:52 AM Proven Provenzano
> > >  wrote:
> > >
> > > > HI,
> > > >
> > > > I've downloaded, built from source and then validated JBOD with KRaft
> > > works
> > > > along with migrating a cluster with JBOD from ZK to KRaft works.
> > > >
> > > > +1 (nonbinding) from me.
> > > >
> > > > --Proven
> > > >
> > > > On Tue, Feb 20, 2024 at 2:13 PM Justine Olshan
> > > > 
> > > > wrote:
> > > >
> > > > > Hey folks,
> > > > >
> > > > > I've done the following to validate the release:
> > > > >
> > > > > -- validated the keys for all the artifacts
> > > > > -- built from source and started a ZK cluster -- ran a few
> workloads on
> > > > it.
> > > > > -- ran 2.12 Kraft cluster and ran a few workloads on it
> > > > >
> > > > > I see there is a lot of ongoing discussion about the upgrade
> notes. +1
> > > > > (binding) from me given Mickael is voting +1 as well.
> > > > >
> > > > > Justine
> > > > >
> > > > > On Tue, Feb 20, 2024 at 6:18 AM Divij Vaidya <
> divijvaidy...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > > I am a bit unclear on the precise process regarding what parts
> of
> > > > this
> > > > > > get merged at what time, and whether the release first needs to
> be
> > > done
> > > > > or
> > > > > > not.
> > > > > >
> > > > > > The order is as follows:
> > > > > >
> > > > > > 1. Release approved as part of this vote. After this we follow
> the
> > > > > > steps from here:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Process#ReleaseProcess-Afterthevotepasses
> > > > > >
> > > > > > 2. Upload artifacts to maven etc. These artifacts do not have RC
> > > suffix
> > > > > in
> > > > > > them. You need a PMC member to mark these artifacts as
> "production"
> > > in
> > > > > > apache svn.
> > > > > > 3. Update website changes (docs, blog etc.). This is where your
> PRs
> > > > > > on kafka-site repo get merged.
> > > > > > 4. Send a release announcement by email.
> > > > > >
> > > > > > --
> > > > > > Divij Vaidya
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Feb 20, 2024