Re: AW: How do I resolve an UnwritableMetadataException

[Luke Chen]

Hi all,

Thanks for reporting this issue.
This issue is fixed in KAFKA-16583
 and will be included in
v3.7.1 and v3.8.0, which should be released in June or July.

Thanks.
Luke

On Tue, Jun 4, 2024 at 11:29 PM Sejal Patel 
wrote:

> Thank you, Tobias.
>
> If the Jira ticket is correct, that explains a bunch. This was not
> something we ever noticed in lower environments, but we did only recently
> upgrade to 3.7 about a month ago. We've never had problems with upgrades
> before, and this one seemed to go smoothly as well until this happened.
>
> I've never downgraded before. How hard is it? Can I simply replace the app
> with the older version, or are there additional steps that need to be done
> as well?
>
> Thanks again for your help.
>
> On Tue Jun 4, 2024, 12:46 PM GMT, tobias.b...@peripetie.de  tobias.b...@peripetie.de> wrote:
> > Had the same Issue some Weeks ago, we also did an upgrade to 3.7..
> > Did not receive any hints on this Topic back in the days.
> >
> > There is an open StackOverflow
> https://stackoverflow.com/questions/78334479/unwritablemetadataexception-on-startup-in-apache-kafka
> > There is also an open Kafka Jira Ticket
> https://issues.apache.org/jira/browse/KAFKA-16662
> > We fixed it by downgrading to 3.6.
> >
> > Hope this helps.
> >
> > -Ursprüngliche Nachricht-
> > Von: Sejal Patel 
> > Gesendet: Montag, 3. Juni 2024 08:26
> > An: users@kafka.apache.org
> > Betreff: How do I resolve an UnwritableMetadataException
> >
> > I was expanding my kafka cluster to 24 nodes (from 16 nodes) and
> rebalancing the topics. 1 of the partitions of the topic did not get
> rebalanced as expected (it was taking a million years so I decided to look
> and see what was happening). It turns out that the script for mounting the
> 2nd partition for use as /data did not kick in and thus there simply wasn't
> enough disk space available at the time of the rebalance. The system was
> left with like 5Mb of disk space and the kafka brokers were essentially
> borked at that point.
> >
> > So I had to kill the kafka processes, move the original kafka data
> folder to a /tmp location, mounted the data partition, and migrated the
> /tmp kafka folder back to the original spot. But when I went to startup the
> kafka instance I got this message over and over again every few
> milliseconds.
> >
> > [2024-06-03 06:14:01,503] ERROR Encountered metadata loading fault:
> Unhandled error initializing new publishers
> (org.apache.kafka.server.fault.LoggingFaultHandler)
> org.apache.kafka.image.writer.UnwritableMetadataException: Metadata has
> been lost because the following could not be represented in metadata
> version 3.4-IV0: the directory assign ment state of one or more replicas at
> org.apache.kafka.image.writer.ImageWriterOptions.handleLoss(ImageWriterOptions.java:94)
> at
> org.apache.kafka.metadata.PartitionRegistration.toRecord(PartitionRegistration.java:391)
> at org.apache.kafka.image.TopicImage.write(TopicImage.java:71) at
> org.apache.kafka.image.TopicsImage.write(TopicsImage.java:84) at
> org.apache.kafka.image.MetadataImage.write(MetadataImage.java:155) at
> org.apache.kafka.image.loader.MetadataLoader.initializeNewPublishers(MetadataLoader.java:295)
> at
> org.apache.kafka.image.loader.MetadataLoader.lambda$scheduleInitializeNewPublishers$0(MetadataLoader.java:266)
> at
> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:127)
> at
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:210)
> at
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:181)
> at java.base/java.lang.Thread.run(Thread.java:1583) [2024-06-03
> 06:14:01,556] INFO [BrokerLifecycleManager id=28] The broker is in
> RECOVERY. (kafka.server.BrokerLifecycleManager)
> > Scarier is that if any node that is working gets restarted, they too
> start sending off that message as well.
> >
> > I am using a kraft setup and have within the past month upgraded to
> kafka 3.7 (original setup over a year ago was kafka 3.4). How do I resolve
> this issue? I'm not sure what the problem is or how to fix it.
> >
> > If I restart a kraft server, it dies with the same error message and can
> never get spun up again.
> >
> > Is it possible to recover from this or do I need to start from scratch?
> If I start from scratch, how do I keep the topics? What is the best way to
> proceed from here? I'm unable to find anything related to this problem via
> a google search.
> >
> > I'm at a loss and would appreciate any help you can provide.
> >
> > Thank you.
> >

Re: Kafka 3.7 Documentation

[Luke Chen]

Hi Edgar,

> Is this the correct documentation on how to contribute code changes?
>
https://cwiki.apache.org/confluence/display/KAFKA/Contributing+Code+Changes#ContributingCodeChanges-PullRequest

Yes, it is.

For the KAFKA-15513 <https://issues.apache.org/jira/browse/KAFKA-15513>,
sorry, I don't have much experience on SCRAM.
Let's wait for other experts to reply in JIRA.

Thanks.
Luke

On Tue, May 28, 2024 at 3:28 PM Zubel, Edgar
 wrote:

> Thank you for quick response!
>
> Is this the correct documentation on how to contribute code changes?
>
> https://cwiki.apache.org/confluence/display/KAFKA/Contributing+Code+Changes#ContributingCodeChanges-PullRequest
>
> Also I would like to ask you about another issue that I'm interested in -
> https://issues.apache.org/jira/browse/KAFKA-15513
> Is there any updates so far?
>
> Edgar Zubel
> DevOps Engineer
>
>
> -Original Message-
> From: Luke Chen 
> Sent: Tuesday, May 28, 2024 9:48 AM
> To: users@kafka.apache.org
> Subject: Re: Kafka 3.7 Documentation
>
> Hi Edgar,
>
> Thanks for reporting this.
> Yes, I confirmed the document is not correct.
> What we want is to revert from "Migrating brokers to KRaft", to "Enter
> Migration Mode on the Brokers" state, and in the end, Pure ZK.
> About the authorizer change, I'm fine if we want to mention it.
>
> I've created KAFKA-16848 <
> https://issues.apache.org/jira/browse/KAFKA-16848>.
>
> Welcome to open a PR to fix it. :)
>
> Thanks.
> Luke
>
> On Tue, May 28, 2024 at 2:13 PM Zubel, Edgar 
> 
> wrote:
>
> > Hello,
> >
> >
> >
> > I would like to report a mistake in the *Kafka 3.7 Documentation ->
> > 6.10 KRaft -> ZooKeeper to KRaft Migration -> Reverting to ZooKeeper
> > mode During the Migration*.
> >
> >
> >
> > While migrating my Kafka + Zookeeper cluster to KRaft and testing
> > rollbacks at a different migration stages I have noticed, that
> > “*Directions for reverting*” provided for “*Migrating brokers to KRaft*”
> are wrong.
> >
> > Following the first step provided in documentation you suppose to :
> > *On each broker, remove the process.roles configuration, and restore
> > the zookeeper.connect configuration to its previous value. If your
> > cluster requires other ZooKeeper configurations for brokers, such as
> > zookeeper.ssl.protocol, re-add those configurations as well. Then
> > perform a
> > rolling.*
> >
> >
> > In that case, if you remove *process.roles *configuration and restore
> > * zookeeper.connect *as well as other *ZooKeeper *configuration (If
> > your cluster requires) you will receive an error that looks like this:
> > [2024-05-28 08:09:49,396] lvl=ERROR Exiting Kafka due to fatal
> > exception logger=kafka.Kafka$
> >
> > java.lang.IllegalArgumentException: requirement failed:
> > controller.listener.names must be empty when not running in KRaft mode:
> > [CONTROLLER]
> >
> > at scala.Predef$.require(Predef.scala:337)
> >
> > at
> > kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:2441)
> >
> > at kafka.server.KafkaConfig.(KafkaConfig.scala:2290)
> >
> > at kafka.server.KafkaConfig.(KafkaConfig.scala:1639)
> >
> > at kafka.Kafka$.buildServer(Kafka.scala:71)
> >
> > at kafka.Kafka$.main(Kafka.scala:90)
> >
> > at kafka.Kafka.main(Kafka.scala)
> >
> >
> >
> > However I was able to perform rollback successfully by performing
> > additional steps:
> >
> >- Restore *zookeeper.metadata.migration.enable=true *line in broker
> >configuration;
> >- We are using *authorizer.class.name <http://authorizer.class.name
> >*,
> >so it also had to be reverted:
> >*org.apache.kafka.metadata.authorizer.StandardAuthorizer* ->
> >*kafka.security.authorizer.AclAuthorizer*;
> >
> >
> >
> > I believe that should be mentioned.
> >
> >
> >
> > *Edgar Zubel*
> >
> > DevOps Engineer
> >
> > edgar.zu...@teliacompany.com
> >
> >
> >
> > [image: En bild som visar text, klocka Automatiskt genererad
> > beskrivning]
> >
> >
> >
> >
> >
> > *This email may contain information which is privileged or protected
> > against unauthorized disclosure or communication. If you are not the
> > intended recipient, please notify the sender and delete this message
> > and any attachments from your system without producing, distributing
> > or retaining copi

Re: Kafka 3.7 Documentation

[Luke Chen]

Hi Edgar,

Thanks for reporting this.
Yes, I confirmed the document is not correct.
What we want is to revert from "Migrating brokers to KRaft", to "Enter
Migration Mode on the Brokers" state, and in the end, Pure ZK.
About the authorizer change, I'm fine if we want to mention it.

I've created KAFKA-16848 .

Welcome to open a PR to fix it. :)

Thanks.
Luke

On Tue, May 28, 2024 at 2:13 PM Zubel, Edgar
 wrote:

> Hello,
>
>
>
> I would like to report a mistake in the *Kafka 3.7 Documentation -> 6.10
> KRaft -> ZooKeeper to KRaft Migration -> Reverting to ZooKeeper mode During
> the Migration*.
>
>
>
> While migrating my Kafka + Zookeeper cluster to KRaft and testing
> rollbacks at a different migration stages I have noticed, that “*Directions
> for reverting*” provided for “*Migrating brokers to KRaft*” are wrong.
>
> Following the first step provided in documentation you suppose to : *On
> each broker, remove the process.roles configuration, and restore the
> zookeeper.connect configuration to its previous value. If your cluster
> requires other ZooKeeper configurations for brokers, such as
> zookeeper.ssl.protocol, re-add those configurations as well. Then perform a
> rolling.*
>
>
> In that case, if you remove *process.roles *configuration and restore *
> zookeeper.connect *as well as other *ZooKeeper *configuration (If your
> cluster requires) you will receive an error that looks like this:
> [2024-05-28 08:09:49,396] lvl=ERROR Exiting Kafka due to fatal exception
> logger=kafka.Kafka$
>
> java.lang.IllegalArgumentException: requirement failed:
> controller.listener.names must be empty when not running in KRaft mode:
> [CONTROLLER]
>
> at scala.Predef$.require(Predef.scala:337)
>
> at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:2441)
>
> at kafka.server.KafkaConfig.(KafkaConfig.scala:2290)
>
> at kafka.server.KafkaConfig.(KafkaConfig.scala:1639)
>
> at kafka.Kafka$.buildServer(Kafka.scala:71)
>
> at kafka.Kafka$.main(Kafka.scala:90)
>
> at kafka.Kafka.main(Kafka.scala)
>
>
>
> However I was able to perform rollback successfully by performing
> additional steps:
>
>- Restore *zookeeper.metadata.migration.enable=true *line in broker
>configuration;
>- We are using *authorizer.class.name *,
>so it also had to be reverted:
>*org.apache.kafka.metadata.authorizer.StandardAuthorizer* ->
>*kafka.security.authorizer.AclAuthorizer*;
>
>
>
> I believe that should be mentioned.
>
>
>
> *Edgar Zubel*
>
> DevOps Engineer
>
> edgar.zu...@teliacompany.com
>
>
>
> [image: En bild som visar text, klocka Automatiskt genererad beskrivning]
>
>
>
>
>
> *This email may contain information which is privileged or protected
> against unauthorized disclosure or communication. If you are not the
> intended recipient, please notify the sender and delete this message and
> any attachments from your system without producing, distributing or
> retaining copies thereof or disclosing its contents to any other person.
> Telia Company processes emails and other files that may contain personal
> data in accordance with Telia Company’s Privacy Policy
> .*
>
>
>

Re: Kafka retention bug (?)

[Luke Chen]

Hi Nicholas,

I didn't know anything in v3.7.0 would cause this issue.
It would be good if you could open a JIRA for it.
Some info to be provided:
1. You said "in the past", what version of Kafka was it using?
2. What is your broker configuration?
3. KRaft mode? Combined mode? (controller + broker node?)
4. There's no much info in the gist link. It would be great if you could
attach the brokers logs for investigation.

Thanks.
Luke


On Wed, May 15, 2024 at 2:46 AM Nicholas Feinberg 
wrote:

> Hello!
>
> We recently upgraded our Kafka cluster to 3.7. This cluster's topics are
> set to have four days of retention (34560 ms).
>
> In the past, when we've temporarily lowered retention for ops, we've seen
> disk usage return to normal four days later, as expected.
>
> [image: image.png]
>
> However, after our latest round of ops, we're now seeing disk usage
> *continue* to grow on most brokers after those four days pass, despite a 
> *decrease
> *in incoming data. This usage increased until day six.
>
> [image: kafka-ooms.png]
> On day *six* after 4d retention was restored, several brokers began to
> crash, with the following error:
>
> # There is insufficient memory for the Java Runtime Environment to
>> continue.
>> # Native memory allocation (mmap) failed to map 16384 bytes for
>> committing reserved memory.
>
>
> (Details:
> https://gist.github.com/PleasingFungus/3e0cf6b58a4f3eee2171ff91b1aff42a .)
>
> These hosts had ~170GiB of free memory available. We saw no signs of
> pressure on either system or JVM heap memory before or after they reported
> this error. Committed memory seems to be around 10%, so this doesn't seem
> to be an overcommit issue.
>
> The hosts which crashed in this fashion freed large amounts of disk after
> they came back up. This returned them to the usage that we'd expect.
>
> Manually restarting Kafka on a broker likewise resulted in its disk usage
> dropping to the 4d retention level.
>
> Other brokers' disk usage seems to have stabilized.
>
> I've spent some time searching for bugs in the Jira or other posts which
> describe this behavior, but have come up empty.
>
> *Questions*:
>
>- Has anyone else seen an issue similar to this?
>- What are some ways that we could confirm whether Kafka is failing to
>clear expired logs from disk?
>- What could cause the mmap failures that we saw?
>- Would it be helpful for us to file a Jira issue or issues for this,
>and what details should we include if so?
>
> Cheers,
> Nicholas Feinberg
>

Re: [ANNOUNCE] New committer: Igor Soarez

[Luke Chen]

Congrats, Igor!

On Thu, Apr 25, 2024 at 6:10 AM Matthias J. Sax  wrote:

> Congrats!
>
> On 4/24/24 2:29 PM, Bill Bejeck wrote:
> > Congrats Igor!
> >
> > -Bill
> >
> > On Wed, Apr 24, 2024 at 2:37 PM Tom Bentley  wrote:
> >
> >> Congratulations Igor!
> >>
> >> On Thu, 25 Apr 2024 at 6:27 AM, Chia-Ping Tsai 
> wrote:
> >>
> >>> Congratulations, Igor! you are one of the best Kafka developers!!!
> >>>
> >>> Mickael Maison  於 2024年4月25日 週四 上午2:16寫道：
> >>>
>  Congratulations Igor!
> 
>  On Wed, Apr 24, 2024 at 8:06 PM Colin McCabe 
> >> wrote:
> >
> > Hi all,
> >
> > The PMC of Apache Kafka is pleased to announce a new Kafka committer,
>  Igor Soarez.
> >
> > Igor has been a Kafka contributor since 2019. In addition to being a
>  regular contributor and reviewer, he has made significant
> contributions
> >>> to
>  improving Kafka's JBOD support in KRaft mode. He has also contributed
> >> to
>  discussing and reviewing many KIPs such as KIP-690, KIP-554, KIP-866,
> >> and
>  KIP-938.
> >
> > Congratulations, Igor!
> >
> > Thanks,
> >
> > Colin (on behalf of the Apache Kafka PMC)
> 
> >>>
> >>
> >
>

Re: Kraft controller readiness checks

[Luke Chen]

Hi Frank,

About your question:
> Unless this is already available but not well publicised in the
documentation, ideally there should be protocol working on the controller
ports that answers to operational questions like “are metadata partitions
in sync?”, “has the current controller converged with other members of the
quorum?”.

I'm sorry that KRaft controller is using raft protocol, so there is no such
"in-sync replica" definition like data replication protocol. What we did
for our check is described here
<https://github.com/strimzi/proposals/blob/main/060-kafka-roller-kraft.md#the-new-quorum-check>.
In short, we use `controller.quorum.fetch.timeout.ms` and
`replicaLastCaughtUpTimestamp` to determine if it's safe to roll this
controller pod.

Hope this helps.

Thank you.
Luke




On Fri, Apr 19, 2024 at 5:06 PM Francesco Burato 
wrote:

> Hi Luke,
>
> Thanks for the answers. I understand what you are describing in terms of
> rationale for using just the availability of the controller port to
> determine the readiness of the controller, but that is not fully satisfying
> under an operational perspective, at least based on the lack of sufficient
> documentation on the matter. Based on my understanding of kraft, which I
> admit is not considerable, the controllers will host the cluster metadata
> partitions on disk and make them available for the brokers. So, presumably,
> one of the purposes of the controllers is to ensure that the metadata
> partitions are properly replicated. Hence, what happens even in a non k8s
> environment all controllers go down? What sort of outage does the wider
> cluster experience in that circumstance?
>
> A complete outage on the controllers is of course an extreme scenario, but
> a more likely one is that a disk of the controller goes offline and needs
> to be replaced. In this scenario, the controller will have to re-construct
> from scratch the cluster metadata from the other controllers in the quorum
> but it presumably cannot participate to the quorum until the metadata
> partitions are fully replicated. Based on this assumption, the mere
> availability of the controller port does not necessarily mean that I can
> safely shut down another controller because replication has not completed
> yet.
>
> As I mentioned earlier, I don’t know the details of kraft in sufficient
> details to evaluate if my assumptions are warranted, but the official
> documentation does not seem to go in much detail on how to safely operate a
> cluster in kraft mode while it provides very good information on how to
> safely operate a ZK cluster by highlighting that the URP and leader
> elections must be kept under control during restarts.
>
> Unless this is already available but not well publicised in the
> documentation, ideally there should be protocol working on the controller
> ports that answers to operational questions like “are metadata partitions
> in sync?”, “has the current controller converged with other members of the
> quorum?”.
>
> Goes without saying that if any of these topics are properly covered
> anywhere in the docs, more than happy to be RTFMed to the right place.
>
> As for the other points you raise: we have a very particular set-up for
> our kafka clusters that makes the circumstance you highlight not a problem.
> In particular, our consumer and producers are all internal in a namespace
> and can connect to non-ready brokers. Given the URP script checks for the
> global URP state rather than just the URP state for the individual broker,
> that means that as long as even one broker is marked as ready, that means
> the entire cluster is safe. With the ordered rotation imposed by
> statefulset parallel rolling restart, together with the URP readiness check
> and the PDB, we are guaranteed not to cause any problem read or write
> errors. Rotations are rather long, but we don’t really care about speed.
>
> Thanks,
>
> Frank
>
> --
> Francesco Burato | Software Development Engineer | Adobe |
> bur...@adobe.com<mailto:bur...@adobe.com>  | c. +44 747
> 9029370
>
>
> From: Luke Chen 
> Date: Friday, 19 April 2024 at 05:21
> To: users@kafka.apache.org 
> Subject: Re: Kraft controller readiness checks
> EXTERNAL: Use caution when clicking on links or opening attachments.
>
>
> Hello Frank,
>
> That's a good question.
> I think we all know there is no "correct" answer for this question. But I
> can share with you what our team did for it.
>
> Readiness: controller is listening on the controller.listener.names
>
> The rationale behind it is:
> 1. The last step for the controller node startup is to wait until all the
> SocketServer ports to be open, and the Acceptors to be started, and the
> controller port is

Re: Kraft controller readiness checks

[Luke Chen]

Hello Frank,

That's a good question.
I think we all know there is no "correct" answer for this question. But I
can share with you what our team did for it.

Readiness: controller is listening on the controller.listener.names

The rationale behind it is:
1. The last step for the controller node startup is to wait until all the
SocketServer ports to be open, and the Acceptors to be started, and the
controller port is one of them.
2. This controller listener is used to talk to other controllers (voters)
to form the raft quorum, so if it is not open and listening, the controller
is basically not working at all.
3. The controller listener is also used for brokers (observers) to get the
updated raft quorum info and fetch metadata.

Compared with Zookeeper cluster, which is the KRaft quorum is trying to
replace with, the liveness/readiness probe that recommended in Kubernetes
tutorial

is also doing "ruok" check for the pod. And the handler for this "ruok"
command

in the Zookeeper server side, is returning "imok" directly, which means
it's just doing connection check only. So we think this check makes sense.

Here's our design proposal

for the Liveness and Readiness probes in a KRaft Kafka cluster, FYI.
But again, I still think there's no "correct" answer for it. If you have
any better ideas, please let us know.

However, I have some suggestions for your readiness probe for brokers:

> our brokers are configured to use a script which marks the containers as
unready if under-replicated partitions exist. With this readiness check and
a pod disruption budget of the minimum in sync replica - 1

I understand it works well, but it has some drawbacks, and the biggest
issue I can think of is: it's possible to cause unavailability in some
partitions.
For example: 3 brokers in the cluster: 0, 1, 2, and 10 topic partitions are
hosted in broker 0.
a. Broker 0 is shutting down, all partitions in broker 0 are becoming
follower.
b. Broker 0 is starting up, all the followers are trying to catch up with
the leader.
c. 9 out of 10 partitions are caught up and joined ISR group. At this
point, this pod is still unready because there's still 1 partition is under
replicated.
d. Some of the partitions in broker 0 are becoming leader, for example,
auto leader rebalance is triggered.
e. For the leader partitions in broker 0 are now unavailable because the
pod is not in ready state, it cannot serve incoming requests.

In our team, we use the brokerState metric value = RUNNING state for
readiness probe. In KRaft mode, the broker will enter RUNNING state after
the broker has caught up with the controller for metadata, and start to
serve requests from clients. We think that makes more senses.
Again, for more details, you can check the design proposal

for the Liveness and Readiness probes in a KRaft Kafka cluster.

Finally, I saw you didn't have operators for Kafka clusters.
I don't know how you manage all these kafka clusters manually, but there
must be some cumbersome operations, like rolling pods.
Let's say now you want to roll the pods 1 by 1, which pod will you go
first?
And which pod goes last?
Will you do any check before rolling?
How much time does it take for each rolling?
...

I'm just listing some of the problems they might have. So I would recommend
deploying an operator to help manage the kafka clusters.
This is our design proposal

for Kafka roller in operator for KRaft. FYI.

And now, I'm totally biased, but Stirmzi
 provides an fully
open-source operator to manager kafka cluster on Kubernetes.
Welcome to try it (hopefully it will help you manage kafka clusters), join
the community to ask questions, join discussions, or contribute to it.

Thank you.
Luke













On Fri, Apr 19, 2024 at 4:19 AM Francesco Burato 
wrote:

> Hello,
>
> I have a question regarding the deployment of Kafka using Kraft
> controllers in a Kubernetes environment. Our current Kafka cluster is
> deployed on K8S clusters as statefulsets without operators and our brokers
> are configured to use a script which marks the containers as unready if
> under-replicated partitions exist. With this readiness check and a pod
> disruption budget of the minimum in sync replica - 1, we are able to
> perform rollout restarts of our brokers automatically without ever
> producing consumers and producers errors.
>
> We have started the processes of transitioning to Kraft and based on the
> recommended deployment strategy we are 

Re: production ready for zookeeper to kraft migration

[Luke Chen]

Hi Matthieu,

Yes, the ZK migrating to KRaft feature is already GA in v3.6.0.
Sorry, we forgot to update the document in the Kafka-site repo.
I've filed a PR for it: https://github.com/apache/kafka-site/pull/594

Thanks.
Luke

On Thu, Apr 4, 2024 at 6:14 AM Matthieu Patou  wrote:

> I looked at the notes for 3.7.x and the migration from ZK to Kraft is still
> not marked as production ready.
>
> I'm wondering what are the issues that people could be facing during the
> migration.
>
> If 4.0 is still planned to be the full removal for ZK, is there a plan for
> something after 3.7 to mark ZK migration as production ready ?
>
> Best.
>
> Matthieu.
>

Re: Replicas not equally distributed within rack

[Luke Chen]

Hi Abhishek,

For Zookeeper's mode, there's no workaround, unfortunately.
But you can upgrade your cluster to the latest kafka version (v3.7.0) to
migrate to KRaft mode.
For your information, in KRaft mode, we use `StripedReplicaPlacer`, which
will fix your problem.
ref:
https://github.com/apache/kafka/blob/trunk/metadata/src/main/java/org/apache/kafka/metadata/placement/StripedReplicaPlacer.java#L94

Thank you.
Luke


On Thu, Mar 28, 2024 at 12:55 AM Abhishek Singla <
abhisheksingla...@gmail.com> wrote:

> Yes, it’s similar.
>
> Replicas are evenly distribute among racks but not among brokers within
> rack even if no. of brokers are same in all racks.
>
> Is there a workaround for this?
>
> On Wed, 27 Mar 2024 at 5:36 PM, Chia-Ping Tsai 
> wrote:
>
> > hi Abhishek
> >
> > Is this issue similar to the unbalance you had met?
> >
> > https://issues.apache.org/jira/browse/KAFKA-10368
> >
> > best,
> > chia-ping
> >
> > On 2024/03/23 21:06:59 Abhishek Singla wrote:
> > > Hi Team,
> > >
> > > Kafka version: 2_2.12-2.6.0
> > > Zookeeper version: 3.8.x
> > >
> > > We have a Kafka Cluster of 12 brokers spread equally across 3 racks.
> > Topic
> > > gets auto created with default num.partitions=6 and
> replication_factor=3.
> > > It is observed that replicas are equally distributed over racks but
> > within
> > > the rack the replicas are randomly distributed like sometimes 3,3,0,0
> or
> > > sometimes 3:2:1 or sometime 2,2,1,1
> > >
> > > Is there a configuration to evenly distribute replicas across brokers
> > > within a rack, maybe some sort of round robin strategy 2,2,1,1?
> > >
> > > And also it is observed that over time 1 broker ends up having way more
> > > replicas across topics than the other broker in the same rack. Is
> there a
> > > config for even distribution of replicas across topics also?
> > >
> > > Regards,
> > > Abhishek Singla
> > >
> >
>

Re: Possible bug on Kafka documentation

[Luke Chen]

Hi Federico,

Thanks for reporting the issue.
We've fixed that in v3.5 and later in this PR:
https://github.com/apache/kafka/pull/13115.
But we didn't update for the older versions of docs.
Are you willing to file a PR to kafka-site repo to fix that? Or create a
JIRA issue for it?

Thanks.
Luke


On Wed, Feb 21, 2024 at 4:23 PM Federico Weisse
 wrote:

> In documentation from version 3.1 to version 3.4, it looks like the
> retries explanation has a bug related to
> max.in.flight.request.per.connection related parameter and possible message
> reordering.
> https://kafka.apache.org/31/documentation.html#producerconfigs_retries
> https://kafka.apache.org/32/documentation.html#producerconfigs_retries
> https://kafka.apache.org/33/documentation.html#producerconfigs_retries
> https://kafka.apache.org/34/documentation.html#producerconfigs_retries
>
> in particular, the section
>
> Allowing retries while setting enable.idempotence to false and
> max.in.flight.requests.per.connection to 1 will potentially change the
> ordering of records because if two batches are sent to a single partition,
> and the first fails and is retried but the second succeeds, then the
> records in the second batch may appear first.
>
> Is states
> max.in.flight.requests.per.connection to 1
>
> We think it should said
> max.in.flight.requests.per.connection to greater than  1
>
> That makes the explanation confusing.
>
> This email and any attachments are confidential and must not be disclosed
> to any person other than the intended recipient.
>

Re: Can a controller in a kafka kraft cluster be a bootstrap server

[Luke Chen]

Hi Vikram,

It would be good if you could share client and broker logs for troubleshooting.

Thanks.
Luke

On Wed, Dec 13, 2023 at 1:15 PM Vikram Singh
 wrote:
>
> Hello,
> I have 3 node kafka cluster, when one node goes down for some reason the
> request which are serving on down node is not routing to other running
> node, It takes me always to restart the services,
> Running on kafka version 3.2.1 (kraft mode)
>
> On Mon, Dec 11, 2023 at 12:33 PM Luke Chen  wrote:
>
> > Hi Dima,
> >
> > You can set "process.roles=controller,broker" to get what you want.
> > Otherwise, the controller role cannot be served as a broker.
> >
> > Thanks.
> > Luke
> >
> > On Sat, Dec 9, 2023 at 3:59 AM Dima Brodsky  wrote:
> >
> > > Hello,
> > >
> > > Would the following configuration be valid in a kafka kraft cluster
> > >
> > > So lets say we had the following configs for a controller and a broker:
> > >
> > > === controller -
> > >
> > >
> > https://github.com/apache/kafka/blob/6d1d68617ecd023b787f54aafc24a4232663428d/config/kraft/controller.properties
> > >
> > > process.roles=controller
> > > node.id=1
> > > controller.quorum.voters=1@host1:9093
> > > listeners=CONTROLLER://:9093,BROKER://:9092
> > > controller.listener.names=CONTROLLER
> > >
> > > advertised.listeners=BROKER://:9092,CONTROLLER://:9093
> > > inter.broker.listener.name=BROKER
> > >
> > > === broker -
> > >
> > >
> > https://github.com/apache/kafka/blob/6d1d68617ecd023b787f54aafc24a4232663428d/config/kraft/broker.properties
> > >
> > > process.roles=broker
> > > node.id=2
> > > controller.quorum.voters=1@host1:9093
> > > listeners=BROKER://:9092
> > > inter.broker.listener.name=BROKER
> > > advertised.listeners=BROKER://:9092
> > >
> > > The controller, not only advertises itself as a controller but also as a
> > > broker.  If the controller is contacted by a client as a bootstrap server
> > > will the controller be able to perform the duties of a bootstrap server
> > or
> > > that role has to be left to the brokers.  What I have in green has been
> > > added to the controller config.
> > >
> > > Thanks!
> > > ttyl
> > > Dima
> > >
> >
>
>
> --
> Thanks & Regards
> *VIKRAM S SINGH*

[ANNOUNCE] Apache Kafka 3.5.2

[Luke Chen]

The Apache Kafka community is pleased to announce the release for
Apache Kafka 3.5.2

This is a bugfix release. It contains many bug fixes including
upgrades the Snappy and Rocksdb dependencies.

All of the changes in this release can be found in the release notes:
https://www.apache.org/dist/kafka/3.5.2/RELEASE_NOTES.html


You can download the source and binary release from:
https://kafka.apache.org/downloads#3.5.2

---


Apache Kafka is a distributed streaming platform with four core APIs:


** The Producer API allows an application to publish a stream of records to
one or more Kafka topics.

** The Consumer API allows an application to subscribe to one or more
topics and process the stream of records produced to them.

** The Streams API allows an application to act as a stream processor,
consuming an input stream from one or more topics and producing an
output stream to one or more output topics, effectively transforming the
input streams to output streams.

** The Connector API allows building and running reusable producers or
consumers that connect Kafka topics to existing applications or data
systems. For example, a connector to a relational database might
capture every change to a table.


With these APIs, Kafka can be used for two broad classes of application:

** Building real-time streaming data pipelines that reliably get data
between systems or applications.

** Building real-time streaming applications that transform or react
to the streams of data.


Apache Kafka is in use at large and small companies worldwide, including
Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
Target, The New York Times, Uber, Yelp, and Zalando, among others.

A big thank you for the following contributors to this release!

A. Sophie Blee-Goldman, atu-sharm, bachmanity1, Calvin Liu, Chase
Thomas, Chris Egerton, Colin Patrick McCabe, David Arthur, Divij
Vaidya, Federico Valeri, flashmouse, Florin Akermann, Greg Harris,
hudeqi, José Armando García Sancio, Levani Kokhreidze, Lucas Brutschy,
Luke Chen, Manikumar Reddy, Matthias J. Sax, Mickael Maison, Nick
Telford, Okada Haruki, Omnia G.H Ibrahim, Robert Wagner, Rohan, Said
Boudjelda, sciclon2, Vincent Jiang, Xiaobing Fang, Yash Mayya

We welcome your help and feedback. For more information on how to
report problems, and to get involved, visit the project website at
https://kafka.apache.org/

Thank you!

Regards,
Luke

Re: Kafka frequent disconnection issue

[Luke Chen]

Hi Ankit,

We can't see the log snippet.
But it looks normal to disconnect when connections.max.idle.ms expires.
When increasing the connections.max.idle.ms value, there might be some
activities in the connection during this time (ex: at 5 min 10 sec), so the
idle timer is reset.

Thanks.
Luke

On Fri, Dec 8, 2023 at 11:06 PM Ankit Nigam
 wrote:

> Hi Team,
>
>
>
> We are using Apache Kafka 3.3.1 in our application.  We have created Kafka
> Admin Client , Kafka Producer and Kafka consumer in the application using
> the default properties.
>
>
>
> Once our application starts we are observing below disconnects logs every
> 5 minutes for Admin client and once for Kafka consumer and producer in our
> application. Also at around the same time disconnection logs  are being
> observed in Kafka debug server logs..
>
> Below is log snippet for the same.
>
>
>
> Application logs snippet:-
>
>
>
>
>
> Debug Kafka server.log snippet
>
>
>
>
>
>
>
>
>
> However, if we create Kafka Admin client , producer and consumer by
> overriding default “connections.max.idle.ms” property to a value greater
> default value ( 5 mins ) like 5 mins 2 second or say 10 mins, we observe
> disconnection logs only one disconnection log.
>
> Below are logs for the same.
>
>
>
>
>
> Request you all  to kindly let us know why this disconnection is happening
> every 5 minutes if we don’t override the default “connections.max.idle.ms”
> property and why it disconnects just only once if the time is increased
> beyond 5 mins for the “connections.max.idle.ms” property.?
>
>
>
> Regards,
>
> Ankit Nigam
>
>
>

Re: Can a controller in a kafka kraft cluster be a bootstrap server

[Luke Chen]

Hi Dima,

You can set "process.roles=controller,broker" to get what you want.
Otherwise, the controller role cannot be served as a broker.

Thanks.
Luke

On Sat, Dec 9, 2023 at 3:59 AM Dima Brodsky  wrote:

> Hello,
>
> Would the following configuration be valid in a kafka kraft cluster
>
> So lets say we had the following configs for a controller and a broker:
>
> === controller -
>
> https://github.com/apache/kafka/blob/6d1d68617ecd023b787f54aafc24a4232663428d/config/kraft/controller.properties
>
> process.roles=controller
> node.id=1
> controller.quorum.voters=1@host1:9093
> listeners=CONTROLLER://:9093,BROKER://:9092
> controller.listener.names=CONTROLLER
>
> advertised.listeners=BROKER://:9092,CONTROLLER://:9093
> inter.broker.listener.name=BROKER
>
> === broker -
>
> https://github.com/apache/kafka/blob/6d1d68617ecd023b787f54aafc24a4232663428d/config/kraft/broker.properties
>
> process.roles=broker
> node.id=2
> controller.quorum.voters=1@host1:9093
> listeners=BROKER://:9092
> inter.broker.listener.name=BROKER
> advertised.listeners=BROKER://:9092
>
> The controller, not only advertises itself as a controller but also as a
> broker.  If the controller is contacted by a client as a bootstrap server
> will the controller be able to perform the duties of a bootstrap server or
> that role has to be left to the brokers.  What I have in green has been
> added to the controller config.
>
> Thanks!
> ttyl
> Dima
>

Re: [ANNOUNCE] Apache Kafka 3.6.1

[Luke Chen]

Hi Mickael,

Thanks for running this release!

Luke

On Thu, Dec 7, 2023 at 7:13 PM Mickael Maison  wrote:

> The Apache Kafka community is pleased to announce the release for
> Apache Kafka 3.6.1
>
> This is a bug fix release and it includes fixes and improvements from 30
> JIRAs.
>
> All of the changes in this release can be found in the release notes:
> https://www.apache.org/dist/kafka/3.6.1/RELEASE_NOTES.html
>
> You can download the source and binary release (Scala 2.12 and Scala 2.13)
> from:
> https://kafka.apache.org/downloads#3.6.1
>
>
> ---
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
> ** The Producer API allows an application to publish a stream of records to
> one or more Kafka topics.
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an
> output stream to one or more output topics, effectively transforming the
> input streams to output streams.
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might
> capture every change to a table.
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
> ** Building real-time streaming applications that transform or react
> to the streams of data.
>
>
> Apache Kafka is in use at large and small companies worldwide, including
> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> Target, The New York Times, Uber, Yelp, and Zalando, among others.
>
> A big thank you for the following 39 contributors to this release!
> (Please report an unintended omission)
>
> Anna Sophie Blee-Goldman, Arpit Goyal, atu-sharm, Bill Bejeck, Chris
> Egerton, Colin P. McCabe, David Arthur, David Jacot, Divij Vaidya,
> Federico Valeri, Greg Harris, Guozhang Wang, Hao Li, hudeqi,
> iit2009060, Ismael Juma, Jorge Esteban Quilcate Otoya, Josep Prat,
> Jotaniya Jeel, Justine Olshan, Kamal Chandraprakash, kumarpritam863,
> Levani Kokhreidze, Lucas Brutschy, Luke Chen, Manikumar Reddy,
> Matthias J. Sax, Mayank Shekhar Narula, Mickael Maison, Nick Telford,
> Philip Nee, Qichao Chu, Rajini Sivaram, Robert Wagner, Sagar Rao,
> Satish Duggana, Walker Carlson, Xiaobing Fang, Yash Mayya
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> https://kafka.apache.org/
>
> Thank you!
>
> Regards,
> Mickael
>

Re: Kafka 2.7.2 to 3.5.1 upgrade

[Luke Chen]

Hi Lud,

This is a known issue(KAFKA-15353
) and I've fixed it in
v3.5.2 (will get released soon) and v3.6.0.

Thanks.
Luke

On Mon, Dec 4, 2023 at 6:01 PM Lud Antonie 
wrote:

> Hi Megh,
>
> No, the number of partitions haven't increased.
> The upgrade is done in a test environment, the topic is just created just
> before the upgrade.
>
> As a test I performed a rollback to 2.7.2 and did an upgrade to 3.4.1.
> This upgrade went ok i.e. without under replicated partitions.
>
> Best regards,
> Lud
>
> Op za 2 dec 2023 om 04:08 schreef megh vidani :
>
> > Hi Lud,
> >
> > The topics for which you're seeing under replicated partitions, Did you
> try
> > to increase the number of partitions anytime after creation of those
> topics
> > before the upgrade?
> >
> > We have earlier faced issues with 2.8.0, in which we had increased the
> > number of partitions for some topics, and for those topics we used to see
> > under replicated partitions after every restart.
> >
> > The reason this happened was, there was a bug in Kafka which assigned a
> new
> > topicId (different from the original topicId) to newly added partitions
> in
> > the partition.metadata file, and upon restart of kafka brokers, this
> > topicId didn't reconcile between brokers and ZK.
> >
> > Thanks,
> > Megh
> >
> > On Thu, Nov 30, 2023, 20:10 Lud Antonie 
> > wrote:
> >
> > > Hello,
> > >
> > > After upgrading from 2.7.2 to 3.5.1 some topics are missing a partition
> > for
> > > one or two brokers.
> > > The kafka manager shows "Under replicated%" for the topic.
> > > Looking at the topic for some brokers (of 3) partitions are missing (in
> > my
> > > case 1 partition).
> > > A rollback will restore the "Under replicated%" to 0 again (this is the
> > > wanted number).
> > >
> > > Is this a bug of kafka or the kafka manager?
> > >
> > > Best regards,
> > > Lud Antonie
> > >
> > >
> > > --
> > > Met vriendelijke groet / Kind regards,
> > >
> > > *Lud Antonie*
> > >
> > > 
> > > Kennedyplein 101, 5611 ZS, Eindhoven
> > > +31(0)402492700 <0031402492700>
> > > www.coosto.com
> > >  <
> https://www.facebook.com/coosto.solution
> > >
> > > 
> > > 
> > >
> >
>
>
> --
> Met vriendelijke groet / Kind regards,
>
> *Lud Antonie*
>
> 
> Kennedyplein 101, 5611 ZS, Eindhoven
> +31(0)402492700 <0031402492700>
> www.coosto.com
>  
> 
> 
>

[VOTE] 3.5.2 RC1

[Luke Chen]

Hello Kafka users, developers and client-developers,

This is the first candidate for release of Apache Kafka 3.5.2.

This is a bugfix release with several fixes since the release of 3.5.1,
including dependency version bumps for CVEs.

Release notes for the 3.5.2 release:
https://home.apache.org/~showuon/kafka-3.5.2-rc1/RELEASE_NOTES.html

*** Please download, test and vote by Nov. 28.

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~showuon/kafka-3.5.2-rc1/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~showuon/kafka-3.5.2-rc1/javadoc/

* Tag to be voted upon (off 3.5 branch) is the 3.5.2 tag:
https://github.com/apache/kafka/releases/tag/3.5.2-rc1

* Documentation:
https://kafka.apache.org/35/documentation.html

* Protocol:
https://kafka.apache.org/35/protocol.html

* Successful Jenkins builds for the 3.5 branch:
Unit/integration tests:
https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.5/98/
There are some falky tests, including the testSingleIP test failure. It
failed because of some infra change and we fixed it
 recently.

System tests: running, will update the results later.



Thank you.
Luke

Re: How to dynamically change configurations in the controllers

[Luke Chen]

Hi Jesus,

KIP-919 is what you're looking for:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-919%3A+Allow+AdminClient+to+Talk+Directly+with+the+KRaft+Controller+Quorum+and+add+Controller+Registration

This feature will be included in next release (i.e. Kafka v3.7.0).

Thanks.
Luke

On Thu, Nov 16, 2023 at 9:01 AM Jesus Cea  wrote:

> Kafka 3.6.0.
>
> The tool "kafka-configs.sh" is able to read and change configuration in
> the brokers, but I am unable to read/change configurations in the Kraft
> controllers. How is that done?
>
> I am interested, for instance, in being able to update the TLS
> certificates.
>
> Help!
>
> Thanks.
>
> --
> Jesús Cea Avión _/_/  _/_/_/_/_/_/
> j...@jcea.es - https://www.jcea.es/_/_/_/_/  _/_/_/_/  _/_/
> Twitter: @jcea_/_/_/_/  _/_/_/_/_/
> jabber / xmpp:j...@jabber.org  _/_/  _/_/_/_/  _/_/  _/_/
> "Things are not so easy"  _/_/  _/_/_/_/  _/_/_/_/  _/_/
> "My name is Dump, Core Dump"   _/_/_/_/_/_/  _/_/  _/_/
> "El amor es poner tu felicidad en la felicidad de otro" - Leibniz
>

Re: [ANNOUNCE] Apache Kafka 3.6.0

[Luke Chen]

Hi Ryan,

Thanks for noticing that.
There's already a PR opened for this issue (thanks, Federico!):
https://github.com/apache/kafka/pull/14534

Thank you.
Luke

On Fri, Oct 13, 2023 at 2:08 AM Ryan Leslie (BLP/ NEW YORK (REMOT) <
rles...@bloomberg.net> wrote:

> It seems that the 3.6.0 upgrade documentation is still missing, i.e.
> rolling restart. There is only the Notable Changes section:
>
> https://kafka.apache.org/documentation/#upgrade_3_6_0
>
>
> From: users@kafka.apache.org At: 10/12/23 11:16:20 UTC-4:00To:
> d...@kafka.apache.org
> Cc:  users@kafka.apache.org
> Subject: Re: [ANNOUNCE] Apache Kafka 3.6.0
>
> Congratulations to the community on an exciting release! Special thanks to
> Satish for driving the release and KIP-405. :)
>
> Ismael
>
> On Tue, Oct 10, 2023 at 10:39 PM Satish Duggana 
> wrote:
>
> > The Apache Kafka community is pleased to announce the release for
> > Apache Kafka 3.6.0
> >
> > This is a minor release and it includes fixes and improvements from 238
> > JIRAs.
> >
> > All of the changes in this release can be found in the release notes:
> > https://www.apache.org/dist/kafka/3.6.0/RELEASE_NOTES.html
> >
> > An overview of the release can be found in our announcement blog post:
> > https://kafka.apache.org/blog
> >
> > You can download the source and binary release (Scala 2.12 and Scala
> 2.13)
> > from:
> > https://kafka.apache.org/downloads#3.6.0
> >
> >
> >
>
> 
> ---
> >
> >
> > Apache Kafka is a distributed streaming platform with four core APIs:
> >
> >
> > ** The Producer API allows an application to publish a stream of records
> to
> > one or more Kafka topics.
> >
> > ** The Consumer API allows an application to subscribe to one or more
> > topics and process the stream of records produced to them.
> >
> > ** The Streams API allows an application to act as a stream processor,
> > consuming an input stream from one or more topics and producing an
> > output stream to one or more output topics, effectively transforming the
> > input streams to output streams.
> >
> > ** The Connector API allows building and running reusable producers or
> > consumers that connect Kafka topics to existing applications or data
> > systems. For example, a connector to a relational database might
> > capture every change to a table.
> >
> >
> > With these APIs, Kafka can be used for two broad classes of application:
> >
> > ** Building real-time streaming data pipelines that reliably get data
> > between systems or applications.
> >
> > ** Building real-time streaming applications that transform or react
> > to the streams of data.
> >
> >
> > Apache Kafka is in use at large and small companies worldwide, including
> > Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> > Target, The New York Times, Uber, Yelp, and Zalando, among others.
> >
> > A big thank you for the following 139 contributors to this release!
> > (Please report an unintended omission)
> >
> > This was a community effort, so thank you to everyone who contributed
> > to this release, including all our users and our 139 contributors:
> > A. Sophie Blee-Goldman, Aaron Ai, Abhijeet Kumar, aindriu-aiven,
> > Akhilesh Chaganti, Alexandre Dupriez, Alexandre Garnier, Alok
> > Thatikunta, Alyssa Huang, Aman Singh, Andras Katona, Andrew Schofield,
> > Andrew Grant, Aneel Kumar, Anton Agestam, Artem Livshits, atu-sharm,
> > bachmanity1, Bill Bejeck, Bo Gao, Bruno Cadonna, Calvin Liu, Chaitanya
> > Mukka, Chase Thomas, Cheryl Simmons, Chia-Ping Tsai, Chris Egerton,
> > Christo Lolov, Clay Johnson, Colin P. McCabe, Colt McNealy, d00791190,
> > Damon Xie, Danica Fine, Daniel Scanteianu, Daniel Urban, David Arthur,
> > David Jacot, David Mao, dengziming, Deqi Hu, Dimitar Dimitrov, Divij
> > Vaidya, DL1231, Dániel Urbán, Erik van Oosten, ezio, Farooq Qaiser,
> > Federico Valeri, flashmouse, Florin Akermann, Gabriel Oliveira,
> > Gantigmaa Selenge, Gaurav Narula, GeunJae Jeon, Greg Harris, Guozhang
> > Wang, Hailey Ni, Hao Li, Hector Geraldino, hudeqi, hzh0425, Iblis Lin,
> > iit2009060, Ismael Juma, Ivan Yurchenko, James Shaw, Jason Gustafson,
> > Jeff Kim, Jim Galasyn, John Roesler, Joobi S B, Jorge Esteban Quilcate
> > Otoya, Josep Prat, Joseph (Ting-Chou) Lin, José Armando García Sancio,
> > Jun Rao, Justine Olshan, Kamal Chandraprakash, Keith Wall, Kirk True,
> > Lianet Magrans, LinShunKa

Re: [ANNOUNCE] Apache Kafka 3.6.0

[Luke Chen]

Thanks for running the release, Satish!

BTW, 3.6.0 should be a major release, not a minor one. :)

Luke

On Wed, Oct 11, 2023 at 1:39 PM Satish Duggana  wrote:

> The Apache Kafka community is pleased to announce the release for
> Apache Kafka 3.6.0
>
> This is a minor release and it includes fixes and improvements from 238
> JIRAs.
>
> All of the changes in this release can be found in the release notes:
> https://www.apache.org/dist/kafka/3.6.0/RELEASE_NOTES.html
>
> An overview of the release can be found in our announcement blog post:
> https://kafka.apache.org/blog
>
> You can download the source and binary release (Scala 2.12 and Scala 2.13)
> from:
> https://kafka.apache.org/downloads#3.6.0
>
>
> ---
>
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
>
> ** The Producer API allows an application to publish a stream of records to
> one or more Kafka topics.
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an
> output stream to one or more output topics, effectively transforming the
> input streams to output streams.
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might
> capture every change to a table.
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
> ** Building real-time streaming applications that transform or react
> to the streams of data.
>
>
> Apache Kafka is in use at large and small companies worldwide, including
> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> Target, The New York Times, Uber, Yelp, and Zalando, among others.
>
> A big thank you for the following 139 contributors to this release!
> (Please report an unintended omission)
>
> This was a community effort, so thank you to everyone who contributed
> to this release, including all our users and our 139 contributors:
> A. Sophie Blee-Goldman, Aaron Ai, Abhijeet Kumar, aindriu-aiven,
> Akhilesh Chaganti, Alexandre Dupriez, Alexandre Garnier, Alok
> Thatikunta, Alyssa Huang, Aman Singh, Andras Katona, Andrew Schofield,
> Andrew Grant, Aneel Kumar, Anton Agestam, Artem Livshits, atu-sharm,
> bachmanity1, Bill Bejeck, Bo Gao, Bruno Cadonna, Calvin Liu, Chaitanya
> Mukka, Chase Thomas, Cheryl Simmons, Chia-Ping Tsai, Chris Egerton,
> Christo Lolov, Clay Johnson, Colin P. McCabe, Colt McNealy, d00791190,
> Damon Xie, Danica Fine, Daniel Scanteianu, Daniel Urban, David Arthur,
> David Jacot, David Mao, dengziming, Deqi Hu, Dimitar Dimitrov, Divij
> Vaidya, DL1231, Dániel Urbán, Erik van Oosten, ezio, Farooq Qaiser,
> Federico Valeri, flashmouse, Florin Akermann, Gabriel Oliveira,
> Gantigmaa Selenge, Gaurav Narula, GeunJae Jeon, Greg Harris, Guozhang
> Wang, Hailey Ni, Hao Li, Hector Geraldino, hudeqi, hzh0425, Iblis Lin,
> iit2009060, Ismael Juma, Ivan Yurchenko, James Shaw, Jason Gustafson,
> Jeff Kim, Jim Galasyn, John Roesler, Joobi S B, Jorge Esteban Quilcate
> Otoya, Josep Prat, Joseph (Ting-Chou) Lin, José Armando García Sancio,
> Jun Rao, Justine Olshan, Kamal Chandraprakash, Keith Wall, Kirk True,
> Lianet Magrans, LinShunKang, Liu Zeyu, lixy, Lucas Bradstreet, Lucas
> Brutschy, Lucent-Wong, Lucia Cerchie, Luke Chen, Manikumar Reddy,
> Manyanda Chitimbo, Maros Orsak, Matthew de Detrich, Matthias J. Sax,
> maulin-vasavada, Max Riedel, Mehari Beyene, Michal Cabak (@miccab),
> Mickael Maison, Milind Mantri, minjian.cai, mojh7, Nikolay, Okada
> Haruki, Omnia G H Ibrahim, Owen Leung, Philip Nee, prasanthV, Proven
> Provenzano, Purshotam Chauhan, Qichao Chu, Rajini Sivaram, Randall
> Hauch, Renaldo Baur Filho, Ritika Reddy, Rittika Adhikari, Rohan, Ron
> Dagostino, Sagar Rao, Said Boudjelda, Sambhav Jain, Satish Duggana,
> sciclon2, Shekhar Rajak, Sungyun Hur, Sushant Mahajan, Tanay
> Karmarkar, tison, Tom Bentley, vamossagar12, Victoria Xia, Vincent
> Jiang, vveicc, Walker Carlson, Yash Mayya, Yi-Sheng Lien, Ziming Deng,
> 蓝士钦
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> https://kafka.apache.org/
>
> Thank you!
>
> Regards,
> Satish Duggana
>

Re: Kafka Protocol : Compact Array or Array ?

[Luke Chen]

Hi Neeraj,

Yes, for MetadataRequest, version 0 ~ 8, the topic is ARRAY type. After
version 9, it'll be COMPACT_ARRAY.
It's because of this definition: "flexibleVersions": "9+".
You can check KIP-482 for more information:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-482

Thanks.
Luke

On Mon, Oct 2, 2023 at 5:09 AM Neeraj Vaidya
 wrote:

>  Hi All,
> I've raised this on StackOverflow as well :
> https://stackoverflow.com/questions/77208870/kafka-binary-protocol-array-or-compact-array
> In case someone can help in answering that question.
>
> Regards,
> Neeraj
>  On Sunday, 1 October, 2023 at 11:32:49 am GMT+11, Neeraj Vaidya <
> neeraj.vai...@yahoo.co.in> wrote:
>
>  Hi All,
> There are 2 types of arrays specified in the Kafka protocol documentation
> : ARRAY and COMPACT_ARRAY.
> But in the protocol details for the different messages, it does not
> explicitly specify if the array type is which one of the above.
>
> For example, the BNF grammar for the section for MetadataRequest API is as
> below :
>
> Metadata Request (Version: 0) => [topics]
>   topics => name
> name => STRING
>
> What is the type of [topics] ? Is it ARRAY or COMPACT_ARRAY ?
>
> After playing around with the protocol using some tests, I think for
> Version:0 of this API request, the broker expects this to be of type ARRAY.
>
> But for higher versions, say v9, COMPACT_ARRAY is expected.
>
> I think the protocol really needs to be explicit and is lacking in this
> respect.
>
> Regards,
> Neeraj

Re: [VOTE] 3.6.0 RC2

[Luke Chen]

Hi Satish,

I verified with:
1. Ran quick start in KRaft for scala 2.12 artifact
2. Making sure the checksum are correct
3. Browsing release notes, documents, javadocs, protocols.
4. Verified the tiered storage feature works well.

+1 (binding).

Thanks.
Luke



On Mon, Oct 2, 2023 at 5:23 AM Jakub Scholz  wrote:

> +1 (non-binding). I used the Scala 2.13 binaries and the staged Maven
> artifacts and run my tests. Everything seems to work fine for me.
>
> Thanks
> Jakub
>
> On Fri, Sep 29, 2023 at 8:17 PM Satish Duggana 
> wrote:
>
> > Hello Kafka users, developers and client-developers,
> >
> > This is the third candidate for the release of Apache Kafka 3.6.0.
> > Some of the major features include:
> >
> > * KIP-405 : Kafka Tiered Storage
> > * KIP-868 : KRaft Metadata Transactions
> > * KIP-875: First-class offsets support in Kafka Connect
> > * KIP-898: Modernize Connect plugin discovery
> > * KIP-938: Add more metrics for measuring KRaft performance
> > * KIP-902: Upgrade Zookeeper to 3.8.1
> > * KIP-917: Additional custom metadata for remote log segment
> >
> > Release notes for the 3.6.0 release:
> > https://home.apache.org/~satishd/kafka-3.6.0-rc2/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Tuesday, October 3, 12pm PT
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~satishd/kafka-3.6.0-rc2/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://home.apache.org/~satishd/kafka-3.6.0-rc2/javadoc/
> >
> > * Tag to be voted upon (off 3.6 branch) is the 3.6.0-rc2 tag:
> > https://github.com/apache/kafka/releases/tag/3.6.0-rc2
> >
> > * Documentation:
> > https://kafka.apache.org/36/documentation.html
> >
> > * Protocol:
> > https://kafka.apache.org/36/protocol.html
> >
> > * Successful Jenkins builds for the 3.6 branch:
> > There are a few runs of unit/integration tests. You can see the latest
> > at https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/. We will
> > continue running a few more iterations.
> > System tests:
> > We will send an update once we have the results.
> >
> > Thanks,
> > Satish.
> >
>

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Luke Chen]

Hi Jose,

Sounds good to me.
Let's have further discussion in JIRA/PR, and target to v3.6.1/v3.7.0.

Thanks.
Luke

On Tue, Sep 26, 2023 at 1:35 AM José Armando García Sancio
 wrote:

> On Sat, Sep 23, 2023 at 3:08 AM Luke Chen  wrote:
> >
> > Hi Satish,
> >
> > I found the current KRaft implementation will have "split brain" issue
> when
> > network partition happens, which will cause inconsistent metadata
> returned
> > from the controller.
> > Filed KAFKA-15489 <https://issues.apache.org/jira/browse/KAFKA-15489>
> for
> > this issue, and PR <https://github.com/apache/kafka/pull/14428> is ready
> > for review.
> >
> > Even though this is not a regression issue (this has already existed
> since
> > the 1st release of KRaft feature), I think this is an important issue
> since
> > KRaft is announced production ready.
> > Not sure what other people's thoughts are.
>
> Thanks for the report and PR Luke. This looks related to this issue:
> https://issues.apache.org/jira/browse/KAFKA-13621
>
> Do you agree? We can move our conversation to those issues but I also
> agree that I don't think this issue should be a release blocker.
>
> Thanks!
> -José
>

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Luke Chen]

Hi Satish,

I verified with:
1. Ran quick start in KRaft for scala 2.12 artifact
2. Making sure the checksum are correct
3. Browsing release notes, documents, javadocs, protocols.

I filed KAFKA-15491 <https://issues.apache.org/jira/browse/KAFKA-15491>for
log output improvement while testing stream application.
It won't be blocker in v3.6.0.

For KAFKA-15489 <https://issues.apache.org/jira/browse/KAFKA-15489>, I'm
fine if we decide to fix it in v3.6.1/v3.7.0.

+1 (binding) from me.

Thank you.
Luke

On Sun, Sep 24, 2023 at 3:38 AM Ismael Juma  wrote:

> Given that this is not a regression and there have been no reports for over
> a year, I think it's ok for this to land in 3.6.1.
>
> Ismael
>
> On Sat, Sep 23, 2023 at 9:32 AM Satish Duggana 
> wrote:
>
> > Thanks Luke for reporting KRaft issue[1].
> >
> > I am not sure whether it is a release blocker for 3.6.0. Need input
> > from other KRaft experts also to finalize the decision. Even if we
> > adopt a fix, do not we need to bake it for some time before it is
> > pushed to production to avoid any regressions as this change is in the
> > critical paths?
> >
> > 1. https://issues.apache.org/jira/browse/KAFKA-15489
> >
> > Thanks,
> > Satish.
> >
> > On Sat, 23 Sept 2023 at 03:08, Luke Chen  wrote:
> > >
> > > Hi Satish,
> > >
> > > I found the current KRaft implementation will have "split brain" issue
> > when
> > > network partition happens, which will cause inconsistent metadata
> > returned
> > > from the controller.
> > > Filed KAFKA-15489 <https://issues.apache.org/jira/browse/KAFKA-15489>
> > for
> > > this issue, and PR <https://github.com/apache/kafka/pull/14428> is
> ready
> > > for review.
> > >
> > > Even though this is not a regression issue (this has already existed
> > since
> > > the 1st release of KRaft feature), I think this is an important issue
> > since
> > > KRaft is announced production ready.
> > > Not sure what other people's thoughts are.
> > >
> > > Thank you.
> > > Luke
> > >
> > > On Thu, Sep 21, 2023 at 6:33 PM Josep Prat  >
> > > wrote:
> > >
> > > > Hi Satish,
> > > >
> > > > I ran the following validation steps:
> > > > - Built from source with Java 11 and Scala 2.13
> > > > - Verified Signatures and hashes of the artifacts generated
> > > > - Navigated through Javadoc including links to JDK classes
> > > > - Run the unit tests
> > > > - Run integration tests
> > > > - Run the quickstart in KRaft and Zookeeper mode
> > > >
> > > >
> > > > I +1 this release (non-binding)
> > > >
> > > > Thanks for your efforts!
> > > >
> > > > On Thu, Sep 21, 2023 at 2:59 AM Satish Duggana <
> > satish.dugg...@gmail.com>
> > > > wrote:
> > > >
> > > > > Thanks Greg for verifying the release including the earlier
> > > > > blocker(KAFKA-15473) verification.
> > > > >
> > > > > ~Satish.
> > > > >
> > > > > On Wed, 20 Sept 2023 at 22:30, Greg Harris
> >  > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I verified the functionality of KIP-898 and the recent fix for
> > > > > > KAFKA-15473 with the following steps:
> > > > > >
> > > > > > 1. I started a 3.5.1 broker, and a 3.5.1 worker with most (>400)
> > > > > > publicly available plugins installed
> > > > > > 2. I captured the output of /connector-plugins
> > > > > > 3. I upgraded the worker to 3.6.0-rc1
> > > > > > 4. I captured the output of /connector-plugins with various
> > settings
> > > > > > of plugin.discovery
> > > > > > 5. I ran the migration script to add manifests to my plugins
> > > > > > 6. I captured the output of /connector-plugins with various
> > settings
> > > > > > of plugin.discovery
> > > > > > 7. I downgraded the worker to 3.5.1
> > > > > > 8. I diffed the output of /connector-plugins across the different
> > > > > > cases and observed the expected changes.
> > > > > > a. When plugins are migrated for 3.6.0, all modes produce
> > identical
> > > > > > res

Re: [kafka-clients] [VOTE] 3.6.0 RC1

[Luke Chen]

Hi Satish,

I found the current KRaft implementation will have "split brain" issue when
network partition happens, which will cause inconsistent metadata returned
from the controller.
Filed KAFKA-15489  for
this issue, and PR  is ready
for review.

Even though this is not a regression issue (this has already existed since
the 1st release of KRaft feature), I think this is an important issue since
KRaft is announced production ready.
Not sure what other people's thoughts are.

Thank you.
Luke

On Thu, Sep 21, 2023 at 6:33 PM Josep Prat 
wrote:

> Hi Satish,
>
> I ran the following validation steps:
> - Built from source with Java 11 and Scala 2.13
> - Verified Signatures and hashes of the artifacts generated
> - Navigated through Javadoc including links to JDK classes
> - Run the unit tests
> - Run integration tests
> - Run the quickstart in KRaft and Zookeeper mode
>
>
> I +1 this release (non-binding)
>
> Thanks for your efforts!
>
> On Thu, Sep 21, 2023 at 2:59 AM Satish Duggana 
> wrote:
>
> > Thanks Greg for verifying the release including the earlier
> > blocker(KAFKA-15473) verification.
> >
> > ~Satish.
> >
> > On Wed, 20 Sept 2023 at 22:30, Greg Harris  >
> > wrote:
> >
> > > Hi all,
> > >
> > > I verified the functionality of KIP-898 and the recent fix for
> > > KAFKA-15473 with the following steps:
> > >
> > > 1. I started a 3.5.1 broker, and a 3.5.1 worker with most (>400)
> > > publicly available plugins installed
> > > 2. I captured the output of /connector-plugins
> > > 3. I upgraded the worker to 3.6.0-rc1
> > > 4. I captured the output of /connector-plugins with various settings
> > > of plugin.discovery
> > > 5. I ran the migration script to add manifests to my plugins
> > > 6. I captured the output of /connector-plugins with various settings
> > > of plugin.discovery
> > > 7. I downgraded the worker to 3.5.1
> > > 8. I diffed the output of /connector-plugins across the different
> > > cases and observed the expected changes.
> > > a. When plugins are migrated for 3.6.0, all modes produce identical
> > > results.
> > > b. When plugins are not migrated for 3.6.0, only_scan and
> > > hybrid_warn produce identical results, hybrid_fail crashes, and
> > > service_load is missing plugins
> > > c. When upgrading from 3.5.1 I see that plugins with invalid
> > > constructors are hidden, AK plugins now have versions, multi-interface
> > > plugins now show each interface type, and plugins using AppInfoParser
> > > change versions.
> > > d. The startup logs now include descriptive errors for invalid
> > > plugins that otherwise would have been thrown at runtime
> > > d. The fix for KAFKA-15473 prevents duplicates
> > > e. The output for 3.5.1 after downgrading is identical to before.
> > >
> > > +1 (non-binding)
> > >
> > > Thanks Satish for running the release!
> > >
> > > On Wed, Sep 20, 2023 at 8:36 AM Divij Vaidya 
> wrote:
> > > >
> > > > Hey Satish
> > > >
> > > > My comments about documentation misses from RC0 vote thread [1] are
> > > > still not addressed (such as missing metric documentation, formatting
> > > > problems etc). Could you please mention why we shouldn't consider
> them
> > > > as blockers to make RC1 as the final release?
> > > >
> > > > [1] https://lists.apache.org/thread/cokoxzd0jtgjtrlxoq7kkzmvpm75381t
> > > >
> > > > On Wed, Sep 20, 2023 at 4:53 PM Satish Duggana <
> > satish.dugg...@gmail.com>
> > > wrote:
> > > > >
> > > > > Hello Kafka users, developers and client-developers,
> > > > >
> > > > > This is the second candidate for the release of Apache Kafka 3.6.0.
> > > Some of the major features include:
> > > > >
> > > > > * KIP-405 : Kafka Tiered Storage
> > > > > * KIP-868 : KRaft Metadata Transactions
> > > > > * KIP-875: First-class offsets support in Kafka Connect
> > > > > * KIP-898: Modernize Connect plugin discovery
> > > > > * KIP-938: Add more metrics for measuring KRaft performance
> > > > > * KIP-902: Upgrade Zookeeper to 3.8.1
> > > > > * KIP-917: Additional custom metadata for remote log segment
> > > > >
> > > > > Release notes for the 3.6.0 release:
> > > > >
> https://home.apache.org/~satishd/kafka-3.6.0-rc1/RELEASE_NOTES.html
> > > > >
> > > > > *** Please download, test and vote by Saturday, September 23, 8am
> PT
> > > > >
> > > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > > https://kafka.apache.org/KEYS
> > > > >
> > > > > * Release artifacts to be voted upon (source and binary):
> > > > > https://home.apache.org/~satishd/kafka-3.6.0-rc1/
> > > > >
> > > > > * Maven artifacts to be voted upon:
> > > > >
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > >
> > > > > * Javadoc:
> > > > > https://home.apache.org/~satishd/kafka-3.6.0-rc1/javadoc/
> > > > >
> > > > > * Tag to be voted upon (off 3.6 branch) is the 3.6.0 tag:
> > > > > 

Re: Ruquest for Jira assign permissions

[Luke Chen]

Hi Romulo,

You should be able to assign tasks to yourselves.

Thanks.
Luke

On Sat, Aug 5, 2023 at 9:10 PM Romulo Oliveira 
wrote:

> I added a new Jira task in order to contribute to a new feature for Kafka
> Connectors TimestampConverter transformer:
> https://issues.apache.org/jira/browse/KAFKA-15310.
> I intend to work on it, but I do not have permission to assign this task to
> my user. Can someone add the permission or at least the assign the task to
> me?
>
> Romulo Souza
>

Re: Unable to launch KRaft

[Luke Chen]

Hi Adrian,

Please follow the quick start guide in Kafka doc:
https://kafka.apache.org/quickstart

Thanks.
Luke

On Fri, Jul 28, 2023 at 3:53 PM adrien ruffie 
wrote:

> dear all,
>
> I would like to try KRaft in  but I didn't arrived to finish the
> installation bellow:
>
>
> https://developer.confluent.io/learn/kraft/?utm_medium=sem_source=google_campaign=ch.sem_br.nonbrand_tp.prs_tgt.dsa_mt.dsa_rgn.emea_lng.eng_dv.all_con.confluent-developer_term===c==1=CjwKCAjwzo2mBhAUEiwAf7wjknGluNtB_3DDul3Qrb5Usv_nybLBLkeoqMnczaAmdhRUk6nGh-WisxoCG2wQAvD_BwE
>
>
> When I try ./bin/kafka-server-start.sh -daemon
> ./config/kraft/server.propertiesnothing happen consequently I tried
> without daemon et after that I got the following error:
>
>
>
> I tried after to make a kafka-storage.sh but without providing any
> arguments, the raised error was "Too few arguments" ...
> anyone have a idea how I can launch correctly kafka with KRaft plz ? I'm
> on an hyperv machin with ubuntu 20.04.2 LTS
>
> Thank a lot and best regards
>
> Adrian
>

Re: [VOTE] 3.5.1 RC1

[Luke Chen]

Hi Divij,

I've run:
1. Download kafka_2.12-3.5.1.tgz
2. Run quick start using KRaft mode
3. Verified the checksum
4. Sanity check the javadoc

All looks good.
+1 (binding)

Thanks.
Luke

On Tue, Jul 18, 2023 at 5:15 AM Chris Egerton 
wrote:

> Hi Divij,
>
> Thanks for running this release!
>
> To verify, I:
> - Built from source using Java 11 with both:
> - - the 3.5.1-rc1 tag on GitHub
> - - the kafka-3.5.1-src.tgz artifact from
> https://home.apache.org/~divijv/kafka-3.5.1-rc1/
> - Checked signatures and checksums
> - Ran the quickstart using the kafka_2.13-3.5.1.tgz artifact from
> https://home.apache.org/~divijv/kafka-3.5.1-rc1/ with Java 11 and Scala 13
> in KRaft mode
> - Ran all unit tests
> - Ran all integration tests for Connect and MM2
> - Verified that only version 1.1.10.1 of Snappy is present in the libs/
> directory of the unpacked kafka_2.12-3.5.1.tgz and kafka_2.13-3.5.1.tgz
> artifacts
> - Verified that case-insensitive validation of the security.protocol
> property is restored for Kafka clients by setting it to "pLAiNTexT" with
> the bin/kafka-topics.sh command (using the --command-config option), and
> with a standalone Connect worker (by adjusting the security.protocol,
> consumer.security.protocol, producer.security.protocol, and
> admin.security.protocol properties in the worker config file)
>
> Everything looks good to me!
>
> +1 (binding)
>
> Cheers,
>
> Chris
>
> On Mon, Jul 17, 2023 at 12:29 PM Federico Valeri 
> wrote:
>
> > Hi Divij, I did the following checks:
> >
> > - Checked signature, checksum, licenses
> > - Spot checked documentation and javadoc
> > - Built from source with Java 17 and Scala 2.13
> > - Ran full unit and integration test suites
> > - Ran test Java app using staging Maven artifacts
> >
> > +1 (non binding)
> >
> > Cheers
> > Fede
> >
> > On Mon, Jul 17, 2023 at 10:27 AM Divij Vaidya 
> > wrote:
> > >
> > > Hello Kafka users, developers and client-developers,
> > >
> > > This is the second candidate (RC1) for release of Apache Kafka 3.5.1.
> > First
> > > release candidate (RC0) was discarded due to incorrect license files.
> > They
> > > have been fixed since then.
> > >
> > > This release is a security patch release. It upgrades the dependency,
> > > snappy-java, to a version which is not vulnerable to CVE-2023-34455.
> You
> > > can find more information about the CVE at Kafka CVE list
> > > .
> > >
> > > Additionally, this releases fixes a regression introduced in 3.3.0,
> which
> > > caused security.protocol configuration values to be restricted to upper
> > > case only. With this release, security.protocol values are
> > > case insensitive. See KAFKA-15053
> > >  for details.
> > >
> > > Release notes for the 3.5.1 release:
> > > https://home.apache.org/~divijv/kafka-3.5.1-rc1/RELEASE_NOTES.html
> > >
> > > *** Please download, test and vote by Thursday, July 20, 9am PT
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > https://kafka.apache.org/KEYS
> > >
> > > Release artifacts to be voted upon (source and binary):
> > > https://home.apache.org/~divijv/kafka-3.5.1-rc1/
> > >
> > > Maven artifacts to be voted upon:
> > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >
> > > Javadoc:
> > > https://home.apache.org/~divijv/kafka-3.5.1-rc1/javadoc/
> > >
> > > Tag to be voted upon (off 3.5 branch) is the 3.5.1 tag:
> > > https://github.com/apache/kafka/releases/tag/3.5.1-rc1
> > >
> > > Documentation:
> > > https://kafka.apache.org/35/documentation.html
> > > Please note that documentation will be updated with upgrade notes (
> > >
> >
> https://github.com/apache/kafka/commit/4c78fd64454e25e3536e8c7ed5725d3fbe944a49
> > )
> > > after the release is complete.
> > >
> > > Protocol:
> > > https://kafka.apache.org/35/protocol.html
> > >
> > > Unit/integration tests:
> > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.5/43/ (2
> > failures)
> > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.5/42/ (6
> > failures)
> > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.5/39/ (9
> > failures)
> > >
> > > In all 3 runs above, there are no common tests which are failing, which
> > > leads me to believe that they are flaky. I have also verified that
> > > unit/integration tests on my local machine successfully pass (JDK 17 +
> > > Scala 2.13)
> > >
> > > System tests:
> > > Not planning to run system tests since this is a patch release.
> > >
> > > Thank you.
> > >
> > > --
> > > Divij Vaidya
> > > Release Manager for Apache Kafka 3.5.1
> >
>

Re: Replication factor for a topic Increase

[Luke Chen]

Once the topic is created, you can only increase the replication factor by:
1. delete the topic and re-create the same topic (note: data will be lost
after this step)
or
2. follow the doc to increase the replication factor (data will be kept)

Thank you.
Luke

On Thu, Jul 13, 2023 at 5:23 PM Gaurav Pande  wrote:

> Hi Luke ,
>
> Thanks does that mean that setting default.replication.factor to more than
> 1 and doing Kafka broker restart wouldn't dynamically change the existing
> partitions replication? We have to use cli shell script, as per
> documentation?
> Regards,
> GP
>
> On Thu, 13 Jul, 2023, 13:54 Luke Chen,  wrote:
>
> > Hi Gaurav,
> >
> > > Why do I see topic ReplicationFactor:1 during description of a topic ?
> > I think you should set this config: `default.replication.factor`
> >
> > > And how can we increase replication factor?  I have 3 Kafka brokers
> > running on 2.7.0 version.
> > Please check this doc:
> >
> >
> https://kafka.apache.org/documentation/#basic_ops_increase_replication_factor
> >
> > Thanks.
> > Luke
> >
> > On Thu, Jul 13, 2023 at 12:23 PM Gaurav Pande 
> > wrote:
> >
> > > Hi all,
> > >
> > > I have noticed that when I describe my Kafka topic I see
> > > ReplicationFactor:1 but in my server.properties  file I already have
> > > defined below values i.e a replication factor of 3 :
> > >
> > > num.partions=3
> > > offsets.topic.replication.factor=3
> > > transaction.state.log.replication.factor=3
> > > transaction.state.log.min.isr=1
> > >
> > > Why do I see topic ReplicationFactor:1 during description of a topic ?
> > >
> > > And how can we increase replication factor?  I have 3 Kafka brokers
> > running
> > > on 2.7.0 version.
> > >
> > > Regards
> > > GP
> > >
> >
>

Re: Replication factor for a topic Increase

[Luke Chen]

Hi Gaurav,

> Why do I see topic ReplicationFactor:1 during description of a topic ?
I think you should set this config: `default.replication.factor`

> And how can we increase replication factor?  I have 3 Kafka brokers
running on 2.7.0 version.
Please check this doc:
https://kafka.apache.org/documentation/#basic_ops_increase_replication_factor

Thanks.
Luke

On Thu, Jul 13, 2023 at 12:23 PM Gaurav Pande  wrote:

> Hi all,
>
> I have noticed that when I describe my Kafka topic I see
> ReplicationFactor:1 but in my server.properties  file I already have
> defined below values i.e a replication factor of 3 :
>
> num.partions=3
> offsets.topic.replication.factor=3
> transaction.state.log.replication.factor=3
> transaction.state.log.min.isr=1
>
> Why do I see topic ReplicationFactor:1 during description of a topic ?
>
> And how can we increase replication factor?  I have 3 Kafka brokers running
> on 2.7.0 version.
>
> Regards
> GP
>

Re: [KRaft] Clarification about Deploying Considerations (Kafka docs)

[Luke Chen]

Hi Grigorios

I've asked the author in the JIRA:
https://issues.apache.org/jira/browse/KAFKA-14207

Thanks.
Luke

On Wed, Jul 5, 2023 at 10:31 PM Grigorios Avgitidis 
wrote:

> Hi,
>
> On the KRaft section of the Kafka documentation, related with `Deploying
> Considerations`, in one of the points is stated:
>
>- For redundancy, a Kafka cluster should use 3 controllers. More than
>3 controllers is not recommended in critical environments. In the rare case
>of a partial network failure it is possible for the cluster metadata quorum
>to become unavailable. This limitation will be addressed in a future
>release of Kafka.
>
> Link to the related section on the docs:
> https://kafka.apache.org/documentation/#kraft_deployment
>
>
> Could you please clarify the following part?
> *"More than 3 controllers is not recommended in critical environments"*
>
> Is it indeed the case or it's a typo?
>
> It was never the case with Zookeeper, where a quorum with number higher
> than 3 was recommended to achieve better failover.
>
> How would one ensure production level fault-tolerance with only 3 nodes if
> having more than 3 is not recommended?
>
> Best regards,
> Grigorios
>
> [image: image.png]
>

Re: [ANNOUNCE] Apache Kafka 3.5.0

[Luke Chen]

Thanks for running this release, Mickael!


On Thu, Jun 15, 2023 at 4:27 PM Mickael Maison  wrote:

> The Apache Kafka community is pleased to announce the release for Apache
> Kafka 3.5.0.
>
> This is a minor release and it includes fixes and improvements from 201
> JIRAs.
>
> All of the changes in this release can be found in the release notes:
> https://downloads.apache.org/kafka/3.5.0/RELEASE_NOTES.html
>
> An overview of the release can be found in our announcement blog post:
> https://kafka.apache.org/blog
>
> You can download the source and binary release (Scala 2.12 and Scala 2.13)
> from:
> https://kafka.apache.org/downloads#3.5.0
>
>
> ---
>
> Apache Kafka is a distributed streaming platform with four core APIs:
> ** The Producer API allows an application to publish a stream records to
> one or more Kafka topics.
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an output
> stream to one or more output topics, effectively transforming the input
> streams to output streams.
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might capture
> every change to a table.
>
> With these APIs, Kafka can be used for two broad classes of application:
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
> ** Building real-time streaming applications that transform or react to the
> streams of data.
>
> Apache Kafka is in use at large and small companies worldwide, including
> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> Target, The New York Times, Uber, Yelp, and Zalando, among others.
>
> A big thank you for the following 103 contributors to this release!
> A. Sophie Blee-Goldman, Akhilesh Chaganti, Alex Sorokoumov, Alexandre
> Dupriez, Alyssa Huang, Anastasia Vela, Andreas Maechler, andymg3, Artem
> Livshits, atu-sharm, bachmanity1, Bill Bejeck, Brendan Ribera, Calvin Liu,
> Chaitanya Mukka, Cheryl Simmons, Chia-Ping Tsai, Chris Egerton, Christo
> Lolov, Colin P. McCabe, csolidum, Daniel Scanteianu, David Arthur, David
> Jacot, David Karlsson, David Mao, Dejan Stojadinović, Divij Vaidya, dorwi,
> drgnchan, Dániel Urbán, Edoardo Comar, egyedt, emilnkrastev, Eric Haag,
> Farooq Qaiser, Federico Valeri, Gantigmaa Selenge, Greg Harris, Guozhang
> Wang, Hao Li, Hector Geraldino, Himani Arora, Hoki Min, hudeqi, iamazy,
> Iblis Lin, Ismael Juma, Ivan Yurchenko, Jakub Scholz, Jason Gustafson, Jeff
> Kim, Jim Galasyn, Jorge Esteban Quilcate Otoya, Josep Prat, José Armando
> García Sancio, Juan José Ramos, Junyang Liu, Justine Olshan, Kamal
> Chandraprakash, Kirk True, Kowshik Prakasam, littlehorse-eng, liuzc9, Lucas
> Brutschy, Lucia Cerchie, Luke Chen, Manikumar Reddy, Manyanda Chitimbo,
> Matthew Wong, Matthias J. Sax, Matthias Seiler, Michael Marshall, Mickael
> Maison, nicolasguyomar, Nikolay, Paolo Patierno, Philip Nee, Pierangelo Di
> Pilato, Proven Provenzano, Purshotam Chauhan, Qing, Rajini Sivaram,
> RivenSun, Robert Young, Rohan, Roman Schmitz, Ron Dagostino, Ruslan
> Krivoshein, Satish Duggana, Shay Elkin, Shekhar Rajak, Simon Woodman,
> Spacrocket, stejani-cflt, Terry, Tom Bentley, vamossagar12, Victoria Xia,
> Viktor Somogyi-Vass, Vladimir Korenev, Yash Mayya, Zheng-Xian Li
>
> We welcome your help and feedback. For more information on how to report
> problems, and to get involved, visit the project website at
> https://kafka.apache.org/
>
> Thank you!
>
> Regards,
> Mickael Maison
>

Re: Process to Upgrade Zookeeper from 2.7.0 to 3.4.1

[Luke Chen]

Hi Gaurav,

Please check Zookeeper's doc for upgrading guide.

Thanks.
Luke

On Wed, Jun 14, 2023 at 12:03 PM Gaurav Pande  wrote:

> Hi Guys,
>
> Could anyone help on this query?
>
> Regards,
> Gaurav
>
> On Tue, 13 Jun, 2023, 11:40 Gaurav Pande,  wrote:
>
> > Hello Guys,
> >
> > Iam new in this space, I was going through the documentation of Upgrading
> > Kafka brokers to 3.4 from any previous version  with Zookeeper mode ,
> but I
> > couldn't find any Upgrade process for Zookeeper.
> >
> > Iam using Zookeeper provided by Kafka binary in this case 2.7.0 and not
> > installed zk externally.
> >
> > So what's the process of Upgrading Zookeeper? And should I upgrade
> > Zookeeper first or 3 Kafka brokers?
> >
> > Note - I have single Zookeeper and 3 Kafka brokers at this point.
> >
> > Regards,
> > Gaurav
> >
>

[ANNOUNCE] Apache Kafka 3.4.1

[Luke Chen]

The Apache Kafka community is pleased to announce the release for
Apache Kafka 3.4.1.

This is a bug fix release and it includes fixes and improvements from
58 JIRAs, including a few critical bugs:
- core
KAFKA-14644 Process should stop after failure in raft IO thread
KAFKA-14946 KRaft controller node shutting down while renouncing leadership
KAFKA-14887 ZK session timeout can cause broker to shutdown
- client
KAFKA-14639 Kafka CooperativeStickyAssignor revokes/assigns partition
in one rebalance cycle
- connect
KAFKA-12558 MM2 may not sync partition offsets correctly
KAFKA-14666 MM2 should translate consumer group offsets behind replication flow
- stream
KAFKA-14172 bug: State stores lose state when tasks are reassigned under EOS

All of the changes in this release can be found in the release notes:

https://www.apache.org/dist/kafka/3.4.1/RELEASE_NOTES.html

You can download the source and binary release (Scala 2.12 and Scala 2.13) from:

https://kafka.apache.org/downloads#3.4.1

---

Apache Kafka is a distributed streaming platform with four core APIs:

** The Producer API allows an application to publish a stream records
to one or more Kafka topics.

** The Consumer API allows an application to subscribe to one or more
topics and process the stream of records produced to them.

** The Streams API allows an application to act as a stream processor,
consuming an input stream from one or more topics and producing an
output stream to one or more output topics, effectively transforming
the input streams to output streams.

** The Connector API allows building and running reusable producers or
consumers that connect Kafka topics to existing applications or data
systems. For example, a connector to a relational database might
capture every change to a table.


With these APIs, Kafka can be used for two broad classes of application:

** Building real-time streaming data pipelines that reliably get data
between systems or applications.

** Building real-time streaming applications that transform or react
to the streams of data.

Apache Kafka is in use at large and small companies worldwide,
including Capital One, Goldman Sachs, ING, LinkedIn, Netflix,
Pinterest, Rabobank, Target, The New York Times, Uber, Yelp, and
Zalando, among others.

A big thank you for the following 32 contributors to this release!

atu-sharm, Chia-Ping Tsai, Chris Egerton, Colin Patrick McCabe,
csolidum, David Arthur, David Jacot, Divij Vaidya, egyedt,
emilnkrastev, Eric Haag, Greg Harris, Guozhang Wang, Hector Geraldino,
hudeqi, Jason Gustafson, Jeff Kim, Jorge Esteban Quilcate Otoya, José
Armando García Sancio, Lucia Cerchie, Luke Chen, Manikumar Reddy,
Matthias J. Sax, Mickael Maison, Philip Nee, Purshotam Chauhan, Rajini
Sivaram, Ron Dagostino, Terry, Victoria Xia, Viktor Somogyi-Vass, Yash
Mayya

We welcome your help and feedback. For more information on how to
report problems, and to get involved, visit the project website at
https://kafka.apache.org/


Thank you!

Regards,
Luke

Re: kafka 3.4.0 sasl_PLAINTEXT kafka-metadata-quorum.sh Unexpected Kafka request of type METADATA during SASL handshake

[Luke Chen]

Hi,

I've replied you in your previous email.
https://lists.apache.org/thread/ogzpxtnsyklp7q82xh5t0y58rgtpc18x
Please take a look.

Thanks.
Luke

On Tue, Jun 6, 2023 at 5:13 PM Yj Jia  wrote:

> 
>
> kafka 3.4.0 sasl_PLAINTEXT exec kafka-metadata-quorum.sh Unexpected Kafka
> request of type METADATA during SASL handshake.
>
> 1、kafka 3.4.0 config broker sasl_PLAINTEXT,
>
> ./kafka-metadata-quorum.sh --bootstrap-server 192.168.3.138:9092
> --command-config kafka_server_config.conf describe --status
>
> kafkalog stdout :
>
> [2023-05-30 16:21:36,542] INFO [SocketServer listenerType=BROKER,
> nodeId=2] Failed authentication with /192.168.3.138
> (channelId=192.168.3.139:9092-192.168.3.138:36074-41) (Unexpected
> Kafka request of type METADATA during SASL handshake.)
> (org.apache.kafka.common.network.Selector)
>
> 2、kafka server server.properties config
>
> process.roles=broker,controllernode.id=1
> controller.quorum.voters=1@192.168.3.138:9093,2@192.168.3.139:9093,
> 3@192.168.3.140:9093
> listeners=BROKER://:9092,CONTROLLER://:9093inter.broker.listener.name
> =BROKER
> controller.listener.names=CONTROLLER
> listener.security.protocol.map=BROKER:SASL_PLAINTEXT,CONTROLLER:PLAINTEXT
> sasl.mechanism.inter.broker.protocol=PLAIN
> sasl.enabled.mechanisms=PLAIN
>
> 3、kafka_server_jaas.conf config
>
> broker.KafkaServer {
> org.apache.kafka.common.security.plain.PlainLoginModule required
> username="admin"
> password="admin-secret"
> user_admin="admin-secret"
> user_alice="alice-secret";
> };
>
> 4、kafka_server_config.conf config
>
> sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule
> required \
> username="admin" \
> password="admin-secret";
> security.protocol=SASL_PLAINTEXT
> sasl.mechanism=PLAIN
>
> 5、 use kafka_server_config.conf create topic or descirbe topic info
>
> [root@opensource01 kafka_2.13-3.4.0]# bin/kafka-topics.sh --describe
> --topic enmotech  --bootstrap-server 192.168.3.138:9092
> --command-config kafka_server_config.conf
> Topic: enmotechTopicId: ZXVFSBuUT7e_xYWKk4rV9APartitionCount:
> 5ReplicationFactor: 3Configs: segment.bytes=1073741824
> Topic: enmotechPartition: 0Leader: 3Replicas: 3,1,2
> Isr: 3,1,2
> Topic: enmotechPartition: 1Leader: 1Replicas: 1,2,3
> Isr: 3,1,2
> Topic: enmotechPartition: 2Leader: 2Replicas: 2,3,1
> Isr: 3,1,2
> Topic: enmotechPartition: 3Leader: 3Replicas: 3,1,2
> Isr: 3,1,2
> Topic: enmotechPartition: 4Leader: 1Replicas: 1,2,3
> Isr: 3,1,2
>
> PS： don't enable sasl_plaintext ,exec kafka-metadata-quorum.sh,the output
> is :
>
> [root@opensource02 bin]# ./kafka-metadata-quorum.sh
> --bootstrap-server 192.168.3.139:9092  describe --replication
> NodeIdLogEndOffsetLagLastFetchTimestamp
> LastCaughtUpTimestampStatus
> 1 96191   0  1685433493877 1685433493877
>  Leader
> 2 96191   0  1685433493482 1685433493482
>  Follower
> 3 96191   0  1685433493474 1685433493474
>  Follower
>
>
>
> how can i fix this problem
>

Re: [VOTE] 3.5.0 RC1

[Luke Chen]

Hi Mickael,

I ran the following validation steps:
- Built from source with Java 17 and Scala 2.13
- Signatures and hashes of the artifacts generated
- Navigated through Javadoc including links to JDK classes
- Run the quickstart in KRaft and Zookeeper mode

+1 (binding) from me.
Thanks for running the release.

Luke

On Tue, Jun 6, 2023 at 4:23 AM Josep Prat 
wrote:

> Hi Mickael,
>
> I ran the following validation steps:
> - Built from source with Java 11 and Scala 2.13
> - Signatures and hashes of the artifacts generated
> - Navigated through Javadoc including links to JDK classes
> - Run the unit tests
> - Run integration tests
> - Run the quickstart in KRaft and Zookeeper mode
> -- For KRaft, I looked at the process running Kafka and confirmed that the
> spamming log message is not present anymore ("Generated a metadata delta
> between...")
> - Checked the contents of LICENSE-binary against the lib folder
> -- I found that the LICENSE-binary has a reference to classgraph-4.8.138
> but I fail to find this library in the lib folder. Trying to backtrack when
> it was added, it seems it was done here:
>
> https://github.com/apache/kafka/commit/3a2ac267178c5464e41b94fcbb2dd897212812bd
> but I fail to find this library in the 3.3.1 lib folder (3.3.0 was a dead
> release).
>
> I'm not sure this qualifies as a blocker for the release though, I leave it
> up to you.
>
> Best,
>
> On Mon, Jun 5, 2023 at 3:39 PM Mickael Maison  wrote:
>
> > Hello Kafka users, developers and client-developers,
> >
> > This is the second candidate for release of Apache Kafka 3.5.0. Some
> > of the major features include:
> > - KIP-710: Full support for distributed mode in dedicated MirrorMaker
> > 2.0 clusters
> > - KIP-881: Rack-aware Partition Assignment for Kafka Consumers
> > - KIP-887: Add ConfigProvider to make use of environment variables
> > - KIP-889: Versioned State Stores
> > - KIP-894: Use incrementalAlterConfig for syncing topic configurations
> > - KIP-900: KRaft kafka-storage.sh API additions to support SCRAM for
> > Kafka Brokers
> >
> > Release notes for the 3.5.0 release:
> > https://home.apache.org/~mimaison/kafka-3.5.0-rc1/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Friday June 9, 5pm PT
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~mimaison/kafka-3.5.0-rc1/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://home.apache.org/~mimaison/kafka-3.5.0-rc1/javadoc/
> >
> > * Tag to be voted upon (off 3.5 branch) is the 3.5.0 tag:
> > https://github.com/apache/kafka/releases/tag/3.5.0-rc1
> >
> > * Documentation:
> > https://kafka.apache.org/35/documentation.html
> >
> > * Protocol:
> > https://kafka.apache.org/35/protocol.html
> >
> > * Successful Jenkins builds for the 3.5 branch:
> > Unit/integration tests: I'm struggling to get all tests to pass in the
> > same build. I'll run a few more builds to ensure each test pass at
> > least once in the CI. All tests passed locally.
> > System tests: The build is still running, I'll send an update once I
> > have the results.
> >
> > Thanks,
> > Mickael
> >
>
>
> --
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |    >
>      <
> https://twitter.com/aiven_io>
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B
>

Re: [VOTE] 3.4.1 RC3

[Luke Chen]

Hi Tom,

Thanks for the vote.
I've re-run the 3.4 jenkins build, and the
`DynamicBrokerReconfigurationTest.testAdvertisedListenerUpdate` test still
pass.
https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.4/142/

And now, I've got:
Binding +1 PMC votes:
* Chris Egerton
* Mickael Maison
* Tom Bentley

Non-binding votes:
* Federico Valeri
* Jakub Scholz
* Josep Prat

I will close this vote thread and go ahead to complete the release process.


Thank you.
Luke

On Fri, Jun 2, 2023 at 5:06 PM Josep Prat 
wrote:

> Hi Tom,
> it failed for me a couple of times, I rebooted and things suddenly worked.
> So maybe there was a dangling process holding a port from previous test
> failures.
>
> On Fri, Jun 2, 2023 at 10:52 AM Tom Bentley  wrote:
>
> > Hi Luke,
> >
> > Thanks for running the release.
> >
> > I've checked signatures, eyeballed the Javadocs, built from source and
> run
> > the unit and integration tests.
> > DynamicBrokerReconfigurationTest.testAdvertisedListenerUpdate fails for
> me
> > repeatedly. I opened https://issues.apache.org/jira/browse/KAFKA-15049
> for
> > it since I couldn't find an existing issue for this one. I note that
> others
> > seem to have run the integration tests without problems, so I don't think
> > this is a blocker. I also did the Kafka, Connect and Streams quickstarts.
> >
> > +1 binding.
> >
> > Tom
> >
> >
> >
> > On Thu, 1 Jun 2023 at 08:46, Luke Chen  wrote:
> >
> > > Hi all,
> > >
> > > Thanks to everyone who has tested and voted for the RC3 so far!
> > > Currently, I've got 2 binding votes and 3 non-binding votes:
> > >
> > > Binding +1 PMC votes:
> > > * Chris Egerton
> > > * Mickael Maison
> > >
> > > Non-binding votes:
> > > * Federico Valeri
> > > * Jakub Scholz
> > > * Josep Prat
> > >
> > > If anyone is available (especially PMC members :)), please help verify
> > the
> > > RC build.
> > >
> > > Thank you.
> > > Luke
> > >
> > > On Wed, May 31, 2023 at 1:53 AM Chris Egerton  >
> > > wrote:
> > >
> > > > Hi Luke,
> > > >
> > > > Many thanks for your continued work on this release!
> > > >
> > > > To verify, I:
> > > > - Built from source using Java 11 with both:
> > > > - - the 3.4.1-rc3 tag on GitHub
> > > > - - the kafka-3.4.1-src.tgz artifact from
> > > > https://home.apache.org/~showuon/kafka-3.4.1-rc3/
> > > > - Checked signatures and checksums
> > > > - Ran the quickstart using the kafka_2.13-3.4.1.tgz artifact from
> > > > https://home.apache.org/~showuon/kafka-3.4.1-rc3/ with Java 11 and
> > Scala
> > > > 13
> > > > in KRaft mode
> > > > - Ran all unit tests
> > > > - Ran all integration tests for Connect and MM2
> > > >
> > > > +1 (binding)
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Tue, May 30, 2023 at 11:16 AM Mickael Maison <
> > > mickael.mai...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Luke,
> > > > >
> > > > > I built from source with Java 11 and Scala 2.13 and ran the unit
> and
> > > > > integration tests. It took a few retries to get some of them to
> pass.
> > > > > I verified signatures and hashes and also ran the zookeeper
> > quickstart.
> > > > >
> > > > > +1 (binding)
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > > > > On Sat, May 27, 2023 at 12:58 PM Jakub Scholz 
> > wrote:
> > > > > >
> > > > > > +1 (non-binding) ... I used the staged binaries and Maven
> artifacts
> > > to
> > > > > run
> > > > > > my tests and all seems to work fine.
> > > > > >
> > > > > > Thanks for running the release.
> > > > > >
> > > > > > Jakub
> > > > > >
> > > > > > On Fri, May 26, 2023 at 9:34 AM Luke Chen 
> > wrote:
> > > > > >
> > > > > > > Hello Kafka users, developers and client-developers,
> > > > > > >
> > > > > > > This is the 4th candidate for release of Apache Kafka 3.4.1.
> > > > > > >
> > > > > > > This is a bu

Re: kafka 3.4.0 sasl_PLAINTEXT exec kafka-metadata-quorum.sh Unexpected Kafka request of type METADATA during SASL handshake.

[Luke Chen]

Hi Yj,

Thanks for reporting the issue.
This is a known issue will be fixed in v3.4.1, which is planning to release
next week (hopefully).
JIRA: https://issues.apache.org/jira/browse/KAFKA-14711

Thanks.
Luke

On Thu, Jun 1, 2023 at 9:02 PM Yj Jia  wrote:

> kafka 3.4.0 sasl_PLAINTEXT exec kafka-metadata-quorum.sh Unexpected Kafka
> request of type METADATA during SASL handshake.
>
> 1、kafka 3.4.0 config broker sasl_PLAINTEXT,
>
> ./kafka-metadata-quorum.sh --bootstrap-server 192.168.3.138:9092
> --command-config kafka_server_config.conf describe --status
>
> kafkalog stdout :
>
> [2023-05-30 16:21:36,542] INFO [SocketServer listenerType=BROKER,
> nodeId=2] Failed authentication with /192.168.3.138
> (channelId=192.168.3.139:9092-192.168.3.138:36074-41) (Unexpected
> Kafka request of type METADATA during SASL handshake.)
> (org.apache.kafka.common.network.Selector)
>
> 2、kafka server server.properties config
>
> process.roles=broker,controllernode.id=1
> controller.quorum.voters=1@192.168.3.138:9093,2@192.168.3.139:9093,
> 3@192.168.3.140:9093
> listeners=BROKER://:9092,CONTROLLER://:9093inter.broker.listener.name
> =BROKER
> controller.listener.names=CONTROLLER
> listener.security.protocol.map=BROKER:SASL_PLAINTEXT,CONTROLLER:PLAINTEXT
> sasl.mechanism.inter.broker.protocol=PLAIN
> sasl.enabled.mechanisms=PLAIN
>
> 3、kafka_server_jaas.conf config
>
> broker.KafkaServer {
> org.apache.kafka.common.security.plain.PlainLoginModule required
> username="admin"
> password="admin-secret"
> user_admin="admin-secret"
> user_alice="alice-secret";
> };
>
> 4、kafka_server_config.conf config
>
> sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule
> required \
> username="admin" \
> password="admin-secret";
> security.protocol=SASL_PLAINTEXT
> sasl.mechanism=PLAIN
>
> 5、 use kafka_server_config.conf create topic or descirbe topic info
>
> [root@opensource01 kafka_2.13-3.4.0]# bin/kafka-topics.sh --describe
> --topic enmotech  --bootstrap-server 192.168.3.138:9092
> --command-config kafka_server_config.conf
> Topic: enmotechTopicId: ZXVFSBuUT7e_xYWKk4rV9APartitionCount:
> 5ReplicationFactor: 3Configs: segment.bytes=1073741824
> Topic: enmotechPartition: 0Leader: 3Replicas: 3,1,2
> Isr: 3,1,2
> Topic: enmotechPartition: 1Leader: 1Replicas: 1,2,3
> Isr: 3,1,2
> Topic: enmotechPartition: 2Leader: 2Replicas: 2,3,1
> Isr: 3,1,2
> Topic: enmotechPartition: 3Leader: 3Replicas: 3,1,2
> Isr: 3,1,2
> Topic: enmotechPartition: 4Leader: 1Replicas: 1,2,3
> Isr: 3,1,2
>
> PS： don't enable sasl_plaintext ,exec kafka-metadata-quorum.sh,the output
> is :
>
> [root@opensource02 bin]# ./kafka-metadata-quorum.sh
> --bootstrap-server 192.168.3.139:9092  describe --replication
> NodeIdLogEndOffsetLagLastFetchTimestamp
> LastCaughtUpTimestampStatus
> 1 96191   0  1685433493877 1685433493877
>  Leader
> 2 96191   0  1685433493482 1685433493482
>  Follower
> 3 96191   0  1685433493474 1685433493474
>  Follower
>
>
>
> how can i fix this problem
>

Re: [VOTE] 3.4.1 RC3

[Luke Chen]

Hi all,

Thanks to everyone who has tested and voted for the RC3 so far!
Currently, I've got 2 binding votes and 3 non-binding votes:

Binding +1 PMC votes:
* Chris Egerton
* Mickael Maison

Non-binding votes:
* Federico Valeri
* Jakub Scholz
* Josep Prat

If anyone is available (especially PMC members :)), please help verify the
RC build.

Thank you.
Luke

On Wed, May 31, 2023 at 1:53 AM Chris Egerton 
wrote:

> Hi Luke,
>
> Many thanks for your continued work on this release!
>
> To verify, I:
> - Built from source using Java 11 with both:
> - - the 3.4.1-rc3 tag on GitHub
> - - the kafka-3.4.1-src.tgz artifact from
> https://home.apache.org/~showuon/kafka-3.4.1-rc3/
> - Checked signatures and checksums
> - Ran the quickstart using the kafka_2.13-3.4.1.tgz artifact from
> https://home.apache.org/~showuon/kafka-3.4.1-rc3/ with Java 11 and Scala
> 13
> in KRaft mode
> - Ran all unit tests
> - Ran all integration tests for Connect and MM2
>
> +1 (binding)
>
> Cheers,
>
> Chris
>
> On Tue, May 30, 2023 at 11:16 AM Mickael Maison 
> wrote:
>
> > Hi Luke,
> >
> > I built from source with Java 11 and Scala 2.13 and ran the unit and
> > integration tests. It took a few retries to get some of them to pass.
> > I verified signatures and hashes and also ran the zookeeper quickstart.
> >
> > +1 (binding)
> >
> > Thanks,
> > Mickael
> >
> > On Sat, May 27, 2023 at 12:58 PM Jakub Scholz  wrote:
> > >
> > > +1 (non-binding) ... I used the staged binaries and Maven artifacts to
> > run
> > > my tests and all seems to work fine.
> > >
> > > Thanks for running the release.
> > >
> > > Jakub
> > >
> > > On Fri, May 26, 2023 at 9:34 AM Luke Chen  wrote:
> > >
> > > > Hello Kafka users, developers and client-developers,
> > > >
> > > > This is the 4th candidate for release of Apache Kafka 3.4.1.
> > > >
> > > > This is a bugfix release with several fixes since the release of
> > 3.4.0. A
> > > > few of the major issues include:
> > > > - core
> > > > KAFKA-14644 <https://issues.apache.org/jira/browse/KAFKA-14644>
> > Process
> > > > should stop after failure in raft IO thread
> > > > KAFKA-14946 <https://issues.apache.org/jira/browse/KAFKA-14946>
> KRaft
> > > > controller node shutting down while renouncing leadership
> > > > KAFKA-14887 <https://issues.apache.org/jira/browse/KAFKA-14887> ZK
> > session
> > > > timeout can cause broker to shutdown
> > > > - client
> > > > KAFKA-14639 <https://issues.apache.org/jira/browse/KAFKA-14639>
> Kafka
> > > > CooperativeStickyAssignor revokes/assigns partition in one rebalance
> > cycle
> > > > - connect
> > > > KAFKA-12558 <https://issues.apache.org/jira/browse/KAFKA-12558> MM2
> > may
> > > > not
> > > > sync partition offsets correctly
> > > > KAFKA-14666 <https://issues.apache.org/jira/browse/KAFKA-14666> MM2
> > should
> > > > translate consumer group offsets behind replication flow
> > > > - stream
> > > > KAFKA-14172 <https://issues.apache.org/jira/browse/KAFKA-14172> bug:
> > State
> > > > stores lose state when tasks are reassigned under EOS
> > > >
> > > >
> > > > Release notes for the 3.4.1 release:
> > > > https://home.apache.org/~showuon/kafka-3.4.1-rc3/RELEASE_NOTES.html
> > > >
> > > > *** Please download, test and vote by Jun 2, 2023
> > > >
> > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > https://kafka.apache.org/KEYS
> > > >
> > > > * Release artifacts to be voted upon (source and binary):
> > > > https://home.apache.org/~showuon/kafka-3.4.1-rc3/
> > > >
> > > > * Maven artifacts to be voted upon:
> > > >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > >
> > > > * Javadoc:
> > > > https://home.apache.org/~showuon/kafka-3.4.1-rc3/javadoc/
> > > >
> > > > * Tag to be voted upon (off 3.4 branch) is the 3.4.1 tag:
> > > > https://github.com/apache/kafka/releases/tag/3.4.1-rc3
> > > >
> > > > * Documentation: (will be updated after released)
> > > > https://kafka.apache.org/34/documentation.html
> > > >
> > > > * Protocol: (will be updated after released)
> > > > https://kafka.apache.org/34/protocol.html
> > > >
> > > > The most recent build has had test failures. These all appear to be
> > due to
> > > > flakiness, but it would be nice if someone more familiar with the
> > failed
> > > > tests could confirm this. I may update this thread with passing build
> > links
> > > > if I can get one, or start a new release vote thread if test failures
> > must
> > > > be addressed beyond re-running builds until they pass.
> > > >
> > > > Unit/integration tests:
> > > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.4/141/
> > > >
> > > > System tests:
> > > > Will update the results later
> > > >
> > > > Thank you
> > > > Luke
> > > >
> >
>

Re: client.dns.lookup=”use_all_dns_ips” cache ttl

[Luke Chen]

Hi George,

I believe the JVM will have cache to this DNS records.
And maybe OS will also cache it.
Had a quick look, Kafka doesn't cache it.

Thanks.
Luke

On Thu, Jun 1, 2023 at 1:21 AM George Goh  wrote:

> Hi!
>
> I’m considering to use the following configuration for my producers and
> consumers:
>
> client.dns.lookup=”use_all_dns_ips”
>
> So that I only have a single DNS entry to manage the list of brokers.
>
> I also have another cluster connected via MirrorMaker 2, which serves as a
> failover.
>
> My question is, if I was to change the DNS entry to point to the 2nd
> cluster, is there a delay before clients pick up the new A Records due to
> caching? If yes, can caching be disabled?
>
> Thank you.
>

[VOTE] 3.4.1 RC3

[Luke Chen]

Hello Kafka users, developers and client-developers,

This is the 4th candidate for release of Apache Kafka 3.4.1.

This is a bugfix release with several fixes since the release of 3.4.0. A
few of the major issues include:
- core
KAFKA-14644  Process
should stop after failure in raft IO thread
KAFKA-14946  KRaft
controller node shutting down while renouncing leadership
KAFKA-14887  ZK session
timeout can cause broker to shutdown
- client
KAFKA-14639  Kafka
CooperativeStickyAssignor revokes/assigns partition in one rebalance cycle
- connect
KAFKA-12558  MM2 may not
sync partition offsets correctly
KAFKA-14666  MM2 should
translate consumer group offsets behind replication flow
- stream
KAFKA-14172  bug: State
stores lose state when tasks are reassigned under EOS


Release notes for the 3.4.1 release:
https://home.apache.org/~showuon/kafka-3.4.1-rc3/RELEASE_NOTES.html

*** Please download, test and vote by Jun 2, 2023

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~showuon/kafka-3.4.1-rc3/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~showuon/kafka-3.4.1-rc3/javadoc/

* Tag to be voted upon (off 3.4 branch) is the 3.4.1 tag:
https://github.com/apache/kafka/releases/tag/3.4.1-rc3

* Documentation: (will be updated after released)
https://kafka.apache.org/34/documentation.html

* Protocol: (will be updated after released)
https://kafka.apache.org/34/protocol.html

The most recent build has had test failures. These all appear to be due to
flakiness, but it would be nice if someone more familiar with the failed
tests could confirm this. I may update this thread with passing build links
if I can get one, or start a new release vote thread if test failures must
be addressed beyond re-running builds until they pass.

Unit/integration tests:
https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.4/141/

System tests:
Will update the results later

Thank you
Luke

[VOTE] 3.4.1 RC2

[Luke Chen]

Hello Kafka users, developers and client-developers,

This is the 3rd candidate for release of Apache Kafka 3.4.1.

This is a bugfix release with several fixes since the release of 3.4.0. A
few of the major issues include:
- core
KAFKA-14644  Process
should stop after failure in raft IO thread
KAFKA-14946  KRaft
controller node shutting down while renouncing leadership
KAFKA-14887  ZK session
timeout can cause broker to shutdown
- client
KAFKA-14639  Kafka
CooperativeStickyAssignor revokes/assigns partition in one rebalance cycle
- connect
KAFKA-12558  MM2 may not
sync partition offsets correctly
KAFKA-14666  MM2 should
translate consumer group offsets behind replication flow
- stream
KAFKA-14172  bug: State
stores lose state when tasks are reassigned under EOS


Release notes for the 3.4.1 release:
https://home.apache.org/~showuon/kafka-3.4.1-rc2/RELEASE_NOTES.html

*** Please download, test and vote by May 31, 2023

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~showuon/kafka-3.4.1-rc2/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~showuon/kafka-3.4.1-rc2/javadoc/

* Tag to be voted upon (off 3.4 branch) is the 3.4.1 tag:
https://github.com/apache/kafka/releases/tag/3.4.1-rc2

* Documentation: (will be updated after released)
https://kafka.apache.org/34/documentation.html

* Protocol: (will be updated after released)
https://kafka.apache.org/34/protocol.html
The most recent build has had test failures. These all appear to be due to
flakiness, but it would be nice if someone more familiar with the failed
tests could confirm this. I may update this thread with passing build links
if I can get one, or start a new release vote thread if test failures must
be addressed beyond re-running builds until they pass.

Unit/integration tests:
https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.4/138/

System tests:
Will update the results later

Hope this will be the final RC for v3.4.1!

Thank you.
Luke

Re: Re: Re: Kafka 3.2.1 performance issue with JDK 11

[Luke Chen]

Hi Vic,

Yes, JDK 8 is still supported in kafka 3.x.x.
It'll become unsupported after kafka 4.0.0.

Thanks.
Luke

On Tue, May 23, 2023 at 9:56 AM Vic Xu  wrote:

> Hi Greg
>
> I know JDK 8 will solve the problem certainly, but I wondering if JDK8 has
> been deprecated after Kafka 3? Can I use JDK 8 with Kafka 3.2.1? Thank you.
>
>
> On 2023/05/22 20:55:01 Greg Harris wrote:
> > Vic,
> >
> > While you can certainly try that, I don't know if that will solve the
> problem.
> > The reason why JDK11 appears relevant in this context is that a class
> > was removed between JDK8 and JDK11. I don't know if a replacement
> > stack inspector with better performance was added to JDK17 and used
> > within log4j2.
> > If you were to try to solve this with a JDK version change, a
> > downgrade to 8 may solve the problem, since the log4j library would
> > use a different stack inspector.
> >
> > Greg
> >
> > On Sun, May 21, 2023 at 11:30 PM Vic Xu  wrote:
> > >
> > > Hi Greg,
> > >
> > > I found another possible solution that is upgrade JDK from 11 to 17.
> Do you recommend this solution?
> > >
> > > On 2023/05/21 17:58:42 Greg Harris wrote:
> > > > Vic,
> > > >
> > > > I found an open JIRA issue that previously reported this problem:
> > > > https://issues.apache.org/jira/browse/KAFKA-10877 .
> > > > I believe one workaround is to use log4j 1.x, such as reload4j. Kafka
> > > > still relies on log4j 1.x until the planned upgrade to log4j 2.x in
> > > > kafka 4.0 https://issues.apache.org/jira/browse/KAFKA-9366 .
> > > > I will look into reviving or replacing the performance patch for 3.x.
> > > >
> > > > Hope this helps,
> > > > Greg Harris
> > > >
> > > > On Sun, May 21, 2023 at 6:31 AM Vic Xu  wrote:
> > > > >
> > > > > Hello all,  I have a Kafka cluster deployed with version 3.2.1 ,
> JDK 11 and log4j 2.18.0. I built my own Kafka image. One of my Kafka
> brokers is experiencing CPU issues, and based on the jstack information, it
> seems that log4j is causing the problem due to its usage of StackWalker.
> How to solve this issue?
> > > > >
> > > > > Here is jstack information:
> > > > > "data-plane-kafka-request-handler-6" #59 daemon prio=5 os_prio=0
> cpu=86381259.23ms elapsed=1948787.21s tid=0x7f8939c04800 nid=0x190
> runnable  [0x7f883f6f5000]
> > > > >java.lang.Thread.State: RUNNABLE
> > > > > at
> java.lang.StackStreamFactory$AbstractStackWalker.fetchStackFrames(java.base@11.0.9/Native
> Method)
> > > > > at
> java.lang.StackStreamFactory$AbstractStackWalker.fetchStackFrames(java.base@11.0.9/Unknown
> Source)
> > > > > at
> java.lang.StackStreamFactory$AbstractStackWalker.getNextBatch(java.base@11.0.9/Unknown
> Source)
> > > > > at
> java.lang.StackStreamFactory$AbstractStackWalker.peekFrame(java.base@11.0.9/Unknown
> Source)
> > > > > at
> java.lang.StackStreamFactory$AbstractStackWalker.hasNext(java.base@11.0.9/Unknown
> Source)
> > > > > at
> java.lang.StackStreamFactory$StackFrameTraverser.tryAdvance(java.base@11.0.9/Unknown
> Source)
> > > > > at
> java.util.stream.ReferencePipeline.forEachWithCancel(java.base@11.0.9/Unknown
> Source)
> > > > > at
> java.util.stream.AbstractPipeline.copyIntoWithCancel(java.base@11.0.9/Unknown
> Source)
> > > > > at
> java.util.stream.AbstractPipeline.copyInto(java.base@11.0.9/Unknown
> Source)
> > > > > at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(java.base@11.0.9/Unknown
> Source)
> > > > > at
> java.util.stream.FindOps$FindOp.evaluateSequential(java.base@11.0.9/Unknown
> Source)
> > > > > at
> java.util.stream.AbstractPipeline.evaluate(java.base@11.0.9/Unknown
> Source)
> > > > > at
> java.util.stream.ReferencePipeline.findFirst(java.base@11.0.9/Unknown
> Source)
> > > > > at
> org.apache.logging.log4j.util.StackLocator.lambda$getCallerClass$2(StackLocator.java:57)
> > > > > at
> org.apache.logging.log4j.util.StackLocator$$Lambda$117/0x0008001a6c40.apply(Unknown
> Source)
> > > > > at
> java.lang.StackStreamFactory$StackFrameTraverser.consumeFrames(java.base@11.0.9/Unknown
> Source)
> > > > > at
> java.lang.StackStreamFactory$AbstractStackWalker.doStackWalk(java.base@11.0.9/Unknown
> Source)
> > > > > at
> java.lang.StackStreamFactory$AbstractStackWalker.callStackWalk(java.base@11.0.9/Native
> Method)
> > > > > at
> java.lang.StackStreamFactory$AbstractStackWalker.beginStackWalk(java.base@11.0.9/Unknown
> Source)
> > > > > at
> java.lang.StackStreamFactory$AbstractStackWalker.walk(java.base@11.0.9/Unknown
> Source)
> > > > > at java.lang.StackWalker.walk(java.base@11.0.9/Unknown
> Source)
> > > > > at
> org.apache.logging.log4j.util.StackLocator.getCallerClass(StackLocator.java:51)
> > > > > at
> org.apache.logging.log4j.util.StackLocatorUtil.getCallerClass(StackLocatorUtil.java:104)
> > > > > at
> 

[VOTE] 3.4.1 RC1

[Luke Chen]

Hello Kafka users, developers and client-developers,

This is the 2nd candidate for release of Apache Kafka 3.4.1.

This is a bugfix release with several fixes since the release of 3.4.0. A
few of the major issues include:
- core
KAFKA-14644  Process
should stop after failure in raft IO thread
KAFKA-14946  KRaft
controller node shutting down while renouncing leadership
KAFKA-14887  ZK session
timeout can cause broker to shutdown
- client
KAFKA-14639  Kafka
CooperativeStickyAssignor revokes/assigns partition in one rebalance cycle
- connect
KAFKA-12558  MM2 may not
sync partition offsets correctly
KAFKA-14666  MM2 should
translate consumer group offsets behind replication flow
- stream
KAFKA-14172  bug: State
stores lose state when tasks are reassigned under EOS

Release notes for the 3.4.1 release:
https://home.apache.org/~showuon/kafka-3.4.1-rc0/RELEASE_NOTES.html

*** Please download, test and vote by May 29, 2023
Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~showuon/kafka-3.4.1-rc0/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~showuon/kafka-3.4.1-rc0/javadoc/

* Tag to be voted upon (off 3.4 branch) is the 3.4.1 tag:
https://github.com/apache/kafka/releases/tag/3.4.1-rc0

* Documentation: (will be updated after released)
https://kafka.apache.org/34/documentation.html

* Protocol: (will be updated after released)
https://kafka.apache.org/34/protocol.html

The most recent build has had test failures. These all appear to be due to
flakiness, but it would be nice if someone more familiar with the failed
tests could confirm this. I may update this thread with passing build links
if I can get one, or start a new release vote thread if test failures must
be addressed beyond re-running builds until they pass.

Unit/integration tests:
https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.4/135/


System tests:
Will update the results later

Confirmed Maven artifacts are in staging repository.

Thank you.
Luke

Re: [VOTE] 3.4.1 RC0

[Luke Chen]

Hi Matthias,

Yes, I agree we should get this hotfix into 3.4.1.
I've backported into the 3.4 branch.
I'll create a new RC for 3.4.1.

Thanks.
Luke

On Mon, May 22, 2023 at 5:13 AM Matthias J. Sax  wrote:

> Hi Luke,
>
> RC0 for 3.4.1 includes a fix for
> https://issues.apache.org/jira/browse/KAFKA-14862. We recently
> discovered that tge fix itself introduces a regression. We have already
> a PR to fix-forward the regression:
> https://github.com/apache/kafka/pull/13734
>
> I think we should get the open PR merged, and back part not only to 3.5,
> but also to 3.4.1, and get a new RC for 3.4.1.
>
> Thoughts?
>
>
> -Matthias
>
>
> On 5/19/23 6:12 AM, Josep Prat wrote:
> > Hi Luke,
> > This gets a +1 from my end. I believe non-binding because if I understand
> > it correctly, binding votes for releases are only issued by PMCs (
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Process#ReleaseProcess-Afterthevotepasses
> > ).
> >
> > I did the following validations:
> > - Verified signatures and checksums for all the generated artifacts
> > - Built from source with Java 11 and Scala 2.13.10
> > - Run unit tests
> > - Run integration tests
> > - Run the quickstart with Zookeeper and KRaft
> >
> > Best,
> >
> > On Wed, May 17, 2023 at 2:11 PM Josep Prat  wrote:
> >
> >> Hi Luke,
> >>
> >> I ran the tests from the source package you created and I didn't get any
> >> of the test failures you had on your CI build. I got other flaky tests
> >> though, that after being run in isolation ran successfully. I'll try to
> run
> >> signature validation, and some further testing later today or later this
> >> week.
> >>
> >> Best,
> >>
> >> On Wed, May 17, 2023 at 12:43 PM Federico Valeri 
> >> wrote:
> >>
> >>> Hi Luke, thanks for running the release.
> >>>
> >>> Looks like the Maven artifacts are not in staging:
> >>>
> >>>
> https://repository.apache.org/content/groups/staging/org/apache/kafka/kafka-clients/3.4.1/
> >>>
> >>> Documentation still has 3.4.0, instead of 3.4.1 (not sure if this will
> >>> be aligned later):
> >>> https://kafka.apache.org/34/documentation.html#producerapi
> >>>
> >>> Br
> >>> Fede
> >>>
> >>>
> >>> On Wed, May 17, 2023 at 5:24 AM Luke Chen  wrote:
> >>>>
> >>>> Hello Kafka users, developers and client-developers,
> >>>>
> >>>> This is the first candidate for release of Apache Kafka 3.4.1.
> >>>>
> >>>> This is a bugfix release with several fixes since the release of
> 3.4.0.
> >>> A
> >>>> few of the major issues include:
> >>>> - core
> >>>> KAFKA-14644 <https://issues.apache.org/jira/browse/KAFKA-14644>
> Process
> >>>> should stop after failure in raft IO thread
> >>>> KAFKA-14946 <https://issues.apache.org/jira/browse/KAFKA-14946> KRaft
> >>>> controller node shutting down while renouncing leadership
> >>>> KAFKA-14887 <https://issues.apache.org/jira/browse/KAFKA-14887> ZK
> >>> session
> >>>> timeout can cause broker to shutdown
> >>>> - client
> >>>> KAFKA-14639 <https://issues.apache.org/jira/browse/KAFKA-14639> Kafka
> >>>> CooperativeStickyAssignor revokes/assigns partition in one rebalance
> >>> cycle
> >>>> - connect
> >>>> KAFKA-12558 <https://issues.apache.org/jira/browse/KAFKA-12558> MM2
> >>> may not
> >>>> sync partition offsets correctly
> >>>> KAFKA-14666 <https://issues.apache.org/jira/browse/KAFKA-14666> MM2
> >>> should
> >>>> translate consumer group offsets behind replication flow
> >>>> - stream
> >>>> KAFKA-14172 <https://issues.apache.org/jira/browse/KAFKA-14172> bug:
> >>> State
> >>>> stores lose state when tasks are reassigned under EOS
> >>>>
> >>>> Release notes for the 3.4.1 release:
> >>>> https://home.apache.org/~showuon/kafka-3.4.1-rc0/RELEASE_NOTES.html
> >>>>
> >>>> *** Please download, test and vote by May 24, 2023
> >>>> Kafka's KEYS file containing PGP keys we use to sign the release:
> >>>> https://kafka.apache.org/KEYS
> >>>>
> >>>> * Release artifacts to be voted 

Re: kafka acl issue

[Luke Chen]

Hi Hari,

You might want to ask in the client repo (kafkajs?)
They should be able to help you.

Thanks.
Luke

On Fri, May 19, 2023 at 3:00 PM HariBabu kuruva 
wrote:

> Hi All,
>
> I am trying to implement kafka acl for one of the topics.
> it's a kafka cluster with 1 broker.
>
> Below are the ACL's applied on the topic
>
> Current ACLs for resource `ResourcePattern(resourceType=TOPIC,
> name=ibxkb.test.topic, patternType=LITERAL)`:
> (principal=User:kafkauser, host=*, operation=WRITE,
> permissionType=ALLOW)
> (principal=User:kafkauser, host=*, operation=CREATE,
> permissionType=ALLOW)
> (principal=User:kafkauser, host=*, operation=DESCRIBE,
> permissionType=ALLOW)
> (principal=User:kafkauser, host=*, operation=READ,
> permissionType=ALLOW)
> ---
> When the producer is trying to connect using the below script, it throws
> the error as shown below .
>
> *Producer Script:*
> import { Kafka, logLevel } from 'kafkajs';
>
>
> (async () => {
>
> const kafka = new Kafka({
> clientId: 'saurabhs-program',
> brokers: ['broker.corp.equinix.com:9092'],
> // authenticationTimeout: 1,
> // reauthenticationThreshold: 1,
> //ssl: true,
>
> sasl: {
> mechanism: 'PLAIN', // scram-sha-256 or scram-sha-512
> username: 'kafkauser',
> password: 'kafkauser',
> //group: 'test-app'
>
> },
> });
> kafka.logger().setLogLevel(logLevel.DEBUG);
>
>
> const producer = kafka.producer();
> producer.logger().setLogLevel(logLevel.DEBUG);
>
> await producer.connect();
>
> const response = await producer.send({
> topic: 'ibxkb.test.topic',
> messages: [
> { value: 'Auth Test' },
> ],
> });
>
> console.log(response);
>
> })();
>
>
> *ERROR:*
>
> *KafkaJSProtocolError: Request is not valid given the current SASL state*
> at createErrorFromCode
>
> (C:\Hari\Equinix\EISP\node-utils\node-utils2\node-utils2\node_modules\kafkajs\src\protocol\error.js:581:10)
> at Object.parse
>
> (C:\Hari\Equinix\EISP\node-utils\node-utils2\node-utils2\node_modules\kafkajs\src\protocol\requests\saslHandshake\v0\response.js:24:11)
> at Connection.send
>
> (C:\Hari\Equinix\EISP\node-utils\node-utils2\node-utils2\node_modules\kafkajs\src\network\connection.js:433:35)
> at process.processTicksAndRejections
> (node:internal/process/task_queues:95:5)
> at async SASLAuthenticator.authenticate
>
> (C:\Hari\Equinix\EISP\node-utils\node-utils2\node-utils2\node_modules\kafkajs\src\broker\saslAuthenticator\index.js:35:23)
> at async
>
> C:\Hari\Equinix\EISP\node-utils\node-utils2\node-utils2\node_modules\kafkajs\src\network\connection.js:139:9
> at async Connection.authenticate
>
> (C:\Hari\Equinix\EISP\node-utils\node-utils2\node-utils2\node_modules\kafkajs\src\network\connection.js:315:5)
> at async Broker.connect
>
> (C:\Hari\Equinix\EISP\node-utils\node-utils2\node-utils2\node_modules\kafkajs\src\broker\index.js:111:7)
> at async
>
> C:\Hari\Equinix\EISP\node-utils\node-utils2\node-utils2\node_modules\kafkajs\src\cluster\brokerPool.js:93:9
> at async
>
> C:\Hari\Equinix\EISP\node-utils\node-utils2\node-utils2\node_modules\kafkajs\src\cluster\index.js:107:14
> {
>   retriable: false,
>   helpUrl: undefined,
> *  type: 'ILLEGAL_SASL_STATE',*
>   code: 34,
>   [cause]: undefined
>
>
> Please give me some advice. Let me know if you need any more information.
> --
>
> Thanks and Regards,
>  Hari
> Mobile:9790756568
>

[VOTE] 3.4.1 RC0

[Luke Chen]

Hello Kafka users, developers and client-developers,

This is the first candidate for release of Apache Kafka 3.4.1.

This is a bugfix release with several fixes since the release of 3.4.0. A
few of the major issues include:
- core
KAFKA-14644  Process
should stop after failure in raft IO thread
KAFKA-14946  KRaft
controller node shutting down while renouncing leadership
KAFKA-14887  ZK session
timeout can cause broker to shutdown
- client
KAFKA-14639  Kafka
CooperativeStickyAssignor revokes/assigns partition in one rebalance cycle
- connect
KAFKA-12558  MM2 may not
sync partition offsets correctly
KAFKA-14666  MM2 should
translate consumer group offsets behind replication flow
- stream
KAFKA-14172  bug: State
stores lose state when tasks are reassigned under EOS

Release notes for the 3.4.1 release:
https://home.apache.org/~showuon/kafka-3.4.1-rc0/RELEASE_NOTES.html

*** Please download, test and vote by May 24, 2023
Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~showuon/kafka-3.4.1-rc0/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~showuon/kafka-3.4.1-rc0/javadoc/

* Tag to be voted upon (off 3.4 branch) is the 3.4.1 tag:
https://github.com/apache/kafka/releases/tag/3.4.1-rc0

* Documentation:
https://kafka.apache.org/34/documentation.html

* Protocol:
https://kafka.apache.org/34/protocol.html

The most recent build has had test failures. These all appear to be due to
flakiness, but it would be nice if someone more familiar with the failed
tests could confirm this. I may update this thread with passing build links
if I can get one, or start a new release vote thread if test failures must
be addressed beyond re-running builds until they pass.

Unit/integration tests:
https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.4/133/

System tests:
Will update the results later

Thank you.
Luke

Re: Trying to reduce the number of replicates from 9 to 3

[Luke Chen]

Hi Mich,

You might want to take a look at this section: "Increasing replication
factor" in the doc:
https://kafka.apache.org/documentation/#basic_ops_increase_replication_factor

Simply put, the json file provided in kafka-reassign-partitions.sh should
show the final replicas assignment after this operation.
In your case, the size of each replica should be 3, but I saw you put 9
replicas there.

Hope it helps.
Luke

On Sat, May 13, 2023 at 7:20 PM Mich Talebzadeh 
wrote:

> Hi,
>
> From the following list
>
>  kafka-topics.sh --describe --bootstrap-server rhes75:9092 --topic md
>
> Topic: md   TopicId: UfQly87bQPCbVKoH-PQheg PartitionCount: 9
> ReplicationFactor: 9Configs:
> segment.bytes=1073741824,retention.bytes=1073741824
> Topic: md   Partition: 0Leader: 12  Replicas:
> 12,10,8,2,9,11,1,7,3  Isr: 10,1,9,2,12,7,3,11,8
> Topic: md   Partition: 1Leader: 9   Replicas:
> 9,8,2,12,11,1,7,3,10  Isr: 10,1,9,2,12,7,3,11,8
> Topic: md   Partition: 2Leader: 11  Replicas:
> 11,2,12,9,1,7,3,10,8  Isr: 10,1,9,2,12,7,3,11,8
> Topic: md   Partition: 3Leader: 1   Replicas:
> 1,12,9,11,7,3,10,8,2  Isr: 10,1,9,2,12,7,3,11,8
> Topic: md   Partition: 4Leader: 7   Replicas:
> 7,9,11,1,3,10,8,2,12  Isr: 10,1,9,2,12,7,3,11,8
> Topic: md   Partition: 5Leader: 3   Replicas:
> 3,11,1,7,10,8,2,12,9  Isr: 10,1,9,2,12,7,3,11,8
> Topic: md   Partition: 6Leader: 10  Replicas:
> 10,1,7,3,8,2,12,9,11  Isr: 10,1,9,2,12,7,3,11,8
> Topic: md   Partition: 7Leader: 8   Replicas:
> 8,7,3,10,2,12,9,11,1  Isr: 10,1,9,2,12,7,3,11,8
> Topic: md   Partition: 8Leader: 2   Replicas:
> 2,3,10,8,12,9,11,1,7  Isr: 10,1,9,2,12,7,3,11,8
>
> so for topic md I have 9 Partitions and 9 Replication
>
> As for redundancy and prevent data loss I only need 3 replicas (the leader
> and 2 followers) , so I use the following to reduce the number of replicas
> to 3
>
>
> {
>  "version":1,
>  "partitions":[
> {"topic":"md","partition":0,"replicas":[12,10,8,2,9,11,1,7,3]},
> {"topic":"md","partition":1,"replicas":[9,8,2,12,11,1,7,3,10]},
> {"topic":"md","partition":2,"replicas":[11,2,12,9,1,7,3,10,8]}
> ]
> }
> with the following command
>
> kafka-reassign-partitions.sh --bootstrap-server rhes75:9092
> --reassignment-json-file ./reduce_replication_factor2.json --execute
>
> and this is the output
>
> Current partition replica assignment
>
>
> {"version":1,"partitions":[{"topic":"md","partition":0,"replicas":[12,10,8,2,9,11,1,7,3],"log_dirs":["any","any","any","any","any","any","any","any","any"]},{"topic":"md","partition":1,"replicas":[9,8,2,12,11,1,7,3,10],"log_dirs":["any","any","any","any","any","any","any","any","any"]},{"topic":"md","partition":2,"replicas":[11,2,12,9,1,7,3,10,8],"log_dirs":["any","any","any","any","any","any","any","any","any"]}]}
>
> Save this to use as the --reassignment-json-file option during rollback
>
> *Successfully started partition reassignments for md-0,md-1,md-2*
>
>
> It says it is doing it, but nothing is happening!
>
>
> This is the size of Kafka Topic in MB per each per partition remaining:
>
>
> kafka-log-dirs.sh --bootstrap-server rhes75:9092 --topic-list md --describe
> | grep -oP '(?<=size":)\d+' | awk '{ sum += $1 } END { print
> sum/1024/1024/9 }'
>
>
> Which comes back with 81.5 MB
>
>
> Will this work as I have stopped the queue but still data there. In short,
> is downsizing practical?
>
>
> Thanks
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>

Re: CVEs related to Kafka

[Luke Chen]

Hi Sahil,

> in which version of Kafka these will be fixed

https://issues.apache.org/jira/browse/KAFKA-14320
https://issues.apache.org/jira/browse/KAFKA-14107
https://issues.apache.org/jira/browse/KAFKA-14256

Maybe you can try to search the JIRA first next time. :)

Thank you.
Luke

On Wed, May 10, 2023 at 12:33 PM Sahil Sharma D
 wrote:

> Hi team,
>
> By when we can expect reply reg this, any idea?
>
> Regards,
> Sahil
>
> -Original Message-
> From: Tauzell, Dave 
> Sent: 09 May 2023 11:29 PM
> To: users@kafka.apache.org
> Subject: Re: CVEs related to Kafka
>
> Consider purchasing support from Confluent to get this sort of request
> answered quickly.
>
>
> From: Sahil Sharma D 
> Date: Tuesday, May 9, 2023 at 12:40 PM
> To: users@kafka.apache.org 
> Subject: [EXTERNAL] RE: CVEs related to Kafka Gentle reminder-2 !
>
> -Original Message-
> From: Sahil Sharma D 
> Sent: 03 May 2023 04:34 PM
> To: users@kafka.apache.org
> Subject: RE: CVEs related to Kafka
>
> Gentle reminder!
>
> From: Sahil Sharma D
> Sent: 03 May 2023 08:57 AM
> To: 'users@kafka.apache.org' 
> Subject: RE: CVEs related to Kafka
> Importance: High
>
> Hi Team,
>
> We have found few more Vulnerabilities on Kafka, below are the list:
>
> CVE-2022-36944<
> https://urldefense.com/v3/__https://nvd.nist.gov/vuln/detail/CVE-2022-36944__;!!K_cMf-SQz-o!dVr1QtyQ4T63P401cAOMN0HLlWf5PlvvulL4LX7JGrbK8mSYbhhN6Snv5XQ7NzMg6wcdPjVpi6k_LPbS9gMBugc0ucxXd2_9ywOkKoY$
> > Scala 2.13.x before 2.13.9 has a Java deserialization chain in its JAR
> file. On its own, it cannot be exploited. There is only a risk in
> conjunction with Java object deserialization within an application. In such
> situations, it allows attackers to erase contents of arbitrary files, make
> network connections, or possibly run arbitrary code (specifically,
> Function0 functions) via a gadget chain
>
> CVE-2023-26048<
> https://urldefense.com/v3/__https://nvd.nist.gov/vuln/detail/CVE-2023-26048__;!!K_cMf-SQz-o!dVr1QtyQ4T63P401cAOMN0HLlWf5PlvvulL4LX7JGrbK8mSYbhhN6Snv5XQ7NzMg6wcdPjVpi6k_LPbS9gMBugc0ucxXd2_9GQ1_xXo$
> > Jetty is a java based web server and servlet engine. In affected versions
> servlets with multipart support (e.g. annotated with `@MultipartConfig`)
> that call `HttpServletRequest.getParameter()` or
> `HttpServletRequest.getParts()` may cause `OutOfMemoryError` when the
> client sends a multipart request with a part that has a name but no
> filename and very large content. This happens even with the default
> settings of `fileSizeThreshold=0` which should stream the whole part
> content to disk. An attacker client may send a large multipart request and
> cause the server to throw `OutOfMemoryError`. However, the server may be
> able to recover after the `OutOfMemoryError` and continue its service --
> although it may take some time. This issue has been patched in versions
> 9.4.51, 10.0.14, and 11.0.14. Users are advised to upgrade. Users unable to
> upgrade may set the multipart parameter `maxRequestSize` which must be set
> to a non-negative value, so the whole multipart content is limited
> (although still read into memory).
>
> CVE-2023-26049<
> https://urldefense.com/v3/__https://nvd.nist.gov/vuln/detail/CVE-2023-26049__;!!K_cMf-SQz-o!dVr1QtyQ4T63P401cAOMN0HLlWf5PlvvulL4LX7JGrbK8mSYbhhN6Snv5XQ7NzMg6wcdPjVpi6k_LPbS9gMBugc0ucxXd2_9K3-reco$
> > Jetty is a java based web server and servlet engine. Nonstandard cookie
> parsing in Jetty may allow an attacker to smuggle cookies within other
> cookies, or otherwise perform unintended behavior by tampering with the
> cookie parsing mechanism. If Jetty sees a cookie VALUE that starts with `"`
> (double quote), it will continue to read the cookie string until it sees a
> closing quote -- even if a semicolon is encountered. So, a cookie header
> such as: `DISPLAY_LANGUAGE="b; JSESSIONID=1337; c=d"` will be parsed as one
> cookie, with the name DISPLAY_LANGUAGE and a value of b; JSESSIONID=1337;
> c=d instead of 3 separate cookies. This has security implications because
> if, say, JSESSIONID is an HttpOnly cookie, and the DISPLAY_LANGUAGE cookie
> value is rendered on the page, an attacker can smuggle the JSESSIONID
> cookie into the DISPLAY_LANGUAGE cookie and thereby exfiltrate it. This is
> significant when an intermediary is enacting some policy based on cookies,
> so a smuggled cookie can bypass that policy yet still be seen by the Jetty
> server or its logging system. This issue has been addressed in versions
> 9.4.51, 10.0.14, 11.0.14, and 12.0.0.beta0 and users are advised to
> upgrade. There are no known workarounds for this issue.
>
> Kindly confirm about the mitigation plan and impact of these CVEs.
>
> Regards,
> Sahil
>
> From: Sahil Sharma D
> Sent: 02 May 2023 02:16 PM
> To: users@kafka.apache.org
> Subject: CVEs related to Kafka
> Importance: High
>
> Hi team,
>
> We have got below two vulnerabilities on Kafka 3PP.
>
> CVE-2022-42003<
> 

Re: KRaft with even number of controllers

[Luke Chen]

Hi De Gao,

Usually, we won't compare the performance with even/odd cases.
I think the most important reason we don't recommend using even number of
nodes is the node failure tolerance.

In 1 nodes case, the majority is 1, so we can't lose any node
In 2 nodes case, the majority is 2, so we can't lose any node
In 3 nodes case, the majority is 2, so we can tolerate 1 node failure.
In 4 nodes case, the majority is 3, so we can tolerate 1 node failure.
...

So, you should be able to see, if you can allow single point of failure,
why use 2 nodes instead of 1?
If you can tolerate 1 node failure, why use 4 nodes instead of 2? and so on.

Thanks.
Luke

On Fri, May 5, 2023 at 2:05 PM Gao De  wrote:

> Hi All:
>
> In raft algorhithm even number of node should still work. Just the leader
> election could take longer time. Did anybody ever use even number of kraft
> controllers in real application environment? Or anybody ever compared this
> kind of setup with odd number of controllers? Are there any significant
> performance difference?
>
> Thanks.
>
> De Gao
>

Re: Kafka topic __consumer_offsets replication issue

[Luke Chen]

Hi kiran,

I would check the log end offset of the in-sync partition first, and check
the lags of the offsets with the leader offset. (You can check by metric:
`kafka.log:type=Log,name=LogEndOffset,topic=xxx, partition=xx`)
Then, I would check if the follower is doing fetching without error by
checking the logs.
Basically, if the network is good, and storage system is good, and it's
doing fetching correctly, it should join in-sync soon.

Hope it works.

Thanks.
Luke

On Sun, Apr 30, 2023 at 10:58 AM kiran kumar  wrote:

> Hello All,
>
> Any insights on this issue?
>
> Thanks,
> Kiran
>
> On Sun, 23 Apr 2023 at 1:58 PM, kiran kumar 
> wrote:
>
> > Hello Kafka Folks,
> >
> > I have encountered an issue with Kafka 3.2 version.
> > Environment:
> > OS: RHEL 7.9
> > Java: 1.8
> > Kafka: 3.2
> >
> > There seems to one of the replica is stuck in __consumer_offsets topic.
> >
> > I have the replication factor set as 3 and its 5 node kafka cluster. I
> > could see that out of 2 replicas one replica is not getting in sync with
> > rest of the two replicas. I have trued to delete the partition folder and
> > performed the rolling restart of Kafka. I have also tried to reassign the
> > partition to another bode but that node as well in same state. I have
> tried
> > to expand the replicas to 5 partition reassignment tool but all the new
> > replicas are at the same state.
> > Even though, there two nodes which are good, the third node is unable to
> > completed sync and partitions to 5 nodes are also stuck at the same state
> > as the other faulty node.
> >
> > Is there any way to recover the Kafka partition replica and get it sync
> > with other two of the good nodes ?
> >
> > Thanks and regards
> > Kiran
> >
> > --
> > G.Kiran Kumar
> >
> --
> G.Kiran Kumar
>

Re: Kafka support for IPV6 only Networking stack

[Luke Chen]

Hi Senjoorian,

I've never tested it, but AFAIK, Kafka supports IPv6.
There's also a KIP to allow Kafka to accept duplicate listener on port for
IPv4/IPv6:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=195726330

Maybe you'll be interested.

Thank you.
Luke

On Thu, Apr 27, 2023 at 4:19 PM Senjoorian Murugan (Nokia) <
senjoorian.muru...@nokia.com> wrote:

> Dear Team ,
>
> We would like to check if Kafka support IPv6 only configuration . In this
> configuration customer will have only IPv6 Networking for the operating
> system where Kafka is going to be installed .
>
> Regards,
> Senjoorian
>
>

Re: Addition to Kafka contributor group

[Luke Chen]

Hi Atul,

I've added you to the contribution group in JIRA.
I didn't find your account in wiki system.
Let me know if you need wiki permission to create a KIP.

Thank you.
Luke

On Fri, Apr 21, 2023 at 10:15 PM Atul Sharma 
wrote:

> Hi Team,
>
> I would like to contribute to open source Kafka, can you please add me to
> Kafka contributor group.
>
> Username: atusharm
> Email: atulsharma...@gmail.com
>
> Thanks
>

Re: Kafka Node Shutting Down Automatically

[Luke Chen]

Hi Akshay,

Thanks for reporting the issue.
It looks like a bug.
Could you open a JIRA  ticket
to track it?

Thank you.
Luke


On Fri, Apr 21, 2023 at 10:16 PM Akshay Kumar 
wrote:

> Hello team,
>
>- We are using the zookeeper less Kafka (kafka Kraft).
>- The cluster is having 3 nodes.
>- One of the nodes gets automatically shut down randomly.
>- Checked the logs but didn't get the exact reason.
>- Sharing the logs below. Kafka version - 3.3.1
>
> *Logs - *
>
> [2023-04-13 01:49:17,411] WARN [Controller 1] Renouncing the leadership
> due to a metadata log event. We were the leader at epoch 37110, but in the
> new epoch 37111, the leader is (none). Reverting to last committed offset
> 28291464. (org.apache.kafka.controller.QuorumController)
> [2023-04-13 01:49:17,531] INFO [RaftManager nodeId=1] Completed transition
> to Unattached(epoch=37112, voters=[1, 2, 3], electionTimeoutMs=982)
> (org.apache.kafka.raft.QuorumState)
>
> [2023-04-13 02:00:33,902] WARN [Controller 1] Renouncing the leadership
> due to a metadata log event. We were the leader at epoch 37116, but in the
> new epoch 37117, the leader is (none). Reverting to last committed offset
> 28292807. (org.apache.kafka.controller.QuorumController)
> [2023-04-13 02:00:33,936] INFO [RaftManager nodeId=1] Completed transition
> to Unattached(epoch=37118, voters=[1, 2, 3], electionTimeoutMs=1497)
> (org.apache.kafka.raft.QuorumState)
>
> [2023-04-13 02:00:35,014] ERROR [Controller 1] processBrokerHeartbeat:
> unable to start processing because of NotControllerException.
> (org.apache.kafka.controller.QuorumController)
>
> [2023-04-13 02:12:21,883] WARN [Controller 1] Renouncing the leadership
> due to a metadata log event. We were the leader at epoch 37129, but in the
> new epoch 37131, the leader is (none). Reverting to last committed offset
> 28294206. (org.apache.kafka.controller.QuorumController)
>
> [2023-04-13 02:13:41,328] WARN [Controller 1] Renouncing the leadership
> due to a metadata log event. We were the leader at epoch 37141, but in the
> new epoch 37142, the leader is (none). Reverting to last committed offset
> 28294325. (org.apache.kafka.controller.QuorumController)
>
> [2023-04-13 02:13:41,328] INFO [Controller 1] writeNoOpRecord: failed with
> NotControllerException in 16561838 us
> (org.apache.kafka.controller.QuorumController)
>
> [2023-04-13 02:13:41,328] INFO [Controller 1] maybeFenceReplicas: failed
> with NotControllerException in 8520846 us
> (org.apache.kafka.controller.QuorumController)
>
> [2023-04-13 02:13:41,328] INFO [BrokerToControllerChannelManager broker=1
> name=heartbeat] Client requested disconnect from node 1
> (org.apache.kafka.clients.NetworkClient)
> [2023-04-13 02:13:41,329] INFO [BrokerLifecycleManager id=1] Unable to
> send a heartbeat because the RPC got timed out before it could be sent.
> (kafka.server.BrokerLifecycleManager)
> [2023-04-13 02:13:41,351] ERROR Encountered fatal fault: exception while
> renouncing leadership
> (org.apache.kafka.server.fault.ProcessExitingFaultHandler)
> java.lang.NullPointerException
> at
> org.apache.kafka.timeline.SnapshottableHashTable$HashTier.mergeFrom(SnapshottableHashTable.java:125)
> at org.apache.kafka.timeline.Snapshot.mergeFrom(Snapshot.java:68)
> at
> org.apache.kafka.timeline.SnapshotRegistry.deleteSnapshot(SnapshotRegistry.java:236)
> at
> org.apache.kafka.timeline.SnapshotRegistry$SnapshotIterator.remove(SnapshotRegistry.java:67)
> at
> org.apache.kafka.timeline.SnapshotRegistry.revertToSnapshot(SnapshotRegistry.java:214)
> at
> org.apache.kafka.controller.QuorumController.renounce(QuorumController.java:1232)
> at
> org.apache.kafka.controller.QuorumController.access$3300(QuorumController.java:150)
> at
> org.apache.kafka.controller.QuorumController$QuorumMetaLogListener.lambda$handleLeaderChange$3(QuorumController.java:1076)
> at
> org.apache.kafka.controller.QuorumController$QuorumMetaLogListener.lambda$appendRaftEvent$4(QuorumController.java:1101)
> at
> org.apache.kafka.controller.QuorumController$ControlEvent.run(QuorumController.java:496)
> at
> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:121)
> at
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:200)
> at
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:173)
> at java.lang.Thread.run(Thread.java:750)
> [2023-04-13 02:13:41,385] INFO [BrokerServer id=1] Transition from STARTED
> to SHUTTING_DOWN (kafka.server.BrokerServer)
>
>
>
> Regards,
> *[image: Inline image 1] *
> *Akshay Kumar*
>
> *Senior Software Engineer ll |  AMEYO
> +91-8556063696*
> *[image: Facebook]  [image: Twitter]
>  

Re: About CVE-2023-25194

[Luke Chen]

Hi,

This is the commit to fix the CVE:
https://github.com/apache/kafka/commit/ae22ec1a0ea005664439c3f45111aa34390ecaa1
2.x upgrades to 3.x includes a major version upgrade, so it'll have some
compatibility issues.
Please check the notable changes for v3.0 here:
https://kafka.apache.org/documentation/#upgrade_300_notable

Thank you.
Luke

On Wed, Mar 29, 2023 at 10:18 PM zjfpla...@hotmail.com <
zjfpla...@hotmail.com> wrote:

> Hi,
> Our kafka version is 2.x. I would like to ask everyone, is it
> risky to upgrade to version 3.4.0 in order to fix CVE-2023-25194? Because
> there are already customers using our products.
>  Also, I would like to ask you how to fix CVE-2023-25194 on
> version 2.x. I did not find the corresponding commit in the historical
> commit of 3.4.0. Can someone help me find the corresponding commit record?
>
>
>
> zjfpla...@hotmail.com
>

Re: Questions about creating jira account

[Luke Chen]

Hi Jimmy,

You can create an account request via
https://selfserve.apache.org/jira-account.html

Thank you.
Luke

On Sat, Mar 18, 2023 at 12:03 AM zw  wrote:

> Hi,
> It seems that JIRA access are disabled by default and I can't create an
> account by myself.
>
> I would appreciate that if anyone could create an account for me. I am
> interested and want to
> contribute to the following ticket:
> https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-10897
>
>
> Thank!
> Regards,
>
> Jimmy Wang

Re: Question about KRaft

[Luke Chen]

For questions related to confluent, I think you'd better ask in their
channel.

Luke

On Fri, Mar 10, 2023 at 12:54 PM sunil chaudhari <
sunilmchaudhar...@gmail.com> wrote:

> Hi Luke,
> This docu is good.
> Does it apply for confluent as well?
>
>
>
> On Fri, 10 Mar 2023 at 8:47 AM, Luke Chen  wrote:
>
> > Hi Zhenyu,
> >
> > Answering your question:
> >
> > > Should I simply
> > 1. download 3.4 binary
> > 2. stop ZK & Kafka service
> > 3. upgrade Kafka to 3.4
> > 4. start only Kafka service with KRaft server.properties
> >
> > That is not migrating, actually. That is just creating another kafka
> > cluster in KRaft mode.
> > The point for migration is to move metadata in ZK into KRaft controllers.
> > You can follow the guide here to do migration:
> > https://kafka.apache.org/documentation/#kraft_zk_migration
> >
> > Thank you.
> > Luke
> >
> > On Tue, Mar 7, 2023 at 11:07 PM Zhenyu Wang 
> > wrote:
> >
> > > Hi Sunil,
> > >
> > > As mentioned earlier in my question, I have only one "combined" node as
> > > both controller and broker, and I totally accept downtime (stop
> service)
> > >
> > > So just want to ask for my case, single node, if I want to upgrade to
> 3.4
> > > then start service under KRaft (get rid of ZK), what would be the
> steps?
> > >
> > > Thanks~
> > >
> > > On Mon, Mar 6, 2023 at 11:49 PM sunil chaudhari <
> > > sunilmchaudhar...@gmail.com>
> > > wrote:
> > >
> > > > How will you achieve zero downtime of you stop zookeeper and kafka?
> > > > There must be some standard steps so that stop zookeeper one by one
> and
> > > > start kraft same time so that it will be migrated gradually.
> > > >
> > > >
> > > >
> > > > On Tue, 7 Mar 2023 at 9:26 AM, Zhenyu Wang 
> > > wrote:
> > > >
> > > > > Hi team,
> > > > >
> > > > > Here is a question about KRaft from normal user, who starts to use
> > and
> > > > > learn Kafka since 3.2
> > > > >
> > > > > Last month Kafka 3.4, the first bridge release was available, and I
> > am
> > > > > considering to have a plan to use KRaft (get rid of ZK) since this
> > > > version
> > > > >
> > > > > Currently I am using 3.3.2 (upgrade from 3.2) with only one node,
> > which
> > > > is
> > > > > both controller & broker, even ZK is installed on this node too
> > (sorry
> > > I
> > > > > know it is not distributed and I will try to improve it with more
> > > > knowledge
> > > > > learned in future)
> > > > >
> > > > > When I read KIP-866, ZK to KRaft migration, from section Migration
> > > > > Overview, seems like the document is for multi-nodes with no or
> > almost
> > > no
> > > > > downtime, enable KRaft node by node; however my case accepts
> downtime
> > > > (one
> > > > > node -_-!!), just want to have Kafka upgrade to 3.4 then start
> > service
> > > > > under KRaft mode, make sure everything works well and no log lost
> > > > >
> > > > > Should I simply
> > > > > 1. download 3.4 binary
> > > > > 2. stop ZK & Kafka service
> > > > > 3. upgrade Kafka to 3.4
> > > > > 4. start only Kafka service with KRaft server.properties
> > > > >
> > > > > Or any other thing I need to pay attention to?
> > > > >
> > > > > If there is a documentation as guide that would be quite helpful
> > > > >
> > > > > Really appreciate
> > > > >
> > > >
> > >
> >
>

Re: Question about KRaft

[Luke Chen]

Hi Zhenyu,

Answering your question:

> Should I simply
1. download 3.4 binary
2. stop ZK & Kafka service
3. upgrade Kafka to 3.4
4. start only Kafka service with KRaft server.properties

That is not migrating, actually. That is just creating another kafka
cluster in KRaft mode.
The point for migration is to move metadata in ZK into KRaft controllers.
You can follow the guide here to do migration:
https://kafka.apache.org/documentation/#kraft_zk_migration

Thank you.
Luke

On Tue, Mar 7, 2023 at 11:07 PM Zhenyu Wang  wrote:

> Hi Sunil,
>
> As mentioned earlier in my question, I have only one "combined" node as
> both controller and broker, and I totally accept downtime (stop service)
>
> So just want to ask for my case, single node, if I want to upgrade to 3.4
> then start service under KRaft (get rid of ZK), what would be the steps?
>
> Thanks~
>
> On Mon, Mar 6, 2023 at 11:49 PM sunil chaudhari <
> sunilmchaudhar...@gmail.com>
> wrote:
>
> > How will you achieve zero downtime of you stop zookeeper and kafka?
> > There must be some standard steps so that stop zookeeper one by one and
> > start kraft same time so that it will be migrated gradually.
> >
> >
> >
> > On Tue, 7 Mar 2023 at 9:26 AM, Zhenyu Wang 
> wrote:
> >
> > > Hi team,
> > >
> > > Here is a question about KRaft from normal user, who starts to use and
> > > learn Kafka since 3.2
> > >
> > > Last month Kafka 3.4, the first bridge release was available, and I am
> > > considering to have a plan to use KRaft (get rid of ZK) since this
> > version
> > >
> > > Currently I am using 3.3.2 (upgrade from 3.2) with only one node, which
> > is
> > > both controller & broker, even ZK is installed on this node too (sorry
> I
> > > know it is not distributed and I will try to improve it with more
> > knowledge
> > > learned in future)
> > >
> > > When I read KIP-866, ZK to KRaft migration, from section Migration
> > > Overview, seems like the document is for multi-nodes with no or almost
> no
> > > downtime, enable KRaft node by node; however my case accepts downtime
> > (one
> > > node -_-!!), just want to have Kafka upgrade to 3.4 then start service
> > > under KRaft mode, make sure everything works well and no log lost
> > >
> > > Should I simply
> > > 1. download 3.4 binary
> > > 2. stop ZK & Kafka service
> > > 3. upgrade Kafka to 3.4
> > > 4. start only Kafka service with KRaft server.properties
> > >
> > > Or any other thing I need to pay attention to?
> > >
> > > If there is a documentation as guide that would be quite helpful
> > >
> > > Really appreciate
> > >
> >
>

Re: Kafka metrics - Requests.

[Luke Chen]

Hi David,

It did look like a bug.
Could you file a bug in JIRA?
And if you have time, welcome to investigate and submit PR for it.
(My guess is there are some internal topics are included in
`TotalFetchRequestsPerSec`, but not included in another one, but not sure)

Thank you.
Luke

On Sat, Feb 25, 2023 at 2:31 AM David Ballano Fernandez <
dfernan...@demonware.net> wrote:

> Hi Pere,
>
> Thanks for replying.
>
> so what i can see is:
>
> *kafka.server:type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec*
> and
> *kafka.network:type=RequestMetrics,name=RequestsPerSec,request={Produce}*
> show the same/close number of producer requests  and that makes sense.
>
> but that is not the same with fetch requests, I pull those metrics from
> Kafka jmx "oneMinuterate" without any other calculations. and this is what
> i see:
>
> kafka.server:type=BrokerTopicMetrics,name=TotalFetchRequestsPerSec 
> *3.1Million
> req/s*
> kafka.network:type=RequestMetrics,name=RequestsPerSec,request={Fetch}
> *97k req/s*
> I always trusted "
> *kafka.network:type=RequestMetrics,name=RequestsPerSec,request" *and I
> thought* "TotalFetchRequestsPerSec" *would be similar with the ability
> like you say to see per topic. but the difference is huge and I don't know
> what to trust.
>
> Thanks.
>
>
> On Mon, Feb 20, 2023 at 2:56 AM Pere Urbón Bayes 
> wrote:
>
>> Hi David,these two metrics have two different objectives. > kafka.
>> server: type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec < 
>> Produce request rate per topic. Omitting 'topic=(. . . )' will yield the
>> all-topic rate. *while >kafka. network:
>> type=RequestMetrics,name=RequestsPerSec,request={Produce|FetchConsumer|FetchFollower}
>>
>> ZjQcmQRYFpfptBannerStart
>> This Message Is From an Untrusted Sender
>> You have not previously corresponded with this sender.
>>
>> ZjQcmQRYFpfptBannerEnd
>> Hi David,
>>these two metrics have two different objectives.
>>
>> > kafka.server:type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec <
>>
>> 
>> Produce request rate per topic. Omitting 'topic=(...)' will yield the
>> all-topic rate.
>> *
>>
>> while
>>
>> >kafka.network:type=RequestMetrics,name=RequestsPerSec,request={Produce|FetchConsumer|FetchFollower}
>> <
>>
>> it is going to report as well requests per sec, however, you can filter
>> and select by Produce/Fetch from consumers or Fetch from Followers.
>> Note this is a metric collected at the network type, while the previous
>> is at the server level.
>>
>> The first one will provide you with info per topic, which is a very
>> valuable asset in order to know more about what kind of usage clients are
>> doing from a topic perspective. Keep in mind the second one, will only give
>> you the global request per second.
>>
>> So to your questions:
>>
>> > Shouldn't both metrics indicate the same?
>>
>> They indicate similar data at different levels, so they should not be far
>> away, however, some distance (but minimal) between the two could be
>> expected as they are collected at two different points.
>>
>> > seems to be the accurate one matching the number of messages sent with
>> no batching.
>>
>> In a good monitoring solution, you should have both metrics as one will
>> give you the overall load and the other the topic partition per broker view
>> (aka what are your users doing).
>>
>> what I don't understand is the batching part of your question.
>>
>> If you like to see messages In I would suggest you to use
>>
>> > kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec <
>> Aggregate incoming message rate.
>>
>> Sincerely
>>
>> -- Pere
>>
>>
>> On Sat, Feb 18, 2023 at 12:17 AM David Ballano Fernandez <
>> dfernan...@demonware.net> wrote:
>>
>>> Hi guys,
>>>
>>> I am having some confusion around 2  Kafka metrics:
>>>
>>> *Request rate.*
>>>
>>> kafka.network:type=RequestMetrics,name=RequestsPerSec,request={Produce|FetchConsumer|FetchFollower}
>>>
>>> *Produce request rate.*
>>> kafka.server:type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec
>>>
>>>
>>> https://docs.confluent.io/platform/current/kafka/monitoring.html#server-metrics
>>> 
>>>
>>> They seem to be tracking the same, but when I graph them and say I pick
>>> Producer. The numbers  are different.
>>>
>>> and looks like
>>>
>>> *Produce request rate.*
>>> kafka.server:type=BrokerTopicMetrics,name=TotalProduceRequestsPerSec
>>>
>>> seems to be the accurate one matching the number of messages sent with no
>>> batching.
>>>
>>> Shouldn't both metrics indicate the same?
>>>
>>> Thanks!
>>>
>>
>>
>> --
>> Pere Urbon-Bayes
>> Software Architect
>> https://twitter.com/purbon
>> 

Re: kafka-producer-perf-test.sh maximum throughput limits for broker on single node

[Luke Chen]

Hi Tomasz,

There are some configurations needed to be tuned in producer side.
Try searching kafka producer tuning, you should find many good articles
describing it.

Good luck.

Luke

On Fri, Feb 24, 2023 at 6:16 AM Tomasz Sąciński <
tomasz.sacin...@linuxpolska.pl> wrote:

> Hello users.
>
> I test max throughput to a single kafka broker but results do not match
> speed limits disk or network.
>
> /kafka-producer-perf-test.sh --topic perf-test --num-records 3000
> --record-size 100 --throughput -1 --producer-props
> bootstrap.servers=localhost:9092
>
> 3000 records sent, 98.872850 records/sec (94.29 MB/sec), 13.51 ms avg
> latency, 528.00 ms max latency, 9 ms 50th, 15 ms 95th, 162 ms 99th, 350 ms
> 99.9th.
>
> Smaller record size gets lower throughput.
>
> Is upper limit throughput for broker or something another limit throughput?
>
> What is bottleneck for this?
>
> There are detailed information:
>
> hardware:
> CPU: 6230R 2.1GHz 2x26core,
> RAM: 192 GB,
> disks SATA SSD – 20 x1,92T (test run on single disk)
>
>
> Java version:
> java-11-openjdk-11.0.17.0.8-2.el8_6.x86_64
>
>
> Kafka version: ./kafka-topics.sh --version
>
> 3.3.1 (Commit:e23c59d00e687ff5)
>
> kafka_2.13-3.3.1.jar
>
> Network speed tested by iperf3: 5,46 GB/s (testing kafka both localhost and
> second node)
> Disk speed:
>
> hdparm -tT /dev/mapper/vg1-kafka
>
> /dev/mapper/vg1-kafka:
>
> Timing cached reads:   19176 MB in  2.00 seconds = 9600.40 MB/sec
>
> Timing buffered disk reads: 10338 MB in  3.00 seconds = 3445.37 MB/sec
>
>
>
> writes:
>
> dd if=/dev/zero of=/kafka/test1.img bs=1G count=1 oflag=dsync
>
> 1+0 records in
>
> 1+0 records out
>
> 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.13398 s, 947 MB/s
>
> Testing both on default jvm settings and with 6GB RAM:
>
>-Xmx6g -Xms6g -XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20
>-XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M
>-XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80
>-XX:+ExplicitGCInvokesConcurrent
>
> Kafka installed and ran from instructions
> https://kafka.apache.org/quickstart  on a single node.
>
>
> Pozdrawiam/Regards
>
>
>
> Tomasz Sąciński
>
> --
>  
>
>
> Linux Polska Sp. z o.o.
>
> Al. Jerozolimskie
> 100, 00-807 Warszawa
> <
> https://www.google.com/maps/place/Linux+Polska+Sp.+z+o.+o./@52.221986,20.9744718,17z/data=!3m1!4b1!4m5!3m4!1s0x0:0x5756147639e870d6!8m2!3d52.221986!4d20.9766605?authuser=2
> >
>
> tel. +48 22 213 95 71, fax +48 22 213 96 71
>
> KRS 0326158, Sąd Rejonowy
> dla M. St. Warszawy w Warszawie, XII Wydział Gospodarczy KRS
>
> Kapitał
> zakładowy 1 000 500 PLN wpłacony w całości, NIP 7010181018, REGON 141791601
>
>
>
> www.linuxpolska.pl  |
> 
> 
> 
>
> _
>
>
>
>
> This message may contain confidential information that is covered by
> legal privilege. If you are not the intended recipient or if you have
> received this message by mistake, please notify the sender immediately and
> delete this e-mail and its attachments from your system. Any unauthorized
> copying, disclosure or distribution of the material in this e-mail and its
> attachments is strictly forbidden.
>
>
> --
>  
>

Re: Release plan for kafka-python

[Luke Chen]

Hi Sahil,

You should ask in kafka-python repo, not here.

Thank you.
Luke

On Tue, Feb 14, 2023 at 7:02 PM Sahil Sharma D
 wrote:

> Hi team,
>
> As per understanding its last release was released in 2020, after that
> there is no release. We are planning to use v2.0.2, if we face any issue in
> this version will it fixed in any upcoming release or what should be the
> workaround.
>
> Can you please share the release plan for kafka-python?
>
> Regards,
> Sahil
>

Re: Server running issue

[Luke Chen]

Hi Shivannad,

It said there's some exception during reading/writing data from/to
/tmp/kafka-logs dir.
There should be some errors in the log above these lines.
That should be the root cause.

Luke

On Mon, Feb 13, 2023 at 11:51 PM Shivanand K Badiger 
wrote:

>
> Hi Team,
>
> I am facing some error while running Kafka server on Ubuntu.
>
> The error message as shown below…
> WARN [ReplicaManager broker=0] Stopping serving replicas in dir
> /tmp/kafka-logs (kafka.server.ReplicaManager)
> [2023-02-13 10:34:32,458] WARN [ReplicaManager broker=0] Broker 0 stopped
> fetcher for partitions  and stopped moving logs for partitions  because
> they are in the failed log directory /tmp/kafka-logs.
> (kafka.server.ReplicaManager)
> [2023-02-13 10:34:32,459] WARN Stopping serving logs in dir
> /tmp/kafka-logs (kafka.log.LogManager)
> [2023-02-13 10:34:32,462] ERROR Shutdown broker because all log dirs in
> /tmp/kafka-logs have failed (kafka.log.LogManager)
> [2023-02-13 10:34:32,964] WARN Unexpected exception
> (org.apache.zookeeper.server.NIOServerCnxn)
>
> Please suggest me what can be done to overcome this error.
>
> Thanks and Regards
> Shivannad
>
>
> Sent from Mail for Windows
>
>

Re: RequestsPerSec version

[Luke Chen]

> Do I understand correctly that messages from 9+ do not follow the same
conversion as before?

No, there's no change for messages 9+.

So, I think your understanding is correct. When client and server are using
the same compression codec, it won't do conversion.
What you need to do now is to make sure the "server" and the "topic" is not
using the compression codec.
You can try to describe the topic to check if the compression.type config
is set.
The default value for `compression.type` is `producer`, which means, it'll
store what it got from producer, won't do conversion.
https://kafka.apache.org/documentation/#brokerconfigs_compression.type


Thank you.
Luke


On Fri, Feb 10, 2023 at 5:55 PM Gonzalo Martin Peci 
wrote:

> Thanks again Luke, I tested the following in "change setting, test
> producing, review metrics" fashion:
>
> - Remove the `zstd` setting from the broker
>
> This had no effect; I could still see the conversions happening
>
> - Set the producer to compress to `zstd`
>
> This had an effect, and I could see the metric go down.
>
> My thought here is that maybe if the setting was on when the topic was
> created, maybe I have to explicitly change it in the topic? The strange
> thing is that I also tested this, and I still saw conversion.
>
> Do I understand correctly that messages from 9+ do not follow the same
> conversion as before? If you have a KIP that you know about, Im happy to
> read.
>

Re: Re: RequestsPerSec version

[Luke Chen]

Hi Gonzalo,

So, the message conversion is because you enabled server compression with
type zstd.
If you remove the config ["compression.type" = "zstd"] in broker,
everything should be fine.

> I tried disabling zstd, but ProduceMessageConversionsPerSec remained at
the same level
Maybe you need to run longer to see the difference.
Anyway, from the source code, I can confirm if the broker is not enabling
compression, or source compression type == broker compression type, there
will be no message conversion happening.

Please give it another try.

Thank you.

On Fri, Feb 10, 2023 at 4:51 AM Gonzalo Martin Peci 
wrote:

> Hey Luke, thanks for your reply.
>
> We run:
> - Brokers: 3.3.1
> - Java Clients:
>   - Confluent "7.3.1-ccs"
>   - Kafka official "3.3.2"
>
> We are running on AWS MSK, and using some minimal configuration:
> ```
> # Security
> "allow.everyone.if.no.acl.found" = false
> "auto.create.topics.enable"  = false
>
> # Guardrails
> "default.replication.factor" = 3
> "min.insync.replicas"= 2
>
> # Cluster Optimizations
> "unclean.leader.election.enable" = false
> "group.initial.rebalance.delay.ms"   = 3000
> "leader.imbalance.per.broker.percentage" = 10
>
> # Offset Availability
> "offsets.topic.replication.factor" = 3
>
> # Transaction Availability
> "transaction.state.log.replication.factor" = 3
> "transaction.state.log.min.isr"= 1
>
> # Storage Optimization
> "compression.type" = "zstd"
> ```
>
> I tried disabling zstd, but ProduceMessageConversionsPerSec remained at the
> same level. Client configuration is also default, so nothing about message
> compression or version.
>
> Thanks
>
> PS: Sorry for the delay, for some reason I did not get the reply to my
> emial.
>
> On 2023/02/08 10:46:19 Luke Chen wrote:
> > Hi Gonzalo,
> >
> > For the produce request record version, you should refer to this file:
> >
>
> https://github.com/apache/kafka/blob/trunk/clients/src/main/resources/common/message/ProduceRequest.json#L35
> >
> > But you're right, basically the message conversion happened in a very old
> > produce request version (ex: version 1 or version 2, which I guess is
> > corresponding to Kafka v1.0 or older) sent to a new versioned broker,
> like
> > this comment said in the file above:
> >
> >   // Version 3 adds the transactional ID, which is used for authorization
> > when attempting to write
> >   // transactional data.  *Version 3 also adds support for Kafka Message
> > Format v2.*
> >
> > Could you let us know which kafka version you're using in broker?
> > And which client and version you're using?
> > And the broker configuration and client configuration is also helpful.
> >
> > Thank you
> > Luke
> >
> >
> > On Tue, Feb 7, 2023 at 11:12 PM Gonzalo Martin Peci 
> > wrote:
> >
> > > Hi! We have been trying to figure out why we see a high value of
> > > ProduceMessageConversionsPerSec and potentially high CPU usage. I was
> > > trying to understand what version our producers/consumers were using,
> but I
> > > was unable to grok this. Any help would be appreciated.
> > >
> > > We are seeing values of `version=8` and `version=9` for RequestsPerSec
> > > "request=produce", and we found:
> > >
> > >
>
> https://github.com/apache/kafka/blob/trunk/server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java#L163-L171
> > > which indicate version 3.4 and 3.5.
> > >
> > > Im following what is described in KIP-272. Although I later found
> KIP-511
> > > which seems to expose a new metric, KIP-896 also implies that versions
> are
> > > checked through the RequestsPerSec metric.
> > >
> > > Any guidance will be appreciated.
> > >
> > > References:
> > > -
> > >
> > >
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-272%3A+Add+API+version+tag+to+broker%27s+RequestsPerSec+metric
> > > -
> > >
> > >
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-511%3A+Collect+and+Expose+Client%27s+Name+and+Version+in+the+Brokers
> > > -
> > >
> > >
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0
> > >
> > > Thanks
> > > Gonzalo
> > >
> >
>

Re: RequestsPerSec version

[Luke Chen]

Hi Gonzalo,

For the produce request record version, you should refer to this file:
https://github.com/apache/kafka/blob/trunk/clients/src/main/resources/common/message/ProduceRequest.json#L35

But you're right, basically the message conversion happened in a very old
produce request version (ex: version 1 or version 2, which I guess is
corresponding to Kafka v1.0 or older) sent to a new versioned broker, like
this comment said in the file above:

  // Version 3 adds the transactional ID, which is used for authorization
when attempting to write
  // transactional data.  *Version 3 also adds support for Kafka Message
Format v2.*

Could you let us know which kafka version you're using in broker?
And which client and version you're using?
And the broker configuration and client configuration is also helpful.

Thank you
Luke


On Tue, Feb 7, 2023 at 11:12 PM Gonzalo Martin Peci 
wrote:

> Hi! We have been trying to figure out why we see a high value of
> ProduceMessageConversionsPerSec and potentially high CPU usage. I was
> trying to understand what version our producers/consumers were using, but I
> was unable to grok this. Any help would be appreciated.
>
> We are seeing values of `version=8` and `version=9` for RequestsPerSec
> "request=produce", and we found:
>
> https://github.com/apache/kafka/blob/trunk/server-common/src/main/java/org/apache/kafka/server/common/MetadataVersion.java#L163-L171
> which indicate version 3.4 and 3.5.
>
> Im following what is described in KIP-272. Although I later found KIP-511
> which seems to expose a new metric, KIP-896 also implies that versions are
> checked through the RequestsPerSec metric.
>
> Any guidance will be appreciated.
>
> References:
> -
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-272%3A+Add+API+version+tag+to+broker%27s+RequestsPerSec+metric
> -
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-511%3A+Collect+and+Expose+Client%27s+Name+and+Version+in+the+Brokers
> -
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0
>
> Thanks
> Gonzalo
>

Re: Certificate renewal issue

[Luke Chen]

Hi Sandeep,

Maybe it'd be better to ask in strimzi community?

Luke

On Tue, Jan 10, 2023 at 5:57 PM Sandeep M  wrote:

> Hi all,
>
> We are using strimzi operator version 0.21.1 and Kubernetes 1.23.1. Broker
> certs are not getting renewed as a result we are facing issue. Let me know
> how to resolve this issue.
>
> Regards,
> Sandeep
>

Re: Topic partition Leader: none

[Luke Chen]

Hi,

That sounds similar to this issue: KAFKA-14190
.
Could you help confirm it and comment in the JIRA?
That will let contributors know this is an important issue bothering many
users.

Thank you.
Luke

On Tue, Jan 10, 2023 at 10:08 AM megh vidani  wrote:

> Hi Tom,
>
> We faced similar problem wherein there was an issue with isr and we were
> also getting NotLeaderOrFollowerException on consumer end. Also, it was not
> getting fixed automatically with broker restarts.
>
> We eventually found out that the topicId for a few partitions in the topic
> (in the partition.metadata file) was different from the actual topicId in
> zookeeper. I'd suggest you to check that as well.
>
> The way we fixed it was to remove the partition.metadata file (only this
> file alone!!) from all the partition directories of that topic and then
> restarting the brokers. This was the safest option we found as it doesn't
> incur any data loss. Before figuring this out we used to delete and
> re-create the topic which resulted into data being lost.
>
> Hope this helps.
>
> Thanks,
> Megh
>
> On Mon, 9 Jan 2023, 22:28 Tom Bolitho,  wrote:
>
> > Dear Kafka Community,
> >
> > I'm hoping you can help with kafka topic partition that is missing a
> > leader. The topic in question is the '__consumer_offsets' topic
> >
> > The output of a '--describe' on that topic looks like:
> >
> > Topic: __consumer_offsets   Partition: 7   Leader: none Replicas 5
> Isr:
> > 5
> > Topic: __consumer_offsets   Partition: 11  Leader: none Replicas 5
> Isr:
> > 5
> >
> > The other 48 partitions are all ok and have an assigned leader (some
> with 5
> > as the leader).
> >
> > I have tried running a --reassignment-json-file against the topic .e.g
> >
> > kafka-reassign-partitons.sh --bootstrap-server localhost:9092
> > --reassignment-json-file /.json  --execute
> >
> > but the reassignment just hangs, with the two partitions that are
> missing a
> > leader reporting:
> > 'Reassignment of partition __consumer_offsets-7 is still in progress'
> >
> > I've since had to --cancel that reassignment
> >
> > Can anyone advise on how I can overcome the issue of this missing leader
> > please?
> >
> > My eventual goal is to reassign this __consumer_offsets topic with a
> > replication factor of 3 to increase resiliency now that the cluster is in
> > production. I realise we should have set the
> > offets.topic.replication.factor to a value higher than 1 before we spun
> up
> > the prod cluster but this was missed so we're now looking to manually
> > reassign the __consumer_offsets with a higher replication factor.
> >
> > Any advice on how to overcome this 'Leader: none' issue would be greatly
> > appreciated.
> >
> > Many thanks,
> >
> > Tom
> >
>

Re: log.cleaner.io.max.bytes.per.second

[Luke Chen]

Hi Nicolas,

> Any Idea some may have to limit log compaction disk usage ? That's a
pretty
big issue we're having, compaction of a few topics is using far more IO
than everything else on the broker...

I don't know any other better solution except make the
`log.cleaner.io.max.bytes.per.second` lower than you expected, maybe 512?
As I explained earlier, based on how throttler works currently, there will
be a 300ms time slot of full speed read/write.
And it will make the final IO speed higher than you expected.

However, Apache Kafka is an open source project, so, welcome to make a
contribution to make it better!
Let me know if you're interested.

Thank you.
Luke

On Wed, Dec 28, 2022 at 6:25 PM Nicolas Carlot
 wrote:

> Okay, thx for the explaination Luc.
> When looking for logs on a bigger topic, I can see the IO limitation kinda
> works.
> I have like15MB/s when having configured
> log.cleaner.io.max.bytes.per.second=1024.0
> I'm not very comfortable with this. Setting the limit rate to 1Kb/s, ending
> with 15Mb/s, seems a bit akward.
> Any Idea some may have to limit log compaction disk usage ? That's a pretty
> big issue we're having, compaction of a few topics is using far more IO
> than everything else on the broker...
>
> Log cleaner thread 0 cleaned log
>
> kafka-stream-parcel360.lt_event360.multiprofiles-lteventmulti2-KSTREAM.RETAINER.kafka-stream-parcel360.lt_event360.multiprofiles-lteventmulti2-changelog-8
> (dirty section = [34819497, 44148502])
> 1 372,1 MB of log processed in 182,2 seconds (7,5 MB/sec).
> Indexed 1 143,2 MB in 84,9 seconds (13,5 Mb/sec, 46,6% of total
> time)
> Buffer utilization: 9,2%
> Cleaned 1 372,1 MB in 97,3 seconds (14,1 Mb/sec, 53,4% of total
> time)
> Start size: 1 372,1 MB (10 329 108 messages)
> End size: 230,0 MB (1 000 105 messages)
> 83,2% size reduction (90,3% fewer messages)
>
> Le mer. 28 déc. 2022 à 09:41, Luke Chen  a écrit :
>
> > Hi Nicolas,
> >
> > The throttler in log cleaner is just a simple throttler to control the IO
> > speed by periodically (every 300ms) check the IO rate so far, and sleep
> > some time to slow it down.
> > That is, in your case, it could be:
> > log cleaner runs for 300ms with full IO speed -> check -> throttle (sleep
> > to reach desired rate) -> run 300ms with full speed -> done
> >
> > So, when printing the log, you will see the rate is higher than you
> > expected.
> > At least I ran in kafka v3.3.1 and it works well.
> >
> > Thank you.
> > Luke
> >
> > On Thu, Dec 22, 2022 at 11:31 PM Nicolas Carlot
> >  wrote:
> >
> > > Hello users,
> > >
> > > I'm trying to use the log.cleaner.io.max.bytes.per.second configuration
> > to
> > > limit disk usage when the log cleaner runs, but it doesn't seem to work
> > as
> > > expected.
> > > When looking at the logs, I can see it still uses far more than
> > configured.
> > >
> > > This is the output with log.cleaner.io.max.bytes.per.second=1024
> > >
> > > [2022-12-22 16:14:49,462] INFO [kafka-log-cleaner-thread-0]:
> > > Log cleaner thread 0 cleaned log __consumer_offsets-9 (dirty
> > > section = [13509895330, 13510879425])
> > > 100,2 MB of log processed in 1,7 seconds (60,1 MB/sec).
> > > Indexed 99,7 MB in 0,8 seconds (117,5 Mb/sec, 50,8% of total
> > time)
> > > Buffer utilization: 0,1%
> > > Cleaned 100,2 MB in 0,8 seconds (122,2 Mb/sec, 49,2% of total
> > time)
> > > Start size: 100,2 MB (988642 messages)
> > > End size: 0,5 MB (4547 messages)
> > > 99,5% size reduction (99,5% fewer messages)
> > >
> > > I did update the value trough kafak-config this way:
> > >
> > > kafka-configs.sh --bootstrap-server localhost:59092 --entity-type
> brokers
> > > --command-config ~/server.admin.properties --entity-default --alter
> > > --add-config log.cleaner.io.max.bytes.per.second=1024
> > >
> > > Which shows up correctly in broker config:
> > >  log.cleaner.io.max.bytes.per.second=1024.0 sensitive=false
> > > synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:log.cleaner.io
> > > .max.bytes.per.second=1024,
> > > DEFAULT_CONFIG:log.cleaner.io
> > .max.bytes.per.second=1.7976931348623157E308}
> > >
> > > Any idea ? Something I'm missing ?
> > >
> > > --
> > > [image: Chronopost] <https://www.chronopost.fr/fr?xtatc=INT-149>
> > >
> > >
> > > *Nicolas Carlot*
> > > *Lead de

Re: log.cleaner.io.max.bytes.per.second

[Luke Chen]

Hi Nicolas,

The throttler in log cleaner is just a simple throttler to control the IO
speed by periodically (every 300ms) check the IO rate so far, and sleep
some time to slow it down.
That is, in your case, it could be:
log cleaner runs for 300ms with full IO speed -> check -> throttle (sleep
to reach desired rate) -> run 300ms with full speed -> done

So, when printing the log, you will see the rate is higher than you
expected.
At least I ran in kafka v3.3.1 and it works well.

Thank you.
Luke

On Thu, Dec 22, 2022 at 11:31 PM Nicolas Carlot
 wrote:

> Hello users,
>
> I'm trying to use the log.cleaner.io.max.bytes.per.second configuration to
> limit disk usage when the log cleaner runs, but it doesn't seem to work as
> expected.
> When looking at the logs, I can see it still uses far more than configured.
>
> This is the output with log.cleaner.io.max.bytes.per.second=1024
>
> [2022-12-22 16:14:49,462] INFO [kafka-log-cleaner-thread-0]:
> Log cleaner thread 0 cleaned log __consumer_offsets-9 (dirty
> section = [13509895330, 13510879425])
> 100,2 MB of log processed in 1,7 seconds (60,1 MB/sec).
> Indexed 99,7 MB in 0,8 seconds (117,5 Mb/sec, 50,8% of total time)
> Buffer utilization: 0,1%
> Cleaned 100,2 MB in 0,8 seconds (122,2 Mb/sec, 49,2% of total time)
> Start size: 100,2 MB (988642 messages)
> End size: 0,5 MB (4547 messages)
> 99,5% size reduction (99,5% fewer messages)
>
> I did update the value trough kafak-config this way:
>
> kafka-configs.sh --bootstrap-server localhost:59092 --entity-type brokers
> --command-config ~/server.admin.properties --entity-default --alter
> --add-config log.cleaner.io.max.bytes.per.second=1024
>
> Which shows up correctly in broker config:
>  log.cleaner.io.max.bytes.per.second=1024.0 sensitive=false
> synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:log.cleaner.io
> .max.bytes.per.second=1024,
> DEFAULT_CONFIG:log.cleaner.io.max.bytes.per.second=1.7976931348623157E308}
>
> Any idea ? Something I'm missing ?
>
> --
> [image: Chronopost] 
>
>
> *Nicolas Carlot*
> *Lead dev*Direction des Systèmes d'Information
>
>
> 3 boulevard Romain Rolland
> 75014 Paris
> [image: img] 
> [image:
> img]  [image:
> img]
>  [image: img]
> 
>
> [image: img]
>
> [image: img]
>

Re: [ANNOUNCE] New committer: Josep Prat

[Luke Chen]

Congratulations, Josep!

Luke

On Wed, Dec 21, 2022 at 6:26 AM Viktor Somogyi-Vass
 wrote:

> Congrats Josep!
>
> On Tue, Dec 20, 2022, 21:56 Matthias J. Sax  wrote:
>
> > Congrats!
> >
> > On 12/20/22 12:01 PM, Josep Prat wrote:
> > > Thank you all!
> > >
> > > ———
> > > Josep Prat
> > >
> > > Aiven Deutschland GmbH
> > >
> > > Immanuelkirchstraße 26, 10405 Berlin
> > >
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > >
> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >
> > > m: +491715557497
> > >
> > > w: aiven.io
> > >
> > > e: josep.p...@aiven.io
> > >
> > > On Tue, Dec 20, 2022, 20:42 Bill Bejeck  wrote:
> > >
> > >> Congratulations Josep!
> > >>
> > >> -Bill
> > >>
> > >> On Tue, Dec 20, 2022 at 1:11 PM Mickael Maison <
> > mickael.mai...@gmail.com>
> > >> wrote:
> > >>
> > >>> Congratulations Josep!
> > >>>
> > >>> On Tue, Dec 20, 2022 at 6:55 PM Bruno Cadonna 
> > >> wrote:
> > 
> >  Congrats, Josep!
> > 
> >  Well deserved!
> > 
> >  Best,
> >  Bruno
> > 
> >  On 20.12.22 18:40, Kirk True wrote:
> > > Congrats Josep!
> > >
> > > On Tue, Dec 20, 2022, at 9:33 AM, Jorge Esteban Quilcate Otoya
> wrote:
> > >> Congrats Josep!!
> > >>
> > >> On Tue, 20 Dec 2022, 17:31 Greg Harris,
> > >>  > 
> > >> wrote:
> > >>
> > >>> Congratulations Josep!
> > >>>
> > >>> On Tue, Dec 20, 2022 at 9:29 AM Chris Egerton <
> > >>> fearthecel...@gmail.com>
> > >>> wrote:
> > >>>
> >  Congrats Josep! Well-earned.
> > 
> >  On Tue, Dec 20, 2022, 12:26 Jun Rao 
> > >>> wrote:
> > 
> > > Hi, Everyone,
> > >
> > > The PMC of Apache Kafka is pleased to announce a new Kafka
> > >>> committer
> >  Josep
> > >Prat.
> > >
> > > Josep has been contributing to Kafka since May 2021. He
> > >>> contributed 20
> >  PRs
> > > including the following 2 KIPs.
> > >
> > > KIP-773 Differentiate metric latency measured in ms and ns
> > > KIP-744: Migrate TaskMetadata and ThreadMetadata to an
> interface
> > >>> with
> > > internal implementation
> > >
> > > Congratulations, Josep!
> > >
> > > Thanks,
> > >
> > > Jun (on behalf of the Apache Kafka PMC)
> > >
> > 
> > >>>
> > >>
> > >
> > >>>
> > >>
> > >
> >
>

Re: KRaft? cannot recreate a particular topic

[Luke Chen]

Hi Simon,

This is a known issue in kafka v3.3.1, and will be fixed in kafka v3.3.2
and v3.4.0.
Here's the JIRA issue: https://issues.apache.org/jira/browse/KAFKA-14337

Thank you
Luke

On Tue, Dec 20, 2022 at 7:13 PM Simon Dahlbacka 
wrote:

> I am running a single node cluster in KRaft mode on
> confluentinc/cp-kafka:7.3.0-1-ubi8
> aka kafka 3.3.x
>
> I cannot recreate a topic 'base.m3.ocusma' (but I can create,delete,create
> other topics just fine.
>
> the topic is deleted:
>
> ls /opt/kafka/data-0/logs/base.m3.ocusma
>
> ls: cannot access '/opt/kafka/data-0/logs/base.m3.ocusma': No such file or
> directory
>
>
> When I try to create the topic I see the following in the kafka log:
>
> [2022-12-20 09:38:15,882] WARN [Controller 0] createTopics: failed with
> unknown server exception NoSuchElementException at epoch 93 in 1311 us.
> Renouncing leadership and reverting to the last committed offset 2037058.
> (org.apache.kafka.controller.QuorumController)
> Tue, Dec 20 2022 11:38:15 am java.util.NoSuchElementException
> Tue, Dec 20 2022 11:38:15 am at
>
> org.apache.kafka.timeline.SnapshottableHashTable$CurrentIterator.next(SnapshottableHashTable.java:167)
> Tue, Dec 20 2022 11:38:15 am at
>
> org.apache.kafka.timeline.SnapshottableHashTable$CurrentIterator.next(SnapshottableHashTable.java:139)
> Tue, Dec 20 2022 11:38:15 am at
>
> org.apache.kafka.timeline.TimelineHashSet$ValueIterator.next(TimelineHashSet.java:120)
> Tue, Dec 20 2022 11:38:15 am at
>
> org.apache.kafka.controller.ReplicationControlManager.validateNewTopicNames(ReplicationControlManager.java:799)
> Tue, Dec 20 2022 11:38:15 am at
>
> org.apache.kafka.controller.ReplicationControlManager.createTopics(ReplicationControlManager.java:567)
> Tue, Dec 20 2022 11:38:15 am at
>
> org.apache.kafka.controller.QuorumController.lambda$createTopics$7(QuorumController.java:1832)
> Tue, Dec 20 2022 11:38:15 am at
>
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:767)
> Tue, Dec 20 2022 11:38:15 am at
>
> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:121)
> Tue, Dec 20 2022 11:38:15 am at
>
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:200)
> Tue, Dec 20 2022 11:38:15 am at
>
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:173)
> Tue, Dec 20 2022 11:38:15 am at
> java.base/java.lang.Thread.run(Thread.java:829)
> Tue, Dec 20 2022 11:38:15 am [2022-12-20 09:38:15,884] INFO [RaftManager
> nodeId=0] Received user request to resign from the current epoch 93
> (org.apache.kafka.raft.KafkaRaftClient)
> Tue, Dec 20 2022 11:38:15 am [2022-12-20 09:38:15,889] INFO [RaftManager
> nodeId=0] Completed transition to ResignedState(localId=0, epoch=93,
> voters=[0], electionTimeoutMs=1515, unackedVoters=[],
> preferredSuccessors=[]) (org.apache.kafka.raft.QuorumState)
> Tue, Dec 20 2022 11:38:16 am [2022-12-20 09:38:16,837] ERROR [Controller 0]
> processBrokerHeartbeat: unable to start processing because of
> NotControllerException. (org.apache.kafka.controller.QuorumController)
> Tue, Dec 20 2022 11:38:16 am [2022-12-20 09:38:16,840] INFO
> [BrokerToControllerChannelManager broker=0 name=heartbeat] Client requested
> disconnect from node 0 (org.apache.kafka.clients.NetworkClient)
> Tue, Dec 20 2022 11:38:16 am [2022-12-20 09:38:16,840] INFO
> [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
> controller, from now on will use broker
> kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
> (kafka.server.BrokerToControllerRequestThread)
> Tue, Dec 20 2022 11:38:16 am [2022-12-20 09:38:16,892] INFO
> [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
> controller, from now on will use broker
> kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
> (kafka.server.BrokerToControllerRequestThread)
> Tue, Dec 20 2022 11:38:16 am [2022-12-20 09:38:16,924] ERROR [Controller 0]
> processBrokerHeartbeat: unable to start processing because of
> NotControllerException. (org.apache.kafka.controller.QuorumController)
> Tue, Dec 20 2022 11:38:16 am [2022-12-20 09:38:16,926] INFO
> [BrokerToControllerChannelManager broker=0 name=heartbeat] Client requested
> disconnect from node 0 (org.apache.kafka.clients.NetworkClient)
> Tue, Dec 20 2022 11:38:16 am [2022-12-20 09:38:16,927] INFO
> [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
> controller, from now on will use broker
> kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
> (kafka.server.BrokerToControllerRequestThread)
> Tue, Dec 20 2022 11:38:16 am [2022-12-20 09:38:16,978] INFO
> [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
> controller, from now on will use broker
> kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
> (kafka.server.BrokerToControllerRequestThread)
> Tue, Dec 20 2022 11:38:17 am [2022-12-20 

Re: Granting permission for Create KIP and contribute to kafka

[Luke Chen]

Hi t-mac,

I've granted your JIRA account.
But I can't find the wiki id: "ws"  in the wiki system.
Are your sure you registered with "ws" here
?

Any more info you can provide to us, like account full name?

Thank you.
Luke

On Mon, Nov 28, 2022 at 12:00 AM t-mac  wrote:

> Hi, All:
>  I'd like to request permission to contribute to kafka~
>
>  Wiki Id:ws 
>  Jira Id:yws  
>  Jira Username: ws
>  Jira Full name: yws
>  
>
>
>
>
>
>
>
>
>
> Thanks a lot

Re: Leader election strategy

[Luke Chen]

Hi Pierre,

Try using kafka-reassign-partitions.sh to reassign partitions to different
replicas you like.
ref:  https://kafka.apache.org/documentation/#basic_ops_automigrate

Luke

On Mon, Nov 14, 2022 at 3:55 PM Pierre Coquentin 
wrote:

> Hello,
> We have a Kafka cluster (2.4.1) with a replication factor of 3. I notice
> when we stop a broker that only one broker takes all the load from the
> missing broker and becomes the leader to all partitions.
> I would have thought that Kafka would split the load evenly among the
> remaining brokers.
>
> So if I have this kind of configuration
> Topic: test
> Partition 0 - Leader: 1 - Replicas: 1,2,3 - Isr: 1,2,3
> Partition 1 - Leader: 2 - Replicas: 2,3,1 - Isr: 1,2,3
> Partition 2 - Leader: 3 - Replicas: 3,1,2 - Isr: 1,2,3
> Partition 3 - Leader: 1 - Replicas: 1,2,3 - Isr: 1,2,3
> Partition 4 - Leader: 2 - Replicas: 2,3,1 - Isr: 1,2,3
> Partition 5 - Leader: 3 - Replicas: 3,1,2 - Isr: 1,2,3
>
> If I stop broker 1, I want something like this (load is split evenly among
> broker 2 and 3):
> Topic: test
> Partition 0 - Leader: 2 - Replicas: 1,2,3 - Isr: 2,3
> Partition 1 - Leader: 2 - Replicas: 2,3,1 - Isr: 2,3
> Partition 2 - Leader: 3 - Replicas: 3,1,2 - Isr: 2,3
> Partition 3 - Leader: 3 - Replicas: 1,2,3 - Isr: 2,3
> Partition 4 - Leader: 2 - Replicas: 2,3,1 - Isr: 2,3
> Partition 5 - Leader: 3 - Replicas: 3,1,2 - Isr: 2,3
>
> What I observe is currently this (broker 2 takes all the load from broker
> 1):
> Partition 0 - Leader: 2 - Replicas: 1,2,3 - Isr: 2,3
> Partition 1 - Leader: 2 - Replicas: 2,3,1 - Isr: 2,3
> Partition 2 - Leader: 3 - Replicas: 3,1,2 - Isr: 2,3
> Partition 3 - Leader: 2 - Replicas: 1,2,3 - Isr: 2,3
> Partition 4 - Leader: 2 - Replicas: 2,3,1 - Isr: 2,3
> Partition 5 - Leader: 3 - Replicas: 3,1,2 - Isr: 2,3
>
> My concern here is that at all times, a broker should not exceed 50% of its
> network bandwidth which could be a problem in my case.
> Is there a way to change this behavior (manually by forcing a leader,
> programmatically, or by configuration)?
> From my understanding, the script kafka-leader-election.sh allows only to
> set the preferred (the first in the list of replicas) or uncleaned
> (replicas not in sync can become a leader).
> Regards,
>
> Pierre
>

Re: Connection issue

[Luke Chen]

Hi shikha,

I think you should ask in Spark community.

Thanks
Luke

On Tue, Nov 15, 2022 at 3:17 AM shikha sharma 
wrote:

> Hello,
>
> I am trying to connect to kafka using this command:
> orderRawData = spark.readStream \
> .format("kafka") \
> .option("kafka.bootstrap.servers", "18.211.252.152:9092") \
> .option("startingOffsets","earliest") \
> .option("failOnDataLoss", "false") \
> .option("subscribe", "real-time-project") \
> .load()
>
> It is giving me error as:
>
> 'Failed to find data source: kafka. Please deploy the application as
> per the deployment section of "Structured Streaming + Kafka
> Integration Guide".;'
> Traceback (most recent call last):
>   File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/streaming.py",
> line 400, in load
> return self._df(self._jreader.load())
>   File
> "/usr/lib/spark/python/lib/py4j-0.10.7-src.zip/py4j/java_gateway.py",
> line 1257, in __call__
> answer, self.gateway_client, self.target_id, self.name)
>   File "/usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/utils.py",
> line 69, in deco
> raise AnalysisException(s.split(': ', 1)[1], stackTrace)
> pyspark.sql.utils.AnalysisException: 'Failed to find data source:
> kafka. Please deploy the application as per the deployment section of
> "Structured Streaming + Kafka Integration Guide".;'
>
>
>
>
> could you please help me with this.
>

Re: [ANNOUNCE] New Kafka PMC Member: Bruno Cadonna

[Luke Chen]

Congrats Bruno!
Well deserved!

Luke

On Wed, Nov 2, 2022 at 10:07 AM John Roesler  wrote:

> Congratulations, Bruno!!!
>
> On Tue, Nov 1, 2022, at 15:16, Lucas Brutschy wrote:
> > Wow, congratulations!
> >
> > On Tue, Nov 1, 2022 at 8:55 PM Chris Egerton 
> wrote:
> >>
> >> Congrats!
> >>
> >> On Tue, Nov 1, 2022, 15:44 Bill Bejeck 
> wrote:
> >>
> >> > Congrats Bruno! Well deserved.
> >> >
> >> > -Bill
> >> >
> >> > On Tue, Nov 1, 2022 at 3:36 PM Guozhang Wang 
> wrote:
> >> >
> >> > > Hi everyone,
> >> > >
> >> > > I'd like to introduce our new Kafka PMC member, Bruno.
> >> > >
> >> > > Bruno has been a committer since April. 2021 and has been very
> active in
> >> > > the community. He's a key contributor to Kafka Streams, and also
> helped
> >> > > review a lot of horizontal improvements such as Mockito. It is my
> >> > pleasure
> >> > > to announce that Bruno has agreed to join the Kafka PMC.
> >> > >
> >> > > Congratulations, Bruno!
> >> > >
> >> > > -- Guozhang Wang, on behalf of Apache Kafka PMC
> >> > >
> >> >
>

Re: Help with MM2 active/passive configuration

[Luke Chen]

This is what Ryanne mentioned doc:
https://github.com/apache/kafka/tree/trunk/connect/mirror

Thanks.
Luke

On Mon, Oct 24, 2022 at 6:38 PM Chris Peart  wrote:

>
>
> Hi Ryanne,
>
> I cannot find what you mentioned, all i'm looking for are some example
> configurations for and active/passive setup.
>
> Many Thanks,
>
> Chris
>
> On 2022-10-19 18:32, Chris Peart wrote:
>
> > Thanks Ryan, can you provide a link to the readme please.
> > Many Thanks
> > Chris
> >
> > On 19 Oct 2022, at 6:15 pm, Ryanne Dolan  wrote:
> >
> > Hey Chris, check out the readme in connect/mirror for examples on how
> > to
> > run mirror maker as a standalone process. It's similar to how
> > Cloudera's
> > mirror maker is configured. If you've got a Connect cluster already, I
> > recommend using that and manually configuring the
> > MirrorSourceConnector.
> >
> > Ryanne
> >
> > On Wed, Oct 19, 2022, 5:26 AM Chris Peart  wrote:
> >
> > Hi All,
> >
> > I'm new to MirrorMaker and have a production cluster running version
> > 2.8.1 and have a development cluster running the same version.
> >
> > Our prod cluster has 4 brokers and our dev cluster has 3 brokers, both
> > have a replication factor of 3.
> >
> > I would like to setup an active/passive replication using MM2, we used
> > to do this with Cloudera but have we have decommissioned Cloudera and
> > would like to know how to configure topic replication using MM2.
> >
> > I believe i need a mm2.properties file to achieve this. i do see what
> > looks like a configuration in
> > /opt/kafka_2.13-2.8.1/config/connect-mirror-maker.properties but i'm
> > not
> > sure if this is an active/passive configuration.
> >
> > Ideally an example file would be ideal if possible?
> >
> > Many Thanks,
> >
> > Chris

Re: in case of disk failure,why not recover from middle position of the log file

[Luke Chen]

Hi Zhang,

Yes, when doing log recovery, we'll start from the checkpoint for the
partition in `recovery-point-offset-checkpoint` file, which will be updated
at runtime. Is that what you expected?

Check LogManager#loadLogs method for more implementation detail.

Thanks.
Luke

On Thu, Oct 13, 2022 at 7:26 PM zhang meng  wrote:

> can we write the byte position of .log 、.index file to a
> position-checkpoint file, and recover from the position to rebuild the
> index files?
>

Re: consumer loses offset

[Luke Chen]

Hi Lorenzo,

In theory, it should commit every 5 secs IF you keep polling the server.
But I saw you "stopped" the consumer for some hours, which means the commit
won't happen during this period.
So, if it exceeds the retention period, it'll get deleted.
That's my assumption. You need to check the logs (both client and server)
to find the clues.

Luke

On Mon, Oct 10, 2022 at 5:53 PM Lorenzo Rovere  wrote:

> Hi, thanks for your response.
>
> Is there any chance the offset is never committed on the
> "__consumer_offsets” topic, although auto commit is enabled every 5 seconds?
> We are checking daily and the offset is always set to NULL.
>
>
>
> Lorenzo Rovere
>
> Technology Reply
> Via Avogadri, 2
> 31057 - Silea (TV) - ITALY
> phone: +39 0422 1836521
> l.rov...@reply.it
> www.reply.it
> -Original Message-
> From: Luke Chen 
> Sent: 7 October, 2022 2:15 PM
> To: users@kafka.apache.org
> Subject: Re: consumer loses offset
>
> Hi Lorenzo,
>
> Sounds like it is caused by this bug:
> https://issues.apache.org/jira/browse/KAFKA-13636
> If you're not in the versions of fixed version list or newer, please try
> to upgrade it.
>
> Thanks.
> Luke
>
> On Fri, Oct 7, 2022 at 5:36 PM Lorenzo Rovere  wrote:
>
> > Hi everyone, I have a simple question about Kafka offsets.
> >
> > We have 1 producer and 1 consumer.
> >
> > Imagine the consumer reading till offset 5 (for example) and then
> > suddenly stops for some hours. In the meantime the producer keeps
> > writing messages and the offset of the last message is 10 (always for
> > example). When we restart the microservice that contains the consumer,
> > where does it start to read from, if we have auto.offset.reset=latest?
> >
> >
> >
> > I ask this because one of our costumers complains about starting
> > reading from 10, thus losing messages between 5 and 10. Is that correct?
> >
> >
> >
> > Other configs:
> > enable.auto.commit = true
> >
> > auto.commit.interval.ms = 5000
> >
> > log.retention.hours=168
> >
> > offsets.retention.minutes (7d)
> >
> >
> >
> > We also noticed that on the "__consumer_offsets” topic offset is
> > always set to NULL
> >
> > [consumer_group, topic_name,partition]::NULL
> >
> >
> >
> > Can you help me understanding what’s happening? Thanks a lot
> >
> >
> > Lorenzo Rovere
> >
> > Technology Reply
> > Via Avogadri, 2
> > 31057 - Silea (TV) - ITALY
> > phone: +39 0422 1836521
> > l.rov...@reply.it
> > www.reply.it
> >
> > [image: Technology Reply]
> >
>

Re: Apply to be a contributor for Kafka

[Luke Chen]

Hi Divya,

Done.
Thanks for the interest in Apache Kafka.

Luke

On Mon, Oct 10, 2022 at 6:20 AM Divya A L  wrote:

> I’m a developer of kafka, and I want to contribute for the project. Can I
> be added as a contributor, as I would like to pick up an issue from the
> jira board.
>
> jira ID : divyaal22
>
> Thanks!
> Divya
>

Re: consumer loses offset

[Luke Chen]

Hi Lorenzo,

Sounds like it is caused by this bug:
https://issues.apache.org/jira/browse/KAFKA-13636
If you're not in the versions of fixed version list or newer, please try to
upgrade it.

Thanks.
Luke

On Fri, Oct 7, 2022 at 5:36 PM Lorenzo Rovere  wrote:

> Hi everyone, I have a simple question about Kafka offsets.
>
> We have 1 producer and 1 consumer.
>
> Imagine the consumer reading till offset 5 (for example) and then suddenly
> stops for some hours. In the meantime the producer keeps writing messages
> and the offset of the last message is 10 (always for example). When we
> restart the microservice that contains the consumer, where does it start to
> read from, if we have auto.offset.reset=latest?
>
>
>
> I ask this because one of our costumers complains about starting reading
> from 10, thus losing messages between 5 and 10. Is that correct?
>
>
>
> Other configs:
> enable.auto.commit = true
>
> auto.commit.interval.ms = 5000
>
> log.retention.hours=168
>
> offsets.retention.minutes (7d)
>
>
>
> We also noticed that on the "__consumer_offsets” topic offset is always
> set to NULL
>
> [consumer_group, topic_name,partition]::NULL
>
>
>
> Can you help me understanding what’s happening? Thanks a lot
>
>
> Lorenzo Rovere
>
> Technology Reply
> Via Avogadri, 2
> 31057 - Silea (TV) - ITALY
> phone: +39 0422 1836521
> l.rov...@reply.it
> www.reply.it
>
> [image: Technology Reply]
>

Re: Metadata Refresh and TimeoutException when MAX_BLOCK_MS_CONFIG set 0

[Luke Chen]

Hi Bhavesh,

I understand your point.
There was an old KIP with the similar idea which was not accepted by the
community in the end.
Maybe you can try to bring it back to the community again, or try to
propose your own KIP for this idea?
https://cwiki.apache.org/confluence/display/KAFKA/KIP-286%3A+producer.send%28%29+should+not+block+on+metadata+update

Thank you.
Luke

On Sat, Sep 24, 2022 at 6:36 AM Bhavesh Mistry 
wrote:

> Hello Kafka Team,
>
> I would appreciate any insight into how to distinguish between Brocker Down
> vs Metadata Refresh not available due to timing issues.
>
> Thanks,
>
> Bhavesh
>
> On Mon, Sep 19, 2022 at 12:50 PM Bhavesh Mistry <
> mistry.p.bhav...@gmail.com>
> wrote:
>
> > Hello Kafka Team,
> >
> >
> >
> > We have an environment where Kafka Broker can go down for whatever
> reason.
> >
> >
> >
> > Hence, we had configured MAX_BLOCK_MS_CONFIG=0 because we wanted to drop
> > messages when brokers were NOT available.
> >
> >
> >
> > Now the issue is we get data loss due to METADATA not being available and
> > get this exception “*Topic  not present in metadata after 0 ms.”.
> > *This is due to the fast metadata has expired and the next request to
> > send an event does not have metadata.
> >
> >
> >
> > Why does Kafka have his design?  Why can’t Kafka distinguish between
> > Broker down vs metadata refresh not available?  Is it reasonable to
> expect
> > metadata would refresh BEFORE it expires so metadata refresh doesn’t need
> > before it expires? Have Metadata ready before expires?  Any particular
> > reason send() has wait for metadata refresh vs background thread that
> > automatically refreshes metadata before it expires, hence send() method
> > never incur wait().
> >
> >
> > Let me know what suggestion you have to prevent the application thread
> > from blocking (MAX_BLOCK_MS_CONFIG) when the Kafka brokers are DOWN vs
> > metadata is NOT available due to expiration.
> >
> >
> >
> > Let me know your suggestions and what you think about metadata refresh.
> > Should Kafka Producer be proactively refreshing metadata intelligently
> > rather than what the producer does today?
> >
> >
> >
> >
> >
> > Thanks,
> > Bhavesh
> >
>

Re: Kafka consumer ensure offsets committed before partition rebalance to avoid duplicates

[Luke Chen]

Hi

1. I was under impression, from documentation, that close method waits for
30 seconds to complete processing of any in-flight events and then commits
offsets of last poll. Isn't that true? what does timeout of 30 seconds mean?

-> 30 seconds timeout is to have a buffer for graceful closing, ex: commit
offsets, leave groups,...
It won't complete processing any in-flight "fetch" events during closing.

2. How does CoperativeStickyAssignor solve my problem when partitions move
out to newly added consumer pod. i.e. consumer1 has polled 100 records from
partition1 and is midway processing those i.e. 50 completed, 50 remaining
and new consumer is added so partition1 has to move to new consumer2. Since
auto.commit is enabled, offsets of all 100 polled records will be committed
only during next poll. 2. How does CoperativeStickyAssignor solve my
problem when partitions move
out to newly added consumer pod. i.e. consumer1 has polled 100 records from
partition1 and is midway processing those i.e. 50 completed, 50 remaining
and new consumer is added so partition1 has to move to new consumer2. Since
auto.commit is enabled, offsets of all 100 polled records will be committed
only during next poll.

-> You're right about the process.

So how does CooperativeStickyAssignore help here to
wait for process of 100 records and commit their offsets before moving the
partition to new consumer? Looks like i am missing something
Looks like i am missing something

-> CooperativeStickyAssignore does the same thing to it, except it will
keep all the partitions "during rebalancing".
So, the issue is:
In eagar protocol (ex: RangeAssignor)
consumer prepare rebalancing -> commit offsets -> revoke all owned
partitions -> rebalancing -> received new assignment -> start fetch data
In cooperative protocol (ex: CooperativeStickyAssignore)
consumer prepare rebalancing -> commit offsets (but no revoke) ->
rebalancing -> received new assignment -> revoke partitions not owned
anymore

So you can see, in cooperative protocol, since it didn't revoke any
partition before rebalancing, it might fetch more data after offset commits.

Hope that's clear
Luke

On Tue, Sep 20, 2022 at 9:36 PM Pushkar Deole  wrote:

> Thanks Luke..
>
> 1. I was under impression, from documentation, that close method waits for
> 30 seconds to complete processing of any in-flight events and then commits
> offsets of last poll. Isn't that true? what does timeout of 30 seconds
> mean?
>
> 2. How does CoperativeStickyAssignor solve my problem when partitions move
> out to newly added consumer pod. i.e. consumer1 has polled 100 records from
> partition1 and is midway processing those i.e. 50 completed, 50 remaining
> and new consumer is added so partition1 has to move to new consumer2. Since
> auto.commit is enabled, offsets of all 100 polled records will be committed
> only during next poll. So how does CooperativeStickyAssignore help here to
> wait for process of 100 records and commit their offsets before moving the
> partition to new consumer? Looks like i am missing something
>
> On Fri, Sep 16, 2022 at 7:59 AM Luke Chen  wrote:
>
> > Hi Pushkar,
> >
> > Here's the answer to your questions:
> >
> > > 1. During scale-down operation, I am adding a shutdown hook to the Java
> > Runtime, and calling close on the consumer. As per kafka docs, close
> > provides 30 sec to commit current offsets if auto.commit is enabled: so,
> i
> > assume that it will process the current batch of polled records within 30
> > sec timeout before committing offsets and then close the consumer. Is my
> > understanding correct?
> >
> > No, close() method is only doing some cleanup and offset commit if
> needed.
> > It won't care if the polled records are processed or not.
> > So, to be clear, the 30 seconds is for consumer to do:
> > (1) commit offset if auto.commit is enabled (2) leave consumer group (3)
> > other cleanup
> >
> > > 2. During scale out operation, new pod (consumer) will be added to the
> > consumer group, so partitions of existing consumers will be rebalanced to
> > new consumer. In this case, I want to ensure that the current batch of
> > records polled and being processed by the consumer is processed and
> offsets
> > are committed before partition rebalance happens to new consumer.
> > How can I ensure this with auto-commit enabled?
> >
> > It depends on which version of Kafka you're running, and which
> > `partition.assignment.strategy` you are setting.
> > In Kafka v3.2.1, we found a bug that it'll have chance to process
> duplicate
> > records during rebalance: KAFKA-14196
> > <https://issues.apache.org/jira/browse/KAFKA-14196>
> > So, assuming you're using def

Re: CVE-2022-34917: Unauthenticated clients may cause OutOfMemoryError on Apache Kafka Brokers

[Luke Chen]

What a great finding!
Thanks Mickael Maison, Tom Bentley and Daniel Collins!

And thanks for all the release managers who help drive all these security
patch releases!

Luke


On Mon, Sep 19, 2022 at 11:53 PM Manikumar  wrote:

> Severity: High
>
> Description:
>
> A security vulnerability has been identified in Apache Kafka. It
> affects all releases since 2.8.0. The vulnerability allows malicious
> unauthenticated clients to allocate large amounts of memory on
> brokers. This can lead to brokers hitting OutOfMemoryException and
> causing denial of service.
>
> Example scenarios:
> - Kafka cluster without authentication: Any clients able to establish
> a network connection to a broker can trigger the issue.
> - Kafka cluster with SASL authentication: Any clients able to
> establish a network connection to a broker, without the need for valid
> SASL credentials, can trigger the issue.
> - Kafka cluster with TLS authentication: Only clients able to
> successfully authenticate via TLS can trigger the issue.
>
> We advise the users to upgrade the Kafka installations to one of the
> 3.2.3, 3.1.2, 3.0.2, 2.8.2 versions.
>
> Credit:
>
> Apache Kafka would like to thank Mickael Maison, Tom Bentley and
> Daniel Collins for reporting this issue.
>
> References:
>
> https://kafka.apache.org/cve-list
>

Re: Kafka consumer ensure offsets committed before partition rebalance to avoid duplicates

[Luke Chen]

Hi Pushkar,

Here's the answer to your questions:

> 1. During scale-down operation, I am adding a shutdown hook to the Java
Runtime, and calling close on the consumer. As per kafka docs, close
provides 30 sec to commit current offsets if auto.commit is enabled: so, i
assume that it will process the current batch of polled records within 30
sec timeout before committing offsets and then close the consumer. Is my
understanding correct?

No, close() method is only doing some cleanup and offset commit if needed.
It won't care if the polled records are processed or not.
So, to be clear, the 30 seconds is for consumer to do:
(1) commit offset if auto.commit is enabled (2) leave consumer group (3)
other cleanup

> 2. During scale out operation, new pod (consumer) will be added to the
consumer group, so partitions of existing consumers will be rebalanced to
new consumer. In this case, I want to ensure that the current batch of
records polled and being processed by the consumer is processed and offsets
are committed before partition rebalance happens to new consumer.
How can I ensure this with auto-commit enabled?

It depends on which version of Kafka you're running, and which
`partition.assignment.strategy` you are setting.
In Kafka v3.2.1, we found a bug that it'll have chance to process duplicate
records during rebalance: KAFKA-14196

So, assuming you're using default `partition.assignment.strategy` setting,
and not in v3.2.1, we can ensure it will not have duplicated consumption.
If you set the `partition.assignment.strategy` to
cooperativeStickyAssignor, there's a bug that we're still working on:
KAFKA-14224 

Thank you.
Luke

On Wed, Sep 14, 2022 at 3:09 PM Pushkar Deole  wrote:

> Hi All,
>
> I am hosting kafka consumers inside microservice hosted as kubernetes pods,
> 3 consumers in a consumer group.
> There is a requirement to add auto-scaling where there will be a single pod
> which will be auto-scaled out or scaled-in based on the load on
> microservice.
> So, 1 pod can be scaled out to 2 or 3 pods, and similarly 3 pods can be
> scaled down to 2 or 1 pod.
>
> Currently, I am using enabled.auto.commit set to 'true' in the consumers
> and during scale out or scale-in, i want to commit offset of polled and
> processed records so duplicates won't occur.
> I have narrowed the problem to 2 scenarios:
>
> 1. During scale-down operation, I am adding a shutdown hook to the Java
> Runtime, and calling close on the consumer. As per kafka docs, close
> provides 30 sec to commit current offsets if auto.commit is enabled: so, i
> assume that it will process the current batch of polled records within 30
> sec timeout before committing offsets and then close the consumer. Is my
> understanding correct?
>
> public void close()
>
> Close the consumer, waiting for up to the default timeout of 30 seconds for
> any needed cleanup. If auto-commit is enabled, this will commit the current
> offsets if possible within the default timeout. See close(Duration)
> <
> https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#close-java.time.Duration-
> >
> for
> details. Note that wakeup()
> <
> https://kafka.apache.org/25/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#wakeup--
> >
> cannot
> be used to interrupt close.
>
> 2. During scale out operation, new pod (consumer) will be added to the
> consumer group, so partitions of existing consumers will be rebalanced to
> new consumer. In this case, I want to ensure that the current batch of
> records polled and being processed by the consumer is processed and offsets
> are committed before partition rebalance happens to new consumer.
> How can I ensure this with auto-commit enabled?
>

Re: Add to kafka contributor list

[Luke Chen]

Hi Vinay,

Done.
Thanks for the interest in Apache Kafka.

Luke

On Mon, Sep 5, 2022 at 1:42 PM vinay kumar 
wrote:

> Hi Bill,
>
> My jira user : vvakati
>
> Can you add me to contributor list please?
>
> Thanks,
> Vinay
>
> On Fri, 2 Sep 2022 at 12:09 AM, Bill Bejeck 
> wrote:
>
> > Hi Vinay,
> >
> > Can you confirm your Jira username? I was unable to locate your account
> on
> > Jira.
> >
> > Thanks!
> > Bill
> >
> > On Thu, Sep 1, 2022 at 6:23 AM vinay kumar 
> > wrote:
> >
> > > Hi,
> > >
> > > Could you add me to Kafka contributor list please?
> > >
> > >
> > > I want to start contributing to Kafka
> > >
> > > Jira user : Vakati
> > >
> > > Thanks,
> > > Vinay
> > >
> >
>

Re: Unable to reset kafka offset in a consumer group

[Luke Chen]

Hi Chris,

Great to hear you found a way to work around it now.
Could you share your solution here in case other people got stuck on the
same issue?

Thank you
Luke

On Fri, Aug 26, 2022 at 11:57 PM Chris Peart  wrote:

>
> Thanks Luke,
> The problem I was having was that the consumer group was not inactive even
> with all the consumers stopped.
> Managed to work around this now.
>
> Many Thanks
> Chris
>
> > On 25 Aug 2022, at 10:12 am, Luke Chen  wrote:
> > Hi Chris,
> >
> >> is there a way to force the offset forward by an increment of 1 just for
> > this topic?
> > kafka-consumer-groups.sh script has an option `--shift-by` to shift
> current
> > offset by 'n'.
> > I think this is what you are looking for.
> >
> > Thank you
> > Luke
> >
> > On Thu, Aug 25, 2022 at 4:25 PM Chris Peart  wrote:
> >
> >>
> >>
> >> Hi All,
> >>
> >> I'm trying to reset a kafka offset for a topic in a consumer group, i
> >> have stopped all the consumers using the consumer group but i'm still
> >> receiving the message that the current state is stable.
> >>
> >> Is there a way to put the consumer group to an inactive state after
> >> stopping all the consumers or is there a way to force the offset forward
> >> by an increment of 1 just for this topic?
> >>
> >> Many Thanks,
> >>
> >> Chris
>
>

Re: Unable to reset kafka offset in a consumer group

[Luke Chen]

Hi Chris,

> is there a way to force the offset forward by an increment of 1 just for
this topic?
kafka-consumer-groups.sh script has an option `--shift-by` to shift current
offset by 'n'.
I think this is what you are looking for.

Thank you
Luke

On Thu, Aug 25, 2022 at 4:25 PM Chris Peart  wrote:

>
>
> Hi All,
>
> I'm trying to reset a kafka offset for a topic in a consumer group, i
> have stopped all the consumers using the consumer group but i'm still
> receiving the message that the current state is stable.
>
> Is there a way to put the consumer group to an inactive state after
> stopping all the consumers or is there a way to force the offset forward
> by an increment of 1 just for this topic?
>
> Many Thanks,
>
> Chris

Re: Regarding kafka 2.3.0

[Luke Chen]

1. Is kafka 2.3.0 going end of life ,If yes then what is the expected date?
-> Kafka supports last 3 releases.

REF:
https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan#TimeBasedReleasePlan-WhatIsOurEOLPolicy
?

2. Is kafka 3.1.0 backward compatible to 2.3.0?
-> Since 2.3 to 3.1 has one major release (through 3.0), some deprecated
features are removed. You can refer to this doc for upgrade guide:
https://kafka.apache.org/documentation/#upgrade_3_1_0 , and check for
release note for each release.

Thanks
Luke

On Thu, Aug 25, 2022 at 3:42 PM Fred Bai  wrote:

> +1
> Me too, We consider upgrading Kafka to 3.X from Kafka 2.X, but don't know
> the compatibility.
>
> thx
>
> Ankit Saran  于2022年8月23日周二 22:21写道：
>
> > Hi Team,
> > We are planning to upgrade kafka version from 2.3.0 to 3.1.0 , We have
> > below queries regarding the same
> >
> > 1. Is kafka 2.3.0 going end of life ,If yes then what is the expected
> date?
> > 2. Is kafka 3.1.0 backward compatible to 2.3.0?
> >
> > Please help us with the above queries, Thanks in advance.
> >
> > Regards,
> > Ankit Saran
> >
>

Re: Fw: An existing connection was forcibly closed by the remote host

[Luke Chen]

Hi Podunk,

If you're saying the error: ERROR Exiting JVM with code 0
(org.apache.zookeeper.util.ServiceUtils), I think this is a bug in Kafka.
I guess it's because we didn't close ZooKeeperAdmin before exit.
Please open a JIRA ticket for it, and welcome to file a PR for it!

But for the ZK shell, it works well.
It returned [0] to you since you only have one broker in the cluster with
id 0.
For me, I used to enter the ZK shell, and type command, like this:

> bin/zookeeper-shell.sh localhost:2181
Connecting to localhost:2181
Welcome to ZooKeeper!


# then, type commands like in normal shell
ls /brokers
[ids, seqid, topics]
ls /brokers/ids
[0]
...

Hope that helps.

Luke

On Wed, Aug 3, 2022 at 7:32 PM  wrote:

> Anyone?
>
>
>
>
> Sent: Saturday, July 30, 2022 at 9:14 PM
> From: pod...@gmx.com
> To: users@kafka.apache.org
> Subject: An existing connection was forcibly closed by the remote host
>
>
> Hi,
>
> I start to learn Kafka and.. I have first problem.
> I started Zookeeper, I started Kafka - both on localhost. Seems all is
> fine. But when I try to execute command:
>
> zookeeper-shell.bat localhost:2181 ls /brokers/ids
>
> I get error:
>
> Connecting to localhost:2181
> WATCHER::
> WatchedEvent state:SyncConnected type:None path:null
> [0]
> [2022-07-30 19:54:22,524] ERROR Exiting JVM with code 0
> (org.apache.zookeeper.util.ServiceUtils)
>
> And in Zookeeper:
>
> [2022-07-30 19:54:53,335] INFO Expiring session 0x1190aa20001, timeout
> of 3ms exceeded (org.apache.zookeeper.server.ZooKeeperServer)
> [2022-07-30 19:58:05,393] WARN Close of session 0x1190aa20002
> (org.apache.zookeeper.server.NIOServerCnxn)
> java.io.IOException: An existing connection was forcibly closed by the
> remote host
> at java.base/sun.nio.ch.SocketDispatcher.read0(Native Method)
> at java.base/sun.nio.ch
> .SocketDispatcher.read(SocketDispatcher.java:43)
> at
> java.base/sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276)
> at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:245)
> at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:223)
> at java.base/sun.nio.ch
> .SocketChannelImpl.read(SocketChannelImpl.java:358)
> at
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:324)
> at
> org.apache.zookeeper.server.NIOServerCnxnFactory$IOWorkRequest.doWork(NIOServerCnxnFactory.java:522)
> at
> org.apache.zookeeper.server.WorkerService$ScheduledWorkRequest.run(WorkerService.java:154)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
>
>
> server.properties:
> listeners=PLAINTEXT://localhost:9092
> advertised.listeners=PLAINTEXT://localhost:9092
>
> zookeeper.connection.timeout.ms=18000
>
> zookeeper.properties:
> clientPort=2181
> maxSessionTimeout=18000
>
> What can be the reason?
>

Re: [ANNOUNCE] New Kafka PMC Member: A. Sophie Blee-Goldman

[Luke Chen]

Congrats Sophie! :)

Luke

On Tue, Aug 2, 2022 at 7:56 AM Adam Bellemare 
wrote:

> Congratulations Sophie! I’m glad to see you made as a PMC member! Well
> earned.
>
> > On Aug 1, 2022, at 6:42 PM, Guozhang Wang  wrote:
> >
> > Hi everyone,
> >
> > I'd like to introduce our new Kafka PMC member, Sophie. She has been a
> > committer since Oct. 2020 and has been contributing to the community
> > consistently, especially around Kafka Streams and Kafka java consumer.
> She
> > has also presented about Kafka Streams at Kafka Summit London this year.
> It
> > is my pleasure to announce that Sophie agreed to join the Kafka PMC.
> >
> > Congratulations, Sophie!
> >
> > -- Guozhang Wang, on behalf of Apache Kafka PMC
>

Re: Kafka certificate monitoring

[Luke Chen]

Hi Sandeep,

AFAIK, Kafka doesn't expose this kind of metrics.
I did a quick search, and found there's a similar request in Strimzi.
https://github.com/strimzi/strimzi-kafka-operator/issues/3761

Maybe you can help contribute it? Either to Kafka or to Strimzi? :)

Thank you.
Luke

On Wed, Jul 27, 2022 at 11:28 PM Sandeep M  wrote:

> Hi Team,
>
> I using Kafka with strimzi as kafka operator. I wanted to monitor kafka
> certificate expiry. We have Rancher kubernetes management tool which has
> built in Prometheus and Grafana. How could we monitor certificate expiry
> with current setup.
>
> Regards,
> Sandeep
>

Re: [ANNOUNCE] New Committer: Chris Egerton

[Luke Chen]

Congratulations Chris! Well deserved!

Luke

On Tue, Jul 26, 2022 at 5:39 AM Anna McDonald 
wrote:

> Congratulations Chris! Time to Cellobrate!
>
> anna
>
> On Mon, Jul 25, 2022 at 4:23 PM Martin Gainty  wrote:
>
> > Congratulations Chris!
> >
> > martin~
> > 
> > From: Mickael Maison 
> > Sent: Monday, July 25, 2022 12:25 PM
> > To: dev ; Users 
> > Subject: [ANNOUNCE] New Committer: Chris Egerton
> >
> > Hi all,
> >
> > The PMC for Apache Kafka has invited Chris Egerton as a committer, and
> > we are excited to announce that he accepted!
> >
> > Chris has been contributing to Kafka since 2017. He has made over 80
> > commits mostly around Kafka Connect. His most notable contributions
> > include KIP-507: Securing Internal Connect REST Endpoints and KIP-618:
> > Exactly-Once Support for Source Connectors.
> >
> > He has been an active participant in discussions and reviews on the
> > mailing lists and on Github.
> >
> > Thanks for all of your contributions Chris. Congratulations!
> >
> > -- Mickael, on behalf of the Apache Kafka PMC
> >
>

Re: NoAuth for /brokers/ids

[Luke Chen]

Hi Evgeny

You can check the zookeeper log to see if there are logs about why the
error happened.

Thanks.
Luke

On Tue, Jul 5, 2022 at 1:34 AM Ivanov, Evgeny 
wrote:

> Hi everyone,
>
> could you please advise how to fix the problem below ?
>
> I'm trying to run Zookeeper with mTLS to authenticate Kafka broker on
> Zookeeper by SSL certificate.
> Both Zookeeper and Kafka broker are located on the same server, so I use
> the same keystore and trustore for them.
>
> Here is the error in Kafka server.log when Kafka starts:
>
> [2022-07-01 19:16:44,157] DEBUG [id: 0x7b9f05b5, L:/10.76.196.200:53876 -
> R:smsk01ap437u/10.76.196.200:2182] HANDSHAKEN: protocol:TLSv1.2 cipher
> suite:TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
> (io.netty.handler.ssl.SslHandler)
> [2022-07-01 19:16:44,206] INFO Session establishment complete on server
> smsk01ap437u/10.76.196.200:2182, session id = 0x100bb14c3bf,
> negotiated timeout = 18000 (org.apache.zookeeper.ClientCnxn)
> [2022-07-01 19:16:44,210] DEBUG [ZooKeeperClient Kafka server] Received
> event: WatchedEvent state:SyncConnected type:None path:null
> (kafka.zookeeper.ZooKeeperClient)
> [2022-07-01 19:16:44,210] INFO [ZooKeeperClient Kafka server] Connected.
> (kafka.zookeeper.ZooKeeperClient)
> [2022-07-01 19:16:44,320] DEBUG Reading reply session id:
> 0x100bb14c3bf, packet:: clientPath:/consumers serverPath:/consumers
> finished:false header:: 1,1  replyHeader:: 1,77309411356,-110  request::
> '/consumers,,v{s{31,s{'world,'anyone}}},0  response::
>  (org.apache.zookeeper.ClientCnxn)
> [2022-07-01 19:16:44,346] DEBUG Reading reply session id:
> 0x100bb14c3bf, packet:: clientPath:/brokers/ids serverPath:/brokers/ids
> finished:false header:: 2,1  replyHeader:: 2,77309411357,-102  request::
> '/brokers/ids,,v{s{31,s{'world,'anyone}}},0  response::
>  (org.apache.zookeeper.ClientCnxn)
> [2022-07-01 19:16:44,358] ERROR Fatal error during KafkaServer startup.
> Prepare to shutdown (kafka.server.KafkaServer)
> org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode =
> NoAuth for /brokers/ids
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:120)
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
> at
> kafka.zookeeper.AsyncResponse.maybeThrow(ZooKeeperClient.scala:566)
> at kafka.zk.KafkaZkClient.createRecursive(KafkaZkClient.scala:1729)
> at
> kafka.zk.KafkaZkClient.makeSurePersistentPathExists(KafkaZkClient.scala:1627)
> at
> kafka.zk.KafkaZkClient.$anonfun$createTopLevelPaths$1(KafkaZkClient.scala:1619)
> at
> kafka.zk.KafkaZkClient.$anonfun$createTopLevelPaths$1$adapted(KafkaZkClient.scala:1619)
> at scala.collection.immutable.List.foreach(List.scala:333)
> at
> kafka.zk.KafkaZkClient.createTopLevelPaths(KafkaZkClient.scala:1619)
> at kafka.server.KafkaServer.initZkClient(KafkaServer.scala:492)
> at kafka.server.KafkaServer.startup(KafkaServer.scala:201)
> at kafka.Kafka$.main(Kafka.scala:109)
> at kafka.Kafka.main(Kafka.scala)
> [2022-07-01 19:16:44,359] INFO shutting down (kafka.server.KafkaServer)
>
> Here are the configs.
>
> Zoo.cfg:
>
> secureClientPort=2182
> serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
>
> authProvider.x509=org.apache.zookeeper.server.auth.X509AuthenticationProvider
> ssl.keyStore.location=/app/kafka/certs/server/server.keystore.jks
> ssl.keyStore.password=Moscow123
> ssl.trustStore.location=/app/kafka/certs/server/server.truststore.jks
> ssl.trustStore.password=Moscow123
>
> server.properties:
>
> zookeeper.connect=server_hostname:2182
> zookeeper.ssl.client.enable=true
> zookeeper.clientCnxnSocket=org.apache.zookeeper.ClientCnxnSocketNetty
> zookeeper.ssl.keystore.location=/app/kafka/certs/server/server.keystore.jks
> zookeeper.ssl.keystore.password=Moscow123
> zookeeper.ssl.truststore.location=kafka/certs/server/server.truststore.jks
> zookeeper.ssl.truststore.password=Moscow123
>
> Best regards,
> Evgeny
>
>
> 
>
> This email message (and any attachments) is confidential and may be
> privileged or otherwise protected from disclosure by applicable law. If you
> are not the intended recipient or have received this in error please notify
> the system manager, postmas...@vtbcapital.ru and remove this message and
> any attachments from your system. Any unauthorized dissemination, copying
> or other use of this message and/or any attachments is strictly prohibited
> and may constitute a breach of civil or criminal law.
> JSC VTB Capital may monitor email traffic data and also the content of
> email.
>

Re: a little problem in quickstart

[Luke Chen]

Thanks Mason & Chris,

The change to current doc is in kafka-site repo:
https://github.com/apache/kafka-site
I've commented on the PR https://github.com/apache/kafka/pull/12252 to ask
for submitting another PR in kafka-site repo.

Thank you.
Luke

On Mon, Jun 27, 2022 at 12:28 AM Chris Egerton 
wrote:

> Hi Mason,
>
> You're correct that the quickstart should use 'libs' instead of 'lib'.
> This has already been fixed in the docs for the upcoming 3.3.0 release with
> https://github.com/apache/kafka/pull/12252. We might consider backporting
> that change; I've CC'd Luke Chen, who merged that fix and might be able to
> help with backporting it (I'd take it on myself but I'm not well-versed in
> how the site docs work, especially with making changes for already-released
> versions).
>
> Cheers,
>
> Chris
>
> On Sun, Jun 26, 2022 at 11:12 AM Men Lim  wrote:
>
>> You don't need to put in the jar file name in the plug in.path variable.
>> Something like plugin.path=/kafka/plugin. Then have the jar file in that
>> plugin folder. Restart the worker and it will pick it up.
>>
>> On Sun, Jun 26, 2022 at 8:04 AM mason lee  wrote:
>>
>> >  Hi I’m new to Kafka and i can not pass step 6 in
>> > https://kafka.apache.org/quickstart, finally I found that the word
>> ‘lib’
>> > in
>> > 'echo "plugin.path=lib/connect-file-3.2.0.jar’ should be ‘libs’.
>> > It bothered me for a while, I think a change would be better.
>> >
>>
>

Re: Granting contributor permission for Jira

[Luke Chen]

Hi,

Done.
Thanks for the interest in Apache Kafka.

Luke

On Sat, Jun 18, 2022 at 10:17 AM lith_angelo 
wrote:

> Please add my Jira ID to contributor list, so that I'd be able to handle
> Jira ticket, thanks.
> Jira ID: lith_angelo

Re: Broker allows transactions with generation.id -1 and could lead to duplicates

[Luke Chen]

Hi Gabriel,

Sounds like a bug to me (although we didn't document anywhere about the
generation id will always start from 0).
You can file a jira and we can discuss it there.

Thank you.
Luke

On Fri, Jun 10, 2022 at 9:35 PM Gabriel Giussi 
wrote:

> I did the following test that allowed me to introduce a duplicate message
> in the output topic.
>
>
> 1. Client A starts the consumer and the producer and holds a reference to
> the current groupMetadata wich has generation.id -1 since the consumer
> didn't join the group yet
> 2. Client A joins the group and gets assigned partition 0 and 1
> 3. Client A polls a message with offset X from partition 1, produces to
> output topic and enters a long gc pause (before calling
> sendOffsetsToTransation)
> 4. Client B starts the consumer and the producer, also getting a reference
> to groupMetadata with generation.id -1
> 5. Client B joins the group and gets assigned partition 1
> 6. Client B polls a message with offset X from partition 1, produces to
> output topic, sends offset with generation.id -1, and commits
> successfully.
> 7. Client A comes back and send offsets with generation.id -1 and commits
> successfully
>
> I did this test because it wasn't so clear for me at which moment I had to
> get the meta and this seems to be a bug to me, since it shouldn't allow
> sending offsets with generation.id -1.
> I know that the right way to do it is to ask for the meta after each poll,
> in that way we always have the generation.id corresponding to the moment
> where the messages were polled from the broker, but it would be nice to
> have an error if we send generation.id -1. WDYT?
>
> Thanks.
>

Re: Random continuous TimeoutException with Topic not present on one KafkaProducer out of many in multithreaded env

[Luke Chen]

Hi Deepak,

So, if you change the value in max.block.ms to default 1 minute, does the
timeout exception still exist?
I think the timeoutException is complaining the 250ms is not a good
configuration for your environment.

Thank you.
Luke

On Tue, Jun 7, 2022 at 11:23 AM Deepak Jain 
wrote:

> Hi,
>
> Thanks for the quick reply.
>
> We are already using the config max.block.ms (alongwith with other
> recommended config like request.timeout.ms and others). Although the
> value we are using is very less at 250 ms but since we have 5 different
> KafkaProducer running in each individual thread out of which 4 are working
> without any issue and only 1 is throwing the TimeOutException, so this does
> not seems to be the issue,
>
> Please else us know if anybody had came across this type of behaviour by
> Kafka. If yes, please help in finding out the root cause and resolving it.
>
> Regards,
> Deepak
>
> -Original Message-
> From: 张晓寅 
> Sent: 06 June 2022 19:10
> To: users@kafka.apache.org
> Cc: Luke Chen 
> Subject: Re: Random continuous TimeoutException with Topic not present on
> one KafkaProducer out of many in multithreaded env
>
> Caution: From Cumulus Systems – IT Department, this email originated from
> outside of the organization. Please call and confirm with the sender before
> opening attachments or clicking links inside the email.
>
>
> maybe you can add producer "max.block.ms" config,but you should test your
> broker look up some logs  about leader change ,producer performance,like
> traffic ,produce "buffer" and "batch.size"
>
> On Mon, Jun 6, 2022 at 6:53 PM Deepak Jain <
> deepak.j...@cumulus-systems.com>
> wrote:
>
> > Hello All,
> >
> > Please help me out in this regard as the Customer has reported this on
> > their production environment and waiting for our reply ASAP.
> >
> > Regards,
> > Deepak
> >
> > From: Deepak Jain
> > Sent: 02 June 2022 20:53
> > To: 'users@kafka.apache.org' 
> > Cc: 'Luke Chen' ; Alap Patwardhan <
> > a...@cumulus-systems.com>; Bhushan Patil <
> > bhushan.pa...@cumulus-systems.com>
> > Subject: Random continuous TimeoutException with Topic not present on
> > one KafkaProducer out of many in multithreaded env
> >
> > Hello Everyone,
> >
> > We are using Kafka 2.8.1 Broker/Client system in our prod env.
> >
> > Getting following exception randomly after 1 hour or so for one
> > Realtime transfer from Kafka Producer to broker out of 5. (Rest 4 are
> > working fine.)
> >
> > java.util.concurrent.ExecutionException:
> > org.apache.kafka.common.errors.TimeoutException: Topic
> > realtimeImport_1 not present in metadata after 250 ms.
> > at
> >
> org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.(KafkaProducer.java:1316)
> > at
> >
> org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:985)
> > at
> >
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:885)
> > at
> > org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.jav
> > a:773)
> >
> > We are using multithreaded KafkaProducer with their each unique topic
> > sending data to single broker. Here, we notice that this exception
> > comes when we reconnect to Kafka using close() (void
> > org > C/platform%5C/common%5C/tools%5C/lib%5C/kafka-clients-2.8.1.jar%3Corg>
> > .apache > pp%5C/platform%5C/common%5C/tools%5C/lib%5C/kafka-clients-2.8.1.jar%3C
> > org.apache>.kafka > megha%5C/app%5C/platform%5C/common%5C/tools%5C/lib%5C/kafka-clients-2.
> > 8.1.jar%3Corg.apache.kafka>.clients > 5C/git%5C/hdca%5C/megha%5C/app%5C/platform%5C/common%5C/tools%5C/lib%5
> > C/kafka-clients-2.8.1.jar%3Corg.apache.kafka.clients>.producer > -javadoc:%E2%98%82=hdca/D:%5C/git%5C/hdca%5C/megha%5C/app%5C/platform%
> > 5C/common%5C/tools%5C/lib%5C/kafka-clients-2.8.1.jar%3Corg.apache.kafk
> > a.clients.producer>.KafkaProducer > /git%5C/hdca%5C/megha%5C/app%5C/platform%5C/common%5C/tools%5C/lib%5C/
> > kafka-clients-2.8.1.jar%3Corg.apache.kafka.clients.producer(KafkaProdu
> > cer.class%E2%98%83KafkaProducer>.close())
> > and
> > org > C/platform%5C/common%5C/tools%5C/lib%5C/kafka-clients-2.8.1.jar%3Corg>
> > .apache > pp%5C/platform%5C/common%5C/tools%5C/lib%5C/kafka-clients-2.8.1.jar%3C
> > org.apache>.kafka > megha%5C/app%5C/platform%5C/common%5C/tools%5C/lib%5C/kafka-clients-2.
> > 8.1.jar%3Corg.apache.kafka>.clients > 5C/git%5C/hdca%5C/megha%5C/app%5C/pla

Re: Request to include me to contributors list

[Luke Chen]

Hi,

I've added you to the contribution list, and assigned this ticket to you.
Thanks for the interest in Apache Kafka.

Luke

On Mon, May 23, 2022 at 12:52 AM Kumud Kumar Srivatsava Tirupati <
kumudkumartirup...@gmail.com> wrote:

> Hi team,
> I am willing to work on https://issues.apache.org/jira/browse/KAFKA-13926
> please add me to the contributors list so that I can assign the ticket to
> myself.
> *---*
> *Thanks and Regards,*
> *Kumud Kumar Srivatsava Tirupati*
>

Re: Increased latency when consuming from the closest ISR

[Luke Chen]

Hi,

We have some improvement for the preferred read replica configured case.
Ex:
https://github.com/apache/kafka/pull/11942
https://github.com/apache/kafka/pull/11965

I know one improvement will be included in the v3.2.0 release, which will
be released soon.
Maybe you can give it a try to see if it improves the throughput.

Thank you.
Luke

On Wed, May 11, 2022 at 2:56 PM benitocm  wrote:

> Hi,
>
> We are using the functionality provided by KIP-392 (a consumer can fetch
> the data from a ISR replica instead of the partition leader) in a Kafka
> cluster stretched between two very close DCs (average round-trip latency
> about 2 milliseconds).
>
> What we have seen is that, on average, when the consumer is in the same DC
> (configured by rack.id) as the partition leader (i.e. the consumer will
> consume from the leader), the time that takes the message to get to the
> consumer is close to 20 milliseconds. However, when the consumer is in a
> different DC than the partition leader (the consumer will consume from a
> replica that is in the same DC as the consumer) that latency goes to around
> 400 milliseconds.
>
> We have also checked that if we dont configure  the rack.id in a consumer
> to force  it to consume from the leader although the partition leader is a
> different DC (i.e. the consumer is in DC1 and the partition leader is in
> DC2 so the consumer goes from a DC to the other DC) , the latency is
> reduced to the 20 milliseconds.
>
> From those tests, we have concluded that consuming from a ISR replica
> implies to have higher latencies.
>
> Please does anybody share any thoughts on this?
>
> Thanks in advance
>

Re: Topic without Leader / ISR

[Luke Chen]

Hi Nicolas,

This is the log in "client" side.
I think the broker side log will have more info.
Also, which Kafka version are you using?

Thank you.
Luke

On Tue, May 10, 2022 at 8:17 PM Nicolas Carlot
 wrote:

> Not a single line regarding this topic, but at creation time :/
> [2022-05-04 14:42:28,091] INFO Creating topic
> douane.request.amendment.traitementdecla-to-custmestranss with
> configuration {delete.retention.ms=60480, retention.ms=60480,
> cleanup.policy=delete, compression.type=lz4} and initial partition
> assignment HashMap(0 -> ArrayBuffer(3, 1, 2), 1 -> ArrayBuffer(1, 2, 3), 2
> -> ArrayBuffer(2, 3, 1), 3 -> ArrayBuffer(3, 2, 1), 4 -> ArrayBuffer(1, 3,
> 2), 5 -> ArrayBuffer(2, 1, 3), 6 -> ArrayBuffer(3, 1, 2), 7 ->
> ArrayBuffer(1, 2, 3), 8 -> ArrayBuffer(2, 3, 1), 9 -> ArrayBuffer(3, 2, 1))
> (kafka.zk.AdminZkClient)
>
> Le mar. 10 mai 2022 à 05:15, Luke Chen  a écrit :
>
> > Hi Nicolas,
> >
> > Could you check the logs in broker side?
> > There must be some errors while electing leaders or something.
> >
> > Thank you.
> > Luke
> >
> > On Mon, May 9, 2022 at 11:31 PM Nicolas Carlot
> >  wrote:
> >
> > > Hello,
> > >
> > > I have a situation where when creating a new topic it stays in the
> > > following state, having not a single ISR, no leader for any partition,
> > > nothing.
> > > Of course I cannot  neitherconsume not produce from/to this topic.
> > >
> > > Topic: douane.request.amendment.traitementdecla-to-custmestrans
> > > Partition: 0Leader: noneReplicas: 1,2,3 Isr:
> > > Topic: douane.request.amendment.traitementdecla-to-custmestrans
> > > Partition: 1Leader: noneReplicas: 2,3,1 Isr:
> > > Topic: douane.request.amendment.traitementdecla-to-custmestrans
> > > Partition: 2Leader: noneReplicas: 3,1,2 Isr:
> > > Topic: douane.request.amendment.traitementdecla-to-custmestrans
> > > Partition: 3Leader: noneReplicas: 1,3,2 Isr:
> > > Topic: douane.request.amendment.traitementdecla-to-custmestrans
> > > Partition: 4Leader: noneReplicas: 2,1,3 Isr:
> > > Topic: douane.request.amendment.traitementdecla-to-custmestrans
> > > Partition: 5Leader: noneReplicas: 3,2,1 Isr:
> > > Topic: douane.request.amendment.traitementdecla-to-custmestrans
> > > Partition: 6Leader: noneReplicas: 1,2,3 Isr:
> > > Topic: douane.request.amendment.traitementdecla-to-custmestrans
> > > Partition: 7Leader: noneReplicas: 2,3,1 Isr:
> > > Topic: douane.request.amendment.traitementdecla-to-custmestrans
> > > Partition: 8Leader: noneReplicas: 3,1,2 Isr:
> > > Topic: douane.request.amendment.traitementdecla-to-custmestrans
> > > Partition: 9Leader: noneReplicas: 1,3,2 Isr:
> > >
> > > Any idea of what may be happening ?
> > >
> > > --
> > > [image: img] <https://www.chronopost.fr/fr?xtatc=INT-149>
> > >
> > >
> > > *Nicolas Carlot*
> > > *Lead dev*Direction des Systèmes d'Information
> > >
> > >
> > > 3 boulevard Romain Rolland
> > > 75014 Paris
> > > [image: img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3A4PQ>
> > > [image:
> > > img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pzPQ> [image:
> > > img]
> > > <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pRPQ> [image: img]
> > > <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pVPQ>
> > >
> > > [image: img]
> > > <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3B3PQ-K1pRPQ>
> > >
> > > [image: img]
> > >
> >
>
>
> --
> [image: img] <https://www.chronopost.fr/fr?xtatc=INT-149>
>
>
> *Nicolas Carlot*
> *Lead dev*Direction des Systèmes d'Information
>
>
> 3 boulevard Romain Rolland
> 75014 Paris
> [image: img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3A4PQ>
> [image:
> img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pzPQ> [image:
> img]
> <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pRPQ> [image: img]
> <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pVPQ>
>
> [image: img]
> <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3B3PQ-K1pRPQ>
>
> [image: img]
>