Re: [VOTE] 3.1.1 RC1

[Luke Chen]

Hi Tom,

I did:
1. check the signature and checksums
2. ran quick start with java17 + sacla2.12
3. browse java docs/documentations

+1 (non-binding)

Thanks for running the release.

Luke

On Thu, May 5, 2022 at 12:43 AM Bill Bejeck  wrote:

> Hi Tom,
>
> Thanks for running the release!
>
> I did the following checks:
>
>1. Validated all the checksum and signatures
>2. Built from source and ran the unit tests (Java 11)
>3. Ran the quickstart and the Kafka Streams demo (Java 11)
>4. Did a quick scan of the Javadoc and the documentation
>
> I noticed the same issues as Mickael on the website, but otherwise, it
> looks good.
> +1 (binding)
>
> Thanks,
> Bill
>
> On Tue, May 3, 2022 at 10:17 AM Jakub Scholz  wrote:
>
> > +1 (non-binding). I used the Scala 2.13 binaries and staged Maven
> artifacts
> > and ran various tests with them. Thanks for doing the release.
> >
> > Jakub
> >
> > On Fri, Apr 29, 2022 at 8:16 PM Mickael Maison 
> > wrote:
> >
> > > Hi Tom,
> > >
> > > Thanks for running this release!
> > >
> > > I've done the following:
> > > - Checked signatures and checksums
> > > - Checked javadocs/maven artifacts
> > > - Built from source and run all tests with Java 11
> > > - Ran quickstart on Scala 2.13 artifact with Java 11
> > >
> > > It looks like the website has not been updated yet, I still only see
> > > 3.1.0. When you'll add 3.1.1, let's make sure we mention reload4j in
> > > the notable changes section.
> > >
> > > +1 (binding)
> > >
> > > Thanks,
> > > Mickael
> > >
> > >
> > >
> > > On Fri, Apr 29, 2022 at 11:12 AM Tom Bentley 
> > wrote:
> > > >
> > > > Hello Kafka users, developers and client-developers,
> > > >
> > > > This is the first candidate for release of Apache Kafka 3.1.1.
> > > >
> > > > Apache Kafka 3.1.1 is a bugfix release and 30 issues have been fixed
> > > > since 3.1.0.
> > > >
> > > > Release notes for the 3.1.1 release:
> > > >
> https://home.apache.org/~tombentley/kafka-3.1.1-rc1/RELEASE_NOTES.html
> > > >
> > > > *** Please download, test and vote by 09:00 UTC, Friday 6th May
> > > >
> > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > https://kafka.apache.org/KEYS
> > > >
> > > > * Release artifacts to be voted upon (source and binary):
> > > > https://home.apache.org/~tombentley/kafka-3.1.1-rc1/
> > > >
> > > > * Maven artifacts to be voted upon:
> > > >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > >
> > > > * Javadoc:
> > > > https://home.apache.org/~tombentley/kafka-3.1.1-rc1/javadoc/
> > > >
> > > > * Tag to be voted upon (off 3.1 branch) is the 3.1.1 tag:
> > > > https://github.com/apache/kafka/releases/tag/3.1.1-rc1
> > > >
> > > > * Documentation:
> > > > https://kafka.apache.org/31/documentation.html
> > > >
> > > > * Protocol:
> > > > https://kafka.apache.org/31/protocol.html
> > > >
> > > > * Successful Jenkins builds for the 3.1 branch:
> > > > I will share a link one the build is complete.
> > > >
> > > > /**
> > > >
> > > > Thanks,
> > > > Tom
> > >
> >
>

Re: [VOTE] 3.2.0 RC1

[Luke Chen]

Hi Bruno,

I did:
1. check the signature and checksums
2. ran quick start with java17 + sacla2.12
3. browse java docs/documentations

+1 (non-binding)

Thanks for running the release.

Luke

On Thu, May 5, 2022 at 1:23 AM Mickael Maison  wrote:

> Hi Bruno,
>
> I checked signatures and checksums, built from source, ran the unit
> tests and ran the quickstart. Javadocs and the website look fine too.
> +1 (binding)
>
> Thanks for running the release!
>
> Mickael
>
> On Wed, May 4, 2022 at 4:48 PM Bruno Cadonna  wrote:
> >
> > Hello Kafka users, developers and client-developers,
> >
> > I've got a green build with unit/integration tests:
> > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.2/45/
> >
> > Best,
> > Bruno
> >
> > On 03.05.22 16:07, Bruno Cadonna wrote:
> > > Hello Kafka users, developers and client-developers,
> > >
> > > This is the second candidate for release of Apache Kafka 3.2.0.
> > >
> > > * log4j 1.x is replaced with reload4j (KAFKA-9366)
> > > * StandardAuthorizer for KRaft (KIP-801)
> > > * Send a hint to the partition leader to recover the partition
> (KIP-704)
> > > * Top-level error code field in DescribeLogDirsResponse (KIP-784)
> > > * kafka-console-producer writes headers and null values (KIP-798 and
> > > KIP-810)
> > > * JoinGroupRequest and LeaveGroupRequest have a reason attached
> (KIP-800)
> > > * Static membership protocol lets the leader skip assignment (KIP-814)
> > > * Rack-aware standby task assignment in Kafka Streams (KIP-708)
> > > * Interactive Query v2 (KIP-796, KIP-805, and KIP-806)
> > > * Connect APIs list all connector plugins and retrieve their
> > > configuration (KIP-769)
> > > * TimestampConverter SMT supports different unix time precisions
> (KIP-808)
> > > * Connect source tasks handle producer exceptions (KIP-779)
> > >
> > >
> > > Release notes for the 3.2.0 release:
> > > https://home.apache.org/~cadonna/kafka-3.2.0-rc1/RELEASE_NOTES.html
> > >
> > > *** Please download, test and vote by Tuesday, May 10th, 9am PDT
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > https://kafka.apache.org/KEYS
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > https://home.apache.org/~cadonna/kafka-3.2.0-rc1/
> > >
> > > * Maven artifacts to be voted upon:
> > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >
> > > * Javadoc:
> > > https://home.apache.org/~cadonna/kafka-3.2.0-rc1/javadoc/
> > >
> > > * Tag to be voted upon (off 3.2 branch) is the 3.2.0 tag:
> > > https://github.com/apache/kafka/releases/tag/3.2.0-rc1
> > >
> > > * Documentation:
> > > https://kafka.apache.org/32/documentation.html
> > >
> > > * Protocol:
> > > https://kafka.apache.org/32/protocol.html
> > >
> > > * Successful Jenkins builds for the 3.2 branch:
> > > Unit/integration tests: I'll share a link once the builds complete
> > > System tests:
> > > https://jenkins.confluent.io/job/system-test-kafka/job/3.2/30/
> > >
> > > /**
> > >
> > > Thanks,
> > > Bruno
>

[jira] [Created] (KAFKA-13875) update docs to include topoicId for kafka-topics.sh --describe output

[Luke Chen (Jira)]

Luke Chen created KAFKA-13875:
-

 Summary: update docs to include topoicId for kafka-topics.sh 
--describe output
 Key: KAFKA-13875
 URL: https://issues.apache.org/jira/browse/KAFKA-13875
 Project: Kafka
  Issue Type: Improvement
  Components: admin
Affects Versions: 3.2.0
Reporter: Luke Chen


The topic describe output in quickstart doc here: 
[https://kafka.apache.org/quickstart] should be updated now.
{code:java}
bin/kafka-topics.sh --describe --topic quickstart-events --bootstrap-server 
localhost:9092
Topic:quickstart-events  PartitionCount:1ReplicationFactor:1 Configs:
Topic: quickstart-events Partition: 0Leader: 0   Replicas: 0 Isr: 
0{code}
After Topic Id implementation, we included the topic id info in the output now.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready

[Ismael Juma]

Yes, all features supported by zk mode will be available in kraft mode in
the 3.x series.

Ismael

On Wed, May 4, 2022, 5:28 PM Israel Ekpo  wrote:

> Ismael,
>
> I like the timeline. However, does this or will this also account for
> features  users rely on today in Zookeeper mode being available when
> Zookeeper is dropped?
>
> That’s my main concern
>
> On Wed, May 4, 2022 at 8:12 PM Ismael Juma  wrote:
>
> > Hi Colin,
> >
> > Thanks for the KIP, this is exciting. Trying to balance progress and
> > compatibility, how about the following?
> >
> > 1. 3.3 (~August 2022): kraft is production ready for new clusters
> > 2. 3.4 (~December 2022/January 2023): migration from zk to kraft is
> > production ready and zk mode is deprecated
> > 3. 3.5 (~April 2023): buffer release
> > 4. 4.0 (~August 2023): kraft mode is on by default and zk mode is removed
> >
> > This would mean about 1 year from kraft being production ready to zk
> > removal and 8 months from zk deprecation to zk removal.
> >
> > If necessary (due to important bugs or security issues), we can do a
> couple
> > of additional bug fix releases in the 3.5 series after 4.0 is released.
> >
> > Thoughts?
> >
> > Ismael
> >
> > On Tue, May 3, 2022, 6:03 PM Colin McCabe  wrote:
> >
> > > Hi all,
> > >
> > > I've written a KIP for marking KRaft as production ready. Please take a
> > > look if you have a chance:
> > >
> > > https://cwiki.apache.org/confluence/x/8xKhD
> > >
> > > thanks,
> > > Colin
> > >
> >
> --
> Israel Ekpo
> Lead Instructor, IzzyAcademy.com
> https://www.youtube.com/c/izzyacademy
> https://izzyacademy.com/
>

[jira] [Created] (KAFKA-13874) Avoid synchronization in SocketServer metrics

[Colin McCabe (Jira)]

Colin McCabe created KAFKA-13874:


 Summary: Avoid synchronization in SocketServer metrics
 Key: KAFKA-13874
 URL: https://issues.apache.org/jira/browse/KAFKA-13874
 Project: Kafka
  Issue Type: Improvement
Reporter: Colin McCabe


For performance reasons, we should avoid synchronization in SocketServer 
metrics like NetworkProcessorAvgIdlePercent



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready

[Israel Ekpo]

Ismael,

I like the timeline. However, does this or will this also account for
features  users rely on today in Zookeeper mode being available when
Zookeeper is dropped?

That’s my main concern

On Wed, May 4, 2022 at 8:12 PM Ismael Juma  wrote:

> Hi Colin,
>
> Thanks for the KIP, this is exciting. Trying to balance progress and
> compatibility, how about the following?
>
> 1. 3.3 (~August 2022): kraft is production ready for new clusters
> 2. 3.4 (~December 2022/January 2023): migration from zk to kraft is
> production ready and zk mode is deprecated
> 3. 3.5 (~April 2023): buffer release
> 4. 4.0 (~August 2023): kraft mode is on by default and zk mode is removed
>
> This would mean about 1 year from kraft being production ready to zk
> removal and 8 months from zk deprecation to zk removal.
>
> If necessary (due to important bugs or security issues), we can do a couple
> of additional bug fix releases in the 3.5 series after 4.0 is released.
>
> Thoughts?
>
> Ismael
>
> On Tue, May 3, 2022, 6:03 PM Colin McCabe  wrote:
>
> > Hi all,
> >
> > I've written a KIP for marking KRaft as production ready. Please take a
> > look if you have a chance:
> >
> > https://cwiki.apache.org/confluence/x/8xKhD
> >
> > thanks,
> > Colin
> >
>
-- 
Israel Ekpo
Lead Instructor, IzzyAcademy.com
https://www.youtube.com/c/izzyacademy
https://izzyacademy.com/

Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready

[Ismael Juma]

Hi Colin,

Thanks for the KIP, this is exciting. Trying to balance progress and
compatibility, how about the following?

1. 3.3 (~August 2022): kraft is production ready for new clusters
2. 3.4 (~December 2022/January 2023): migration from zk to kraft is
production ready and zk mode is deprecated
3. 3.5 (~April 2023): buffer release
4. 4.0 (~August 2023): kraft mode is on by default and zk mode is removed

This would mean about 1 year from kraft being production ready to zk
removal and 8 months from zk deprecation to zk removal.

If necessary (due to important bugs or security issues), we can do a couple
of additional bug fix releases in the 3.5 series after 4.0 is released.

Thoughts?

Ismael

On Tue, May 3, 2022, 6:03 PM Colin McCabe  wrote:

> Hi all,
>
> I've written a KIP for marking KRaft as production ready. Please take a
> look if you have a chance:
>
> https://cwiki.apache.org/confluence/x/8xKhD
>
> thanks,
> Colin
>

Re: [VOTE] 3.1.1 RC0

[Ismael Juma]

Hi Dongjoon,

Any updates on your testing?

Ismael

On Fri, Apr 29, 2022, 12:04 PM Dongjoon Hyun  wrote:

> Thank you for the recommendation and the releasing, Tom and Ismael.
>
> There are some concerns about Apache Kafka 3.1.x still in the Apache Spark
> dev mailing list.
>
> I'm going to proceed to test new Apache Kafka 3.1.1 RC1 in Apache Spark
> 3.3.0 preparation.
>
> Thanks,
> Dongjoon.
>
> On 2022/04/28 14:05:49 Ismael Juma wrote:
> > Hi Tom,
> >
> > I merged and cherry-picked the fix.
> >
> > Ismael
> >
> > On Thu, Apr 28, 2022 at 12:33 AM Tom Bentley 
> wrote:
> >
> > > Hi Dongjoon,
> > >
> > > I apologise, I should have been a bit more communicative. I was
> waiting for
> > > a better fix to the issue previously highlighted by David. Ismael has
> > > kindly provided a patch [1], so I will roll RC1 once this is merged and
> > > cherry-picked.
> > >
> > > Kind regards,
> > >
> > > Tom
> > >
> > > [1]: https://github.com/apache/kafka/pull/12096
> > >
> > >
> > > On Thu, 28 Apr 2022 at 04:38, Luke Chen  wrote:
> > >
> > > > Hi Dongjoon,
> > > >
> > > > The Apache Kafka community doesn't recommend users to use which
> version
> > > of
> > > > Kafka.
> > > > The 2 releases v3.1.1 and v3.2.0 are running in parallel, and we
> don't
> > > > guarantee which version will be released earlier.
> > > >
> > > > Thank you.
> > > > Luke
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Apr 28, 2022 at 4:54 AM Dongjoon Hyun 
> > > wrote:
> > > >
> > > > > Hi, All.
> > > > >
> > > > > It seems that Apache Kafka 3.2.0 RC0 vote started already instead
> of
> > > > > Apache Kafka 3.1.1 release.
> > > > >
> > > > > Does Apache Kafka community recommend to use Apache Kafka 3.2.0
> instead
> > > > of
> > > > > Apache Kafka 3.1.1?
> > > > >
> > > > > Dongjoon.
> > > > >
> > > > > On 2022/04/14 01:00:40 Ismael Juma wrote:
> > > > > > I added a comment to that PR. Let's figure out if we need an
> > > additional
> > > > > > change before doing the next RC.
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Tue, Apr 12, 2022 at 7:47 PM Luke Chen 
> wrote:
> > > > > >
> > > > > > > Thanks for pointing that out, David.
> > > > > > > +1 to include this PR since we've already included the first
> fix
> > > for
> > > > > > > KAFKA-13794, and this is a follow up fix for it.
> > > > > > >
> > > > > > > Thank you.
> > > > > > > Luke
> > > > > > >
> > > > > > > On Wed, Apr 13, 2022 at 2:31 AM David Jacot
> > > > > 
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Tom,
> > > > > > > >
> > > > > > > > Thanks for running the release. I wonder if we should
> include:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> https://github.com/apache/kafka/commit/134c432d6452de1bfb99d0f6b455a58c16bc626a
> > > > > > > > .
> > > > > > > >
> > > > > > > > This is a follow up of KAFKA-13794. What do you think?
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > David
> > > > > > > >
> > > > > > > > On Fri, Apr 8, 2022 at 6:18 PM Tom Bentley <
> tbent...@redhat.com>
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hello Kafka users, developers and client-developers,
> > > > > > > > >
> > > > > > > > > This is the first candidate for release of Apache Kafka
> 3.1.1.
> > > > > > > > >
> > > > > > > > > Apache Kafka 3.1.1 is a bugfix release and 29 issues have
> been
> > > > > fixed
> > > > > > > > > since 3.1.0.
> > > > > > > > >
> > > > > > > > > Release notes for the 3.1.1 release:
> > > > > > > > >
> > > > >
> https://home.apache.org/~tombentley/kafka-3.1.1-rc0/RELEASE_NOTES.html
> > > > > > > > >
> > > > > > > > > *** Please download, test and vote by Friday 15 April,
> 12:00
> > > UTC
> > > > > > > > >
> > > > > > > > > Kafka's KEYS file containing PGP keys we use to sign the
> > > release:
> > > > > > > > > https://kafka.apache.org/KEYS
> > > > > > > > >
> > > > > > > > > * Release artifacts to be voted upon (source and binary):
> > > > > > > > > https://home.apache.org/~tombentley/kafka-3.1.1-rc0/
> > > > > > > > >
> > > > > > > > > * Maven artifacts to be voted upon:
> > > > > > > > >
> > > > >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > > > > > >
> > > > > > > > > * Javadoc:
> > > > > > > > >
> https://home.apache.org/~tombentley/kafka-3.1.1-rc0/javadoc/
> > > > > > > > >
> > > > > > > > > * Tag to be voted upon (off 3.1 branch) is the 3.1.1 tag:
> > > > > > > > > https://github.com/apache/kafka/releases/tag/3.1.1-rc0
> > > > > > > > >
> > > > > > > > > * Documentation:
> > > > > > > > > https://kafka.apache.org/31/documentation.html
> > > > > > > > >
> > > > > > > > > * Protocol:
> > > > > > > > > https://kafka.apache.org/31/protocol.html
> > > > > > > > >
> > > > > > > > > * Successful Jenkins builds for the 3.1 branch:
> > > > > > > > > I will share a link one the build is complete.
> > > > > > > > >
> > > > > > > > > /**
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Tom
> > > > > > > >
> > > > > > 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #908

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-832 Allow creating a producer/consumer using a producer/consumer config

[François Rosière]

Hi all,

KIP-832 has been created to allow implementing Spring managed interceptors
for Producers and Consumers.

At the moment, interceptors are given as configuration classes to the
producer and consumer configurations. So, the idea here would be to create
2 new constructors to allow using a Producer and Consumer configuration
instead of properties or a key value map of configurations entries.
Interceptors could then be given as instances by overriding a config method.
More details can be found in the Spring issue.
https://github.com/spring-projects/spring-kafka/issues/2244

Any feedback, proposal, vote for this KIP would be more than welcome.

Kind regards,

Francois R.

Le lun. 2 mai 2022 à 21:05, François Rosière  a
écrit :

> Kip link:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211882578
>
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #907

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-831: Add metric for log recovery progress

[James Cheng]

Hi Luke,

Thanks for adding RemainingSegmentsToRecovery.

Another thought: different topics can have different segment sizes. I don't 
know how common it is, but it is possible. Some topics might want small segment 
sizes to more granular expiration of data.

The downside of RemainingLogsToRecovery and RemainingSegmentsToRecovery is that 
the rate that they will decrement depends on the configuration and patterns of 
the topics and partitions and segment sizes. If someone is monitoring those 
metrics, they might see times where the metric decrements slowly, followed by a 
burst where it decrements quickly.

What about RemainingBytesToRecovery? This would not depend on the configuration 
of the topic or of the data. It would actually be a pretty good metric, because 
I think that this metric would change at a constant rate (based on the disk I/O 
speed that the broker allocates to recovery). Because it changes at a constant 
rate, you would be able to use the rate-of-change to predict when it hits zero, 
which will let you know when the broker is going to start up. Like, I would 
imagine if we graphed RemainingBytesToRecovery that we'd see a fairly straight 
line that is decrementing at a steady rate towards zero.

What do you think about adding RemainingBytesToRecovery? 

Or, what would you think about making the primary metric be 
RemainingBytesToRecovery, and getting rid of the others?

I don't know if I personally would rather have all 3 metrics, or would just use 
RemainingBytesToRecovery. I'd too would like more community input on which of 
those metrics would be useful to people.

About the JMX metrics, you said that if num.recovery.threads.per.data.dir=2, 
that there might be a separate RemainingSegmentsToRecovery counter for each 
thread. Is that actually how the data is structured within the Kafka recovery 
threads? Does each thread get a fixed set of partitions, or is there just one 
big pool of partitions that the threads all work on?

As a more concrete example:
* If I have 9 small partitions and 1 big partition, and 
num.recovery.threads.per.data.dir=2
Does each thread get 5 partitions, which means one thread will finish much 
sooner than the other?
OR
Do both threads just work on the set of 10 partitions, which means likely 1 
thread will be busy with the big partition, while the other one ends up plowing 
through the 9 small partitions?

If each thread gets assigned 5 partitions, then it would make sense that each 
thread has its own counter.
If the threads works on a single pool of 10 partitions, then it would probably 
mean that the counter is on the pool of partitions itself, and not on each 
thread.

-James

> On May 4, 2022, at 5:55 AM, Luke Chen  wrote:
> 
> Hi devs,
> 
> If there are no other comments, I'll start a vote tomorrow.
> 
> Thank you.
> Luke
> 
> On Sun, May 1, 2022 at 5:08 PM Luke Chen  wrote:
> 
>> Hi James,
>> 
>> Sorry for the late reply.
>> 
>> Yes, this is a good point, to know how many segments to be recovered if
>> there are some large partitions.
>> I've updated the KIP, to add a `*RemainingSegmentsToRecover*` metric for
>> each log recovery thread, to show the value.
>> The example in the Proposed section here
>> 
>> shows what it will look like.
>> 
>> Thanks for the suggestion.
>> 
>> Thank you.
>> Luke
>> 
>> 
>> 
>> On Sat, Apr 23, 2022 at 8:54 AM James Cheng  wrote:
>> 
>>> The KIP describes RemainingLogsToRecovery, which seems to be the number
>>> of partitions in each log.dir.
>>> 
>>> We have some partitions which are much much larger than others. Those
>>> large partitions have many many more segments than others.
>>> 
>>> Is there a way the metric can reflect partition size? Could it be
>>> RemainingSegmentsToRecover? Or even RemainingBytesToRecover?
>>> 
>>> -James
>>> 
>>> Sent from my iPhone
>>> 
 On Apr 20, 2022, at 2:01 AM, Luke Chen  wrote:
 
 Hi all,
 
 I'd like to propose a KIP to expose a metric for log recovery progress.
 This metric would let the admins have a way to monitor the log recovery
 progress.
 Details can be found here:
 
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-831%3A+Add+metric+for+log+recovery+progress
 
 Any feedback is appreciated.
 
 Thank you.
 Luke
>>> 
>> 

[jira] [Resolved] (KAFKA-13815) Avoid reinitialization for a replica that is being deleted

[Jun Rao (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-13815.
-
Fix Version/s: 3.3.0
   Resolution: Fixed

merged the PR to trunk

> Avoid reinitialization for a replica that is being deleted
> --
>
> Key: KAFKA-13815
> URL: https://issues.apache.org/jira/browse/KAFKA-13815
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Lucas Wang
>Assignee: Lucas Wang
>Priority: Major
> Fix For: 3.3.0
>
>
> https://issues.apache.org/jira/browse/KAFKA-10002
> identified that deletion of replicas can be slow when a StopReplica request 
> is being
> processed, and has implemented a change to improve the efficiency.
> We found that the efficiency can be further improved by avoiding the 
> reinitialization of the
> leader epoch cache and partition metadata for a replica that needs to be 
> deleted.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Resolved] (KAFKA-13861) validateOnly request field does not work for CreatePartition requests in Kraft mode.

[Akhilesh Chaganti (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akhilesh Chaganti resolved KAFKA-13861.
---
Resolution: Fixed

> validateOnly request field does not work for CreatePartition requests in 
> Kraft mode.
> 
>
> Key: KAFKA-13861
> URL: https://issues.apache.org/jira/browse/KAFKA-13861
> Project: Kafka
>  Issue Type: Bug
>Reporter: Akhilesh Chaganti
>Assignee: Akhilesh Chaganti
>Priority: Major
>
> `ControllerApis` ignores the validateOnly field and the `QuorumController` 
> does not have any logic to handle the `validateOnly` requests for 
> `CreatePartitions.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[DISCUSS] KIP-834: Pause / Resume KafkaStreams Topologies

[Jim Hughes]

Hi all,

I have written up a KIP for adding the ability to pause and resume the
processing of a topology in AK Streams.  The KIP is here:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211882832

Thanks in advance for your feedback!

Cheers,

Jim

[jira] [Created] (KAFKA-13873) Add ability to Pause / Resume KafkaStreams Topologies

[Jim Hughes (Jira)]

Jim Hughes created KAFKA-13873:
--

 Summary: Add ability to Pause / Resume KafkaStreams Topologies
 Key: KAFKA-13873
 URL: https://issues.apache.org/jira/browse/KAFKA-13873
 Project: Kafka
  Issue Type: New Feature
Reporter: Jim Hughes


In order to reduce resources used or modify data pipelines, users may want to 
pause processing temporarily.  Presently, this would require stopping the 
entire KafkaStreams instance (or instances).  

This work would add the ability to pause and resume topologies.  When the need 
to pause processing has passed, then users should be able to resume processing.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [VOTE] 3.2.0 RC1

[Mickael Maison]

Hi Bruno,

I checked signatures and checksums, built from source, ran the unit
tests and ran the quickstart. Javadocs and the website look fine too.
+1 (binding)

Thanks for running the release!

Mickael

On Wed, May 4, 2022 at 4:48 PM Bruno Cadonna  wrote:
>
> Hello Kafka users, developers and client-developers,
>
> I've got a green build with unit/integration tests:
> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.2/45/
>
> Best,
> Bruno
>
> On 03.05.22 16:07, Bruno Cadonna wrote:
> > Hello Kafka users, developers and client-developers,
> >
> > This is the second candidate for release of Apache Kafka 3.2.0.
> >
> > * log4j 1.x is replaced with reload4j (KAFKA-9366)
> > * StandardAuthorizer for KRaft (KIP-801)
> > * Send a hint to the partition leader to recover the partition (KIP-704)
> > * Top-level error code field in DescribeLogDirsResponse (KIP-784)
> > * kafka-console-producer writes headers and null values (KIP-798 and
> > KIP-810)
> > * JoinGroupRequest and LeaveGroupRequest have a reason attached (KIP-800)
> > * Static membership protocol lets the leader skip assignment (KIP-814)
> > * Rack-aware standby task assignment in Kafka Streams (KIP-708)
> > * Interactive Query v2 (KIP-796, KIP-805, and KIP-806)
> > * Connect APIs list all connector plugins and retrieve their
> > configuration (KIP-769)
> > * TimestampConverter SMT supports different unix time precisions (KIP-808)
> > * Connect source tasks handle producer exceptions (KIP-779)
> >
> >
> > Release notes for the 3.2.0 release:
> > https://home.apache.org/~cadonna/kafka-3.2.0-rc1/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Tuesday, May 10th, 9am PDT
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~cadonna/kafka-3.2.0-rc1/
> >
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >
> > * Javadoc:
> > https://home.apache.org/~cadonna/kafka-3.2.0-rc1/javadoc/
> >
> > * Tag to be voted upon (off 3.2 branch) is the 3.2.0 tag:
> > https://github.com/apache/kafka/releases/tag/3.2.0-rc1
> >
> > * Documentation:
> > https://kafka.apache.org/32/documentation.html
> >
> > * Protocol:
> > https://kafka.apache.org/32/protocol.html
> >
> > * Successful Jenkins builds for the 3.2 branch:
> > Unit/integration tests: I'll share a link once the builds complete
> > System tests:
> > https://jenkins.confluent.io/job/system-test-kafka/job/3.2/30/
> >
> > /**
> >
> > Thanks,
> > Bruno

Re: [VOTE] 3.1.1 RC1

[Bill Bejeck]

Hi Tom,

Thanks for running the release!

I did the following checks:

   1. Validated all the checksum and signatures
   2. Built from source and ran the unit tests (Java 11)
   3. Ran the quickstart and the Kafka Streams demo (Java 11)
   4. Did a quick scan of the Javadoc and the documentation

I noticed the same issues as Mickael on the website, but otherwise, it
looks good.
+1 (binding)

Thanks,
Bill

On Tue, May 3, 2022 at 10:17 AM Jakub Scholz  wrote:

> +1 (non-binding). I used the Scala 2.13 binaries and staged Maven artifacts
> and ran various tests with them. Thanks for doing the release.
>
> Jakub
>
> On Fri, Apr 29, 2022 at 8:16 PM Mickael Maison 
> wrote:
>
> > Hi Tom,
> >
> > Thanks for running this release!
> >
> > I've done the following:
> > - Checked signatures and checksums
> > - Checked javadocs/maven artifacts
> > - Built from source and run all tests with Java 11
> > - Ran quickstart on Scala 2.13 artifact with Java 11
> >
> > It looks like the website has not been updated yet, I still only see
> > 3.1.0. When you'll add 3.1.1, let's make sure we mention reload4j in
> > the notable changes section.
> >
> > +1 (binding)
> >
> > Thanks,
> > Mickael
> >
> >
> >
> > On Fri, Apr 29, 2022 at 11:12 AM Tom Bentley 
> wrote:
> > >
> > > Hello Kafka users, developers and client-developers,
> > >
> > > This is the first candidate for release of Apache Kafka 3.1.1.
> > >
> > > Apache Kafka 3.1.1 is a bugfix release and 30 issues have been fixed
> > > since 3.1.0.
> > >
> > > Release notes for the 3.1.1 release:
> > > https://home.apache.org/~tombentley/kafka-3.1.1-rc1/RELEASE_NOTES.html
> > >
> > > *** Please download, test and vote by 09:00 UTC, Friday 6th May
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > https://kafka.apache.org/KEYS
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > https://home.apache.org/~tombentley/kafka-3.1.1-rc1/
> > >
> > > * Maven artifacts to be voted upon:
> > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >
> > > * Javadoc:
> > > https://home.apache.org/~tombentley/kafka-3.1.1-rc1/javadoc/
> > >
> > > * Tag to be voted upon (off 3.1 branch) is the 3.1.1 tag:
> > > https://github.com/apache/kafka/releases/tag/3.1.1-rc1
> > >
> > > * Documentation:
> > > https://kafka.apache.org/31/documentation.html
> > >
> > > * Protocol:
> > > https://kafka.apache.org/31/protocol.html
> > >
> > > * Successful Jenkins builds for the 3.1 branch:
> > > I will share a link one the build is complete.
> > >
> > > /**
> > >
> > > Thanks,
> > > Tom
> >
>

[jira] [Created] (KAFKA-13872) Partitions are truncated when leader is replaced

[Francois Visconte (Jira)]

Francois Visconte created KAFKA-13872:
-

 Summary: Partitions are truncated when leader is replaced
 Key: KAFKA-13872
 URL: https://issues.apache.org/jira/browse/KAFKA-13872
 Project: Kafka
  Issue Type: Bug
Affects Versions: 2.7.2
Reporter: Francois Visconte
 Attachments: extract-2022-05-04T15_50_34.110Z.csv

Sample setup:
 * a topic with one partition and RF=3
 * a producer using acks=1
 * min.insync.replicas to 1
 * 3 brokers 1,2,3
 * Preferred leader of the partition is brokerId 0

 

Steps to reproduce the issue
 * Producer keeps producing to the partition, leader is brokerId=0
 * At some point, replicas 1 and 2 are falling behind and removed from the ISR
 * The leader broker 0 has an hardware failure
 * Partition transition to offline
 * This leader is replaced with a new broker with an empty disk and the same 
broker id 0
 * Partition transition from offline to online with leader 0, ISR = 0
 * Followers see the leader offset is 0 and decide to truncate their partitions 
to 0, ISR=0,1,2

Attached some of the relevant logs. I can provide more logs if necessary



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [VOTE] 3.2.0 RC1

[Bruno Cadonna]

Hello Kafka users, developers and client-developers,

I've got a green build with unit/integration tests:
https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.2/45/

Best,
Bruno

On 03.05.22 16:07, Bruno Cadonna wrote:

Hello Kafka users, developers and client-developers,

This is the second candidate for release of Apache Kafka 3.2.0.

* log4j 1.x is replaced with reload4j (KAFKA-9366)
* StandardAuthorizer for KRaft (KIP-801)
* Send a hint to the partition leader to recover the partition (KIP-704)
* Top-level error code field in DescribeLogDirsResponse (KIP-784)
* kafka-console-producer writes headers and null values (KIP-798 and 
KIP-810)

* JoinGroupRequest and LeaveGroupRequest have a reason attached (KIP-800)
* Static membership protocol lets the leader skip assignment (KIP-814)
* Rack-aware standby task assignment in Kafka Streams (KIP-708)
* Interactive Query v2 (KIP-796, KIP-805, and KIP-806)
* Connect APIs list all connector plugins and retrieve their 
configuration (KIP-769)

* TimestampConverter SMT supports different unix time precisions (KIP-808)
* Connect source tasks handle producer exceptions (KIP-779)


Release notes for the 3.2.0 release:
https://home.apache.org/~cadonna/kafka-3.2.0-rc1/RELEASE_NOTES.html

*** Please download, test and vote by Tuesday, May 10th, 9am PDT

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~cadonna/kafka-3.2.0-rc1/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~cadonna/kafka-3.2.0-rc1/javadoc/

* Tag to be voted upon (off 3.2 branch) is the 3.2.0 tag:
https://github.com/apache/kafka/releases/tag/3.2.0-rc1

* Documentation:
https://kafka.apache.org/32/documentation.html

* Protocol:
https://kafka.apache.org/32/protocol.html

* Successful Jenkins builds for the 3.2 branch:
Unit/integration tests: I'll share a link once the builds complete
System tests: 
https://jenkins.confluent.io/job/system-test-kafka/job/3.2/30/


/**

Thanks,
Bruno

Jenkins build is back to stable : Kafka » Kafka Branch Builder » 3.2 #45

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready

[Israel Ekpo]

Thanks Colin for the update and clarification and the additional details
you added.

I second Igor's thoughts that we should not deprecate ZK before having
feature parity.

However, I think marking KRaft mode as production-ready is something that
can be done independent of the status of feature parity (between ZK-mode
and KRaft-mode).

Maybe we can mark KRaft as production ready for now (in 3.3) then work on
getting to feature parity in 3.4 and once feature parity is attained, then
we deprecate ZK-mode in 3.4 and remove it in the next release after 3.4.

That would be fair from my perspective.

It is a bit hard to ask users to jump off the boat when the dock is still
far away and they are unable to swim that far.

Some of the missing features are still in the critical path for some users
and we should include them in the project before the ZK-mode deprecation or
removal.

These are my initial thoughts about this and I believe users may not feel
supported if the cord is just ripped off with features still missing and
Zookeeper is gone.

That could send a wrong message to the community of current and prospective
users.

Israel Ekpo
Lead Instructor, IzzyAcademy.com
https://www.youtube.com/c/izzyacademy
https://izzyacademy.com/


On Wed, May 4, 2022 at 5:58 AM Igor Soarez  wrote:

> Hi Colin,
>
> It's very exciting to see KRaft ready for production!
>
> I see the value of marking it ready for production even with the current
> missing features.
>
> However, I'm worried about marking ZK mode as deprecated before these
> missing features are ready. It might be hard to estimate right now how much
> work is actually necessary to close that gap, particularly the upgrade from
> ZK. Deprecating ZK before providing users with an upgrade path might be
> sending the wrong message - when something is deprecated we want users to
> move away from it, so that should be possible. It might well be more than a
> few months to close the gap.
>
> Best,
>
> --
> Igor
>
> On Wed, May 4, 2022, at 7:22 AM, Colin McCabe wrote:
> > On Tue, May 3, 2022, at 23:16, Colin McCabe wrote:
> >>
> >> To be clear, the proposal here is to have the bridge release be 3.4 and
> >> then move on to a ZK-free 4.0. With a 3.5 release as an option (but not
> >> requirement) if we can't finish everything in time after 3.4. So that
> >> would be two more releases with ZK, or dropping ZK a little bit less
> >> than a year from now.
> >>
> >
> > Sigh. This should read "To be clear, the proposal here is to have the
> > bridge release be 3.3 and then move on to a ZK-free 4.0. With a 3.4
> > release as an option (but not requirement) if we can't finish
> > everything in time after 3.3." So much for being clear, I guess :)
> >
> > cheers,
> > Colin
> >
> >
> >> best,
> >> Colin
> >>
> >>>
> >>> [1]
> >>>
> https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan
> >>> [2] https://kafka.apache.org/downloads
> >>>
> >>>
> >>> Israel Ekpo
> >>> Lead Instructor, IzzyAcademy.com
> >>> https://www.youtube.com/c/izzyacademy
> >>> https://izzyacademy.com/
> >>>
> >>>
> >>> On Tue, May 3, 2022 at 10:32 PM Luke Chen  wrote:
> >>>
>  Hi Colin,
> 
>  So exciting to see the KIP to mark the Kraft production ready!
> 
>  Just one comment: We should make sure the period between ZK
> deprecation
>  (i.e. v3.4.0) to ZK removal (i.e. v4.0.0) is not too short.
>  Do we have any expectation for the deprecation period?
>  After all, this is not a small feature change, and users need more
> time to
>  do the migration.
>  But to be honest, I don't know how long is long enough.
>  Maybe at least half a year? Or more? Thoughts?
> 
> 
>  Thank you.
>  Luke
> 
>  On Wed, May 4, 2022 at 9:10 AM Colin McCabe 
> wrote:
> 
>  > Hi all,
>  >
>  > I've written a KIP for marking KRaft as production ready. Please
> take a
>  > look if you have a chance:
>  >
>  > https://cwiki.apache.org/confluence/x/8xKhD
>  >
>  > thanks,
>  > Colin
>  >
> 
>

[jira] [Created] (KAFKA-13871) The documentation for the configuration item QUORUM_FETCH_TIMEOUT of the RaftConfig class is incorrect

[yingquan he (Jira)]

yingquan he created KAFKA-13871:
---

 Summary: The documentation for the configuration item 
QUORUM_FETCH_TIMEOUT of the RaftConfig class is incorrect
 Key: KAFKA-13871
 URL: https://issues.apache.org/jira/browse/KAFKA-13871
 Project: Kafka
  Issue Type: Improvement
  Components: kraft
Reporter: yingquan he
Assignee: yingquan he


The syntax of the field QUORUM_FETCH_TIMEOUT_MS_DOC is incorrect. `a election`  
should be changed to `an election`.

 
{code:java}
public static final String QUORUM_FETCH_TIMEOUT_MS_DOC = "Maximum time without 
a successful fetch from " +
"the current leader before becoming a candidate and triggering a election 
for voters; Maximum time without " +
"receiving fetch from a majority of the quorum before asking around to see 
if there's a new epoch for leader"; {code}
 



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [VOTE] KIP-797: Accept duplicate listener on port for IPv4/IPv6

[Igor Soarez]

Hi Matthew,

Thanks for submitting this KIP.
This is a useful improvement.
+1 non binding

Best,

--
Igor

On Fri, Apr 22, 2022, at 12:01 PM, Tom Bentley wrote:
> Hi Matthew,
>
> Thanks for the KIP, +1 (binding).
>
> Kind regards,
>
> Tom
>
> On Thu, 14 Apr 2022 at 12:15, Matthew de Detrich
>  wrote:
>
>> Hi David,
>>
>> Thanks for the response.
>>
>> > 1. In the public interface section, could we spell out
>> the configurations that we are changing with this
>> KIP? The name does not change but the semantic is
>> so it is good to be clear.
>>
>> Done
>>
>> > 2. In the proposed changes section, I would rather
>> mention the configuration that we need to change the
>> validation for instead of saying "loosening the validation
>> on listenerListToEndPoints in kafka.utils.CoreUtils.scala"
>> as this is specific to the implementation.
>>
>> This is already done with examples later down in the same section, or am I
>> missing something? Would you like me to just remove the
>> kafka.utils.CoreUtils.scala reference so its not implying an implementation
>> detail?
>>
>> > 3. For my understanding, using the same port with two
>> different DNS entries would fail, right? e.g.
>> "PLAINTEXT://foo:9092,PLAINTEXT://bar:9092"
>>
>> Correct, the idea is that it checks that the listener host is an IP address
>> and if it's not then it doesn't even consider it (i.e. it short circuits to
>> what is current behaviour). The proposed KIP changes only apply if
>> hostnames in the listener are IP address's otherwise no change is
>> observable.
>>
>> Regards
>>
>> On Mon, Feb 21, 2022 at 10:42 AM David Jacot 
>> wrote:
>>
>> > Hi Matthew,
>> >
>> > Thanks for the KIP. I have a few minor comments:
>> >
>> > 1. In the public interface section, could we spell out
>> > the configurations that we are changing with this
>> > KIP? The name does not change but the semantic is
>> > so it is good to be clear.
>> >
>> > 2. In the proposed changes section, I would rather
>> > mention the configuration that we need to change the
>> > validation for instead of saying "loosening the validation
>> > on listenerListToEndPoints in kafka.utils.CoreUtils.scala"
>> > as this is specific to the implementation.
>> >
>> > 3. For my understanding, using the same port with two
>> > different DNS entries would fail, right? e.g.
>> > "PLAINTEXT://foo:9092,PLAINTEXT://bar:9092"
>> >
>> > Best,
>> > David
>> >
>> > On Fri, Feb 11, 2022 at 10:35 AM Luke Chen  wrote:
>> > >
>> > > Hi Matthew,
>> > >
>> > > Thanks for the update.
>> > > I'm +1 (binding)
>> > >
>> > > Thank you.
>> > > Luke
>> > >
>> > > On Fri, Feb 11, 2022 at 3:32 PM Matthew de Detrich
>> > >  wrote:
>> > >
>> > > > Hi Luke,
>> > > >
>> > > > I have just updated the KIP with the changes you requested.
>> > > >
>> > > > Regards
>> > > >
>> > > > On Fri, Feb 11, 2022 at 4:47 AM Luke Chen  wrote:
>> > > >
>> > > > > Hi Matthew,
>> > > > >
>> > > > > I checked again the KIP, and it LGTM.
>> > > > >
>> > > > > Just a minor comment:
>> > > > > Maybe add some examples into the KIP to show how users can set both
>> > IPv4
>> > > > > and IPv6 on the same port.
>> > > > > And some examples to show how the validation will fail like you
>> > listed in
>> > > > > `Proposed Changes`.
>> > > > >
>> > > > > Thank you.
>> > > > > Luke
>> > > > >
>> > > > >
>> > > > > On Fri, Feb 11, 2022 at 8:54 AM Matthew de Detrich
>> > > > >  wrote:
>> > > > >
>> > > > > > Hello everyone
>> > > > > >
>> > > > > > I have just updated/rebased the PR against the latest Kafka
>> trunk.
>> > Let
>> > > > me
>> > > > > > know if anything else is required/missing.
>> > > > > >
>> > > > > > Regards
>> > > > > >
>> > > > > > On Thu, Jan 13, 2022 at 10:28 AM Matthew de Detrich <
>> > > > > > matthew.dedetr...@aiven.io> wrote:
>> > > > > >
>> > > > > > > Does anyone have any additional comments/regards to help get
>> > this PR
>> > > > > > voted
>> > > > > > > through?
>> > > > > > >
>> > > > > > > On Tue, Nov 23, 2021 at 7:46 AM Josep Prat
>> > > > > > > > > >
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > >> Hi Matthew,
>> > > > > > >>
>> > > > > > >> Thank you for the PR.
>> > > > > > >>
>> > > > > > >> +1 (non binding) from my side.
>> > > > > > >>
>> > > > > > >>
>> > > > > > >> Best,
>> > > > > > >>
>> > > > > > >> ———
>> > > > > > >> Josep Prat
>> > > > > > >>
>> > > > > > >> Aiven Deutschland GmbH
>> > > > > > >>
>> > > > > > >> Immanuelkirchstraße 26, 10405 Berlin
>> > > > > > >>
>> > > > > > >> Amtsgericht Charlottenburg, HRB 209739 B
>> > > > > > >>
>> > > > > > >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> > > > > > >>
>> > > > > > >> m: +491715557497
>> > > > > > >>
>> > > > > > >> w: aiven.io
>> > > > > > >>
>> > > > > > >> e: josep.p...@aiven.io
>> > > > > > >>
>> > > > > > >> On Tue, Nov 23, 2021, 07:11 Ivan Yurchenko <
>> > > > ivan0yurche...@gmail.com>
>> > > > > > >> wrote:
>> > > > > > >>
>> > > > > > >> > Hi,
>> > > > > > >> >
>> > > > > > >> > Thank you for the KIP.
>> > > > > > >> >
>> > > > > > >> 

Re: [DISCUSS] KIP-831: Add metric for log recovery progress

[Luke Chen]

Hi devs,

If there are no other comments, I'll start a vote tomorrow.

Thank you.
Luke

On Sun, May 1, 2022 at 5:08 PM Luke Chen  wrote:

> Hi James,
>
> Sorry for the late reply.
>
> Yes, this is a good point, to know how many segments to be recovered if
> there are some large partitions.
> I've updated the KIP, to add a `*RemainingSegmentsToRecover*` metric for
> each log recovery thread, to show the value.
> The example in the Proposed section here
> 
> shows what it will look like.
>
> Thanks for the suggestion.
>
> Thank you.
> Luke
>
>
>
> On Sat, Apr 23, 2022 at 8:54 AM James Cheng  wrote:
>
>> The KIP describes RemainingLogsToRecovery, which seems to be the number
>> of partitions in each log.dir.
>>
>> We have some partitions which are much much larger than others. Those
>> large partitions have many many more segments than others.
>>
>> Is there a way the metric can reflect partition size? Could it be
>> RemainingSegmentsToRecover? Or even RemainingBytesToRecover?
>>
>> -James
>>
>> Sent from my iPhone
>>
>> > On Apr 20, 2022, at 2:01 AM, Luke Chen  wrote:
>> >
>> > Hi all,
>> >
>> > I'd like to propose a KIP to expose a metric for log recovery progress.
>> > This metric would let the admins have a way to monitor the log recovery
>> > progress.
>> > Details can be found here:
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-831%3A+Add+metric+for+log+recovery+progress
>> >
>> > Any feedback is appreciated.
>> >
>> > Thank you.
>> > Luke
>>
>

Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API

[Igor Soarez]

Hi Mickael,

Thanks for writing this KIP. This would be a very useful improvement!

--
Igor

On Thu, Apr 7, 2022, at 10:16 AM, Mickael Maison wrote:
> Hi,
>
> I wrote a small KIP to expose the total and usable space of logdirs
> via the DescribeLogDirs API:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API
>
> Please take a look and let me know if you have any feedback.
>
> Thanks,
> Mickael

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.2 #44

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready

[Igor Soarez]

Hi Colin,

It's very exciting to see KRaft ready for production! 

I see the value of marking it ready for production even with the current 
missing features.

However, I'm worried about marking ZK mode as deprecated before these missing 
features are ready. It might be hard to estimate right now how much work is 
actually necessary to close that gap, particularly the upgrade from ZK. 
Deprecating ZK before providing users with an upgrade path might be sending the 
wrong message - when something is deprecated we want users to move away from 
it, so that should be possible. It might well be more than a few months to 
close the gap. 

Best,

--
Igor

On Wed, May 4, 2022, at 7:22 AM, Colin McCabe wrote:
> On Tue, May 3, 2022, at 23:16, Colin McCabe wrote:
>>
>> To be clear, the proposal here is to have the bridge release be 3.4 and 
>> then move on to a ZK-free 4.0. With a 3.5 release as an option (but not 
>> requirement) if we can't finish everything in time after 3.4. So that 
>> would be two more releases with ZK, or dropping ZK a little bit less 
>> than a year from now.
>>
>
> Sigh. This should read "To be clear, the proposal here is to have the 
> bridge release be 3.3 and then move on to a ZK-free 4.0. With a 3.4 
> release as an option (but not requirement) if we can't finish 
> everything in time after 3.3." So much for being clear, I guess :)
>
> cheers,
> Colin
>
>
>> best,
>> Colin
>>
>>>
>>> [1]
>>> https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan
>>> [2] https://kafka.apache.org/downloads
>>>
>>>
>>> Israel Ekpo
>>> Lead Instructor, IzzyAcademy.com
>>> https://www.youtube.com/c/izzyacademy
>>> https://izzyacademy.com/
>>>
>>>
>>> On Tue, May 3, 2022 at 10:32 PM Luke Chen  wrote:
>>>
 Hi Colin,

 So exciting to see the KIP to mark the Kraft production ready!

 Just one comment: We should make sure the period between ZK deprecation
 (i.e. v3.4.0) to ZK removal (i.e. v4.0.0) is not too short.
 Do we have any expectation for the deprecation period?
 After all, this is not a small feature change, and users need more time to
 do the migration.
 But to be honest, I don't know how long is long enough.
 Maybe at least half a year? Or more? Thoughts?


 Thank you.
 Luke

 On Wed, May 4, 2022 at 9:10 AM Colin McCabe  wrote:

 > Hi all,
 >
 > I've written a KIP for marking KRaft as production ready. Please take a
 > look if you have a chance:
 >
 > https://cwiki.apache.org/confluence/x/8xKhD
 >
 > thanks,
 > Colin
 >

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #906

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready

[Colin McCabe]

On Tue, May 3, 2022, at 23:16, Colin McCabe wrote:
>
> To be clear, the proposal here is to have the bridge release be 3.4 and 
> then move on to a ZK-free 4.0. With a 3.5 release as an option (but not 
> requirement) if we can't finish everything in time after 3.4. So that 
> would be two more releases with ZK, or dropping ZK a little bit less 
> than a year from now.
>

Sigh. This should read "To be clear, the proposal here is to have the bridge 
release be 3.3 and then move on to a ZK-free 4.0. With a 3.4 release as an 
option (but not requirement) if we can't finish everything in time after 3.3." 
So much for being clear, I guess :)

cheers,
Colin


> best,
> Colin
>
>>
>> [1]
>> https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan
>> [2] https://kafka.apache.org/downloads
>>
>>
>> Israel Ekpo
>> Lead Instructor, IzzyAcademy.com
>> https://www.youtube.com/c/izzyacademy
>> https://izzyacademy.com/
>>
>>
>> On Tue, May 3, 2022 at 10:32 PM Luke Chen  wrote:
>>
>>> Hi Colin,
>>>
>>> So exciting to see the KIP to mark the Kraft production ready!
>>>
>>> Just one comment: We should make sure the period between ZK deprecation
>>> (i.e. v3.4.0) to ZK removal (i.e. v4.0.0) is not too short.
>>> Do we have any expectation for the deprecation period?
>>> After all, this is not a small feature change, and users need more time to
>>> do the migration.
>>> But to be honest, I don't know how long is long enough.
>>> Maybe at least half a year? Or more? Thoughts?
>>>
>>>
>>> Thank you.
>>> Luke
>>>
>>> On Wed, May 4, 2022 at 9:10 AM Colin McCabe  wrote:
>>>
>>> > Hi all,
>>> >
>>> > I've written a KIP for marking KRaft as production ready. Please take a
>>> > look if you have a chance:
>>> >
>>> > https://cwiki.apache.org/confluence/x/8xKhD
>>> >
>>> > thanks,
>>> > Colin
>>> >
>>>

Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready

[Colin McCabe]

On Tue, May 3, 2022, at 19:32, Luke Chen wrote:
> Hi Colin,
>
> So exciting to see the KIP to mark the Kraft production ready!
>
> Just one comment: We should make sure the period between ZK deprecation
> (i.e. v3.4.0) to ZK removal (i.e. v4.0.0) is not too short.
> Do we have any expectation for the deprecation period?
> After all, this is not a small feature change, and users need more time to
> do the migration.
> But to be honest, I don't know how long is long enough.
> Maybe at least half a year? Or more? Thoughts?
>

Hi Luke,

Like we discussed below, in chronological terms a Kafka release is about 4 or 5 
months. So if we have 3.3, followed by a 4.0 where we drop ZK, that would be 
about 8 to 10 months. If we need more time we can add a 3.4 which will add 
another 4-5 months in between.

best,
Colin

>
> Thank you.
> Luke
>
> On Wed, May 4, 2022 at 9:10 AM Colin McCabe  wrote:
>
>> Hi all,
>>
>> I've written a KIP for marking KRaft as production ready. Please take a
>> look if you have a chance:
>>
>> https://cwiki.apache.org/confluence/x/8xKhD
>>
>> thanks,
>> Colin
>>

Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready

[Colin McCabe]

On Tue, May 3, 2022, at 20:48, Israel Ekpo wrote:
> Hi Colin,
>
> Thanks for this KIP. I am very excited to see the proposal.
>
> A lot in the community have been asking when KRaft mode will be ready and I
> think this KIP will provide a lot of the answers and guidance desperately
> needed.
>
> Thanks for working on it.
>
> I also have some of the questions from Luke regarding the duration of the
> deprecation period and the release where ZK will be completely dropped.
>
> As per our time-based release plan [1], the release should be every 4
> months, but it has been approximately 5 months between releases from the
> downloads page [2]
>
> So, right now we are about to push out 3.2.0 and 3.4.0 is approximately 10
> months from now based on the schedule/plan.
>

Hi Israel,

Sorry, I meant to write "the next release, 3.3" not "the next release, 3.4." I 
fixed the KIP text now.

As you noted, 3.3 should be about 4 or 5 months from now.

> I wonder if this will be enough time to complete all the features needed to
> bring KRaft mode to parity with Legacy/Zookeeper mode.
>
> The missing features you have highlighted and any other critical features
> not working in KRaft mode should be implemented fully before marking KRaft
> mode as production ready and deprecating Legacy mode.
>

As I wrote in "potential objections and responses," I don't think we need full 
feature parity before marking KRaft mode as production ready. It's customary 
for Kafka to bring new features to production incrementally.

>
> The main question here is would we have Kafka versions 3.6.0, 3.7.0, 3.8.0
> and 3.9.0 because this will add approximately 20 additional months after
> 3.4.0 which should be enough time to have ZK-mode deprecated and tested
> thoroughly before removal completely. I feel we should not mark ZK mode as
> deprecated until all the features in KRaft mode are at parity with ZK-mode.
>
> If we are going to have 20 months after 3.4 to work with a deprecated
> ZK-mode, then it could be possible to even drop ZK fully before 4.0
>
> Let's move the boat closer to the dock before we jump out of the boat. It
> will be very stressful if we jump out of the boat and have to swim several
> miles to shore.
>

That is a lot of Kafka versions! I don't think anyone wants 3.x to go on that 
long, or the ZooKeeper metadata implementation either. Ultimately I believe 
such a very long multi-year transition period would make the code quality and 
project stability worse, since we'd have to split our efforts across the two 
implementations.

To be clear, the proposal here is to have the bridge release be 3.4 and then 
move on to a ZK-free 4.0. With a 3.5 release as an option (but not requirement) 
if we can't finish everything in time after 3.4. So that would be two more 
releases with ZK, or dropping ZK a little bit less than a year from now.

best,
Colin

>
> [1]
> https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan
> [2] https://kafka.apache.org/downloads
>
>
> Israel Ekpo
> Lead Instructor, IzzyAcademy.com
> https://www.youtube.com/c/izzyacademy
> https://izzyacademy.com/
>
>
> On Tue, May 3, 2022 at 10:32 PM Luke Chen  wrote:
>
>> Hi Colin,
>>
>> So exciting to see the KIP to mark the Kraft production ready!
>>
>> Just one comment: We should make sure the period between ZK deprecation
>> (i.e. v3.4.0) to ZK removal (i.e. v4.0.0) is not too short.
>> Do we have any expectation for the deprecation period?
>> After all, this is not a small feature change, and users need more time to
>> do the migration.
>> But to be honest, I don't know how long is long enough.
>> Maybe at least half a year? Or more? Thoughts?
>>
>>
>> Thank you.
>> Luke
>>
>> On Wed, May 4, 2022 at 9:10 AM Colin McCabe  wrote:
>>
>> > Hi all,
>> >
>> > I've written a KIP for marking KRaft as production ready. Please take a
>> > look if you have a chance:
>> >
>> > https://cwiki.apache.org/confluence/x/8xKhD
>> >
>> > thanks,
>> > Colin
>> >
>>