Re: [VOTE] 3.1.1 RC1

[Dongjoon Hyun]

Thank you for sharing your assessment, Tom and Chris.

Apache Spark community had a discussion thread on March.

https://lists.apache.org/thread/yq6f5gv7gtdbk1ynwpd9hnc547bgz03m
"[DISCUSS] Migration guide on upgrading Kafka to 3.1 in Spark 3.3"

As a person who initiated that upgrade via SPARK-36837, I need to answer it 
before Apache Spark 3.3 RC2. In the worst case, Apache Spark community may 
decide to wait for next Kafka versions instead of upgrading.

Thank you again all people to help this release!

Bests,
Dongjoon.

On 2022/05/12 01:39:16 Chris Egerton wrote:
> It's worth noting that one of the most frequent offenders
> (RebalanceSourceConnectorsIntegrationTest.testDeleteConnector), which
> failed in five of the nine runs from 111 through 119, is actually disabled
> on 3.2 and trunk as it's caused by a known and not-yet-addressed bug in
> rebalancing logic. Since it's not a regression we can safely ignore those
> failures, but if we plan on putting out another 3.1 release we should
> probably disable that test on 3.1 as well.
> 
> On Wed, May 11, 2022 at 3:32 AM Tom Bentley  wrote:
> 
> > Hi Dongjoon,
> >
> > I've been trying to get a green build of the 3.1 branch from Jenkins (
> > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.1/) since my
> > original email promised to provide this. Individually each stage between
> > builds 111 and 119 has passed in at least one run, but there have been no
> > runs where all stages have passed.
> >
> > However, as you note, the vote has the necessary 3 binding votes, so
> > assuming the voters don't wish to change their votes I can proceed with the
> > release. If this is not the case please respond on the thread. If I hear
> > nothing in the next 24 hours I will assume the voters are OK with this and
> > proceed with the rest of the release process.
> >
> > Apologies for the delay,
> >
> > Kind regards,
> >
> > Tom
> >
> >
> > On Wed, 11 May 2022 at 08:15, Dongjoon Hyun  wrote:
> >
> > > Hi, Tom.
> > >
> > > Could you conclude this vote as a release manager? :)
> > >
> > > Dongjoon.
> > >
> > > On 2022/05/06 13:31:15 Michal Tóth wrote:
> > > > Hello
> > > >
> > > > I have executed some produce/consume system tests which all passed.
> > > > Also everything passed from
> > > https://github.com/tombentley/kafka-verify-rc
> > > > - checking signatures, checksums, with gradle unit & integration tests,
> > > etc.
> > > >
> > > > Good from me (non-binding).
> > > >
> > > >
> > > >
> > > > pi 6. 5. 2022 o 14:30 David Jacot 
> > > napísal(a):
> > > >
> > > > > Thanks for running the release, Tom.
> > > > >
> > > > > I performed the following validations:
> > > > > * Verified all checksums and signatures.
> > > > > * Built from source and ran unit tests.
> > > > > * Ran the first quickstart steps for both ZK and KRaft.
> > > > > * Spotchecked the Javadocs.
> > > > >
> > > > > I noticed the same issues as others on the website. I checked
> > > > > the doc in git and it looks good.
> > > > >
> > > > > +1 (binding)
> > > > >
> > > > > Best,
> > > > > David
> > > > >
> > > > > On Thu, May 5, 2022 at 7:52 PM Dongjoon Hyun 
> > > wrote:
> > > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > RC1 was tested with Apache Spark tests
> > > > > >
> > > > > > - https://github.com/apache/spark/pull/36135
> > > > > >
> > > > > > Thanks,
> > > > > > Dongjoon.
> > > > > >
> > > > > > On 2022/05/05 03:25:25 Luke Chen wrote:
> > > > > > > Hi Tom,
> > > > > > >
> > > > > > > I did:
> > > > > > > 1. check the signature and checksums
> > > > > > > 2. ran quick start with java17 + sacla2.12
> > > > > > > 3. browse java docs/documentations
> > > > > > >
> > > > > > > +1 (non-binding)
> > > > > > >
> > > > > > > Thanks for running the release.
> > > > > > >
> > > > > > > Luke
> > > > > > >
> > > > > > > On Thu, May 5, 2022 at 12:43 AM Bill Bejeck 
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi Tom,
> > > > > > > >
> > > > > > > > Thanks for running the release!
> > > > > > > >
> > > > > > > > I did the following checks:
> > > > > > > >
> > > > > > > >1. Validated all the checksum and signatures
> > > > > > > >2. Built from source and ran the unit tests (Java 11)
> > > > > > > >3. Ran the quickstart and the Kafka Streams demo (Java 11)
> > > > > > > >4. Did a quick scan of the Javadoc and the documentation
> > > > > > > >
> > > > > > > > I noticed the same issues as Mickael on the website, but
> > > otherwise,
> > > > > it
> > > > > > > > looks good.
> > > > > > > > +1 (binding)
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Bill
> > > > > > > >
> > > > > > > > On Tue, May 3, 2022 at 10:17 AM Jakub Scholz 
> > > > > wrote:
> > > > > > > >
> > > > > > > > > +1 (non-binding). I used the Scala 2.13 binaries and staged
> > > Maven
> > > > > > > > artifacts
> > > > > > > > > and ran various tests with them. Thanks for doing the
> > release.
> > > > > > > > >
> > > > > > > > > Jakub
> > > > > > > > >
> > > > > > > > > On Fri, Apr 29, 2022 at 8:16 PM 

[jira] [Resolved] (KAFKA-13763) Improve unit testing coverage and flexibility for IncrementalCooperativeAssignor

[Chris Egerton (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Egerton resolved KAFKA-13763.
---
Resolution: Done

> Improve unit testing coverage and flexibility for 
> IncrementalCooperativeAssignor
> 
>
> Key: KAFKA-13763
> URL: https://issues.apache.org/jira/browse/KAFKA-13763
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Reporter: Chris Egerton
>Assignee: Chris Egerton
>Priority: Minor
>
> The 
> [tests|https://github.com/apache/kafka/blob/dcd09de1ed84b43f269eb32fc2baf589a791d468/connect/runtime/src/test/java/org/apache/kafka/connect/runtime/distributed/IncrementalCooperativeAssignorTest.java]
>  for the {{IncrementalCooperativeAssignor}} class provide a moderate level of 
> coverage and cover some non-trivial cases, but there are some areas for 
> improvement that will allow us to iterate on the assignment logic for Kafka 
> Connect faster and with greater confidence.
> These improvements include:
>  * Adding reusable utility methods to assert that a cluster's assignment is 
> *balanced* (the difference in the number of connectors and tasks assigned to 
> any two workers is at most one) and *complete* (all connectors and tasks are 
> assigned to a worker)
>  * Removing the existing 
> [assertAssignment|https://github.com/apache/kafka/blob/dcd09de1ed84b43f269eb32fc2baf589a791d468/connect/runtime/src/test/java/org/apache/kafka/connect/runtime/distributed/IncrementalCooperativeAssignorTest.java#L1373-L1405]
>  methods and replacing them with a more fine-grained alternative that allows 
> for more granular assertions about the number of tasks/connectors 
> assigned/revoked from each worker during a round of rebalance, instead of the 
> total for the entire cluster
>  * Adding a reusable utility method to assert the current distribution of 
> connectors and tasks across the cluster
>  * Decomposing large portions of repeated code for simulating a round of 
> rebalancing into a reusable utility method
>  * Renaming variable names to improve accuracy/readability (the 
> {{expectedMemberConfigs}} field, for example, is pretty poorly named)
> But other improvements may be added in a pull request that addresses the 
> above as they come up.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [VOTE] 3.1.1 RC1

[Chris Egerton]

It's worth noting that one of the most frequent offenders
(RebalanceSourceConnectorsIntegrationTest.testDeleteConnector), which
failed in five of the nine runs from 111 through 119, is actually disabled
on 3.2 and trunk as it's caused by a known and not-yet-addressed bug in
rebalancing logic. Since it's not a regression we can safely ignore those
failures, but if we plan on putting out another 3.1 release we should
probably disable that test on 3.1 as well.

On Wed, May 11, 2022 at 3:32 AM Tom Bentley  wrote:

> Hi Dongjoon,
>
> I've been trying to get a green build of the 3.1 branch from Jenkins (
> https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.1/) since my
> original email promised to provide this. Individually each stage between
> builds 111 and 119 has passed in at least one run, but there have been no
> runs where all stages have passed.
>
> However, as you note, the vote has the necessary 3 binding votes, so
> assuming the voters don't wish to change their votes I can proceed with the
> release. If this is not the case please respond on the thread. If I hear
> nothing in the next 24 hours I will assume the voters are OK with this and
> proceed with the rest of the release process.
>
> Apologies for the delay,
>
> Kind regards,
>
> Tom
>
>
> On Wed, 11 May 2022 at 08:15, Dongjoon Hyun  wrote:
>
> > Hi, Tom.
> >
> > Could you conclude this vote as a release manager? :)
> >
> > Dongjoon.
> >
> > On 2022/05/06 13:31:15 Michal Tóth wrote:
> > > Hello
> > >
> > > I have executed some produce/consume system tests which all passed.
> > > Also everything passed from
> > https://github.com/tombentley/kafka-verify-rc
> > > - checking signatures, checksums, with gradle unit & integration tests,
> > etc.
> > >
> > > Good from me (non-binding).
> > >
> > >
> > >
> > > pi 6. 5. 2022 o 14:30 David Jacot 
> > napísal(a):
> > >
> > > > Thanks for running the release, Tom.
> > > >
> > > > I performed the following validations:
> > > > * Verified all checksums and signatures.
> > > > * Built from source and ran unit tests.
> > > > * Ran the first quickstart steps for both ZK and KRaft.
> > > > * Spotchecked the Javadocs.
> > > >
> > > > I noticed the same issues as others on the website. I checked
> > > > the doc in git and it looks good.
> > > >
> > > > +1 (binding)
> > > >
> > > > Best,
> > > > David
> > > >
> > > > On Thu, May 5, 2022 at 7:52 PM Dongjoon Hyun 
> > wrote:
> > > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > RC1 was tested with Apache Spark tests
> > > > >
> > > > > - https://github.com/apache/spark/pull/36135
> > > > >
> > > > > Thanks,
> > > > > Dongjoon.
> > > > >
> > > > > On 2022/05/05 03:25:25 Luke Chen wrote:
> > > > > > Hi Tom,
> > > > > >
> > > > > > I did:
> > > > > > 1. check the signature and checksums
> > > > > > 2. ran quick start with java17 + sacla2.12
> > > > > > 3. browse java docs/documentations
> > > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > Thanks for running the release.
> > > > > >
> > > > > > Luke
> > > > > >
> > > > > > On Thu, May 5, 2022 at 12:43 AM Bill Bejeck 
> > > > wrote:
> > > > > >
> > > > > > > Hi Tom,
> > > > > > >
> > > > > > > Thanks for running the release!
> > > > > > >
> > > > > > > I did the following checks:
> > > > > > >
> > > > > > >1. Validated all the checksum and signatures
> > > > > > >2. Built from source and ran the unit tests (Java 11)
> > > > > > >3. Ran the quickstart and the Kafka Streams demo (Java 11)
> > > > > > >4. Did a quick scan of the Javadoc and the documentation
> > > > > > >
> > > > > > > I noticed the same issues as Mickael on the website, but
> > otherwise,
> > > > it
> > > > > > > looks good.
> > > > > > > +1 (binding)
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Bill
> > > > > > >
> > > > > > > On Tue, May 3, 2022 at 10:17 AM Jakub Scholz 
> > > > wrote:
> > > > > > >
> > > > > > > > +1 (non-binding). I used the Scala 2.13 binaries and staged
> > Maven
> > > > > > > artifacts
> > > > > > > > and ran various tests with them. Thanks for doing the
> release.
> > > > > > > >
> > > > > > > > Jakub
> > > > > > > >
> > > > > > > > On Fri, Apr 29, 2022 at 8:16 PM Mickael Maison <
> > > > mimai...@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Tom,
> > > > > > > > >
> > > > > > > > > Thanks for running this release!
> > > > > > > > >
> > > > > > > > > I've done the following:
> > > > > > > > > - Checked signatures and checksums
> > > > > > > > > - Checked javadocs/maven artifacts
> > > > > > > > > - Built from source and run all tests with Java 11
> > > > > > > > > - Ran quickstart on Scala 2.13 artifact with Java 11
> > > > > > > > >
> > > > > > > > > It looks like the website has not been updated yet, I still
> > only
> > > > see
> > > > > > > > > 3.1.0. When you'll add 3.1.1, let's make sure we mention
> > > > reload4j in
> > > > > > > > > the notable changes section.
> > > > > > > > >
> > > > > > > > > +1 (binding)
> > > > > > > > >
> > > > > > > > > Thanks,
> > 

Kafka Consumer Partition Non Uniform Data Fetch/Pull

[Muthuswamy S]

Hi,
We are using open source Confluent Kafka deployed in Kubernetes. We have
less consumers than the partitions and as such we have multiple Kafka
partitions assigned to each consumer pod. We observed some of these
partitions have higher lag than the other, it seems the consumer is
prioritizing some partitions over others. Our messages are different in
sizes and we have set max bytes per message and overall partition fetch
size too.

How can we ensure to process Round Robin mode in Kafka Consumer with the
partitions assigned to it? Can you pls help?

-- 
thanks,
Muthuswamy.S

Re: [DISCUSS] KIP-832 Allow creating a producer/consumer using a producer/consumer config

[Chris Egerton]

Hi Francois,

This is a bit of an unusual situation. I think a separate KIP may be the
safest route here since this one has already gotten the votes necessary to
be included in the next release (as long as it can be implemented and
reviewed in time), but it'd probably be best if only one of this KIP and
the other one get approved and released. Perhaps you could mention in the
separate KIP that it's meant to supersede KIP-832 if it can be approved in
time for the 3.3.0 release, and otherwise, we'll move forward with KIP-832?
Of course, this is all assuming that you'd like to see this new API
introduced in the 3.3.0 release; if not, then it should be fine to stick
with this KIP and keep the discussion here.

On the topic of the builder pattern:

1. I agree that with single-class properties (like key/value
(de)serializers) that instances provided to the builder should take
priority over any classes specified in the client config. However, with
multi-class properties (like interceptors), it may be better to combine the
lists in order to accommodate the example provided in the KIP with
the SpringAwareProducerConfig. If combination isn't desirable, it's simple
enough for developers to nullify the relevant property in the client
config. On the other hand, if the client doesn't combine builder-provided
and config-provided plugin classes, it'd be trickier to pre-instantiate the
ones in the client config, manually combine them with the ones that are
already instantiated, and then pass those to the builder.

2. I'd opt to keep the Map/Properties argument in the builder constructor
instead of in the build() method, as it's a required parameter and IME
those tend to be included in constructors instead of follow-up methods. But
it's just a slight preference, both should be fine.

3. Unless we want to make the API and implementation more complex, it'd
probably be best to note that any Closeable objects provided to a builder
will still be closed once the client is closed, and as a result, it'd be
highly recommended to only use a builder once. We might even go so far as
to throw an exception if build() is invoked multiple times.

4. We should also decide whether objects that implement the Configurable
interface have their configure() method invoked by the builder/client. I
suspect we'd want to refrain from invoking it as that would provide more
flexibility (you can now customize the instance with a completely separate
config from what you give the client) and it would align more closely with
the example provided in the KIP, but an argument could be made the other
way as it would be slightly more convenient to not have to remember to
invoke configure() yourself, and it would probably be less surprising to
developers.

5. I see your point about enriching the builder API over adding a general
"withConfiguredInstance" method. The reason I originally proposed this is
that it'd be easier to maintain in the future and we wouldn't have to
remember to expand the builder API whenever adding a new plugin class
property to our clients, and the only reason that (de)serializers and
interceptors were given special treatment was to address generic type
safety holes in the existing API. After sleeping on it, I think I'm
convinced that your more-enriched API is better; it's simpler, easier to
understand, and addresses non-generics-related type safety holes with the
current API (how do I know that the class name for the
"partition.assignment.strategy" property actually refers to a class that
implements the ConsumerPartitionAssignor interface?). Thumbs up!

6. You mentioned earlier that "Having an access to the config also gives a
way to also fine tune other aspects such as the logging related to the
config. Log unused, skip some properties, etc.". As someone who's wrangled
with the unused property logging recently [1] I can definitely sympathize
with that motivation, although I really wish we could "do things the
right way" instead of giving developers a way to work around our
well-intentioned-but-not-completely-accurate logic. Overall I think it
might be acceptable to address these concerns separately instead of
coupling them with this KIP (or a KIP to introduce the builder pattern for
clients), but if any of these are pressing for you, we should make note of
that and see if there's a way to handle them.

7. Although I'd love it if we could whittle things down to one constructor
per client, I think deprecation of the existing constructors is a bit too
aggressive of a step for now. We could mention in the KIP that it's an
option for the future if this new API is well-accepted, but for now, we'll
simply be adding a new constructor and continuing to support all the
existing ones as well.

Lastly--apologies to the Streams people involved here. I've been saying
"client" and "clients" when, most of the time, I should have been appending
"and/or Streams" to that as well :)
And yes, I definitely do support including KafkaStreams with the builder

Contributor Permission

[Jack Burgoyne]

Hello,

I'd like to start contributing to Kafka, could I have contributor
permission for my Jira username: jcbu2718?

Thank you,
Jack

Re: [DISCUSS] KIP-835: Monitor KRaft Controller Quorum Health

[José Armando García Sancio]

Thanks for all of the feedback. Some comments below:

Luke wrote:
> 1. Jason has asked but you didn't answer: What is the default value for `
> metadata.monitor.write.interval.ms`?

Thanks for asking again. Looks like I missed this in my previous
reply. In the implementation I am currently working on, I have it at 1
second. Colin is suggesting 500ms. I should point out that the fetch
timeout default is currently 2 seconds. I'll go with Colin's
suggestion and make it 500ms.

> 2. The `noopRecord` API key is `TBD`. Why can't we put the "currently used
> API Key nums + 1" into it? Any concern?

Yes. It will be the highest currently used API key for metadata
records plus one. I didn't specify it yet since the code hasn't been
committed. I usually update the KIP once the code is committed to
trunk.

> 3. typo: name=MetadataWriteOffse[t]s

Fixed.

> 4. I don't understand the difference between MetadataLastCommittedOffset
> and MetadataWriteOffsets metrics. I think the 2 values will always be
> dentical, unless the controller metadata write failed, is that correct?

The active controller tracks two offsets, the writeOffset and the
lastCommittedOffset. The writeOffset is the largest offset that was
sent to the KRaft layer to get committed. The lastCommittedOffset is
the largest offset that the controller knows got committed. The
controller increases the lastCommittedOffset as the KRaft layer
replicates records, commits records and finally the controller handles
the committed record. Colin suggested calling this the
last-applied-offset-*`. I agree with this rename as it more accurately
represents the state being reported.

I updated the description of those metrics. Please take a look.

David wrote:
> 1. Based on the config name "metadata.monitor.write.interval.ms" I'm
> guessing the intention is to have a regularly scheduled write. If the
> quorum is busy with lots of the writes, we wouldn't need this NoopRecord
> ight? Maybe we can instead write a NoopRecord only after some amount of
> idle time.

Correct. I suspect that the first implementation will always write
this record in the interval configured. Future implementation can
optimize this and only write the NoopRecord if the active controller
didn't already write a record in this time interval.

> 2. A small naming suggestion, what about "NoOpRecord"?

Sounds good to me. I changed the record to NoOpRecord.

> 4. We should consider omitting these records from the log dump tool, or at
> least adding an option to skip over them.

Thanks for pointing this out. DumpLogSegments uses MetadataRecordSerde
so it should just work as long as it is the same software version as
the Kafka node. The issue is if the user uses an old DumpLogSegments
on a metadata log that supports this feature.

I updated the "Compatibility, Deprecation, and Migration Plan" section
to document this

> 5. If (somehow) one of these NoopRecord appeared in the snapshot, it sounds
> like the broker/controller would skip over them. We should specify this
> behavior in the KIP

Excluding bugs they should not be part of the metadata snapshot
because the controllers will not store this information in the
"timeline" data types and the brokers will not include this
information in the "metadata image". In other words both the
controller and brokers will skip this record when replaying the log
and snapshot. Both `handleCommit` and `handleSnapshot` eventually
execute the same "replay" code.

Colin wrote:
> I had the same question as Luke and Jason: what's the default here for the 
> NoOpRecord time? :) We should add a value here even if we think we'll adjust 
> it later, just to give a feeling for how much traffic this would create. 
> Perhaps 500 ms?

I agree. I updated the KIP to document this as 500ms.

> Also how about naming the time configuration something like 
> "metadata.max.idle.interval.ms", to make it clear that this is the maximum 
> time we can go between writes to the metadata log. I don't get that meaning 
> out of "monitor interval"...

I like this suggestion. I think this name makes it less implementation
specific and allows us to not write NoOpRecords if the active
controller already wrote a record in the configured time interval.

> I'd suggest renaming this as "last-applied-offset-*" to make it clear that 
> the offset we're measuring is the last one that the broker applied. The last 
> committed offset is something else, a more log-layer-centric concept.

I agree. I change the KIP to prefix these metrics with "last applied
record ...".

> (It would also be good to have a separate metric which purely reflected the 
> last metadata offset we've fetched and not applied, so we can see if the 
> delta between that and last-applied-offset increases...)

In KIP-595 we have this metric which reports this information:
1. log-end-offset, type=raft-manager, dynamic gauge

Here are the changes to the KIP:
https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=211883219=6=5

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.2 #50

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-13894) Extend Kafka kerberos auth support to beyond only hostname

[Yiming Zang (Jira)]

Yiming Zang created KAFKA-13894:
---

 Summary: Extend Kafka kerberos auth support to beyond only hostname
 Key: KAFKA-13894
 URL: https://issues.apache.org/jira/browse/KAFKA-13894
 Project: Kafka
  Issue Type: Improvement
Reporter: Yiming Zang


{*}Problem{*}:

Currently Kafka client only support using the Kafka broker hostname in the 
kerberos authentication process ([Source 
Code|https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/network/SaslChannelBuilder.java#L231]).

However, not all companies support per-host based keytabs. It is a common 
practice that a keytabs which contains a shared identity name is used instead. 
To support this kind of Kerberos set ups, we need to make some changes to make 
Kafka support a customized service name apart from just using the hostname for 
authentication.

{*}Proposal{*}:

To address this issue, we propose to add an extra client side configuration for 
Kerberos authentication. If user provide that configuration, we will use 
whatever is provided to replace the hostname, otherwise we will default back to 
use hostnames. Here's an example:

 
{code:java}
String kerberosServiceNameFromConfig = 
(String)configs.get(SaslConfigs.SASL_KERBEROS_SERVICE_NAME);

String hostnameOrServiceName = (kerberosServiceNameFromConfig == null || 
kerberosServiceNameFromConfig.trim().isEmpty()) ? 
socket.getInetAddress().getHostName() : kerberosServiceNameFromConfig;

authenticatorCreator = () -> buildClientAuthenticator(configs,
  saslCallbackHandlers.get(clientSaslMechanism),
  id,
  hostnameOrServiceName,
  loginManager.serviceName(),
  transportLayer,
  subjects.get(clientSaslMechanism));{code}
 

 

 



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

[Omnia Ibrahim]

Hi Colin,
I don't mind the idea of MM2 users implementing the AdminClient interface.
However, there're two disadvantages to this.

   1. Having around 70 methods definitions to have "NotImplemented" is one
   downside, and keep up with these if the AdminClient interface changes.
   2. It makes it hard to list what admin functionality MM2 uses as MM2
   interactions with AdminClient in the codebase are in many places.

I guess it's OK for MM2 users who want to build their admin client to carry
this burden, as I explained in my previous response to the discussion
thread. And we can do some cleanup to the codebase to have all Admin
interactions in MM2 in a utils class or something like that to make it
easier to navigate what MM2 needs from the Admin interface.

Maybe I'm misunderstanding the use-case you're describing here. But it
> seems to me that if you create a proxy that has the ability to do any admin
> operation, and give MM2 access to that proxy, the security model is the
> same as just giving MM2 admin access. (Or it may be worse if the sysadmin
> doesn't know what this proxy is doing, and doesn't lock it down...)
>

MM2 runs with the assumption that it has

   - "CREATE" ACLs for topics on the source clusters to create `heartbeat`
   topics.
   - "CREATE"  and "ALTER" ACLs to create topics, add partitions, update
   topics' config and topics' ACLs (in future, will also include group ACLS as
   Mikael mentioned before in the thread) on the destination clusters.

Most organisations have some resource management or federated solutions
(some would even have a budget system as part of these systems) to manage
Kafka resources, and these systems are usually the only application allowed
to initializing a client with "CREATE" and "ALTER" ACLs. They don't grant
these ACLs to any other teams/groups/applications to create such a client
outside these systems, so assuming MM2 can bypass these systems and use the
AdminClient directly to create/update resources isn't valid. This is the
primary concern here.

The KIP is trying to give MM2 more flexibility to allow organisations to
integrate MM2 with their resource management system as they see fit without
forcing them to disable most MM2 features.

Hope this make sense and clear it up.


On Wed, May 11, 2022 at 9:09 PM Colin McCabe  wrote:

> Hi Omnia Ibrahim,
>
> I'm sorry, but I am -1 on adding competing Admin interfaces. This would
> create confusion and a heavier maintenance burden for the project.
>
> Since the org.apache.kafka.clients.admin.Admin interface is a Java
> interface, any third-party software that wants to insert its own
> implementation of the interface can do so already.
>
> A KIP to make the Admin class used pluggable for MM2 would be reasonable.
> Adding a competing admin API is not.
>
> It's true that there are many Admin methods, but you do not need to
> implement all of them -- just the ones that MirrorMaker uses. The other
> ones can throw a NotImplementedException or similar.
>
> > The current approach also assumes that the user running MM2 has the
> Admin right to
> > create/update topics, which is only valid if the user who runs MM2 also
> manages both
> > source and destination clusters.
>
> Maybe I'm misunderstanding the use-case you're describing here. But it
> seems to me that if you create a proxy that has the ability to do any admin
> operation, and give MM2 access to that proxy, the security model is the
> same as just giving MM2 admin access. (Or it may be worse if the sysadmin
> doesn't know what this proxy is doing, and doesn't lock it down...)
>
> best,
> Colin
>
>
> On Mon, May 9, 2022, at 13:21, Omnia Ibrahim wrote:
> > Hi, I gave the KIP another look after talking to some people at the Kafka
> > Summit in London. And I would like to clear up the motivation of this
> KIP.
> >
> >
> > At the moment, MM2 has some opinionated decisions that are creating
> issues
> > for teams that use IaC, federated solutions or have a capacity/budget
> > planning system for Kafka destination clusters. To explain it better,
> let's
> > assume we have MM2 with the following configurations to highlight these
> > problems.
> >
> > ```
> >
> > topics = .*
> >
> > refresh.topics.enabled = true
> >
> > sync.topic.configs.enabled = true
> >
> > sync.topic.acls.enabled = true
> >
> > // Maybe in futrue we can have sync.group.acls.enabled = true
> >
> > ```
> >
> >
> > These configurations allow us to run MM2 with the value of its full
> > features. However, there are two main concerns when we run on a scale
> with
> > these configs:
> >
> > 1. *Capacity/Budgeting Planning:*
> >
> > Functionality or features that impact capacity planning using MM2 are:
> >
> >1. MM2 automatically creates topics (breaking the rule of
> >`auto.create.topics.enable=false`) and creates topic partitions on
> >destination clusters if the number of partitions increases on the
> source.
> >In the previous example, this functionality will apply to any topic
> 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #924

[Apache Jenkins Server]

See 

Re: [DISCUSS] Apache Kafka 3.3.0 Release

[Chris Egerton]

Hi José,

Could we add KIP-618 (
https://cwiki.apache.org/confluence/display/KAFKA/KIP-618%3A+Exactly-Once+Support+for+Source+Connectors)
to the release plan? It's been blocked on review for almost a year now. If
we can't make time to review it before the 3.3.0 release I'll probably
close my PRs and abandon the feature.

Best,

Chris

On Wed, May 11, 2022 at 4:44 PM José Armando García Sancio
 wrote:

> Great.
>
> I went ahead and created a release page for 3.3.0:
> https://cwiki.apache.org/confluence/x/-xahD
>
> The planned KIP content is based on the list of KIPs targeting 3.3.0
> in https://cwiki.apache.org/confluence/x/4QwIAw. Please take a look at
> the list and let me know if I missed your KIP.
>
> The 3.3.0 release plan page also enumerates the potential release
> dates. Here is a summary of them:
> 1. KIP Freeze June 15th, 2022
> 2. Feature Freeze July 6th, 2022
> 3. Code Freeze July 20th, 2022
>
> Thanks and let me know what you think,
> -José
>

Re: Request to be added to Kafka Jira project

[Mickael Maison]

Hi,

Welcome and thanks for your interest in Kafka!
What is your JIRA id?

Thanks,
Mickael

On Wed, May 11, 2022 at 10:45 PM Lim Qingwei  wrote:
>
> Hi Dev at kafka,
>
> I wish to try to contribute to the project, can I be added to Kafka Jira
> project so that I can pick up ticket?
>
> I am following this guide: https://kafka.apache.org/contributing
>
> I wish to pick up ticket from this list:
> https://issues.apache.org/jira/browse/KAFKA-13882?jql=project%20%3D%20KAFKA%20AND%20labels%20%3D%20newbie%20AND%20status%20%3D%20Open

Request to be added to Kafka Jira project

[Lim Qingwei]

Hi Dev at kafka,

I wish to try to contribute to the project, can I be added to Kafka Jira
project so that I can pick up ticket?

I am following this guide: https://kafka.apache.org/contributing

I wish to pick up ticket from this list:
https://issues.apache.org/jira/browse/KAFKA-13882?jql=project%20%3D%20KAFKA%20AND%20labels%20%3D%20newbie%20AND%20status%20%3D%20Open

Re: [DISCUSS] Apache Kafka 3.3.0 Release

[José Armando García Sancio]

Great.

I went ahead and created a release page for 3.3.0:
https://cwiki.apache.org/confluence/x/-xahD

The planned KIP content is based on the list of KIPs targeting 3.3.0
in https://cwiki.apache.org/confluence/x/4QwIAw. Please take a look at
the list and let me know if I missed your KIP.

The 3.3.0 release plan page also enumerates the potential release
dates. Here is a summary of them:
1. KIP Freeze June 15th, 2022
2. Feature Freeze July 6th, 2022
3. Code Freeze July 20th, 2022

Thanks and let me know what you think,
-José

Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready

[Andrew Otto]

> Stretch clusters are possible with the KRaft architecture. For example,
if you had a cluster with nodes in us-west-1, us-west-2 and us-central-1,
you could put a KRaft controller node in each region. This is similar to
how with ZK you'd put a ZK node in each region.

Ah ha! TIL about process.roles
.
Thank you!



On Wed, May 11, 2022 at 4:15 PM Colin McCabe  wrote:

> On Thu, May 5, 2022, at 15:57, Israel Ekpo wrote:
> > Thanks Colin.
> >
> > I think we may need to update the KIP name to reflect the intent of the
> KIP
> > and convey everything it’s about if all the 3 action items will be
> covered
> > by the same KIP
> >
> > It contains three parts:
> >
> > Marking KRraft as Production Ready
> > Deprecating ZK Mode and
> > Removing Zookeeper Mode
> >
> > Should this be broken up in three separate KIPs since it will be done in
> > multiple releases?
>
> Hi Israel,
>
> I think these three things are closely related, and it makes sense to have
> them in a single KIP. Obviously we cannot remove ZK before deprecating it,
> or deprecate it before marking it as production-ready. So if we had 3 KIPs
> the discussion would get quite confusing since we'd constantly be
> referencing the other discussion thread(s).
>
> >
> > The current name of the KIP only conveys the first item and not the next
> > two
> >
> > It is just a thought. I will like to get your perspective
> >
>
> Names are always a hard part :) I didn't want the name to get too long.
> Maybe if others want to see a longer name I can change it -- I'm open.
>
> best,
> Colin
>
> >
> >
> > On Thu, May 5, 2022 at 1:19 PM Colin McCabe  wrote:
> >
> >> Hi all,
> >>
> >> Thanks for the comments. I agree that we should split out declaring
> KRaft
> >> going production for new clusters from deprecating ZK. We can do the
> former
> >> in the next release, 3.3, and the latter in the release after that, 3.4.
> >>
> >> I also talked offline with some of the people working on upgrade from ZK
> >> and it seems like 3.4 is a more realistic target for that work. Partly
> this
> >> is because 3.3 will be a bit earlier than I originally thought (for some
> >> reason I thought it might be October, but Ismael pointed out it's
> planned
> >> for August)
> >>
> >> I also agree that it will probably be useful to have a 3.5 release
> >> following the 3.4 one, which will also support ZK. Hopefully we will not
> >> need a 3.6, but we don't have to decide that now.
> >>
> >> I added a timeline section to the KIP to make this all clearer. To be
> >> clear, it is a preliminary timeline, which may change. It's difficult to
> >> fully plan out the next 1.5 years of Apache Kafka releases right now --
> and
> >> obviously, there are things which may come up to change our plans.
> However,
> >> I think it is still helpful to have the section to give us a feeling for
> >> the general roadmap.
> >>
> >> When you read the KIP, please consider it all speculative except for the
> >> three proposed changes at the top:
> >>
> >> 1. Mark KRaft as production-ready for new clusters in the upcoming Kafka
> >> 3.3 release.
> >> 2. Deprecate ZooKeeper mode in the upcoming Kafka 3.4 release
> >> 3. Plan to remove ZooKeeper mode entirely in Kafka 4.0.
> >>
> >> best,
> >> Colin
> >>
> >>
> >> On Wed, May 4, 2022, at 19:31, Ismael Juma wrote:
> >> > Yes, all features supported by zk mode will be available in kraft
> mode in
> >> > the 3.x series.
> >> >
> >> > Ismael
> >> >
> >> > On Wed, May 4, 2022, 5:28 PM Israel Ekpo 
> wrote:
> >> >
> >> >> Ismael,
> >> >>
> >> >> I like the timeline. However, does this or will this also account for
> >> >> features  users rely on today in Zookeeper mode being available when
> >> >> Zookeeper is dropped?
> >> >>
> >> >> That’s my main concern
> >> >>
> >> >> On Wed, May 4, 2022 at 8:12 PM Ismael Juma 
> wrote:
> >> >>
> >> >> > Hi Colin,
> >> >> >
> >> >> > Thanks for the KIP, this is exciting. Trying to balance progress
> and
> >> >> > compatibility, how about the following?
> >> >> >
> >> >> > 1. 3.3 (~August 2022): kraft is production ready for new clusters
> >> >> > 2. 3.4 (~December 2022/January 2023): migration from zk to kraft is
> >> >> > production ready and zk mode is deprecated
> >> >> > 3. 3.5 (~April 2023): buffer release
> >> >> > 4. 4.0 (~August 2023): kraft mode is on by default and zk mode is
> >> removed
> >> >> >
> >> >> > This would mean about 1 year from kraft being production ready to
> zk
> >> >> > removal and 8 months from zk deprecation to zk removal.
> >> >> >
> >> >> > If necessary (due to important bugs or security issues), we can do
> a
> >> >> couple
> >> >> > of additional bug fix releases in the 3.5 series after 4.0 is
> >> released.
> >> >> >
> >> >> > Thoughts?
> >> >> >
> >> >> > Ismael
> >> >> >
> >> >> > On Tue, May 3, 2022, 6:03 PM Colin McCabe 
> wrote:
> >> >> >
> >> >> > > Hi all,
> >> >> > >
> >> >> > > I've written a KIP for marking KRaft 

Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready

[Colin McCabe]

On Thu, May 5, 2022, at 15:57, Israel Ekpo wrote:
> Thanks Colin.
>
> I think we may need to update the KIP name to reflect the intent of the KIP
> and convey everything it’s about if all the 3 action items will be covered
> by the same KIP
>
> It contains three parts:
>
> Marking KRraft as Production Ready
> Deprecating ZK Mode and
> Removing Zookeeper Mode
>
> Should this be broken up in three separate KIPs since it will be done in
> multiple releases?

Hi Israel,

I think these three things are closely related, and it makes sense to have them 
in a single KIP. Obviously we cannot remove ZK before deprecating it, or 
deprecate it before marking it as production-ready. So if we had 3 KIPs the 
discussion would get quite confusing since we'd constantly be referencing the 
other discussion thread(s).

>
> The current name of the KIP only conveys the first item and not the next
> two
>
> It is just a thought. I will like to get your perspective
>

Names are always a hard part :) I didn't want the name to get too long. Maybe 
if others want to see a longer name I can change it -- I'm open.

best,
Colin

>
>
> On Thu, May 5, 2022 at 1:19 PM Colin McCabe  wrote:
>
>> Hi all,
>>
>> Thanks for the comments. I agree that we should split out declaring KRaft
>> going production for new clusters from deprecating ZK. We can do the former
>> in the next release, 3.3, and the latter in the release after that, 3.4.
>>
>> I also talked offline with some of the people working on upgrade from ZK
>> and it seems like 3.4 is a more realistic target for that work. Partly this
>> is because 3.3 will be a bit earlier than I originally thought (for some
>> reason I thought it might be October, but Ismael pointed out it's planned
>> for August)
>>
>> I also agree that it will probably be useful to have a 3.5 release
>> following the 3.4 one, which will also support ZK. Hopefully we will not
>> need a 3.6, but we don't have to decide that now.
>>
>> I added a timeline section to the KIP to make this all clearer. To be
>> clear, it is a preliminary timeline, which may change. It's difficult to
>> fully plan out the next 1.5 years of Apache Kafka releases right now -- and
>> obviously, there are things which may come up to change our plans. However,
>> I think it is still helpful to have the section to give us a feeling for
>> the general roadmap.
>>
>> When you read the KIP, please consider it all speculative except for the
>> three proposed changes at the top:
>>
>> 1. Mark KRaft as production-ready for new clusters in the upcoming Kafka
>> 3.3 release.
>> 2. Deprecate ZooKeeper mode in the upcoming Kafka 3.4 release
>> 3. Plan to remove ZooKeeper mode entirely in Kafka 4.0.
>>
>> best,
>> Colin
>>
>>
>> On Wed, May 4, 2022, at 19:31, Ismael Juma wrote:
>> > Yes, all features supported by zk mode will be available in kraft mode in
>> > the 3.x series.
>> >
>> > Ismael
>> >
>> > On Wed, May 4, 2022, 5:28 PM Israel Ekpo  wrote:
>> >
>> >> Ismael,
>> >>
>> >> I like the timeline. However, does this or will this also account for
>> >> features  users rely on today in Zookeeper mode being available when
>> >> Zookeeper is dropped?
>> >>
>> >> That’s my main concern
>> >>
>> >> On Wed, May 4, 2022 at 8:12 PM Ismael Juma  wrote:
>> >>
>> >> > Hi Colin,
>> >> >
>> >> > Thanks for the KIP, this is exciting. Trying to balance progress and
>> >> > compatibility, how about the following?
>> >> >
>> >> > 1. 3.3 (~August 2022): kraft is production ready for new clusters
>> >> > 2. 3.4 (~December 2022/January 2023): migration from zk to kraft is
>> >> > production ready and zk mode is deprecated
>> >> > 3. 3.5 (~April 2023): buffer release
>> >> > 4. 4.0 (~August 2023): kraft mode is on by default and zk mode is
>> removed
>> >> >
>> >> > This would mean about 1 year from kraft being production ready to zk
>> >> > removal and 8 months from zk deprecation to zk removal.
>> >> >
>> >> > If necessary (due to important bugs or security issues), we can do a
>> >> couple
>> >> > of additional bug fix releases in the 3.5 series after 4.0 is
>> released.
>> >> >
>> >> > Thoughts?
>> >> >
>> >> > Ismael
>> >> >
>> >> > On Tue, May 3, 2022, 6:03 PM Colin McCabe  wrote:
>> >> >
>> >> > > Hi all,
>> >> > >
>> >> > > I've written a KIP for marking KRaft as production ready. Please
>> take a
>> >> > > look if you have a chance:
>> >> > >
>> >> > > https://cwiki.apache.org/confluence/x/8xKhD
>> >> > >
>> >> > > thanks,
>> >> > > Colin
>> >> > >
>> >> >
>> >> --
>> >> Israel Ekpo
>> >> Lead Instructor, IzzyAcademy.com
>> >> https://www.youtube.com/c/izzyacademy
>> >> https://izzyacademy.com/
>> >>
>>
> -- 
> Israel Ekpo
> Lead Instructor, IzzyAcademy.com
> https://www.youtube.com/c/izzyacademy
> https://izzyacademy.com/

Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready

[Colin McCabe]

Hi Andrew,

Stretch clusters are possible with the KRaft architecture. For example, if you 
had a cluster with nodes in us-west-1, us-west-2 and us-central-1, you could 
put a KRaft controller node in each region. This is similar to how with ZK 
you'd put a ZK node in each region.

best,
Colin


On Mon, May 9, 2022, at 05:56, Andrew Otto wrote:
>> Deprecating ZK Mode and
>> Removing Zookeeper Mode
>
> I'm excited about KRaft, but quick Q.  I'm researching Kafka 'stretch'
> cluster deployments, and as far as I can tell stretch clusters require
> Zookeeper to function properly, is this correct?  If so, we might want to
> solve that before Deprecating ZK Mode.
>
> Thanks!
>
>
>
> On Thu, May 5, 2022 at 6:57 PM Israel Ekpo  wrote:
>
>> Thanks Colin.
>>
>> I think we may need to update the KIP name to reflect the intent of the KIP
>> and convey everything it’s about if all the 3 action items will be covered
>> by the same KIP
>>
>> It contains three parts:
>>
>> Marking KRraft as Production Ready
>> Deprecating ZK Mode and
>> Removing Zookeeper Mode
>>
>> Should this be broken up in three separate KIPs since it will be done in
>> multiple releases?
>>
>> The current name of the KIP only conveys the first item and not the next
>> two
>>
>> It is just a thought. I will like to get your perspective
>>
>>
>>
>> On Thu, May 5, 2022 at 1:19 PM Colin McCabe  wrote:
>>
>> > Hi all,
>> >
>> > Thanks for the comments. I agree that we should split out declaring KRaft
>> > going production for new clusters from deprecating ZK. We can do the
>> former
>> > in the next release, 3.3, and the latter in the release after that, 3.4.
>> >
>> > I also talked offline with some of the people working on upgrade from ZK
>> > and it seems like 3.4 is a more realistic target for that work. Partly
>> this
>> > is because 3.3 will be a bit earlier than I originally thought (for some
>> > reason I thought it might be October, but Ismael pointed out it's planned
>> > for August)
>> >
>> > I also agree that it will probably be useful to have a 3.5 release
>> > following the 3.4 one, which will also support ZK. Hopefully we will not
>> > need a 3.6, but we don't have to decide that now.
>> >
>> > I added a timeline section to the KIP to make this all clearer. To be
>> > clear, it is a preliminary timeline, which may change. It's difficult to
>> > fully plan out the next 1.5 years of Apache Kafka releases right now --
>> and
>> > obviously, there are things which may come up to change our plans.
>> However,
>> > I think it is still helpful to have the section to give us a feeling for
>> > the general roadmap.
>> >
>> > When you read the KIP, please consider it all speculative except for the
>> > three proposed changes at the top:
>> >
>> > 1. Mark KRaft as production-ready for new clusters in the upcoming Kafka
>> > 3.3 release.
>> > 2. Deprecate ZooKeeper mode in the upcoming Kafka 3.4 release
>> > 3. Plan to remove ZooKeeper mode entirely in Kafka 4.0.
>> >
>> > best,
>> > Colin
>> >
>> >
>> > On Wed, May 4, 2022, at 19:31, Ismael Juma wrote:
>> > > Yes, all features supported by zk mode will be available in kraft mode
>> in
>> > > the 3.x series.
>> > >
>> > > Ismael
>> > >
>> > > On Wed, May 4, 2022, 5:28 PM Israel Ekpo  wrote:
>> > >
>> > >> Ismael,
>> > >>
>> > >> I like the timeline. However, does this or will this also account for
>> > >> features  users rely on today in Zookeeper mode being available when
>> > >> Zookeeper is dropped?
>> > >>
>> > >> That’s my main concern
>> > >>
>> > >> On Wed, May 4, 2022 at 8:12 PM Ismael Juma  wrote:
>> > >>
>> > >> > Hi Colin,
>> > >> >
>> > >> > Thanks for the KIP, this is exciting. Trying to balance progress and
>> > >> > compatibility, how about the following?
>> > >> >
>> > >> > 1. 3.3 (~August 2022): kraft is production ready for new clusters
>> > >> > 2. 3.4 (~December 2022/January 2023): migration from zk to kraft is
>> > >> > production ready and zk mode is deprecated
>> > >> > 3. 3.5 (~April 2023): buffer release
>> > >> > 4. 4.0 (~August 2023): kraft mode is on by default and zk mode is
>> > removed
>> > >> >
>> > >> > This would mean about 1 year from kraft being production ready to zk
>> > >> > removal and 8 months from zk deprecation to zk removal.
>> > >> >
>> > >> > If necessary (due to important bugs or security issues), we can do a
>> > >> couple
>> > >> > of additional bug fix releases in the 3.5 series after 4.0 is
>> > released.
>> > >> >
>> > >> > Thoughts?
>> > >> >
>> > >> > Ismael
>> > >> >
>> > >> > On Tue, May 3, 2022, 6:03 PM Colin McCabe 
>> wrote:
>> > >> >
>> > >> > > Hi all,
>> > >> > >
>> > >> > > I've written a KIP for marking KRaft as production ready. Please
>> > take a
>> > >> > > look if you have a chance:
>> > >> > >
>> > >> > > https://cwiki.apache.org/confluence/x/8xKhD
>> > >> > >
>> > >> > > thanks,
>> > >> > > Colin
>> > >> > >
>> > >> >
>> > >> --
>> > >> Israel Ekpo
>> > >> Lead Instructor, IzzyAcademy.com
>> > >> 

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

[Colin McCabe]

Hi Omnia Ibrahim,

I'm sorry, but I am -1 on adding competing Admin interfaces. This would create 
confusion and a heavier maintenance burden for the project.

Since the org.apache.kafka.clients.admin.Admin interface is a Java interface, 
any third-party software that wants to insert its own implementation of the 
interface can do so already.

A KIP to make the Admin class used pluggable for MM2 would be reasonable. 
Adding a competing admin API is not.

It's true that there are many Admin methods, but you do not need to implement 
all of them -- just the ones that MirrorMaker uses. The other ones can throw a 
NotImplementedException or similar.

> The current approach also assumes that the user running MM2 has the Admin 
> right to
> create/update topics, which is only valid if the user who runs MM2 also 
> manages both
> source and destination clusters.

Maybe I'm misunderstanding the use-case you're describing here. But it seems to 
me that if you create a proxy that has the ability to do any admin operation, 
and give MM2 access to that proxy, the security model is the same as just 
giving MM2 admin access. (Or it may be worse if the sysadmin doesn't know what 
this proxy is doing, and doesn't lock it down...)

best,
Colin


On Mon, May 9, 2022, at 13:21, Omnia Ibrahim wrote:
> Hi, I gave the KIP another look after talking to some people at the Kafka
> Summit in London. And I would like to clear up the motivation of this KIP.
>
>
> At the moment, MM2 has some opinionated decisions that are creating issues
> for teams that use IaC, federated solutions or have a capacity/budget
> planning system for Kafka destination clusters. To explain it better, let's
> assume we have MM2 with the following configurations to highlight these
> problems.
>
> ```
>
> topics = .*
>
> refresh.topics.enabled = true
>
> sync.topic.configs.enabled = true
>
> sync.topic.acls.enabled = true
>
> // Maybe in futrue we can have sync.group.acls.enabled = true
>
> ```
>
>
> These configurations allow us to run MM2 with the value of its full
> features. However, there are two main concerns when we run on a scale with
> these configs:
>
> 1. *Capacity/Budgeting Planning:*
>
> Functionality or features that impact capacity planning using MM2 are:
>
>1. MM2 automatically creates topics (breaking the rule of
>`auto.create.topics.enable=false`) and creates topic partitions on
>destination clusters if the number of partitions increases on the source.
>In the previous example, this functionality will apply to any topic that
>matches the regex of the `topics` config.
>2. Sync topic configs include configurations that impact capacity like `
>retention.ms` and `retention.bytes`.
>
> These 2 points lead to adding new untracked capacity to destination
> clusters without a way to count for them up-front or safeguard the cluster.
> The team that runs the cluster will only see the capacity issue when their
> disk usage hits the threshold for their alerts. The desk capacity issue can
> be avoided if MM2 is flexible enough to
>
>- have a way for teams that run their ecosystem to have MM2 behave
>within their system.
>- disable the auto-creation and avoid syncing configs that impact
>capacity
>
>
> 2. *Provisioning conflict:*
>
> In the previous MM2 configurations; we ended up with conflict as MM2 used
> `AdminClient` directly to perform the following functionality
>
>-  Create a Kafka topic (no way to disable this at the moment)
>-  Add new Kafka partitions (no way to disable this at the moment)
>-  Sync Kafka Topic configurations (can be disabled, but then this
>reduces the value of MM2 potential for users)
>-  Sync Kafka topic's ACLs (can be disabled, but this reduces the users'
>value). Disabling this feature also means that users must ensure they have
>the right ACLs to the mirrored topics on the destination cluster before
>switching their consumers, especially when MM2 is used for disaster
>recovery. It may lead to extra downtime for them.
>
>
> All these functionalities are using AdminClient; which causes an issue with
> teams that
>
>- Manage their Kafka resources using tools like Strimizi or custom
>federated solutions. For example, Strimizi's UserOperator doesn't sync the
>topic ACLs when MM2 is enabled. Strimzi documentation mentions that users
>must to disable MM2 `sync.topic.acls.enabled` if they use `UserOperator`.
>On the other hand, Strimizi's TopicOperator doesn't have the same issue
>because it has a bi-directional reconciliation process that watches the
>topics state on the Kafka cluster and updates KafkaTopic resources for
>Strimzi. This design works fine with Kafka MM2 for Topics but not for
>syncing ACLs. Strimizi TopicOperator also doesn't have a way to stop
>syncing config that impact capacity for example retention configs.
>
>
>- Teams that run MM2 but don't own the destination cluster. In this
>

Re: [DISCUSS] KIP-836: Addition of Information in DescribeQuorumResponse about Voter Lag

[Colin McCabe]

Thanks, Niket. I also agree with Jason that this is a public API despite the 
lack of command-line tool, so we do indeed need a KIP. :)

One minor point: I suspect that whatever we end up naming the additional fields 
here, should also be the name of the metrics in KIP-835. So if we go with a 
metric named "last-applied-offset" we'd want a lastAppliedOffset field here, 
and so on.

I also wonder if it makes sense for us to report the timestamp of the latest 
batch that has been fetched (and not necessarily applied) rather than the wall 
clock time at which the leader made the latest fetch. If we take both 
timestamps directly from the metadata log, we know they'll be comparable even 
in the presence of clock skew. And we know because of KIP-835 that the metadata 
log won't go quiet for prolonged periods.

best,
Colin


On Tue, May 10, 2022, at 13:30, Niket Goel wrote:
>> @Niket does it make sense to add the Admin API to this KIP?
>
> Thanks Deng for pointing this out. I agree with Jason's suggestion. I will
> go ahead and add the admin API to this KIP.
>
> - Niket
>
> On Tue, May 10, 2022 at 11:44 AM Jason Gustafson 
> wrote:
>
>> > Hello Niket, currently DescribeQuorumResponse is not a public API, we
>> don’t have a Admin api or shell script to get DescribeQuorumResponse, so
>> it’s unnecessary to submit a KIP to change it, you can just submit a PR to
>> accomplish this.
>>
>> Hey Ziming, I think it is public. It was documented in KIP-595 and we have
>> implemented the API on the server. However, it looks like I never added
>> the Admin API (even though it is assumed by the `kafka-metadata-quorum.sh`
>> tool). @Niket does it make sense to add the Admin API to this KIP?
>>
>> Best,
>> Jason
>>
>> On Mon, May 9, 2022 at 8:09 PM deng ziming 
>> wrote:
>>
>> > Hello Niket, currently DescribeQuorumResponse is not a public API, we
>> > don’t have a Admin api or shell script to get DescribeQuorumResponse, so
>> > it’s unnecessary to submit a KIP to change it, you can just submit a PR
>> to
>> > accomplish this.
>> >
>> > --
>> > Thanks
>> > Ziming
>> >
>> > > On May 10, 2022, at 1:33 AM, Niket Goel 
>> > wrote:
>> > >
>> > > Hi all,
>> > >
>> > > I created a KIP to add some more information to
>> > `DesscribeQuorumResponse` to enable ascertaining voter lag in the quorum
>> a
>> > little better.
>> > > Please see KIP --
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-836%3A+Additional+Information+in+DescribeQuorumResponse+about+Voter+Lag
>> > >
>> > > Thanks for your feedback,
>> > > Niket Goel
>> >
>> >
>>
>
>
> -- 
> - Niket

Re: [DISCUSS] KIP-835: Monitor KRaft Controller Quorum Health

[Colin McCabe]

Hi José,

Thanks for the KIP! I think this will be a nice improvement.

I had the same question as Luke and Jason: what's the default here for the 
NoOpRecord time? :) We should add a value here even if we think we'll adjust it 
later, just to give a feeling for how much traffic this would create. Perhaps 
500 ms?

Also how about naming the time configuration something like 
"metadata.max.idle.interval.ms", to make it clear that this is the maximum time 
we can go between writes to the metadata log. I don't get that meaning out of 
"monitor interval"...

> Broker
> kafka.server:type=broker-metadata-metrics,name=last-committed-offset 
> kafka.server:type=broker-metadata-metrics,name=last-committed-timestamp 
> kafka.server:type=broker-metadata-metrics,name=last-committed-lag-ms

I'd suggest renaming this as "last-applied-offset-*" to make it clear that the 
offset we're measuring is the last one that the broker applied. The last 
committed offset is something else, a more log-layer-centric concept.

(It would also be good to have a separate metric which purely reflected the 
last metadata offset we've fetched and not applied, so we can see if the delta 
between that and last-applied-offset increases...)

regards,
Colin


On Wed, May 11, 2022, at 12:27, David Arthur wrote:
> José, thanks for the KIP! I think this is a good approach for proving
> the liveness of the quorum when metadata is not changing.
>
> 1. Based on the config name "metadata.monitor.write.interval.ms" I'm
> guessing the intention is to have a regularly scheduled write. If the
> quorum is busy with lots of the writes, we wouldn't need this NoopRecord
> right? Maybe we can instead write a NoopRecord only after some amount of
> idle time.
>
> 2. A small naming suggestion, what about "NoOpRecord"?
>
> 3. Typo in one of the metric names: "MetadataWriteOffses"
>
> 4. We should consider omitting these records from the log dump tool, or at
> least adding an option to skip over them.
>
> 5. If (somehow) one of these NoopRecord appeared in the snapshot, it sounds
> like the broker/controller would skip over them. We should specify this
> behavior in the KIP
>
> Cheers,
> David
>
> On Wed, May 11, 2022 at 2:37 AM Luke Chen  wrote:
>
>> Hi José,
>>
>> Thanks for the KIP!
>>
>> Some questions:
>> 1. Jason has asked but you didn't answer: What is the default value for `
>> metadata.monitor.write.interval.ms`?
>> 2. The `noopRecord` API key is `TBD`. Why can't we put the "currently used
>> API Key nums + 1" into it? Any concern?
>> 3. typo: name=MetadataWriteOffse[t]s
>> 4. I don't understand the difference between MetadataLastCommittedOffset
>> and MetadataWriteOffsets metrics. I think the 2 values will always be
>> identical, unless the controller metadata write failed, is that correct?
>>
>>
>> Thank you.
>> Luke
>>
>> On Wed, May 11, 2022 at 5:58 AM José Armando García Sancio
>>  wrote:
>>
>> > Thanks for your feedback Jason, much appreciated.
>> >
>> > Here are the changes to the KIP:
>> >
>> >
>> https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=211883219=5=4
>> >
>> > On Tue, May 10, 2022 at 1:34 PM Jason Gustafson
>> >  wrote:
>> > > The approach sounds reasonable. By the way, I think one of the gaps we
>> > have
>> > > today is when the leader gets partitioned from the remaining voters. I
>> > > believe it continues acting as a leader indefinitely. I was considering
>> > > whether this periodic write can address the issue. Basically it can be
>> > used
>> > > to force a leader to prove it is still the leader by committing some
>> > data.
>> > > Say, for example, that the leader fails to commit the record after the
>> > > fetch timeout expires, then perhaps it could start a new election. What
>> > do
>> > > you think?
>> >
>> > We have an issue for this at
>> > https://issues.apache.org/jira/browse/KAFKA-13621. I updated the issue
>> > with your feedback and included some of my thoughts. Do you mind if we
>> > move this conversation to that issue?
>> >
>> > > A couple additional questions:
>> > >
>> > > - What is the default value for `metadata.monitor.write.interval.ms`?
>> > Also,
>> > > I'm wondering if `controller` would be a more suitable prefix?
>> >
>> > Yeah. I am not sure. Looking at the current configuration we have both
>> > prefixes. For example, with the `controller` prefix we have
>> > `controller.quorum.voters`, `controller.listener.names`,
>> > `controller.quota.window.num`, etc. For the `metadata` prefix we have
>> > `metadata.log.dir`, `metadata.log.*` and `metadat.max.retention.ms`,
>> > etc.
>> > I get the impression that we use `metadata` for things that are kinda
>> > log/disk related and `controller` for things that are not. I am
>> > thinking that the `metadata` prefix is more consistent with the
>> > current situation. What do you think Jason?
>> >
>> > > - Could we avoid letting BrokerMetadataPublisher escape into the metric
>> > > name? Letting the classnames leak into the metrics tends to 

Re: [DISCUSS] Apache Kafka 3.3.0 Release

[Bill Bejeck]

Thanks for volunteering to run the release José!

+1

-Bill

On Wed, May 11, 2022 at 3:26 AM Bruno Cadonna  wrote:

> Thanks José!
>
> +1
>
> Best,
> Bruno
>
> On 11.05.22 04:38, John Roesler wrote:
> > +1 from me! Thanks for volunteering, José.
> > -John
> >
> > On Tue, May 10, 2022, at 17:53, José Armando García Sancio wrote:
> >> Hi all,
> >>
> >> I would like to volunteer for the release of Apache Kafka 3.3.0. If
> >> people agree, I'll start working on the release plan and update this
> >> thread.
> >>
> >> Thanks,
> >> -José
>

Re: [DISCUSS] KIP-835: Monitor KRaft Controller Quorum Health

[David Arthur]

José, thanks for the KIP! I think this is a good approach for proving
the liveness of the quorum when metadata is not changing.

1. Based on the config name "metadata.monitor.write.interval.ms" I'm
guessing the intention is to have a regularly scheduled write. If the
quorum is busy with lots of the writes, we wouldn't need this NoopRecord
right? Maybe we can instead write a NoopRecord only after some amount of
idle time.

2. A small naming suggestion, what about "NoOpRecord"?

3. Typo in one of the metric names: "MetadataWriteOffses"

4. We should consider omitting these records from the log dump tool, or at
least adding an option to skip over them.

5. If (somehow) one of these NoopRecord appeared in the snapshot, it sounds
like the broker/controller would skip over them. We should specify this
behavior in the KIP

Cheers,
David

On Wed, May 11, 2022 at 2:37 AM Luke Chen  wrote:

> Hi José,
>
> Thanks for the KIP!
>
> Some questions:
> 1. Jason has asked but you didn't answer: What is the default value for `
> metadata.monitor.write.interval.ms`?
> 2. The `noopRecord` API key is `TBD`. Why can't we put the "currently used
> API Key nums + 1" into it? Any concern?
> 3. typo: name=MetadataWriteOffse[t]s
> 4. I don't understand the difference between MetadataLastCommittedOffset
> and MetadataWriteOffsets metrics. I think the 2 values will always be
> identical, unless the controller metadata write failed, is that correct?
>
>
> Thank you.
> Luke
>
> On Wed, May 11, 2022 at 5:58 AM José Armando García Sancio
>  wrote:
>
> > Thanks for your feedback Jason, much appreciated.
> >
> > Here are the changes to the KIP:
> >
> >
> https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=211883219=5=4
> >
> > On Tue, May 10, 2022 at 1:34 PM Jason Gustafson
> >  wrote:
> > > The approach sounds reasonable. By the way, I think one of the gaps we
> > have
> > > today is when the leader gets partitioned from the remaining voters. I
> > > believe it continues acting as a leader indefinitely. I was considering
> > > whether this periodic write can address the issue. Basically it can be
> > used
> > > to force a leader to prove it is still the leader by committing some
> > data.
> > > Say, for example, that the leader fails to commit the record after the
> > > fetch timeout expires, then perhaps it could start a new election. What
> > do
> > > you think?
> >
> > We have an issue for this at
> > https://issues.apache.org/jira/browse/KAFKA-13621. I updated the issue
> > with your feedback and included some of my thoughts. Do you mind if we
> > move this conversation to that issue?
> >
> > > A couple additional questions:
> > >
> > > - What is the default value for `metadata.monitor.write.interval.ms`?
> > Also,
> > > I'm wondering if `controller` would be a more suitable prefix?
> >
> > Yeah. I am not sure. Looking at the current configuration we have both
> > prefixes. For example, with the `controller` prefix we have
> > `controller.quorum.voters`, `controller.listener.names`,
> > `controller.quota.window.num`, etc. For the `metadata` prefix we have
> > `metadata.log.dir`, `metadata.log.*` and `metadat.max.retention.ms`,
> > etc.
> > I get the impression that we use `metadata` for things that are kinda
> > log/disk related and `controller` for things that are not. I am
> > thinking that the `metadata` prefix is more consistent with the
> > current situation. What do you think Jason?
> >
> > > - Could we avoid letting BrokerMetadataPublisher escape into the metric
> > > name? Letting the classnames leak into the metrics tends to cause
> > > compatibility issues over time.
> >
> > Good point. For Raft we use `kafka.server:type=raft-metrics,name=...`.
> > I'll change it to
> > `kafka.server:type=broker-metadata-metrics,name=...`.
> >
> > Thanks,
> > -José
> >
>


-- 
David Arthur

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.2 #49

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #923

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-834: Pause / Resume KafkaStreams Topologies

[Leah Thomas]

Thanks Jim, great discussion. +1 from me (non-binding)

Cheers,
Leah

On Wed, May 11, 2022 at 10:14 AM Bill Bejeck  wrote:

> Thanks for the KIP!
>
> +1 (binding)
>
> -Bill
>
> On Wed, May 11, 2022 at 9:36 AM Luke Chen  wrote:
>
> > Hi Jim,
> >
> > I'm +1. (please add some note in KIP about the stream resetting tool
> can't
> > be used in paused state)
> > Thanks for the KIP!
> >
> > Luke
> >
> > On Wed, May 11, 2022 at 9:09 AM Guozhang Wang 
> wrote:
> >
> > > Thanks Jim. +1 from me.
> > >
> > > On Tue, May 10, 2022 at 4:51 PM Matthias J. Sax 
> > wrote:
> > >
> > > > I had one minor question on the discuss thread. It's mainly about
> > > > clarifying and document the user contract. I am fine either way.
> > > >
> > > > +1 (binding)
> > > >
> > > >
> > > > -Matthias
> > > >
> > > > On 5/10/22 12:32 PM, Sophie Blee-Goldman wrote:
> > > > > Thanks for the KIP! +1 (binding)
> > > > >
> > > > > On Tue, May 10, 2022, 12:24 PM Bruno Cadonna 
> > > wrote:
> > > > >
> > > > >> Thanks Jim,
> > > > >>
> > > > >> +1 (binding)
> > > > >>
> > > > >> Best,
> > > > >> Bruno
> > > > >>
> > > > >> On 10.05.22 21:19, John Roesler wrote:
> > > > >>> Thanks Jim,
> > > > >>>
> > > > >>> I’m +1 (binding)
> > > > >>>
> > > > >>> -John
> > > > >>>
> > > > >>> On Tue, May 10, 2022, at 14:05, Jim Hughes wrote:
> > > >  Hi all,
> > > > 
> > > >  I'm asking for a vote on KIP-834:
> > > > 
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211882832
> > > > 
> > > >  Thanks in advance!
> > > > 
> > > >  Jim
> > > > >>
> > > > >
> > > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>

[jira] [Resolved] (KAFKA-13893) Add BrokerApiVersions Api in AdminClient

[dengziming (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

dengziming resolved KAFKA-13893.

Resolution: Invalid

> Add BrokerApiVersions Api in AdminClient
> 
>
> Key: KAFKA-13893
> URL: https://issues.apache.org/jira/browse/KAFKA-13893
> Project: Kafka
>  Issue Type: Task
>Reporter: dengziming
>Assignee: dengziming
>Priority: Major
>  Labels: kip-required
>
> We already have a BrokerApiVersionsCommand to get broker api version, yet we 
> lack similar api in AdminClient.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] KIP-834: Pause / Resume KafkaStreams Topologies

[Jim Hughes]

Hi Luke, John,

Thanks for bringing up this and also sorting it out!

I have added a note to the KIP.

Thanks,

Jim

On Wed, May 11, 2022 at 9:34 AM Luke Chen  wrote:

> Thanks John!
> It makes sense.
> I have no other questions as long as it is documented in the KIP.
>
> Thank you.
> Luke
>
> On Wed, May 11, 2022 at 9:15 PM John Roesler  wrote:
>
> > Hi Luke,
> >
> > It’s not my KIP, but my two cents is that users should not run the reset
> > tool while the application is paused.
> >
> > The reset tool should only be run while the whole app is shut down
> because
> > it messes with a lot of internal state bits without synchronization.
> > Leaving the app running (even while pausing processing) will result in
> the
> > app being in an undefined state, as the members and the tool will be
> > simultaneously trying to set the committed offsets to different values,
> etc.
> >
> > Jim, can you also make it a point to document this? As Luke points out,
> it
> > might be a natural thing to want to do.
> >
> > Thanks,
> > John
> >
> > On Wed, May 11, 2022, at 02:19, Luke Chen wrote:
> > > Hi Jim,
> > >
> > > Thanks for the KIP. Overall LGTM!
> > >
> > > One late question:
> > > Could we run the stream resetter tool (i.e.
> > > kafka-streams-application-reset.sh) during pause state?
> > > I can imagine there's a use case that after pausing for a while, user
> > just
> > > want to continue with the latest offset, and skipping the intermediate
> > > records.
> > >
> > > Thank you.
> > > Luke
> > >
> > > On Wed, May 11, 2022 at 10:12 AM Jim Hughes
>  > >
> > > wrote:
> > >
> > >> Hi Matthias,
> > >>
> > >> I like it.  I've updated the KIP to reflect that detail; I put the
> > details
> > >> in the docs for pause.
> > >>
> > >> Cheers,
> > >>
> > >> Jim
> > >>
> > >> On Tue, May 10, 2022 at 7:51 PM Matthias J. Sax 
> > wrote:
> > >>
> > >> > Thanks for the KIP. Overall LGTM.
> > >> >
> > >> > Can we clarify one question: would it be allowed to call `pause()`
> > >> > before calling `start()`? I don't see any reason why we would need
> to
> > >> > disallow it?
> > >> >
> > >> > It could be helpful to start a KafkaStreams client in paused state
> --
> > >> > otherwise there is a race between calling `start()` and calling
> > >> `pause()`.
> > >> >
> > >> > If we allow it, we should clearly document it.
> > >> >
> > >> >
> > >> > -Matthias
> > >> >
> > >> > On 5/10/22 12:04 PM, Jim Hughes wrote:
> > >> > > Hi Bill, all,
> > >> > >
> > >> > > Thank you.  I've updated the KIP to reflect pausing standby tasks
> as
> > >> > well.
> > >> > > I think all the outstanding points have been addressed and I'm
> > going to
> > >> > > start the vote thread!
> > >> > >
> > >> > > Cheers,
> > >> > >
> > >> > > Jim
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Tue, May 10, 2022 at 2:43 PM Bill Bejeck 
> > wrote:
> > >> > >
> > >> > >> Hi Jim,
> > >> > >>
> > >> > >> After reading the comments on the KIP, I agree that it makes
> sense
> > to
> > >> > pause
> > >> > >> all activities and any changes can be made later on.
> > >> > >>
> > >> > >> Thanks,
> > >> > >> Bill
> > >> > >>
> > >> > >> On Tue, May 10, 2022 at 4:03 AM Bruno Cadonna <
> cado...@apache.org>
> > >> > wrote:
> > >> > >>
> > >> > >>> Hi Jim,
> > >> > >>>
> > >> > >>> Thanks for the KIP!
> > >> > >>>
> > >> > >>> I am fine with the KIP in general.
> > >> > >>>
> > >> > >>> However, I am with Sophie and John to also pause the standbys
> for
> > the
> > >> > >>> reasons they brought up. Is there a specific reason you want to
> > keep
> > >> > >>> standbys going? It feels like premature optimization to me. We
> can
> > >> > still
> > >> > >>> add keeping standby running in a follow up if needed.
> > >> > >>>
> > >> > >>> Best,
> > >> > >>> Bruno
> > >> > >>>
> > >> > >>> On 10.05.22 05:15, Sophie Blee-Goldman wrote:
> > >> >  Thanks Jim, just one note/question on the standby tasks:
> > >> > 
> > >> >  At the minute, my moderately held position is that standby
> tasks
> > >> ought
> > >> > >> to
> > >> > > continue reading and remain caught up.  If standby tasks would
> > run
> > >> > out
> > >> > >>> of
> > >> > > space, there are probably bigger problems.
> > >> > 
> > >> > 
> > >> >  For a single node application, or when the #pause API is
> invoked
> > on
> > >> > all
> > >> >  instances,
> > >> >  then there won't be any further active processing and thus
> > nothing
> > >> to
> > >> > >>> keep
> > >> >  up with,
> > >> >  right? So for that case, it's just a matter of whether any
> > standbys
> > >> > >> that
> > >> >  are lagging
> > >> >  will have the chance to catch up to the (paused) active task
> > state
> > >> > >> before
> > >> >  they stop
> > >> >  as well, in which case having them continue feels fine to me.
> > >> However
> > >> > >>> this
> > >> >  is a
> > >> >  relatively trivial benefit and I would only consider it as a
> > >> deciding
> > >> >  factor when all
> > >> >  

Re: [VOTE] KIP-834: Pause / Resume KafkaStreams Topologies

[Bill Bejeck]

Thanks for the KIP!

+1 (binding)

-Bill

On Wed, May 11, 2022 at 9:36 AM Luke Chen  wrote:

> Hi Jim,
>
> I'm +1. (please add some note in KIP about the stream resetting tool can't
> be used in paused state)
> Thanks for the KIP!
>
> Luke
>
> On Wed, May 11, 2022 at 9:09 AM Guozhang Wang  wrote:
>
> > Thanks Jim. +1 from me.
> >
> > On Tue, May 10, 2022 at 4:51 PM Matthias J. Sax 
> wrote:
> >
> > > I had one minor question on the discuss thread. It's mainly about
> > > clarifying and document the user contract. I am fine either way.
> > >
> > > +1 (binding)
> > >
> > >
> > > -Matthias
> > >
> > > On 5/10/22 12:32 PM, Sophie Blee-Goldman wrote:
> > > > Thanks for the KIP! +1 (binding)
> > > >
> > > > On Tue, May 10, 2022, 12:24 PM Bruno Cadonna 
> > wrote:
> > > >
> > > >> Thanks Jim,
> > > >>
> > > >> +1 (binding)
> > > >>
> > > >> Best,
> > > >> Bruno
> > > >>
> > > >> On 10.05.22 21:19, John Roesler wrote:
> > > >>> Thanks Jim,
> > > >>>
> > > >>> I’m +1 (binding)
> > > >>>
> > > >>> -John
> > > >>>
> > > >>> On Tue, May 10, 2022, at 14:05, Jim Hughes wrote:
> > >  Hi all,
> > > 
> > >  I'm asking for a vote on KIP-834:
> > > 
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211882832
> > > 
> > >  Thanks in advance!
> > > 
> > >  Jim
> > > >>
> > > >
> > >
> >
> >
> > --
> > -- Guozhang
> >
>

[jira] [Created] (KAFKA-13893) Add BrokerApiVersions Api in AdminClient

[dengziming (Jira)]

dengziming created KAFKA-13893:
--

 Summary: Add BrokerApiVersions Api in AdminClient
 Key: KAFKA-13893
 URL: https://issues.apache.org/jira/browse/KAFKA-13893
 Project: Kafka
  Issue Type: Task
Reporter: dengziming
Assignee: dengziming


We already have a BrokerApiVersionsCommand to get broker api version, yet we 
lack similar api in AdminClient.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [VOTE] KIP-834: Pause / Resume KafkaStreams Topologies

[Luke Chen]

Hi Jim,

I'm +1. (please add some note in KIP about the stream resetting tool can't
be used in paused state)
Thanks for the KIP!

Luke

On Wed, May 11, 2022 at 9:09 AM Guozhang Wang  wrote:

> Thanks Jim. +1 from me.
>
> On Tue, May 10, 2022 at 4:51 PM Matthias J. Sax  wrote:
>
> > I had one minor question on the discuss thread. It's mainly about
> > clarifying and document the user contract. I am fine either way.
> >
> > +1 (binding)
> >
> >
> > -Matthias
> >
> > On 5/10/22 12:32 PM, Sophie Blee-Goldman wrote:
> > > Thanks for the KIP! +1 (binding)
> > >
> > > On Tue, May 10, 2022, 12:24 PM Bruno Cadonna 
> wrote:
> > >
> > >> Thanks Jim,
> > >>
> > >> +1 (binding)
> > >>
> > >> Best,
> > >> Bruno
> > >>
> > >> On 10.05.22 21:19, John Roesler wrote:
> > >>> Thanks Jim,
> > >>>
> > >>> I’m +1 (binding)
> > >>>
> > >>> -John
> > >>>
> > >>> On Tue, May 10, 2022, at 14:05, Jim Hughes wrote:
> >  Hi all,
> > 
> >  I'm asking for a vote on KIP-834:
> > 
> > >>
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211882832
> > 
> >  Thanks in advance!
> > 
> >  Jim
> > >>
> > >
> >
>
>
> --
> -- Guozhang
>

Re: [DISCUSS] KIP-834: Pause / Resume KafkaStreams Topologies

[Luke Chen]

Thanks John!
It makes sense.
I have no other questions as long as it is documented in the KIP.

Thank you.
Luke

On Wed, May 11, 2022 at 9:15 PM John Roesler  wrote:

> Hi Luke,
>
> It’s not my KIP, but my two cents is that users should not run the reset
> tool while the application is paused.
>
> The reset tool should only be run while the whole app is shut down because
> it messes with a lot of internal state bits without synchronization.
> Leaving the app running (even while pausing processing) will result in the
> app being in an undefined state, as the members and the tool will be
> simultaneously trying to set the committed offsets to different values, etc.
>
> Jim, can you also make it a point to document this? As Luke points out, it
> might be a natural thing to want to do.
>
> Thanks,
> John
>
> On Wed, May 11, 2022, at 02:19, Luke Chen wrote:
> > Hi Jim,
> >
> > Thanks for the KIP. Overall LGTM!
> >
> > One late question:
> > Could we run the stream resetter tool (i.e.
> > kafka-streams-application-reset.sh) during pause state?
> > I can imagine there's a use case that after pausing for a while, user
> just
> > want to continue with the latest offset, and skipping the intermediate
> > records.
> >
> > Thank you.
> > Luke
> >
> > On Wed, May 11, 2022 at 10:12 AM Jim Hughes  >
> > wrote:
> >
> >> Hi Matthias,
> >>
> >> I like it.  I've updated the KIP to reflect that detail; I put the
> details
> >> in the docs for pause.
> >>
> >> Cheers,
> >>
> >> Jim
> >>
> >> On Tue, May 10, 2022 at 7:51 PM Matthias J. Sax 
> wrote:
> >>
> >> > Thanks for the KIP. Overall LGTM.
> >> >
> >> > Can we clarify one question: would it be allowed to call `pause()`
> >> > before calling `start()`? I don't see any reason why we would need to
> >> > disallow it?
> >> >
> >> > It could be helpful to start a KafkaStreams client in paused state --
> >> > otherwise there is a race between calling `start()` and calling
> >> `pause()`.
> >> >
> >> > If we allow it, we should clearly document it.
> >> >
> >> >
> >> > -Matthias
> >> >
> >> > On 5/10/22 12:04 PM, Jim Hughes wrote:
> >> > > Hi Bill, all,
> >> > >
> >> > > Thank you.  I've updated the KIP to reflect pausing standby tasks as
> >> > well.
> >> > > I think all the outstanding points have been addressed and I'm
> going to
> >> > > start the vote thread!
> >> > >
> >> > > Cheers,
> >> > >
> >> > > Jim
> >> > >
> >> > >
> >> > >
> >> > > On Tue, May 10, 2022 at 2:43 PM Bill Bejeck 
> wrote:
> >> > >
> >> > >> Hi Jim,
> >> > >>
> >> > >> After reading the comments on the KIP, I agree that it makes sense
> to
> >> > pause
> >> > >> all activities and any changes can be made later on.
> >> > >>
> >> > >> Thanks,
> >> > >> Bill
> >> > >>
> >> > >> On Tue, May 10, 2022 at 4:03 AM Bruno Cadonna 
> >> > wrote:
> >> > >>
> >> > >>> Hi Jim,
> >> > >>>
> >> > >>> Thanks for the KIP!
> >> > >>>
> >> > >>> I am fine with the KIP in general.
> >> > >>>
> >> > >>> However, I am with Sophie and John to also pause the standbys for
> the
> >> > >>> reasons they brought up. Is there a specific reason you want to
> keep
> >> > >>> standbys going? It feels like premature optimization to me. We can
> >> > still
> >> > >>> add keeping standby running in a follow up if needed.
> >> > >>>
> >> > >>> Best,
> >> > >>> Bruno
> >> > >>>
> >> > >>> On 10.05.22 05:15, Sophie Blee-Goldman wrote:
> >> >  Thanks Jim, just one note/question on the standby tasks:
> >> > 
> >> >  At the minute, my moderately held position is that standby tasks
> >> ought
> >> > >> to
> >> > > continue reading and remain caught up.  If standby tasks would
> run
> >> > out
> >> > >>> of
> >> > > space, there are probably bigger problems.
> >> > 
> >> > 
> >> >  For a single node application, or when the #pause API is invoked
> on
> >> > all
> >> >  instances,
> >> >  then there won't be any further active processing and thus
> nothing
> >> to
> >> > >>> keep
> >> >  up with,
> >> >  right? So for that case, it's just a matter of whether any
> standbys
> >> > >> that
> >> >  are lagging
> >> >  will have the chance to catch up to the (paused) active task
> state
> >> > >> before
> >> >  they stop
> >> >  as well, in which case having them continue feels fine to me.
> >> However
> >> > >>> this
> >> >  is a
> >> >  relatively trivial benefit and I would only consider it as a
> >> deciding
> >> >  factor when all
> >> >  things are equal otherwise.
> >> > 
> >> >  My concern is the more interesting case: when this feature is
> used
> >> to
> >> > >>> pause
> >> >  only
> >> >  one nodes, or some subset of the overall application. In this
> case,
> >> > >> yes,
> >> >  the standby
> >> >  tasks will indeed fall out of sync. But the only reason I can
> >> imagine
> >> >  someone using
> >> >  the pause feature in such a way is because there is something
> going
> >> > >>> wrong,
> >> >  or about
> >> >  to go 

Re: [HEADS-UP] Modification to KIP Template

[John Roesler]

+1 from me also. This is a great idea. 

Thanks,
John

On Wed, May 11, 2022, at 02:49, Luke Chen wrote:
> Hi Mickael,
>
> +1 to add "test plan" section to KIP template.
> Thanks for the improvement!
>
> Luke
>
> On Tue, May 10, 2022 at 11:21 PM Mickael Maison 
> wrote:
>
>> Hi,
>>
>> I did not see any comments nor concerns so I went ahead and added the
>> Test Plan section to the KIP template.
>>
>> Thanks,
>> Mickael
>>
>> On Fri, Apr 1, 2022 at 5:53 PM Mickael Maison 
>> wrote:
>> >
>> > Hi,
>> >
>> > Unearthing this old thread as today I stumbled on the issue that
>> > Ismael reported. It looks like this was never fixed!
>> >
>> > The "Test Plan" section was only added in the KIP-template page [0]
>> > and not in the actual KIP-template template [1] that is used when
>> > doing `Create -> KIP-Template` or by clicking on `Create KIP` on
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>> >
>> > I think this new section makes sense and it's very easy to add it to
>> > the actual template. Before doing it, I just want to ping the dev list
>> > to see if anybody has suggestions or concerns since this was discussed
>> > many years ago now.
>> >
>> > 0: https://cwiki.apache.org/confluence/display/KAFKA/KIP-Template
>> > 1:
>> https://cwiki.apache.org/confluence/pages/templates2/viewpagetemplate.action?entityId=54329345=KAFKA
>> >
>> > Thanks,
>> > Mickael
>> >
>> > On Fri, May 27, 2016 at 10:55 AM Ismael Juma  wrote:
>> > >
>> > > Hi Gwen,
>> > >
>> > > Thanks for adding the "Test Plans" section. I think it may be worth
>> adding
>> > > a note about performance testing plans too (whenever relevant). By the
>> way,
>> > > even though the following page has the new section, if I use `Create ->
>> > > KIP-Template`, the new section doesn't appear. Do you know why?
>> > >
>> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-Template
>> > >
>> > > Ismael
>> > >
>> > > On Fri, May 27, 2016 at 3:24 AM, Gwen Shapira 
>> wrote:
>> > >
>> > > > Hi Kafka Developers,
>> > > >
>> > > > Just a quick heads-up that I added a new section to the KIP
>> template: "Test
>> > > > Plans".
>> > > > I think its a good habit to think about how a feature will be tested
>> while
>> > > > planning it. I'm talking about high-level notes on system tests, not
>> gritty
>> > > > details.
>> > > >
>> > > > This will apply to new KIPs, not ones in discussion/implementation
>> phases
>> > > > (although if your KIP is under discussion and you want to add test
>> plans,
>> > > > it will be very nice of you).
>> > > >
>> > > > I figured we all agree that thinking a bit about tests is a good
>> idea, so I
>> > > > added it first and started a discussion later. If you strongly
>> object,
>> > > > please respond with strong objections. Wikis are easy to edit :)
>> > > >
>> > > > Gwen
>> > > >
>>

Re: [DISCUSS] KIP-834: Pause / Resume KafkaStreams Topologies

[John Roesler]

Hi Luke,

It’s not my KIP, but my two cents is that users should not run the reset tool 
while the application is paused.

The reset tool should only be run while the whole app is shut down because it 
messes with a lot of internal state bits without synchronization. Leaving the 
app running (even while pausing processing) will result in the app being in an 
undefined state, as the members and the tool will be simultaneously trying to 
set the committed offsets to different values, etc.

Jim, can you also make it a point to document this? As Luke points out, it 
might be a natural thing to want to do.

Thanks,
John

On Wed, May 11, 2022, at 02:19, Luke Chen wrote:
> Hi Jim,
>
> Thanks for the KIP. Overall LGTM!
>
> One late question:
> Could we run the stream resetter tool (i.e.
> kafka-streams-application-reset.sh) during pause state?
> I can imagine there's a use case that after pausing for a while, user just
> want to continue with the latest offset, and skipping the intermediate
> records.
>
> Thank you.
> Luke
>
> On Wed, May 11, 2022 at 10:12 AM Jim Hughes 
> wrote:
>
>> Hi Matthias,
>>
>> I like it.  I've updated the KIP to reflect that detail; I put the details
>> in the docs for pause.
>>
>> Cheers,
>>
>> Jim
>>
>> On Tue, May 10, 2022 at 7:51 PM Matthias J. Sax  wrote:
>>
>> > Thanks for the KIP. Overall LGTM.
>> >
>> > Can we clarify one question: would it be allowed to call `pause()`
>> > before calling `start()`? I don't see any reason why we would need to
>> > disallow it?
>> >
>> > It could be helpful to start a KafkaStreams client in paused state --
>> > otherwise there is a race between calling `start()` and calling
>> `pause()`.
>> >
>> > If we allow it, we should clearly document it.
>> >
>> >
>> > -Matthias
>> >
>> > On 5/10/22 12:04 PM, Jim Hughes wrote:
>> > > Hi Bill, all,
>> > >
>> > > Thank you.  I've updated the KIP to reflect pausing standby tasks as
>> > well.
>> > > I think all the outstanding points have been addressed and I'm going to
>> > > start the vote thread!
>> > >
>> > > Cheers,
>> > >
>> > > Jim
>> > >
>> > >
>> > >
>> > > On Tue, May 10, 2022 at 2:43 PM Bill Bejeck  wrote:
>> > >
>> > >> Hi Jim,
>> > >>
>> > >> After reading the comments on the KIP, I agree that it makes sense to
>> > pause
>> > >> all activities and any changes can be made later on.
>> > >>
>> > >> Thanks,
>> > >> Bill
>> > >>
>> > >> On Tue, May 10, 2022 at 4:03 AM Bruno Cadonna 
>> > wrote:
>> > >>
>> > >>> Hi Jim,
>> > >>>
>> > >>> Thanks for the KIP!
>> > >>>
>> > >>> I am fine with the KIP in general.
>> > >>>
>> > >>> However, I am with Sophie and John to also pause the standbys for the
>> > >>> reasons they brought up. Is there a specific reason you want to keep
>> > >>> standbys going? It feels like premature optimization to me. We can
>> > still
>> > >>> add keeping standby running in a follow up if needed.
>> > >>>
>> > >>> Best,
>> > >>> Bruno
>> > >>>
>> > >>> On 10.05.22 05:15, Sophie Blee-Goldman wrote:
>> >  Thanks Jim, just one note/question on the standby tasks:
>> > 
>> >  At the minute, my moderately held position is that standby tasks
>> ought
>> > >> to
>> > > continue reading and remain caught up.  If standby tasks would run
>> > out
>> > >>> of
>> > > space, there are probably bigger problems.
>> > 
>> > 
>> >  For a single node application, or when the #pause API is invoked on
>> > all
>> >  instances,
>> >  then there won't be any further active processing and thus nothing
>> to
>> > >>> keep
>> >  up with,
>> >  right? So for that case, it's just a matter of whether any standbys
>> > >> that
>> >  are lagging
>> >  will have the chance to catch up to the (paused) active task state
>> > >> before
>> >  they stop
>> >  as well, in which case having them continue feels fine to me.
>> However
>> > >>> this
>> >  is a
>> >  relatively trivial benefit and I would only consider it as a
>> deciding
>> >  factor when all
>> >  things are equal otherwise.
>> > 
>> >  My concern is the more interesting case: when this feature is used
>> to
>> > >>> pause
>> >  only
>> >  one nodes, or some subset of the overall application. In this case,
>> > >> yes,
>> >  the standby
>> >  tasks will indeed fall out of sync. But the only reason I can
>> imagine
>> >  someone using
>> >  the pause feature in such a way is because there is something going
>> > >>> wrong,
>> >  or about
>> >  to go wrong, on that particular node. For example as mentioned
>> above,
>> > >> if
>> >  the user
>> >  wants to cut down on costs without stopping everything, or if the
>> node
>> > >> is
>> >  about to
>> >  run out of disk or needs to be debugged or so on. And in this case,
>> >  continuing to
>> >  process the standby tasks while other instances continue to run
>> would
>> >  pretty much
>> >  defeat the purpose of pausing it entirely, and might have 

Re: [DISCUSS] KIP-781: Allow MirrorMaker 2 to override the client configurations

[Omnia Ibrahim]

Hi Dongjin, nice spot for this bug. I wonder if this really needs a KIP as
it seems like a clear bug in MM2's interpretation for the config. If you're
going to keep the KIP can you please add some details in the Public
interface or Proposed changes to list which part of MM2 codebase will
change to fix this issue?

Thanks

On Mon, Oct 11, 2021 at 9:43 PM Dongjin Lee  wrote:

> Hi Kafka dev,
>
> I hope to initiate the discussion of KIP-781: Allow MirrorMaker 2 to
> override the client configurations.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-781%3A+Allow+MirrorMaker+2+to+override+the+client+configurations
>
> I found this problem while testing the MirrorMaker 2 deployments; in short,
> the configurations like `us-east.producer.acks = all` are not working now.
> It seems like to make it working like the documentation, a configuration
> overriding policy is needed.
>
> All kinds of feedbacks are greatly appreciated!
>
> Thanks,
> Dongjin
>
> --
> *Dongjin Lee*
>
> *A hitchhiker in the mathematical world.*
>
>
>
> *github:  github.com/dongjinleekr
> keybase: https://keybase.io/dongjinleekr
> linkedin: kr.linkedin.com/in/dongjinleekr
> speakerdeck:
> speakerdeck.com/dongjin
> *
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.1 #122

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.1 #121

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-832 Allow creating a producer/consumer using a producer/consumer config

[François Rosière]

To be clear, there is no problem for me to update the current KIP with the
builder approach.
It's not a lot of work in terms of code.
So, up to you. Let me know and I will do the necessary to go in one or the
other direction...
Thanks again for the feedbacks.

Le mer. 11 mai 2022 à 10:52, François Rosière 
a écrit :

> Hello,
>
> Builder is clearly the way to go for future releases of Kafka.
>
> If we align streams, we would have 3 builders
>
> new ConsumerBuilder()
>.withKeyDeserializer()
>.withValueDeserializer()
>.withInterceptors()
>.withMetricsReporter()
>.build();
>
> new ProducerBuilder()
>.withKeySerializer()
>.withValueSerializer()
>.withInterceptors()
>.withPartitioner()
>.withMetricsReporter()
>.build();
>
> new KafkaStreamsBuilder()
>.withProducerInterceptors()
>.withConsumerInterceptors()
>.withTime()
>.withKafkaClientSupplier()
>.withMetricsReporter()
>.build();
>
> The builder property would always override the configuration "instances".
> There is maybe other methods to add to the builders...
> The map, properties or config could be given to the constructor of the
> builder instead of the build method...
> At the end, we may only keep one single constructor in the Producer,
> Consumer and KafkaStreams obects.
>
> @Chris, @Bruno, thank you for your replies and proposals. Do you want I
> create another KIP explaining the builder approach or do you prefer to do
> it?
>
> Kr,
>
> F.
>
>
> Le mer. 11 mai 2022 à 09:46, Bruno Cadonna  a écrit :
>
>> Hi Francois and Chris,
>>
>> I find the idea of the builder interesting.
>>
>> I think we should go ahead with the current KIP as it is to allow
>> Francois to fix his issue soon. If one of you or both want to push
>> forward the builder idea, feel free to create a new KIP and discuss it
>> with the community.
>>
>> Regarding Francois' questions:
>>
>> 3. If the existing constructors should be removed, they need to be
>> marked as deprecated and removed in one of the next major releases.
>>
>> 5. Yes, I think Streams should be aligned.
>>
>> Both questions should be discussed in the context of a new KIP about the
>> builder idea.
>>
>> Best,
>> Bruno
>>
>> On 11.05.22 04:24, Chris Egerton wrote:
>> > Hi Francois,
>> >
>> > Thanks for your thoughts. I think it's worth noting that in regards to
>> item
>> > 2, it's possible to explicitly declare the type parameters for a builder
>> > without capturing it in a variable; for example:
>> >
>> > KafkaProducer p = new Builder(...)
>> >  .withKeySerializer(new StringSerializer())
>> >  .withValueSerializer(new IntegerSerializer())
>> >  .build();
>> >
>> > That aside, given the three binding votes already cast on the vote
>> thread,
>> > it's probably too late to be worth changing direction at this point.
>> Thanks
>> > for entertaining the proposal, and congratulations on your KIP!
>> >
>> > Cheers,
>> >
>> > Chris
>> >
>> > On Tue, May 10, 2022 at 5:33 PM François Rosière <
>> francois.rosi...@gmail.com>
>> > wrote:
>> >
>> >> Hello Chris,
>> >>
>> >> Thanks for the feedback. Builders is definitely the pattern to apply
>> when
>> >> an object needs to be built using different arguments/combinations.
>> >>
>> >> Here are my thoughts about the proposal:
>> >>
>> >> 1. The builder should only expose meaningful methods for the users
>> such as
>> >> the interceptors, the serializer/deserializer, partitioner, etc. A
>> method
>> >> like the configured instances is internal and should not be exposed if
>> we
>> >> don't want to expose the config itself. Using this internal method is
>> the
>> >> only way to solve the issue if the config is exposed.
>> >>
>> >> 2. As the key and value types are not given, a variable will need to be
>> >> created for the builder before being used. Otherwise, there is no way
>> to
>> >> infer the type correctly. Breaks a bit the inline usage with DSL style.
>> >>
>> >> 3. What about existing constructors, they would need to stay to keep
>> >> compatibility with existing o could they be removed in benefit of the
>> >> builder?
>> >>
>> >> 4. Having an access to the config also gives a way to also fine tune
>> other
>> >> aspects such as the logging related to the config. Log unused, skip
>> some
>> >> properties, etc.
>> >>
>> >> 5. What about streams? Shouldn't it be aligned?
>> >>
>> >> So, to summarise, the KIP was a best effort solution to support already
>> >> configured instances related to both the producer and the consumer.
>> >> The builder will work, it's just a matter of deciding the best
>> approach...
>> >> for me, both solutions are fine, I just need a way to inject already
>> >> configured dependencies into the producers and consumers.
>> >>
>> >> If we conclude, I will drop a PR on Github.
>> >>
>> >> Kr,
>> >>
>> >> F.
>> >>
>> >> Le mar. 10 mai 2022 à 15:01, Chris Egerton  a
>> >> écrit :
>> >>
>> >>> Hi Francois,
>> >>>
>> >>> Thanks for the KIP! I sympathize with the issue you're 

Re: [DISCUSS] KIP-832 Allow creating a producer/consumer using a producer/consumer config

[François Rosière]

Hello,

Builder is clearly the way to go for future releases of Kafka.

If we align streams, we would have 3 builders

new ConsumerBuilder()
   .withKeyDeserializer()
   .withValueDeserializer()
   .withInterceptors()
   .withMetricsReporter()
   .build();

new ProducerBuilder()
   .withKeySerializer()
   .withValueSerializer()
   .withInterceptors()
   .withPartitioner()
   .withMetricsReporter()
   .build();

new KafkaStreamsBuilder()
   .withProducerInterceptors()
   .withConsumerInterceptors()
   .withTime()
   .withKafkaClientSupplier()
   .withMetricsReporter()
   .build();

The builder property would always override the configuration "instances".
There is maybe other methods to add to the builders...
The map, properties or config could be given to the constructor of the
builder instead of the build method...
At the end, we may only keep one single constructor in the Producer,
Consumer and KafkaStreams obects.

@Chris, @Bruno, thank you for your replies and proposals. Do you want I
create another KIP explaining the builder approach or do you prefer to do
it?

Kr,

F.


Le mer. 11 mai 2022 à 09:46, Bruno Cadonna  a écrit :

> Hi Francois and Chris,
>
> I find the idea of the builder interesting.
>
> I think we should go ahead with the current KIP as it is to allow
> Francois to fix his issue soon. If one of you or both want to push
> forward the builder idea, feel free to create a new KIP and discuss it
> with the community.
>
> Regarding Francois' questions:
>
> 3. If the existing constructors should be removed, they need to be
> marked as deprecated and removed in one of the next major releases.
>
> 5. Yes, I think Streams should be aligned.
>
> Both questions should be discussed in the context of a new KIP about the
> builder idea.
>
> Best,
> Bruno
>
> On 11.05.22 04:24, Chris Egerton wrote:
> > Hi Francois,
> >
> > Thanks for your thoughts. I think it's worth noting that in regards to
> item
> > 2, it's possible to explicitly declare the type parameters for a builder
> > without capturing it in a variable; for example:
> >
> > KafkaProducer p = new Builder(...)
> >  .withKeySerializer(new StringSerializer())
> >  .withValueSerializer(new IntegerSerializer())
> >  .build();
> >
> > That aside, given the three binding votes already cast on the vote
> thread,
> > it's probably too late to be worth changing direction at this point.
> Thanks
> > for entertaining the proposal, and congratulations on your KIP!
> >
> > Cheers,
> >
> > Chris
> >
> > On Tue, May 10, 2022 at 5:33 PM François Rosière <
> francois.rosi...@gmail.com>
> > wrote:
> >
> >> Hello Chris,
> >>
> >> Thanks for the feedback. Builders is definitely the pattern to apply
> when
> >> an object needs to be built using different arguments/combinations.
> >>
> >> Here are my thoughts about the proposal:
> >>
> >> 1. The builder should only expose meaningful methods for the users such
> as
> >> the interceptors, the serializer/deserializer, partitioner, etc. A
> method
> >> like the configured instances is internal and should not be exposed if
> we
> >> don't want to expose the config itself. Using this internal method is
> the
> >> only way to solve the issue if the config is exposed.
> >>
> >> 2. As the key and value types are not given, a variable will need to be
> >> created for the builder before being used. Otherwise, there is no way to
> >> infer the type correctly. Breaks a bit the inline usage with DSL style.
> >>
> >> 3. What about existing constructors, they would need to stay to keep
> >> compatibility with existing o could they be removed in benefit of the
> >> builder?
> >>
> >> 4. Having an access to the config also gives a way to also fine tune
> other
> >> aspects such as the logging related to the config. Log unused, skip some
> >> properties, etc.
> >>
> >> 5. What about streams? Shouldn't it be aligned?
> >>
> >> So, to summarise, the KIP was a best effort solution to support already
> >> configured instances related to both the producer and the consumer.
> >> The builder will work, it's just a matter of deciding the best
> approach...
> >> for me, both solutions are fine, I just need a way to inject already
> >> configured dependencies into the producers and consumers.
> >>
> >> If we conclude, I will drop a PR on Github.
> >>
> >> Kr,
> >>
> >> F.
> >>
> >> Le mar. 10 mai 2022 à 15:01, Chris Egerton  a
> >> écrit :
> >>
> >>> Hi Francois,
> >>>
> >>> Thanks for the KIP! I sympathize with the issue you're facing and with
> >>> John's reluctance to let perfect be the enemy of good, and if KIP
> freeze
> >>> were tomorrow, I think this would be good enough. Given that we still
> >> have
> >>> some time to work with, I'd like to propose an alternative approach and
> >> see
> >>> what your thoughts are.
> >>>
> >>> There are a few issues with the current client APIs that are closely
> >>> related to the KIP:
> >>> 1. Too many constructors (there are currently four each for
> KafkaProducer
> >>> and KafkaConsumer, 

Re: [DISCUSS] KIP-831: Add metric for log recovery progress

[Luke Chen]

> And if people start using RemainingLogs and RemainingSegments and then
REALLY FEEL like they need RemainingBytes, then we can always add it in the
future.

+1

Thanks James!
Luke

On Wed, May 11, 2022 at 3:57 PM James Cheng  wrote:

> Hi Luke,
>
> Thanks for the detailed explanation. I agree that the current proposal of
> RemainingLogs and RemainingSegments will greatly improve the situation, and
> that we can go ahead with the KIP as is.
>
> If RemainingBytes were straight-forward to implement, then I’d like to
> have it. But we can live without it for now. And if people start using
> RemainingLogs and RemainingSegments and then REALLY FEEL like they need
> RemainingBytes, then we can always add it in the future.
>
> Thanks Luke, for the detailed explanation, and for responding to my
> feedback!
>
> -James
>
> Sent from my iPhone
>
> > On May 10, 2022, at 6:48 AM, Luke Chen  wrote:
> >
> > Hi James and all,
> >
> > I checked again and I can see when creating UnifiedLog, we expected the
> > logs/indexes/snapshots are in good state.
> > So, I don't think we should break the current design to expose the
> > `RemainingBytesToRecovery`
> > metric.
> >
> > If there is no other comments, I'll start a vote within this week.
> >
> > Thank you.
> > Luke
> >
> >> On Fri, May 6, 2022 at 6:00 PM Luke Chen  wrote:
> >>
> >> Hi James,
> >>
> >> Thanks for your input.
> >>
> >> For the `RemainingBytesToRecovery` metric proposal, I think there's one
> >> thing I didn't make it clear.
> >> Currently, when log manager start up, we'll try to load all logs
> >> (segments), and during the log loading, we'll try to recover logs if
> >> necessary.
> >> And the logs loading is using "thread pool" as you thought.
> >>
> >> So, here's the problem:
> >> All segments in each log folder (partition) will be loaded in each log
> >> recovery thread, and until it's loaded, we can know how many segments
> (or
> >> how many Bytes) needed to recover.
> >> That means, if we have 10 partition logs in one broker, and we have 2
> log
> >> recovery threads (num.recovery.threads.per.data.dir=2), before the
> >> threads load the segments in each log, we only know how many logs
> >> (partitions) we have in the broker (i.e. RemainingLogsToRecover metric).
> >> We cannot know how many segments/Bytes needed to recover until each
> thread
> >> starts to load the segments under one log (partition).
> >>
> >> So, the example in the KIP, it shows:
> >> Currently, there are still 5 logs (partitions) needed to recover under
> >> /tmp/log1 dir. And there are 2 threads doing the jobs, where one thread
> has
> >> 1 segments needed to recover, and the other one has 3 segments
> needed
> >> to recover.
> >>
> >>   - kafka.log
> >>  - LogManager
> >> - RemainingLogsToRecover
> >>- /tmp/log1 => 5← there are 5 logs under
> >>/tmp/log1 needed to be recovered
> >>- /tmp/log2 => 0
> >> - RemainingSegmentsToRecover
> >>- /tmp/log1 ← 2 threads are doing log
> >>recovery for /tmp/log1
> >>- 0 => 1 ← there are 1 segments needed to be
> >>   recovered for thread 0
> >>   - 1 => 3
> >>   - /tmp/log2
> >>   - 0 => 0
> >>   - 1 => 0
> >>
> >>
> >> So, after a while, the metrics might look like this:
> >> It said, now, there are only 4 logs needed to recover in /tmp/log1, and
> >> the thread 0 has 9000 segments left, and thread 1 has 5 segments left
> >> (which should imply the thread already completed 2 logs recovery in the
> >> period)
> >>
> >>   - kafka.log
> >>  - LogManager
> >> - RemainingLogsToRecover
> >>- /tmp/log1 => 3← there are 3 logs under
> >>/tmp/log1 needed to be recovered
> >>- /tmp/log2 => 0
> >> - RemainingSegmentsToRecover
> >>- /tmp/log1 ← 2 threads are doing log
> >>recovery for /tmp/log1
> >>- 0 => 9000 ← there are 9000 segments needed to be
> >>   recovered for thread 0
> >>   - 1 => 5
> >>   - /tmp/log2
> >>   - 0 => 0
> >>   - 1 => 0
> >>
> >>
> >> That said, the `RemainingBytesToRecovery` metric is difficult to achieve
> >> as you expected. I think the current proposal with
> `RemainingLogsToRecover`
> >> and `RemainingSegmentsToRecover` should already provide enough info for
> >> the log recovery progress.
> >>
> >> I've also updated the KIP example to make it clear.
> >>
> >>
> >> Thank you.
> >> Luke
> >>
> >>
> >>> On Thu, May 5, 2022 at 3:31 AM James Cheng 
> wrote:
> >>>
> >>> Hi Luke,
> >>>
> >>> Thanks for adding RemainingSegmentsToRecovery.
> >>>
> >>> Another thought: different topics can have different segment sizes. I
> >>> don't know how common it is, but it is possible. Some topics might want
> >>> small segment sizes to more granular 

Re: [DISCUSS] KIP-831: Add metric for log recovery progress

[James Cheng]

Hi Luke,

Thanks for the detailed explanation. I agree that the current proposal of 
RemainingLogs and RemainingSegments will greatly improve the situation, and 
that we can go ahead with the KIP as is.

If RemainingBytes were straight-forward to implement, then I’d like to have it. 
But we can live without it for now. And if people start using RemainingLogs and 
RemainingSegments and then REALLY FEEL like they need RemainingBytes, then we 
can always add it in the future.

Thanks Luke, for the detailed explanation, and for responding to my feedback!

-James

Sent from my iPhone

> On May 10, 2022, at 6:48 AM, Luke Chen  wrote:
> 
> Hi James and all,
> 
> I checked again and I can see when creating UnifiedLog, we expected the
> logs/indexes/snapshots are in good state.
> So, I don't think we should break the current design to expose the
> `RemainingBytesToRecovery`
> metric.
> 
> If there is no other comments, I'll start a vote within this week.
> 
> Thank you.
> Luke
> 
>> On Fri, May 6, 2022 at 6:00 PM Luke Chen  wrote:
>> 
>> Hi James,
>> 
>> Thanks for your input.
>> 
>> For the `RemainingBytesToRecovery` metric proposal, I think there's one
>> thing I didn't make it clear.
>> Currently, when log manager start up, we'll try to load all logs
>> (segments), and during the log loading, we'll try to recover logs if
>> necessary.
>> And the logs loading is using "thread pool" as you thought.
>> 
>> So, here's the problem:
>> All segments in each log folder (partition) will be loaded in each log
>> recovery thread, and until it's loaded, we can know how many segments (or
>> how many Bytes) needed to recover.
>> That means, if we have 10 partition logs in one broker, and we have 2 log
>> recovery threads (num.recovery.threads.per.data.dir=2), before the
>> threads load the segments in each log, we only know how many logs
>> (partitions) we have in the broker (i.e. RemainingLogsToRecover metric).
>> We cannot know how many segments/Bytes needed to recover until each thread
>> starts to load the segments under one log (partition).
>> 
>> So, the example in the KIP, it shows:
>> Currently, there are still 5 logs (partitions) needed to recover under
>> /tmp/log1 dir. And there are 2 threads doing the jobs, where one thread has
>> 1 segments needed to recover, and the other one has 3 segments needed
>> to recover.
>> 
>>   - kafka.log
>>  - LogManager
>> - RemainingLogsToRecover
>>- /tmp/log1 => 5← there are 5 logs under
>>/tmp/log1 needed to be recovered
>>- /tmp/log2 => 0
>> - RemainingSegmentsToRecover
>>- /tmp/log1 ← 2 threads are doing log
>>recovery for /tmp/log1
>>- 0 => 1 ← there are 1 segments needed to be
>>   recovered for thread 0
>>   - 1 => 3
>>   - /tmp/log2
>>   - 0 => 0
>>   - 1 => 0
>> 
>> 
>> So, after a while, the metrics might look like this:
>> It said, now, there are only 4 logs needed to recover in /tmp/log1, and
>> the thread 0 has 9000 segments left, and thread 1 has 5 segments left
>> (which should imply the thread already completed 2 logs recovery in the
>> period)
>> 
>>   - kafka.log
>>  - LogManager
>> - RemainingLogsToRecover
>>- /tmp/log1 => 3← there are 3 logs under
>>/tmp/log1 needed to be recovered
>>- /tmp/log2 => 0
>> - RemainingSegmentsToRecover
>>- /tmp/log1 ← 2 threads are doing log
>>recovery for /tmp/log1
>>- 0 => 9000 ← there are 9000 segments needed to be
>>   recovered for thread 0
>>   - 1 => 5
>>   - /tmp/log2
>>   - 0 => 0
>>   - 1 => 0
>> 
>> 
>> That said, the `RemainingBytesToRecovery` metric is difficult to achieve
>> as you expected. I think the current proposal with `RemainingLogsToRecover`
>> and `RemainingSegmentsToRecover` should already provide enough info for
>> the log recovery progress.
>> 
>> I've also updated the KIP example to make it clear.
>> 
>> 
>> Thank you.
>> Luke
>> 
>> 
>>> On Thu, May 5, 2022 at 3:31 AM James Cheng  wrote:
>>> 
>>> Hi Luke,
>>> 
>>> Thanks for adding RemainingSegmentsToRecovery.
>>> 
>>> Another thought: different topics can have different segment sizes. I
>>> don't know how common it is, but it is possible. Some topics might want
>>> small segment sizes to more granular expiration of data.
>>> 
>>> The downside of RemainingLogsToRecovery and RemainingSegmentsToRecovery
>>> is that the rate that they will decrement depends on the configuration and
>>> patterns of the topics and partitions and segment sizes. If someone is
>>> monitoring those metrics, they might see times where the metric decrements
>>> slowly, followed by a burst where it decrements quickly.
>>> 
>>> What about RemainingBytesToRecovery? This would not depend on the

Re: [HEADS-UP] Modification to KIP Template

[Luke Chen]

Hi Mickael,

+1 to add "test plan" section to KIP template.
Thanks for the improvement!

Luke

On Tue, May 10, 2022 at 11:21 PM Mickael Maison 
wrote:

> Hi,
>
> I did not see any comments nor concerns so I went ahead and added the
> Test Plan section to the KIP template.
>
> Thanks,
> Mickael
>
> On Fri, Apr 1, 2022 at 5:53 PM Mickael Maison 
> wrote:
> >
> > Hi,
> >
> > Unearthing this old thread as today I stumbled on the issue that
> > Ismael reported. It looks like this was never fixed!
> >
> > The "Test Plan" section was only added in the KIP-template page [0]
> > and not in the actual KIP-template template [1] that is used when
> > doing `Create -> KIP-Template` or by clicking on `Create KIP` on
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> >
> > I think this new section makes sense and it's very easy to add it to
> > the actual template. Before doing it, I just want to ping the dev list
> > to see if anybody has suggestions or concerns since this was discussed
> > many years ago now.
> >
> > 0: https://cwiki.apache.org/confluence/display/KAFKA/KIP-Template
> > 1:
> https://cwiki.apache.org/confluence/pages/templates2/viewpagetemplate.action?entityId=54329345=KAFKA
> >
> > Thanks,
> > Mickael
> >
> > On Fri, May 27, 2016 at 10:55 AM Ismael Juma  wrote:
> > >
> > > Hi Gwen,
> > >
> > > Thanks for adding the "Test Plans" section. I think it may be worth
> adding
> > > a note about performance testing plans too (whenever relevant). By the
> way,
> > > even though the following page has the new section, if I use `Create ->
> > > KIP-Template`, the new section doesn't appear. Do you know why?
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-Template
> > >
> > > Ismael
> > >
> > > On Fri, May 27, 2016 at 3:24 AM, Gwen Shapira 
> wrote:
> > >
> > > > Hi Kafka Developers,
> > > >
> > > > Just a quick heads-up that I added a new section to the KIP
> template: "Test
> > > > Plans".
> > > > I think its a good habit to think about how a feature will be tested
> while
> > > > planning it. I'm talking about high-level notes on system tests, not
> gritty
> > > > details.
> > > >
> > > > This will apply to new KIPs, not ones in discussion/implementation
> phases
> > > > (although if your KIP is under discussion and you want to add test
> plans,
> > > > it will be very nice of you).
> > > >
> > > > I figured we all agree that thinking a bit about tests is a good
> idea, so I
> > > > added it first and started a discussion later. If you strongly
> object,
> > > > please respond with strong objections. Wikis are easy to edit :)
> > > >
> > > > Gwen
> > > >
>

Re: [DISCUSS] KIP-832 Allow creating a producer/consumer using a producer/consumer config

[Bruno Cadonna]

Hi Francois and Chris,

I find the idea of the builder interesting.

I think we should go ahead with the current KIP as it is to allow 
Francois to fix his issue soon. If one of you or both want to push 
forward the builder idea, feel free to create a new KIP and discuss it 
with the community.


Regarding Francois' questions:

3. If the existing constructors should be removed, they need to be 
marked as deprecated and removed in one of the next major releases.


5. Yes, I think Streams should be aligned.

Both questions should be discussed in the context of a new KIP about the 
builder idea.


Best,
Bruno

On 11.05.22 04:24, Chris Egerton wrote:

Hi Francois,

Thanks for your thoughts. I think it's worth noting that in regards to item
2, it's possible to explicitly declare the type parameters for a builder
without capturing it in a variable; for example:

KafkaProducer p = new Builder(...)
 .withKeySerializer(new StringSerializer())
 .withValueSerializer(new IntegerSerializer())
 .build();

That aside, given the three binding votes already cast on the vote thread,
it's probably too late to be worth changing direction at this point. Thanks
for entertaining the proposal, and congratulations on your KIP!

Cheers,

Chris

On Tue, May 10, 2022 at 5:33 PM François Rosière 
wrote:


Hello Chris,

Thanks for the feedback. Builders is definitely the pattern to apply when
an object needs to be built using different arguments/combinations.

Here are my thoughts about the proposal:

1. The builder should only expose meaningful methods for the users such as
the interceptors, the serializer/deserializer, partitioner, etc. A method
like the configured instances is internal and should not be exposed if we
don't want to expose the config itself. Using this internal method is the
only way to solve the issue if the config is exposed.

2. As the key and value types are not given, a variable will need to be
created for the builder before being used. Otherwise, there is no way to
infer the type correctly. Breaks a bit the inline usage with DSL style.

3. What about existing constructors, they would need to stay to keep
compatibility with existing o could they be removed in benefit of the
builder?

4. Having an access to the config also gives a way to also fine tune other
aspects such as the logging related to the config. Log unused, skip some
properties, etc.

5. What about streams? Shouldn't it be aligned?

So, to summarise, the KIP was a best effort solution to support already
configured instances related to both the producer and the consumer.
The builder will work, it's just a matter of deciding the best approach...
for me, both solutions are fine, I just need a way to inject already
configured dependencies into the producers and consumers.

If we conclude, I will drop a PR on Github.

Kr,

F.

Le mar. 10 mai 2022 à 15:01, Chris Egerton  a
écrit :


Hi Francois,

Thanks for the KIP! I sympathize with the issue you're facing and with
John's reluctance to let perfect be the enemy of good, and if KIP freeze
were tomorrow, I think this would be good enough. Given that we still

have

some time to work with, I'd like to propose an alternative approach and

see

what your thoughts are.

There are a few issues with the current client APIs that are closely
related to the KIP:
1. Too many constructors (there are currently four each for KafkaProducer
and KafkaConsumer, yet they all do basically the same thing)
2. Lack of type safety with interceptors (you have no way to enforce at
compile time that your ProducerInterceptor is used with

a

Serializer and Serializer, for example)
3. Inflexibility and inconsistency with instantiation of pluggable
interfaces (you can bring your own (de)serializers, but everything else
gets instantiated and configured for you at producer/consumer

construction

time)

The KIP as it exists now will only address item 3, and will exacerbate

item

1.

In addition, there are a few new issues introduced by the KIP as it

exists

now:
1. Tighter coupling between the ProducerConfig/ConsumerConfig classes and
the KafkaProducer/KafkaConsumer classes. Any change we make in the future
that breaks either of these config classes in unexpected ways (but which
does not break the KafkaProducer/KafkaConsumer constructors that do not
accept these classes as parameters) will now have a much higher chance to
also break a user's entire producer/consumer application.
2. Complexity for users like yourself who would like to override behavior
in a ProducerConfig/ConsumerConfig in order to inject pre-instantiated
dependencies. The example in the KIP overrides
AbstractConfig::getConfiguredInstances [1] in order to achieve this. But
there are two other overloaded variants of getConfiguredInstances, and

two

AbstractConfig::getConfiguredInstance methods that also exist. We'd

either

need to establish a dependency graph between these methods (e.g., some
methods are guaranteed to invoke another overloaded variant) as 

Re: [VOTE] 3.1.1 RC1

[Tom Bentley]

Hi Dongjoon,

I've been trying to get a green build of the 3.1 branch from Jenkins (
https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.1/) since my
original email promised to provide this. Individually each stage between
builds 111 and 119 has passed in at least one run, but there have been no
runs where all stages have passed.

However, as you note, the vote has the necessary 3 binding votes, so
assuming the voters don't wish to change their votes I can proceed with the
release. If this is not the case please respond on the thread. If I hear
nothing in the next 24 hours I will assume the voters are OK with this and
proceed with the rest of the release process.

Apologies for the delay,

Kind regards,

Tom


On Wed, 11 May 2022 at 08:15, Dongjoon Hyun  wrote:

> Hi, Tom.
>
> Could you conclude this vote as a release manager? :)
>
> Dongjoon.
>
> On 2022/05/06 13:31:15 Michal Tóth wrote:
> > Hello
> >
> > I have executed some produce/consume system tests which all passed.
> > Also everything passed from
> https://github.com/tombentley/kafka-verify-rc
> > - checking signatures, checksums, with gradle unit & integration tests,
> etc.
> >
> > Good from me (non-binding).
> >
> >
> >
> > pi 6. 5. 2022 o 14:30 David Jacot 
> napísal(a):
> >
> > > Thanks for running the release, Tom.
> > >
> > > I performed the following validations:
> > > * Verified all checksums and signatures.
> > > * Built from source and ran unit tests.
> > > * Ran the first quickstart steps for both ZK and KRaft.
> > > * Spotchecked the Javadocs.
> > >
> > > I noticed the same issues as others on the website. I checked
> > > the doc in git and it looks good.
> > >
> > > +1 (binding)
> > >
> > > Best,
> > > David
> > >
> > > On Thu, May 5, 2022 at 7:52 PM Dongjoon Hyun 
> wrote:
> > > >
> > > > +1 (non-binding)
> > > >
> > > > RC1 was tested with Apache Spark tests
> > > >
> > > > - https://github.com/apache/spark/pull/36135
> > > >
> > > > Thanks,
> > > > Dongjoon.
> > > >
> > > > On 2022/05/05 03:25:25 Luke Chen wrote:
> > > > > Hi Tom,
> > > > >
> > > > > I did:
> > > > > 1. check the signature and checksums
> > > > > 2. ran quick start with java17 + sacla2.12
> > > > > 3. browse java docs/documentations
> > > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > Thanks for running the release.
> > > > >
> > > > > Luke
> > > > >
> > > > > On Thu, May 5, 2022 at 12:43 AM Bill Bejeck 
> > > wrote:
> > > > >
> > > > > > Hi Tom,
> > > > > >
> > > > > > Thanks for running the release!
> > > > > >
> > > > > > I did the following checks:
> > > > > >
> > > > > >1. Validated all the checksum and signatures
> > > > > >2. Built from source and ran the unit tests (Java 11)
> > > > > >3. Ran the quickstart and the Kafka Streams demo (Java 11)
> > > > > >4. Did a quick scan of the Javadoc and the documentation
> > > > > >
> > > > > > I noticed the same issues as Mickael on the website, but
> otherwise,
> > > it
> > > > > > looks good.
> > > > > > +1 (binding)
> > > > > >
> > > > > > Thanks,
> > > > > > Bill
> > > > > >
> > > > > > On Tue, May 3, 2022 at 10:17 AM Jakub Scholz 
> > > wrote:
> > > > > >
> > > > > > > +1 (non-binding). I used the Scala 2.13 binaries and staged
> Maven
> > > > > > artifacts
> > > > > > > and ran various tests with them. Thanks for doing the release.
> > > > > > >
> > > > > > > Jakub
> > > > > > >
> > > > > > > On Fri, Apr 29, 2022 at 8:16 PM Mickael Maison <
> > > mimai...@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Tom,
> > > > > > > >
> > > > > > > > Thanks for running this release!
> > > > > > > >
> > > > > > > > I've done the following:
> > > > > > > > - Checked signatures and checksums
> > > > > > > > - Checked javadocs/maven artifacts
> > > > > > > > - Built from source and run all tests with Java 11
> > > > > > > > - Ran quickstart on Scala 2.13 artifact with Java 11
> > > > > > > >
> > > > > > > > It looks like the website has not been updated yet, I still
> only
> > > see
> > > > > > > > 3.1.0. When you'll add 3.1.1, let's make sure we mention
> > > reload4j in
> > > > > > > > the notable changes section.
> > > > > > > >
> > > > > > > > +1 (binding)
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Mickael
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Apr 29, 2022 at 11:12 AM Tom Bentley <
> > > tbent...@redhat.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hello Kafka users, developers and client-developers,
> > > > > > > > >
> > > > > > > > > This is the first candidate for release of Apache Kafka
> 3.1.1.
> > > > > > > > >
> > > > > > > > > Apache Kafka 3.1.1 is a bugfix release and 30 issues have
> been
> > > fixed
> > > > > > > > > since 3.1.0.
> > > > > > > > >
> > > > > > > > > Release notes for the 3.1.1 release:
> > > > > > > > >
> > > > > >
> > > https://home.apache.org/~tombentley/kafka-3.1.1-rc1/RELEASE_NOTES.html
> > > > > > > > >
> > > > > > > > > *** Please download, test and vote by 09:00 UTC, Friday
> 6th May
> > 

Re: [DISCUSS] KIP-834: Pause / Resume KafkaStreams Topologies

[Luke Chen]

Hi Jim,

Thanks for the KIP. Overall LGTM!

One late question:
Could we run the stream resetter tool (i.e.
kafka-streams-application-reset.sh) during pause state?
I can imagine there's a use case that after pausing for a while, user just
want to continue with the latest offset, and skipping the intermediate
records.

Thank you.
Luke

On Wed, May 11, 2022 at 10:12 AM Jim Hughes 
wrote:

> Hi Matthias,
>
> I like it.  I've updated the KIP to reflect that detail; I put the details
> in the docs for pause.
>
> Cheers,
>
> Jim
>
> On Tue, May 10, 2022 at 7:51 PM Matthias J. Sax  wrote:
>
> > Thanks for the KIP. Overall LGTM.
> >
> > Can we clarify one question: would it be allowed to call `pause()`
> > before calling `start()`? I don't see any reason why we would need to
> > disallow it?
> >
> > It could be helpful to start a KafkaStreams client in paused state --
> > otherwise there is a race between calling `start()` and calling
> `pause()`.
> >
> > If we allow it, we should clearly document it.
> >
> >
> > -Matthias
> >
> > On 5/10/22 12:04 PM, Jim Hughes wrote:
> > > Hi Bill, all,
> > >
> > > Thank you.  I've updated the KIP to reflect pausing standby tasks as
> > well.
> > > I think all the outstanding points have been addressed and I'm going to
> > > start the vote thread!
> > >
> > > Cheers,
> > >
> > > Jim
> > >
> > >
> > >
> > > On Tue, May 10, 2022 at 2:43 PM Bill Bejeck  wrote:
> > >
> > >> Hi Jim,
> > >>
> > >> After reading the comments on the KIP, I agree that it makes sense to
> > pause
> > >> all activities and any changes can be made later on.
> > >>
> > >> Thanks,
> > >> Bill
> > >>
> > >> On Tue, May 10, 2022 at 4:03 AM Bruno Cadonna 
> > wrote:
> > >>
> > >>> Hi Jim,
> > >>>
> > >>> Thanks for the KIP!
> > >>>
> > >>> I am fine with the KIP in general.
> > >>>
> > >>> However, I am with Sophie and John to also pause the standbys for the
> > >>> reasons they brought up. Is there a specific reason you want to keep
> > >>> standbys going? It feels like premature optimization to me. We can
> > still
> > >>> add keeping standby running in a follow up if needed.
> > >>>
> > >>> Best,
> > >>> Bruno
> > >>>
> > >>> On 10.05.22 05:15, Sophie Blee-Goldman wrote:
> >  Thanks Jim, just one note/question on the standby tasks:
> > 
> >  At the minute, my moderately held position is that standby tasks
> ought
> > >> to
> > > continue reading and remain caught up.  If standby tasks would run
> > out
> > >>> of
> > > space, there are probably bigger problems.
> > 
> > 
> >  For a single node application, or when the #pause API is invoked on
> > all
> >  instances,
> >  then there won't be any further active processing and thus nothing
> to
> > >>> keep
> >  up with,
> >  right? So for that case, it's just a matter of whether any standbys
> > >> that
> >  are lagging
> >  will have the chance to catch up to the (paused) active task state
> > >> before
> >  they stop
> >  as well, in which case having them continue feels fine to me.
> However
> > >>> this
> >  is a
> >  relatively trivial benefit and I would only consider it as a
> deciding
> >  factor when all
> >  things are equal otherwise.
> > 
> >  My concern is the more interesting case: when this feature is used
> to
> > >>> pause
> >  only
> >  one nodes, or some subset of the overall application. In this case,
> > >> yes,
> >  the standby
> >  tasks will indeed fall out of sync. But the only reason I can
> imagine
> >  someone using
> >  the pause feature in such a way is because there is something going
> > >>> wrong,
> >  or about
> >  to go wrong, on that particular node. For example as mentioned
> above,
> > >> if
> >  the user
> >  wants to cut down on costs without stopping everything, or if the
> node
> > >> is
> >  about to
> >  run out of disk or needs to be debugged or so on. And in this case,
> >  continuing to
> >  process the standby tasks while other instances continue to run
> would
> >  pretty much
> >  defeat the purpose of pausing it entirely, and might have unpleasant
> >  consequences
> >  for the unsuspecting developer.
> > 
> >  All that said, I don't want to block this KIP so if you have strong
> >  feelings about the
> >  standby behavior I'm happy to back down. I'm only pushing back now
> > >>> because
> >  it
> >  felt like there wasn't any particular motivation for the standbys to
> >  continue processing
> >  or not, and I figured I'd try to fill in this gap with my thoughts
> on
> > >> the
> >  matter :)
> >  Either way we should just make sure that this behavior is documented
> >  clearly,
> >  since it may be surprising if we decide to only pause active
> > processing
> >  (another option
> >  is to rename the method something like #pauseProcessing or
> >  #pauseActiveProcessing
> >  so that it's 

Re: [DISCUSS] Apache Kafka 3.3.0 Release

[Bruno Cadonna]

Thanks José!

+1

Best,
Bruno

On 11.05.22 04:38, John Roesler wrote:

+1 from me! Thanks for volunteering, José.
-John

On Tue, May 10, 2022, at 17:53, José Armando García Sancio wrote:

Hi all,

I would like to volunteer for the release of Apache Kafka 3.3.0. If
people agree, I'll start working on the release plan and update this
thread.

Thanks,
-José

Re: [VOTE] 3.1.1 RC1

[Dongjoon Hyun]

Hi, Tom.

Could you conclude this vote as a release manager? :)

Dongjoon.

On 2022/05/06 13:31:15 Michal Tóth wrote:
> Hello
> 
> I have executed some produce/consume system tests which all passed.
> Also everything passed from https://github.com/tombentley/kafka-verify-rc
> - checking signatures, checksums, with gradle unit & integration tests, etc.
> 
> Good from me (non-binding).
> 
> 
> 
> pi 6. 5. 2022 o 14:30 David Jacot  napísal(a):
> 
> > Thanks for running the release, Tom.
> >
> > I performed the following validations:
> > * Verified all checksums and signatures.
> > * Built from source and ran unit tests.
> > * Ran the first quickstart steps for both ZK and KRaft.
> > * Spotchecked the Javadocs.
> >
> > I noticed the same issues as others on the website. I checked
> > the doc in git and it looks good.
> >
> > +1 (binding)
> >
> > Best,
> > David
> >
> > On Thu, May 5, 2022 at 7:52 PM Dongjoon Hyun  wrote:
> > >
> > > +1 (non-binding)
> > >
> > > RC1 was tested with Apache Spark tests
> > >
> > > - https://github.com/apache/spark/pull/36135
> > >
> > > Thanks,
> > > Dongjoon.
> > >
> > > On 2022/05/05 03:25:25 Luke Chen wrote:
> > > > Hi Tom,
> > > >
> > > > I did:
> > > > 1. check the signature and checksums
> > > > 2. ran quick start with java17 + sacla2.12
> > > > 3. browse java docs/documentations
> > > >
> > > > +1 (non-binding)
> > > >
> > > > Thanks for running the release.
> > > >
> > > > Luke
> > > >
> > > > On Thu, May 5, 2022 at 12:43 AM Bill Bejeck 
> > wrote:
> > > >
> > > > > Hi Tom,
> > > > >
> > > > > Thanks for running the release!
> > > > >
> > > > > I did the following checks:
> > > > >
> > > > >1. Validated all the checksum and signatures
> > > > >2. Built from source and ran the unit tests (Java 11)
> > > > >3. Ran the quickstart and the Kafka Streams demo (Java 11)
> > > > >4. Did a quick scan of the Javadoc and the documentation
> > > > >
> > > > > I noticed the same issues as Mickael on the website, but otherwise,
> > it
> > > > > looks good.
> > > > > +1 (binding)
> > > > >
> > > > > Thanks,
> > > > > Bill
> > > > >
> > > > > On Tue, May 3, 2022 at 10:17 AM Jakub Scholz 
> > wrote:
> > > > >
> > > > > > +1 (non-binding). I used the Scala 2.13 binaries and staged Maven
> > > > > artifacts
> > > > > > and ran various tests with them. Thanks for doing the release.
> > > > > >
> > > > > > Jakub
> > > > > >
> > > > > > On Fri, Apr 29, 2022 at 8:16 PM Mickael Maison <
> > mimai...@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Tom,
> > > > > > >
> > > > > > > Thanks for running this release!
> > > > > > >
> > > > > > > I've done the following:
> > > > > > > - Checked signatures and checksums
> > > > > > > - Checked javadocs/maven artifacts
> > > > > > > - Built from source and run all tests with Java 11
> > > > > > > - Ran quickstart on Scala 2.13 artifact with Java 11
> > > > > > >
> > > > > > > It looks like the website has not been updated yet, I still only
> > see
> > > > > > > 3.1.0. When you'll add 3.1.1, let's make sure we mention
> > reload4j in
> > > > > > > the notable changes section.
> > > > > > >
> > > > > > > +1 (binding)
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Mickael
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Apr 29, 2022 at 11:12 AM Tom Bentley <
> > tbent...@redhat.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > Hello Kafka users, developers and client-developers,
> > > > > > > >
> > > > > > > > This is the first candidate for release of Apache Kafka 3.1.1.
> > > > > > > >
> > > > > > > > Apache Kafka 3.1.1 is a bugfix release and 30 issues have been
> > fixed
> > > > > > > > since 3.1.0.
> > > > > > > >
> > > > > > > > Release notes for the 3.1.1 release:
> > > > > > > >
> > > > >
> > https://home.apache.org/~tombentley/kafka-3.1.1-rc1/RELEASE_NOTES.html
> > > > > > > >
> > > > > > > > *** Please download, test and vote by 09:00 UTC, Friday 6th May
> > > > > > > >
> > > > > > > > Kafka's KEYS file containing PGP keys we use to sign the
> > release:
> > > > > > > > https://kafka.apache.org/KEYS
> > > > > > > >
> > > > > > > > * Release artifacts to be voted upon (source and binary):
> > > > > > > > https://home.apache.org/~tombentley/kafka-3.1.1-rc1/
> > > > > > > >
> > > > > > > > * Maven artifacts to be voted upon:
> > > > > > > >
> > > > >
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > > > > >
> > > > > > > > * Javadoc:
> > > > > > > > https://home.apache.org/~tombentley/kafka-3.1.1-rc1/javadoc/
> > > > > > > >
> > > > > > > > * Tag to be voted upon (off 3.1 branch) is the 3.1.1 tag:
> > > > > > > > https://github.com/apache/kafka/releases/tag/3.1.1-rc1
> > > > > > > >
> > > > > > > > * Documentation:
> > > > > > > > https://kafka.apache.org/31/documentation.html
> > > > > > > >
> > > > > > > > * Protocol:
> > > > > > > > https://kafka.apache.org/31/protocol.html
> > > > > > > >
> > > > > > > > * Successful Jenkins builds for the 3.1 branch:
> > > > > > 

Re: [DISCUSS] KIP-835: Monitor KRaft Controller Quorum Health

[Luke Chen]

Hi José,

Thanks for the KIP!

Some questions:
1. Jason has asked but you didn't answer: What is the default value for `
metadata.monitor.write.interval.ms`?
2. The `noopRecord` API key is `TBD`. Why can't we put the "currently used
API Key nums + 1" into it? Any concern?
3. typo: name=MetadataWriteOffse[t]s
4. I don't understand the difference between MetadataLastCommittedOffset
and MetadataWriteOffsets metrics. I think the 2 values will always be
identical, unless the controller metadata write failed, is that correct?


Thank you.
Luke

On Wed, May 11, 2022 at 5:58 AM José Armando García Sancio
 wrote:

> Thanks for your feedback Jason, much appreciated.
>
> Here are the changes to the KIP:
>
> https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=211883219=5=4
>
> On Tue, May 10, 2022 at 1:34 PM Jason Gustafson
>  wrote:
> > The approach sounds reasonable. By the way, I think one of the gaps we
> have
> > today is when the leader gets partitioned from the remaining voters. I
> > believe it continues acting as a leader indefinitely. I was considering
> > whether this periodic write can address the issue. Basically it can be
> used
> > to force a leader to prove it is still the leader by committing some
> data.
> > Say, for example, that the leader fails to commit the record after the
> > fetch timeout expires, then perhaps it could start a new election. What
> do
> > you think?
>
> We have an issue for this at
> https://issues.apache.org/jira/browse/KAFKA-13621. I updated the issue
> with your feedback and included some of my thoughts. Do you mind if we
> move this conversation to that issue?
>
> > A couple additional questions:
> >
> > - What is the default value for `metadata.monitor.write.interval.ms`?
> Also,
> > I'm wondering if `controller` would be a more suitable prefix?
>
> Yeah. I am not sure. Looking at the current configuration we have both
> prefixes. For example, with the `controller` prefix we have
> `controller.quorum.voters`, `controller.listener.names`,
> `controller.quota.window.num`, etc. For the `metadata` prefix we have
> `metadata.log.dir`, `metadata.log.*` and `metadat.max.retention.ms`,
> etc.
> I get the impression that we use `metadata` for things that are kinda
> log/disk related and `controller` for things that are not. I am
> thinking that the `metadata` prefix is more consistent with the
> current situation. What do you think Jason?
>
> > - Could we avoid letting BrokerMetadataPublisher escape into the metric
> > name? Letting the classnames leak into the metrics tends to cause
> > compatibility issues over time.
>
> Good point. For Raft we use `kafka.server:type=raft-metrics,name=...`.
> I'll change it to
> `kafka.server:type=broker-metadata-metrics,name=...`.
>
> Thanks,
> -José
>

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #922

[Apache Jenkins Server]

See