Re: [ANNOUNCE] New committer: Igor Soarez

[ziming deng]

Congrats Igor!

> On Apr 26, 2024, at 13:53, Christo Lolov  wrote:
> 
> Congratulations Igor :) !
> 
> On Thu, 25 Apr 2024 at 17:07, Igor Soarez  wrote:
> 
>> Thanks everyone, I'm very honoured to join!
>> 
>> --
>> Igor
>> 

Re: [DISCUSS] KIP-996: Pre-Vote

[ziming deng]

Hi Alyssa,

I have a minor question about the description in motivation section

> Pre-Vote (as originally detailed in the Raft paper and in KIP-650)

It seems Pre-vote is not mentioned in Raft paper, can you check out it again 
and rectify it? It would be helpful, thank you!

- 
Thanks,
Ziming


> On Dec 8, 2023, at 16:13, Luke Chen  wrote:
> 
> Hi Alyssa,
> 
> Thanks for the update.
> LGTM now.
> 
> Luke
> 
> On Fri, Dec 8, 2023 at 10:03 AM José Armando García Sancio
>  wrote:
> 
>> Hi Alyssa,
>> 
>> Thanks for the answers and the updates to the KIP. I took a look at
>> the latest version and it looks good to me.
>> 
>> --
>> -José
>> 

Re: [VOTE] KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

[ziming deng]

The KIP is accepted with 4 binding votes (Chirs, Divij Vaidya, David and Luke) 
and 4 unbinding votes(Kamal, Federico, Andrew and Николай Ижиков).
Thank you all !

> On Jan 24, 2024, at 20:44, Николай Ижиков  wrote:
> 
> +1 (non-binding)
> 
>> 24 янв. 2024 г., в 12:48, Chris Egerton  написал(а):
>> 
>> Thanks Ziming! +1 (binding)
>> 
>> On Wed, Jan 24, 2024, 03:23 ziming deng  wrote:
>> 
>>> Thank you for reminding this, David,
>>> 
>>> I have mentioned this in the [Compatibility] section as a following work.
>>> 
>>> --,
>>> Best,
>>> Ziming
>>> 
>>>> On Jan 23, 2024, at 15:17, David Jacot 
>>> wrote:
>>>> 
>>>> Hi Chris, Ziming,
>>>> 
>>>> Thanks for the clarification. I am glad that it does not impact the tool.
>>>> It may be worth adding a note about it in the KIP to avoid the same
>>>> question in the future.
>>>> 
>>>> Otherwise, I am +1 (binding). Thanks for driving this!
>>>> 
>>>> Best,
>>>> David
>>>> 
>>>> On Tue, Jan 23, 2024 at 6:07 AM ziming deng 
>>>> wrote:
>>>> 
>>>>> Hello David,
>>>>> 
>>>>> Thanks for reminding this, as Chirs explained, the tools I’m trying to
>>>>> update only support set/delete configs, and I’m just make a way for
>>>>> append/subtract configs in the future, so this would not be affected by
>>>>> KAFKA-10140, and it would be a little overkill to support
>>> append/subtract
>>>>> configs or solve KAFKA-10140 here, so let’s leave it right now, I'm
>>> happy
>>>>> to pick it after finishing this KIP.
>>>>> 
>>>>> --,
>>>>> Ziming
>>>>> 
>>>>>> On Jan 22, 2024, at 18:23, David Jacot 
>>>>> wrote:
>>>>>> 
>>>>>> Hi Ziming,
>>>>>> 
>>>>>> Thanks for driving this. I wanted to bring KAFKA-10140
>>>>>> <https://issues.apache.org/jira/browse/KAFKA-10140> to your attention.
>>>>> It
>>>>>> looks like the incremental API does not work for configuring plugins. I
>>>>>> think that we need to cover this in the KIP.
>>>>>> 
>>>>>> Best,
>>>>>> David
>>>>>> 
>>>>>> On Mon, Jan 22, 2024 at 10:13 AM Andrew Schofield <
>>>>>> andrew_schofield_j...@outlook.com> wrote:
>>>>>> 
>>>>>>> +1 (non-binding)
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Andrew
>>>>>>> 
>>>>>>>> On 22 Jan 2024, at 07:29, Federico Valeri 
>>>>> wrote:
>>>>>>>> 
>>>>>>>> +1 (non binding)
>>>>>>>> 
>>>>>>>> Thanks.
>>>>>>>> 
>>>>>>>> On Mon, Jan 22, 2024 at 7:03 AM Luke Chen  wrote:
>>>>>>>>> 
>>>>>>>>> Hi Ziming,
>>>>>>>>> 
>>>>>>>>> +1(binding) from me.
>>>>>>>>> 
>>>>>>>>> Thanks.
>>>>>>>>> Luke
>>>>>>>>> 
>>>>>>>>> On Mon, Jan 22, 2024 at 11:50 AM Kamal Chandraprakash <
>>>>>>>>> kamal.chandraprak...@gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>>> +1 (non-binding)
>>>>>>>>>> 
>>>>>>>>>> On Mon, Jan 22, 2024 at 8:34 AM ziming deng <
>>>>> dengziming1...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Hello everyone,
>>>>>>>>>>> I'd like to initiate a vote for KIP-1011.
>>>>>>>>>>> This KIP is about replacing alterConfigs with
>>>>> incrementalAlterConfigs
>>>>>>>>>>> when updating broker configs using kafka-configs.sh, this is
>>> similar
>>>>>>> to
>>>>>>>>>>> what we have done in KIP-894.
>>>>>>>>>>> 
>>>>>>>>>>> KIP link:
>>>>>>>>>>> KIP-1011: Use incrementalAlterConfigs when updating broker configs
>>>>> by
>>>>>>>>>>> kafka-configs.sh - Apache Kafka - Apache Software Foundation
>>>>>>>>>>> <
>>>>>>> 
>>>>> 
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh
>>>>>>>> 
>>>>>>>>>>> cwiki.apache.org
>>>>>>>>>>> <
>>>>>>> 
>>>>> 
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh
>>>>>>>> 
>>>>>>>>>>> [image: favicon.ico]
>>>>>>>>>>> <
>>>>>>> 
>>>>> 
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh
>>>>>>>> 
>>>>>>>>>>> <
>>>>>>> 
>>>>> 
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh
>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> Discussion thread:
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> lists.apache.org
>>>>>>>>>>> <https://lists.apache.org/thread/xd28mgqy75stgsvp6qybzpljzflkqcsy
>>>> 
>>>>>>>>>>> <https://lists.apache.org/thread/xd28mgqy75stgsvp6qybzpljzflkqcsy
>>>> 
>>>>>>>>>>> <https://lists.apache.org/thread/xd28mgqy75stgsvp6qybzpljzflkqcsy
>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> --,
>>>>>>>>>>> Best,
>>>>>>>>>>> Ziming
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>> 
>>>>> 
>>> 
>>> 
> 

Re: [VOTE] KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

[ziming deng]

Thank you for reminding this, David,

I have mentioned this in the [Compatibility] section as a following work.

--,
Best,
Ziming

> On Jan 23, 2024, at 15:17, David Jacot  wrote:
> 
> Hi Chris, Ziming,
> 
> Thanks for the clarification. I am glad that it does not impact the tool.
> It may be worth adding a note about it in the KIP to avoid the same
> question in the future.
> 
> Otherwise, I am +1 (binding). Thanks for driving this!
> 
> Best,
> David
> 
> On Tue, Jan 23, 2024 at 6:07 AM ziming deng 
> wrote:
> 
>> Hello David,
>> 
>> Thanks for reminding this, as Chirs explained, the tools I’m trying to
>> update only support set/delete configs, and I’m just make a way for
>> append/subtract configs in the future, so this would not be affected by
>> KAFKA-10140, and it would be a little overkill to support append/subtract
>> configs or solve KAFKA-10140 here, so let’s leave it right now, I'm happy
>> to pick it after finishing this KIP.
>> 
>> --,
>> Ziming
>> 
>>> On Jan 22, 2024, at 18:23, David Jacot 
>> wrote:
>>> 
>>> Hi Ziming,
>>> 
>>> Thanks for driving this. I wanted to bring KAFKA-10140
>>> <https://issues.apache.org/jira/browse/KAFKA-10140> to your attention.
>> It
>>> looks like the incremental API does not work for configuring plugins. I
>>> think that we need to cover this in the KIP.
>>> 
>>> Best,
>>> David
>>> 
>>> On Mon, Jan 22, 2024 at 10:13 AM Andrew Schofield <
>>> andrew_schofield_j...@outlook.com> wrote:
>>> 
>>>> +1 (non-binding)
>>>> 
>>>> Thanks,
>>>> Andrew
>>>> 
>>>>> On 22 Jan 2024, at 07:29, Federico Valeri 
>> wrote:
>>>>> 
>>>>> +1 (non binding)
>>>>> 
>>>>> Thanks.
>>>>> 
>>>>> On Mon, Jan 22, 2024 at 7:03 AM Luke Chen  wrote:
>>>>>> 
>>>>>> Hi Ziming,
>>>>>> 
>>>>>> +1(binding) from me.
>>>>>> 
>>>>>> Thanks.
>>>>>> Luke
>>>>>> 
>>>>>> On Mon, Jan 22, 2024 at 11:50 AM Kamal Chandraprakash <
>>>>>> kamal.chandraprak...@gmail.com> wrote:
>>>>>> 
>>>>>>> +1 (non-binding)
>>>>>>> 
>>>>>>> On Mon, Jan 22, 2024 at 8:34 AM ziming deng <
>> dengziming1...@gmail.com>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Hello everyone,
>>>>>>>> I'd like to initiate a vote for KIP-1011.
>>>>>>>> This KIP is about replacing alterConfigs with
>> incrementalAlterConfigs
>>>>>>>> when updating broker configs using kafka-configs.sh, this is similar
>>>> to
>>>>>>>> what we have done in KIP-894.
>>>>>>>> 
>>>>>>>> KIP link:
>>>>>>>> KIP-1011: Use incrementalAlterConfigs when updating broker configs
>> by
>>>>>>>> kafka-configs.sh - Apache Kafka - Apache Software Foundation
>>>>>>>> <
>>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh
>>>>> 
>>>>>>>> cwiki.apache.org
>>>>>>>> <
>>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh
>>>>> 
>>>>>>>> [image: favicon.ico]
>>>>>>>> <
>>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh
>>>>> 
>>>>>>>> <
>>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh
>>>>> 
>>>>>>>> 
>>>>>>>> Discussion thread:
>>>>>>>> 
>>>>>>>> 
>>>>>>>> lists.apache.org
>>>>>>>> <https://lists.apache.org/thread/xd28mgqy75stgsvp6qybzpljzflkqcsy>
>>>>>>>> <https://lists.apache.org/thread/xd28mgqy75stgsvp6qybzpljzflkqcsy>
>>>>>>>> <https://lists.apache.org/thread/xd28mgqy75stgsvp6qybzpljzflkqcsy>
>>>>>>>> 
>>>>>>>> 
>>>>>>>> --,
>>>>>>>> Best,
>>>>>>>> Ziming
>>>> 
>>>> 
>>>> 
>> 
>> 

Re: [VOTE] KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

[ziming deng]

Hello Ismael,
I have tested it locally to verified again that this issue only affect 
append/subtract, I also commented in the JIRA ticket to clarify this.

Best,
Ziming


> On Jan 23, 2024, at 23:07, Ismael Juma  wrote:
> 
> Hi Ziming,
> 
> Have you verified that the issue only impacts append/subtract? The way I
> read the JIRA is that the issue currently affects all operations for
> plugins, but it can be fixed to only affect append/subtract (where it would
> be harder to fix). It would be good to verify the current state.
> 
> Ismael
> 
> On Mon, Jan 22, 2024 at 9:07 PM ziming deng 
> wrote:
> 
>> Hello David,
>> 
>> Thanks for reminding this, as Chirs explained, the tools I’m trying to
>> update only support set/delete configs, and I’m just make a way for
>> append/subtract configs in the future, so this would not be affected by
>> KAFKA-10140, and it would be a little overkill to support append/subtract
>> configs or solve KAFKA-10140 here, so let’s leave it right now, I'm happy
>> to pick it after finishing this KIP.
>> 
>> --,
>> Ziming
>> 
>>> On Jan 22, 2024, at 18:23, David Jacot 
>> wrote:
>>> 
>>> Hi Ziming,
>>> 
>>> Thanks for driving this. I wanted to bring KAFKA-10140
>>> <https://issues.apache.org/jira/browse/KAFKA-10140> to your attention.
>> It
>>> looks like the incremental API does not work for configuring plugins. I
>>> think that we need to cover this in the KIP.
>>> 
>>> Best,
>>> David
>>> 
>>> On Mon, Jan 22, 2024 at 10:13 AM Andrew Schofield <
>>> andrew_schofield_j...@outlook.com> wrote:
>>> 
>>>> +1 (non-binding)
>>>> 
>>>> Thanks,
>>>> Andrew
>>>> 
>>>>> On 22 Jan 2024, at 07:29, Federico Valeri 
>> wrote:
>>>>> 
>>>>> +1 (non binding)
>>>>> 
>>>>> Thanks.
>>>>> 
>>>>> On Mon, Jan 22, 2024 at 7:03 AM Luke Chen  wrote:
>>>>>> 
>>>>>> Hi Ziming,
>>>>>> 
>>>>>> +1(binding) from me.
>>>>>> 
>>>>>> Thanks.
>>>>>> Luke
>>>>>> 
>>>>>> On Mon, Jan 22, 2024 at 11:50 AM Kamal Chandraprakash <
>>>>>> kamal.chandraprak...@gmail.com> wrote:
>>>>>> 
>>>>>>> +1 (non-binding)
>>>>>>> 
>>>>>>> On Mon, Jan 22, 2024 at 8:34 AM ziming deng <
>> dengziming1...@gmail.com>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Hello everyone,
>>>>>>>> I'd like to initiate a vote for KIP-1011.
>>>>>>>> This KIP is about replacing alterConfigs with
>> incrementalAlterConfigs
>>>>>>>> when updating broker configs using kafka-configs.sh, this is similar
>>>> to
>>>>>>>> what we have done in KIP-894.
>>>>>>>> 
>>>>>>>> KIP link:
>>>>>>>> KIP-1011: Use incrementalAlterConfigs when updating broker configs
>> by
>>>>>>>> kafka-configs.sh - Apache Kafka - Apache Software Foundation
>>>>>>>> <
>>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh
>>>>> 
>>>>>>>> cwiki.apache.org
>>>>>>>> <
>>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh
>>>>> 
>>>>>>>> [image: favicon.ico]
>>>>>>>> <
>>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh
>>>>> 
>>>>>>>> <
>>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh
>>>>> 
>>>>>>>> 
>>>>>>>> Discussion thread:
>>>>>>>> 
>>>>>>>> 
>>>>>>>> lists.apache.org
>>>>>>>> <https://lists.apache.org/thread/xd28mgqy75stgsvp6qybzpljzflkqcsy>
>>>>>>>> <https://lists.apache.org/thread/xd28mgqy75stgsvp6qybzpljzflkqcsy>
>>>>>>>> <https://lists.apache.org/thread/xd28mgqy75stgsvp6qybzpljzflkqcsy>
>>>>>>>> 
>>>>>>>> 
>>>>>>>> --,
>>>>>>>> Best,
>>>>>>>> Ziming
>>>> 
>>>> 
>>>> 
>> 
>> 

Re: [VOTE] KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

[ziming deng]

Hello David,

Thanks for reminding this, as Chirs explained, the tools I’m trying to update 
only support set/delete configs, and I’m just make a way for append/subtract 
configs in the future, so this would not be affected by KAFKA-10140, and it 
would be a little overkill to support append/subtract configs or solve 
KAFKA-10140 here, so let’s leave it right now, I'm happy to pick it after 
finishing this KIP.

--,
Ziming

> On Jan 22, 2024, at 18:23, David Jacot  wrote:
> 
> Hi Ziming,
> 
> Thanks for driving this. I wanted to bring KAFKA-10140
> <https://issues.apache.org/jira/browse/KAFKA-10140> to your attention. It
> looks like the incremental API does not work for configuring plugins. I
> think that we need to cover this in the KIP.
> 
> Best,
> David
> 
> On Mon, Jan 22, 2024 at 10:13 AM Andrew Schofield <
> andrew_schofield_j...@outlook.com> wrote:
> 
>> +1 (non-binding)
>> 
>> Thanks,
>> Andrew
>> 
>>> On 22 Jan 2024, at 07:29, Federico Valeri  wrote:
>>> 
>>> +1 (non binding)
>>> 
>>> Thanks.
>>> 
>>> On Mon, Jan 22, 2024 at 7:03 AM Luke Chen  wrote:
>>>> 
>>>> Hi Ziming,
>>>> 
>>>> +1(binding) from me.
>>>> 
>>>> Thanks.
>>>> Luke
>>>> 
>>>> On Mon, Jan 22, 2024 at 11:50 AM Kamal Chandraprakash <
>>>> kamal.chandraprak...@gmail.com> wrote:
>>>> 
>>>>> +1 (non-binding)
>>>>> 
>>>>> On Mon, Jan 22, 2024 at 8:34 AM ziming deng 
>>>>> wrote:
>>>>> 
>>>>>> Hello everyone,
>>>>>> I'd like to initiate a vote for KIP-1011.
>>>>>> This KIP is about replacing alterConfigs with incrementalAlterConfigs
>>>>>> when updating broker configs using kafka-configs.sh, this is similar
>> to
>>>>>> what we have done in KIP-894.
>>>>>> 
>>>>>> KIP link:
>>>>>> KIP-1011: Use incrementalAlterConfigs when updating broker configs by
>>>>>> kafka-configs.sh - Apache Kafka - Apache Software Foundation
>>>>>> <
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh
>>> 
>>>>>> cwiki.apache.org
>>>>>> <
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh
>>> 
>>>>>> [image: favicon.ico]
>>>>>> <
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh
>>> 
>>>>>> <
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh
>>> 
>>>>>> 
>>>>>> Discussion thread:
>>>>>> 
>>>>>> 
>>>>>> lists.apache.org
>>>>>> <https://lists.apache.org/thread/xd28mgqy75stgsvp6qybzpljzflkqcsy>
>>>>>> <https://lists.apache.org/thread/xd28mgqy75stgsvp6qybzpljzflkqcsy>
>>>>>> <https://lists.apache.org/thread/xd28mgqy75stgsvp6qybzpljzflkqcsy>
>>>>>> 
>>>>>> 
>>>>>> --,
>>>>>> Best,
>>>>>> Ziming
>> 
>> 
>> 

[VOTE] KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

[ziming deng]

Hello everyone, 
I'd like to initiate a vote for KIP-1011.
This KIP is about replacing alterConfigs with incrementalAlterConfigs when 
updating broker configs using kafka-configs.sh, this is similar to what we have 
done in KIP-894.

KIP link:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh
Discussion thread:


https://lists.apache.org/thread/xd28mgqy75stgsvp6qybzpljzflkqcsy

--,
Best,
Ziming

Re: DISCUSS KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

[ziming deng]

Hello Luke,

thank you for finding this error, I have rectified it, and I will start a vote 
process soon.

--
Best,
Ziming


> On Jan 12, 2024, at 16:32, Luke Chen  wrote:
> 
> Hi Ziming,
> 
> Thanks for the KIP!
> LGTM!
> Using incremental by defaul and fallback automatically if it's not
> supported is a good idea!
> 
> One minor comment:
> 1. so I'm inclined to move it to incrementalAlterConfigs  and "provide a
> flag" to still use alterConfigs  for new client to interact with old
> servers.
> I don't think we will "provide any flag" after the discussion. We should
> remove it.
> 
> Thanks.
> Luke
> 
> On Fri, Jan 12, 2024 at 12:29 PM ziming deng  <mailto:dengziming1...@gmail.com>>
> wrote:
> 
>> Thank you for your clarification, Chris,
>> 
>> I have spent some time to review KIP-894 and I think it's automatic way is
>> better and bring no side effect, and I will also adopt this way here.
>> As you mentioned, the changes in semantics is minor, the most important
>> reason for this change is fixing bug brought by sensitive configs.
>> 
>> 
>>> We
>>> don't appear to support appending/subtracting from list properties via
>> the
>>> CLI for any other entity type right now,
>> You are right about this, I tried and found that we can’t subtract or
>> append configs, I will change the KIP to "making way for
>> appending/subtracting list properties"
>> 
>> --
>> Best,
>> Ziming
>> 
>>> On Jan 6, 2024, at 01:34, Chris Egerton  wrote:
>>> 
>>> Hi all,
>>> 
>>> Can we clarify any changes in the user-facing semantics for the CLI tool
>>> that would come about as a result of this KIP? I think the debate over
>> the
>>> necessity of an opt-in flag, or waiting for 4.0.0, ultimately boils down
>> to
>>> this.
>>> 
>>> My understanding is that the only changes in semantics are fairly minor
>>> (semantic versioning pun intended):
>>> 
>>> - Existing sensitive broker properties no longer have to be explicitly
>>> specified on the command line if they're not being changed
>>> - A small race condition is fixed where the broker config is updated by a
>>> separate operation in between when the CLI reads the existing broker
>> config
>>> and writes the new broker config
>>> - Usage of a new broker API that has been supported since version 2.3.0,
>>> but which does not require any new ACLs and does not act any differently
>>> apart from the two small changes noted above
>>> 
>>> If this is correct, then I'm inclined to agree with Ismael's suggestion
>> of
>>> starting with incrementalAlterConfigs, and falling back on alterConfigs
>> if
>>> the former is not supported by the broker, and do not believe it's
>>> necessary to wait for 4.0.0 or provide opt-in or opt-out flags to release
>>> this change. This would also be similar to changes we made to
>> MirrorMaker 2
>>> in KIP-894 [1], where the default behavior for syncing topic configs is
>> now
>>> to start with incrementalAlterConfigs and fall back on alterConfigs if
>> it's
>>> not supported.
>>> 
>>> If there are other, more significant changes to the user-facing semantics
>>> for the CLI, then these should be called out here and in the KIP, and we
>>> might consider a more cautious approach.
>>> 
>>> [1] -
>>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-894%3A+Use+incrementalAlterConfigs+API+for+syncing+topic+configurations
>>> 
>>> 
>>> Also, regarding this part of the KIP:
>>> 
>>>> incrementalAlterConfigs is more convenient especially for updating
>>> configs of list data type, such as
>> "leader.replication.throttled.replicas"
>>> 
>>> While this is true for the Java admin client and the corresponding broker
>>> APIs, it doesn't appear to be relevant to the kafka-configs.sh CLI tool.
>> We
>>> don't appear to support appending/subtracting from list properties via
>> the
>>> CLI for any other entity type right now, and there's nothing in the KIP
>>> that leads me to believe we'd be adding it for broker configs.
>>> 
>>> Cheers,
>>> 
>>> Chris
>>> 
>>> On Thu, Jan 4, 2024 at 10:12 PM ziming deng > <mailto:dengziming1...@gmail.com>>
>>> wrote:
>>> 
>>>> Hi Ismael,
>>>> I added this automatically appro

Re: DISCUSS KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

[ziming deng]

Thank you for your clarification, Chris,

I have spent some time to review KIP-894 and I think it's automatic way is 
better and bring no side effect, and I will also adopt this way here.
As you mentioned, the changes in semantics is minor, the most important reason 
for this change is fixing bug brought by sensitive configs.


>  We
> don't appear to support appending/subtracting from list properties via the
> CLI for any other entity type right now,
You are right about this, I tried and found that we can’t subtract or append 
configs, I will change the KIP to "making way for appending/subtracting list 
properties"

--
Best,
Ziming

> On Jan 6, 2024, at 01:34, Chris Egerton  wrote:
> 
> Hi all,
> 
> Can we clarify any changes in the user-facing semantics for the CLI tool
> that would come about as a result of this KIP? I think the debate over the
> necessity of an opt-in flag, or waiting for 4.0.0, ultimately boils down to
> this.
> 
> My understanding is that the only changes in semantics are fairly minor
> (semantic versioning pun intended):
> 
> - Existing sensitive broker properties no longer have to be explicitly
> specified on the command line if they're not being changed
> - A small race condition is fixed where the broker config is updated by a
> separate operation in between when the CLI reads the existing broker config
> and writes the new broker config
> - Usage of a new broker API that has been supported since version 2.3.0,
> but which does not require any new ACLs and does not act any differently
> apart from the two small changes noted above
> 
> If this is correct, then I'm inclined to agree with Ismael's suggestion of
> starting with incrementalAlterConfigs, and falling back on alterConfigs if
> the former is not supported by the broker, and do not believe it's
> necessary to wait for 4.0.0 or provide opt-in or opt-out flags to release
> this change. This would also be similar to changes we made to MirrorMaker 2
> in KIP-894 [1], where the default behavior for syncing topic configs is now
> to start with incrementalAlterConfigs and fall back on alterConfigs if it's
> not supported.
> 
> If there are other, more significant changes to the user-facing semantics
> for the CLI, then these should be called out here and in the KIP, and we
> might consider a more cautious approach.
> 
> [1] -
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-894%3A+Use+incrementalAlterConfigs+API+for+syncing+topic+configurations
> 
> 
> Also, regarding this part of the KIP:
> 
>> incrementalAlterConfigs is more convenient especially for updating
> configs of list data type, such as "leader.replication.throttled.replicas"
> 
> While this is true for the Java admin client and the corresponding broker
> APIs, it doesn't appear to be relevant to the kafka-configs.sh CLI tool. We
> don't appear to support appending/subtracting from list properties via the
> CLI for any other entity type right now, and there's nothing in the KIP
> that leads me to believe we'd be adding it for broker configs.
> 
> Cheers,
> 
> Chris
> 
> On Thu, Jan 4, 2024 at 10:12 PM ziming deng  <mailto:dengziming1...@gmail.com>>
> wrote:
> 
>> Hi Ismael,
>> I added this automatically approach to “Rejected alternatives” concerning
>> that we need to unify the semantics between alterConfigs and
>> incrementalAlterConfigs, so I choose to give this privilege to users.
>> 
>> After reviewing these code and doing some tests I found that they
>> following the similar approach, I think the simplest way is to let the
>> client choose the best method heuristically.
>> 
>> Thank you for pointing out this, I will change the KIP later.
>> 
>> Best,
>> Ziming
>> 
>>> On Jan 4, 2024, at 17:28, Ismael Juma  wrote:
>>> 
>>> Hi Ziming,
>>> 
>>> Why is the flag required at all? Can we use incremental and fallback
>> automatically if it's not supported by the broker? At this point, the vast
>> majority of clusters should support it.
>>> 
>>> Ismael
>>> 
>>> On Mon, Dec 18, 2023 at 7:58 PM ziming deng > <mailto:dengziming1...@gmail.com>> wrote:
>>>> 
>>>> Hello, I want to start a discussion on KIP-1011, to make the broker
>> config change path unified with that of user/topic/client-metrics and avoid
>> some bugs.
>>>> 
>>>> Here is the link:
>>>> 
>>>> KIP-1011: Use incrementalAlterConfigs when updating broker configs by
>> kafka-configs.sh - Apache Kafka - Apache Software Foundation
>>>> cwiki.apache.org <http://cwiki.apache.org/>
>>>> 
>>>&g

Re: DISCUSS KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

[ziming deng]

Hi Ismael,
I added this automatically approach to “Rejected alternatives” concerning that 
we need to unify the semantics between alterConfigs and 
incrementalAlterConfigs, so I choose to give this privilege to users.

After reviewing these code and doing some tests I found that they following the 
similar approach, I think the simplest way is to let the client choose the best 
method heuristically.

Thank you for pointing out this, I will change the KIP later.

Best,
Ziming

> On Jan 4, 2024, at 17:28, Ismael Juma  wrote:
> 
> Hi Ziming,
> 
> Why is the flag required at all? Can we use incremental and fallback 
> automatically if it's not supported by the broker? At this point, the vast 
> majority of clusters should support it.
> 
> Ismael
> 
> On Mon, Dec 18, 2023 at 7:58 PM ziming deng  <mailto:dengziming1...@gmail.com>> wrote:
>> 
>> Hello, I want to start a discussion on KIP-1011, to make the broker config 
>> change path unified with that of user/topic/client-metrics and avoid some 
>> bugs.
>> 
>> Here is the link: 
>> 
>> KIP-1011: Use incrementalAlterConfigs when updating broker configs by 
>> kafka-configs.sh - Apache Kafka - Apache Software Foundation
>> cwiki.apache.org
>> 
>>  
>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh>KIP-1011:
>>  Use incrementalAlterConfigs when updating broker configs by 
>> kafka-configs.sh - Apache Kafka - Apache Software Foundation 
>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh>
>> cwiki.apache.org 
>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh>
>>  
>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh>
>> 
>> Best, 
>> Ziming.

Re: DISCUSS KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

[ziming deng]

Hello all,

We have 2 people prefer using "--enable-incremental" and it makes sense to move 
the incompatible process to 4.X, I changed the default way to still use 
`alterConfigs` and use `incrementalAlterConfigs` only when 
"--enable-incremental" is specified.

I will initiate a vote process if no further opinions coming.

Best,
Ziming


> On Dec 26, 2023, at 13:21, Kamal Chandraprakash 
>  wrote:
> 
> Hi Ziming,
> 
> Thanks for the KIP! The proposal LGTM.
> 
> I'm also inclined towards option 2 (i.e. add an explicit
> --enable-incremental flag in 3.X version)
> to avoid any incompatible change in v3.X. As mentioned in this thread, many
> users might be
> using external tools to do the topic rebalance and apply throttle config on
> the brokers, it might
> break for them when the tools are upgraded before the broker binary.
> 
> cruise-control is currently using kafka-version: 3.1
> https://github.com/linkedin/cruise-control/blob/migrate_to_kafka_2_5/gradle.properties#L5
> 
> On Mon, Dec 25, 2023 at 7:15 PM Divij Vaidya 
> wrote:
> 
>> Thank you for the summary, Ziming.
>> 
>> Personally, I would prefer the latter i.e. having the incompatible change
>> in 4.x instead of 3.x. This is because a major version upgrade goes through
>> additional scrutiny by the users and usually comes with inevitable code
>> changes required on the client. Hence, this incompatibility will be part of
>> one amongst many changes that users will perform to upgrade to 4.x. This is
>> unlike a major version change from 3.7 to 3.8 where users expect a simple
>> upgrade without any code changes.
>> 
>> Let's wait and hear what others think about this.
>> 
>> --
>> Divij Vaidya
>> 
>> 
>> 
>> On Mon, Dec 25, 2023 at 1:18 PM ziming deng 
>> wrote:
>> 
>>> Hello Divij Vaidya,
>>> 
>>> You are right that users should update existing scripts to add
>>> ‘—disable-incremental’, and you mentioned another upgrade path which is
>>> similar, the summary of the 2 schemes are:
>>> we change existing scripts to use `incrementalAlterConfigs` and add
>>> "--disable-incremental" flag for old servers in Kafka 3.X, and remove it
>> in
>>> Kafka 4.X.
>>> we keep existing scripts unchanged and add an "--enable-incremental" flag
>>> for new servers in Kafka 3.X, and remove it in Kafka 4.X.
>>> 
>>> I think there will always be an incompatible upgrade process to move
>>> default behavior from `alterConfigs` to `incrementalConfigs`. In the
>> first
>>> scheme we are doing this incompatible upgrade in Kafka 3.X, and in the
>>> second scheme we are moving it to 4.X, I think we should do it as soon as
>>> possible if it's inevitable.
>>> However, I will add this to , and I'm open to this
>>> if more people think it's more suitable.
>>> 
>>> 
>>> --,
>>> Ziming
>>> 
>>>> On Dec 22, 2023, at 18:13, Divij Vaidya 
>> wrote:
>>>> 
>>>> Divij Vaidya
>>> 
>>> 
>> 

Re: [ANNOUNCE] New Kafka PMC Member: Divij Vaidya

[ziming deng]

Congrats Divij!


> On Dec 28, 2023, at 10:59, Satish Duggana  wrote:
> 
> Congratulations Divij!
> 
> On Thu, 28 Dec 2023 at 07:21, Kamal Chandraprakash
>  wrote:
>> 
>> Congrats Divij!
>> 
>> On Thu, Dec 28, 2023, 07:09 Kirk True  wrote:
>> 
>>> Congrats Divij!!!
>>> 
>>> On Wed, Dec 27, 2023, at 1:44 PM, Jorge Esteban Quilcate Otoya wrote:
 Congratulations Divij!!
 
 On Wed 27. Dec 2023 at 14.56, Tom Bentley  wrote:
 
> Congratulations!
> 
> On Thu, 28 Dec 2023 at 06:17, Philip Nee  wrote:
> 
>> congrats divij!
>> 
>> On Wed, Dec 27, 2023 at 8:55 AM Justine Olshan
>> 
>> wrote:
>> 
>>> Congratulations Divij!
>>> 
>>> On Wed, Dec 27, 2023 at 4:20 AM Gaurav Narula 
> wrote:
>>> 
 Congratulations Divij!
 
 Regards,
 Gaurav
 
> On 27-Dec-2023, at 17:44, Mickael Maison <
>>> mickael.mai...@gmail.com
>> 
 wrote:
> 
> Congratulations Divij!
> 
>> On Wed, Dec 27, 2023 at 1:05 PM Sagar <
>>> sagarmeansoc...@gmail.com>
 wrote:
>> 
>> Congrats Divij! Absolutely well deserved !
>> 
>> Thanks!
>> Sagar.
>> 
>>> On Wed, Dec 27, 2023 at 5:15 PM Luke Chen >>> 
>> wrote:
>>> 
>>> Hi, Everyone,
>>> 
>>> Divij has been a Kafka committer since June, 2023. He has
> remained
>>> very
>>> active and instructive in the community since becoming a
> committer.
 It's my
>>> pleasure to announce that Divij is now a member of Kafka PMC.
>>> 
>>> Congratulations Divij!
>>> 
>>> Luke
>>> on behalf of Apache Kafka PMC
>>> 
 
>>> 
>> 
> 
 
>>> 

Re: DISCUSS KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

[ziming deng]

Hello Divij Vaidya,

You are right that users should update existing scripts to add 
‘—disable-incremental’, and you mentioned another upgrade path which is 
similar, the summary of the 2 schemes are: 
we change existing scripts to use `incrementalAlterConfigs` and add 
"--disable-incremental" flag for old servers in Kafka 3.X, and remove it in 
Kafka 4.X.
we keep existing scripts unchanged and add an "--enable-incremental" flag for 
new servers in Kafka 3.X, and remove it in Kafka 4.X.

I think there will always be an incompatible upgrade process to move default 
behavior from `alterConfigs` to `incrementalConfigs`. In the first scheme we 
are doing this incompatible upgrade in Kafka 3.X, and in the second scheme we 
are moving it to 4.X, I think we should do it as soon as possible if it's 
inevitable.
However, I will add this to , and I'm open to this if 
more people think it's more suitable. 


--,
Ziming

> On Dec 22, 2023, at 18:13, Divij Vaidya  wrote:
> 
> Divij Vaidya

Re: DISCUSS KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

[ziming deng]

+1 for adding them to rejected alternatives, These kafka-ui tools should also 
evolve with the iterations of Kafka.

> On Dec 21, 2023, at 16:58, Николай Ижиков  wrote:
> 
>> In fact alterConfig and incrementalAlterConfig have different semantics, we 
>> should pass all configs when using alterConfig and we can update config  
>> incrementally using incrementalAlterConfigs, and is’t not worth doing so 
>> since alterConfig has been deprecated for a long time.
> 
> There can be third-party tools like `kafka-ui` or similar that suffer from 
> the same bug as you fixing.
> If we fix `alterConfig` itself then we fix all tools, scripts that still 
> using alterConfig.
> 
> Anyway, let’s add to the «Rejected alternatives» section reasons - why we 
> keep buggy method as is and fixing only tools.
> 
>> I think your suggestion is nice, it should be marked as deprecated and will 
>> be removed together with `AdminClient.alterConfigs()`
> 
> Is it OK to introduce option that is deprecated from the beginning?
> 
> 
>> 21 дек. 2023 г., в 06:03, ziming deng > <mailto:dengziming1...@gmail.com>> написал(а):
>> 
>>> shouldn't we also introduce --disable-incremental as deprecated?
>> 
>> I think your suggestion is nice, it should be marked as deprecated and will 
>> be removed together with `AdminClient.alterConfigs()`
>> 
>> 
>>> On Dec 19, 2023, at 16:36, Federico Valeri  wrote:
>>> 
>>> HI Ziming, thanks for the KIP. Looks good to me.
>>> 
>>> Just on question: given that alterConfig is deprecated, shouldn't we
>>> also introduce --disable-incremental as deprecated? That way we would
>>> get rid of both in Kafka 4.0. Also see:
>>> https://issues.apache.org/jira/browse/KAFKA-14705.
>>> 
>>> On Tue, Dec 19, 2023 at 9:05 AM ziming deng >> <mailto:dengziming1...@gmail.com><mailto:dengziming1...@gmail.com>> wrote:
>>>> 
>>>> Thank you for mention this Ismael,
>>>> 
>>>> I added this to the motivation section, and I think we can still update 
>>>> configs in this case by passing all sensitive configs, which is weird and 
>>>> not friendly.
>>>> 
>>>> --
>>>> Best,
>>>> Ziming
>>>> 
>>>>> On Dec 19, 2023, at 14:24, Ismael Juma >>>> <mailto:m...@ismaeljuma.com>> wrote:
>>>>> 
>>>>> Thanks for the KIP. I think one of the main benefits of the change isn't 
>>>>> listed: sensitive configs make it impossible to make updates with the 
>>>>> current cli tool because sensitive config values are never returned.
>>>>> 
>>>>> Ismael
>>>>> 
>>>>> On Mon, Dec 18, 2023 at 7:58 PM ziming deng >>>> <mailto:dengziming1...@gmail.com><mailto:dengziming1...@gmail.com> 
>>>>> <mailto:dengziming1...@gmail.com>> wrote:
>>>>>> 
>>>>>> Hello, I want to start a discussion on KIP-1011, to make the broker 
>>>>>> config change path unified with that of user/topic/client-metrics and 
>>>>>> avoid some bugs.
>>>>>> 
>>>>>> Here is the link:
>>>>>> 
>>>>>> KIP-1011: Use incrementalAlterConfigs when updating broker configs by 
>>>>>> kafka-configs.sh - Apache Kafka - Apache Software Foundation
>>>>>> cwiki.apache.org <http://cwiki.apache.org/> <http://cwiki.apache.org/>
>>>>>> 
>>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh>KIP-1011:
>>>>>>  Use incrementalAlterConfigs when updating broker configs by 
>>>>>> kafka-configs.sh - Apache Kafka - Apache Software Foundation 
>>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh>
>>>>>> cwiki.apache.org <http://cwiki.apache.org/> <http://cwiki.apache.org/> 
>>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh>
>>>>>>  
>>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh>
>>>>>> 
>>>>>> Best,
>>>>>> Ziming.

Re: DISCUSS KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

[ziming deng]

> shouldn't we also introduce --disable-incremental as deprecated? 

I think your suggestion is nice, it should be marked as deprecated and will be 
removed together with `AdminClient.alterConfigs()`


> On Dec 19, 2023, at 16:36, Federico Valeri  wrote:
> 
> HI Ziming, thanks for the KIP. Looks good to me.
> 
> Just on question: given that alterConfig is deprecated, shouldn't we
> also introduce --disable-incremental as deprecated? That way we would
> get rid of both in Kafka 4.0. Also see:
> https://issues.apache.org/jira/browse/KAFKA-14705.
> 
> On Tue, Dec 19, 2023 at 9:05 AM ziming deng  <mailto:dengziming1...@gmail.com>> wrote:
>> 
>> Thank you for mention this Ismael,
>> 
>> I added this to the motivation section, and I think we can still update 
>> configs in this case by passing all sensitive configs, which is weird and 
>> not friendly.
>> 
>> --
>> Best,
>> Ziming
>> 
>>> On Dec 19, 2023, at 14:24, Ismael Juma  wrote:
>>> 
>>> Thanks for the KIP. I think one of the main benefits of the change isn't 
>>> listed: sensitive configs make it impossible to make updates with the 
>>> current cli tool because sensitive config values are never returned.
>>> 
>>> Ismael
>>> 
>>> On Mon, Dec 18, 2023 at 7:58 PM ziming deng >> <mailto:dengziming1...@gmail.com> <mailto:dengziming1...@gmail.com>> wrote:
>>>> 
>>>> Hello, I want to start a discussion on KIP-1011, to make the broker config 
>>>> change path unified with that of user/topic/client-metrics and avoid some 
>>>> bugs.
>>>> 
>>>> Here is the link:
>>>> 
>>>> KIP-1011: Use incrementalAlterConfigs when updating broker configs by 
>>>> kafka-configs.sh - Apache Kafka - Apache Software Foundation
>>>> cwiki.apache.org <http://cwiki.apache.org/>
>>>> 
>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh>KIP-1011:
>>>>  Use incrementalAlterConfigs when updating broker configs by 
>>>> kafka-configs.sh - Apache Kafka - Apache Software Foundation 
>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh>
>>>> cwiki.apache.org <http://cwiki.apache.org/> 
>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh>
>>>>  
>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh>
>>>> 
>>>> Best,
>>>> Ziming.

Re: DISCUSS KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

[ziming deng]

Hello, thanks for mention this.

In fact alterConfig and incrementalAlterConfig have different semantics, we 
should pass all configs when using alterConfig and we can update config  
incrementally using incrementalAlterConfigs, and is’t not worth doing so since 
alterConfig has been deprecated for a long time.

--,
Best, 
Ziming

> On Dec 19, 2023, at 16:40, Николай Ижиков  wrote:
> 
> Hello.
> 
> Can we just forward all invocations of alterConfig to incrementalAlterConfig?
> Transform parameters and call correct method.
> 
>> 19 дек. 2023 г., в 11:36, Federico Valeri  написал(а):
>> 
>> HI Ziming, thanks for the KIP. Looks good to me.
>> 
>> Just on question: given that alterConfig is deprecated, shouldn't we
>> also introduce --disable-incremental as deprecated? That way we would
>> get rid of both in Kafka 4.0. Also see:
>> https://issues.apache.org/jira/browse/KAFKA-14705.
>> 
>> On Tue, Dec 19, 2023 at 9:05 AM ziming deng  wrote:
>>> 
>>> Thank you for mention this Ismael,
>>> 
>>> I added this to the motivation section, and I think we can still update 
>>> configs in this case by passing all sensitive configs, which is weird and 
>>> not friendly.
>>> 
>>> --
>>> Best,
>>> Ziming
>>> 
>>>> On Dec 19, 2023, at 14:24, Ismael Juma  wrote:
>>>> 
>>>> Thanks for the KIP. I think one of the main benefits of the change isn't 
>>>> listed: sensitive configs make it impossible to make updates with the 
>>>> current cli tool because sensitive config values are never returned.
>>>> 
>>>> Ismael
>>>> 
>>>> On Mon, Dec 18, 2023 at 7:58 PM ziming deng >>> <mailto:dengziming1...@gmail.com>> wrote:
>>>>> 
>>>>> Hello, I want to start a discussion on KIP-1011, to make the broker 
>>>>> config change path unified with that of user/topic/client-metrics and 
>>>>> avoid some bugs.
>>>>> 
>>>>> Here is the link:
>>>>> 
>>>>> KIP-1011: Use incrementalAlterConfigs when updating broker configs by 
>>>>> kafka-configs.sh - Apache Kafka - Apache Software Foundation
>>>>> cwiki.apache.org
>>>>> 
>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh>KIP-1011:
>>>>>  Use incrementalAlterConfigs when updating broker configs by 
>>>>> kafka-configs.sh - Apache Kafka - Apache Software Foundation 
>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh>
>>>>> cwiki.apache.org 
>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh>
>>>>>  
>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh>
>>>>> 
>>>>> Best,
>>>>> Ziming.
>>> 
> 

Re: DISCUSS KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

[ziming deng]

Thank you for mention this Ismael,

I added this to the motivation section, and I think we can still update configs 
in this case by passing all sensitive configs, which is weird and not friendly.

--
Best,
Ziming

> On Dec 19, 2023, at 14:24, Ismael Juma  wrote:
> 
> Thanks for the KIP. I think one of the main benefits of the change isn't 
> listed: sensitive configs make it impossible to make updates with the current 
> cli tool because sensitive config values are never returned.
> 
> Ismael
> 
> On Mon, Dec 18, 2023 at 7:58 PM ziming deng  <mailto:dengziming1...@gmail.com>> wrote:
>> 
>> Hello, I want to start a discussion on KIP-1011, to make the broker config 
>> change path unified with that of user/topic/client-metrics and avoid some 
>> bugs.
>> 
>> Here is the link: 
>> 
>> KIP-1011: Use incrementalAlterConfigs when updating broker configs by 
>> kafka-configs.sh - Apache Kafka - Apache Software Foundation
>> cwiki.apache.org
>> 
>>  
>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh>KIP-1011:
>>  Use incrementalAlterConfigs when updating broker configs by 
>> kafka-configs.sh - Apache Kafka - Apache Software Foundation 
>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh>
>> cwiki.apache.org 
>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh>
>>  
>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh>
>> 
>> Best, 
>> Ziming.

DISCUSS KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

[ziming deng]

Hello, I want to start a discussion on KIP-1011, to make the broker config 
change path unified with that of user/topic/client-metrics and avoid some bugs.

Here is the link: 

https://cwiki.apache.org/confluence/display/KAFKA/KIP-1011%3A+Use+incrementalAlterConfigs+when+updating+broker+configs+by+kafka-configs.sh

Best, 
Ziming.

Re: Remaining tests that need to support KRaft

[ziming deng]

Hello Sameer, I have created a PR for some test in `kafka.api` and 
`kafka.network`, you can take a review when you are free, thanks.


https://github.com/apache/kafka/pull/14595
MINOR: Enable kraft test in kafka.api and kafka.network by dengziming · Pull 
Request #14595 · apache/kafka
github.com


> On Oct 30, 2023, at 07:26, Sameer Tejani  wrote:
> 
> Hi everyone,
> 
> I worked with Colin who had taken an initial pass at tests that still need
> to be converted to support KRaft.  I created individual Jiras
> 
> for them and have marked them with labels kraft-test.  Some of them should
> be simple enough to implement.  We will need to have them all converted
> before AK 4.0 is released.
> 
> Thx
> 
> -- 
> - Sameer

Re: [ANNOUNCE] New Kafka PMC Member: Satish Duggana

[ziming deng]

Congratulations Satish!

> On Oct 27, 2023, at 23:03, Jun Rao  wrote:
> 
> Hi, Everyone,
> 
> Satish Duggana has been a Kafka committer since 2022. He has been very
> instrumental to the community since becoming a committer. It's my pleasure
> to announce that Satish is now a member of Kafka PMC.
> 
> Congratulations Satish!
> 
> Jun
> on behalf of Apache Kafka PMC

Re: [ANNOUNCE] New Kafka PMC Member: Justine Olshan

[ziming deng]

Congratulations Justine!


> On Sep 25, 2023, at 00:01, Viktor Somogyi-Vass 
>  wrote:
> 
> Congrats Justine!
> 
> On Sun, Sep 24, 2023, 17:45 Kirk True  wrote:
> 
>> Congratulations Justine! Thanks for all your great work!
>> 
>>> On Sep 24, 2023, at 8:37 AM, John Roesler  wrote:
>>> 
>>> Congratulations, Justine!
>>> -John
>>> 
>>> On Sun, Sep 24, 2023, at 05:05, Mickael Maison wrote:
 Congratulations Justine!
 
 On Sun, Sep 24, 2023 at 5:04 AM Sophie Blee-Goldman
  wrote:
> 
> Congrats Justine!
> 
> On Sat, Sep 23, 2023, 4:36 PM Tom Bentley  wrote:
> 
>> Congratulations!
>> 
>> On Sun, 24 Sept 2023 at 12:32, Satish Duggana <
>> satish.dugg...@gmail.com>
>> wrote:
>> 
>>> Congratulations Justine!!
>>> 
>>> On Sat, 23 Sept 2023 at 15:46, Bill Bejeck 
>> wrote:
 
 Congrats Justine!
 
 -Bill
 
 On Sat, Sep 23, 2023 at 6:23 PM Greg Harris
>> >>> 
 wrote:
 
> Congratulations Justine!
> 
> On Sat, Sep 23, 2023 at 5:49 AM Boudjelda Mohamed Said
>  wrote:
>> 
>> Congrats Justin !
>> 
>> On Sat 23 Sep 2023 at 14:44, Randall Hauch 
>> wrote:
>> 
>>> Congratulations, Justine!
>>> 
>>> On Sat, Sep 23, 2023 at 4:25 AM Kamal Chandraprakash <
>>> kamal.chandraprak...@gmail.com> wrote:
>>> 
 Congrats Justine!
 
 On Sat, Sep 23, 2023, 13:28 Divij Vaidya <
>>> divijvaidy...@gmail.com>
>>> wrote:
 
> Congratulations Justine!
> 
> On Sat 23. Sep 2023 at 07:06, Chris Egerton <
> fearthecel...@gmail.com>
> wrote:
> 
>> Congrats Justine!
>> On Fri, Sep 22, 2023, 20:47 Guozhang Wang <
>>> guozhang.wang...@gmail.com>
>> wrote:
>> 
>>> Congratulations!
>>> 
>>> On Fri, Sep 22, 2023 at 8:44 PM Tzu-Li (Gordon) Tai <
> tzuli...@apache.org
>>> 
>>> wrote:
 
 Congratulations Justine!
 
 On Fri, Sep 22, 2023, 19:25 Philip Nee <
>>> philip...@gmail.com>
 wrote:
 
> Congrats Justine!
> 
> On Fri, Sep 22, 2023 at 7:07 PM Luke Chen <
> show...@gmail.com>
> wrote:
> 
>> Hi, Everyone,
>> 
>> Justine Olshan has been a Kafka committer since
>> Dec.
> 2022.
>>> She
> has
>>> been
>> very active and instrumental to the community since
> becoming
>>> a
>>> committer.
>> It's my pleasure to announce that Justine is now a
> member of
> Kafka
>>> PMC.
>> 
>> Congratulations Justine!
>> 
>> Luke
>> on behalf of Apache Kafka PMC
>> 
> 
>>> 
>> 
> 
 
>>> 
> 
>>> 
>>> 
>> 
>> 
>> 

Re: Unable to start the Kafka with Kraft in Windows 11

[ziming deng]

It seems this is related to KAFKA-14273, there is already a pr for this 
problem, but it’s not merged.
 https://github.com/apache/kafka/pull/12763

--
Ziming

> On Sep 6, 2023, at 07:25, Greg Harris  wrote:
> 
> Hey Sumanshu,
> 
> Thanks for trying out Kraft! I hope that you can get it working :)
> 
> I am not familiar with Kraft or Windows, but the error appears to
> mention that the file is already in use by another process so maybe we
> can start there.
> 
> 1. Have you verified that no other Kafka processes are running, such
> as in the background or in another terminal?
> 2. Are you setting up multiple Kafka brokers on the same machine in your test?
> 3. Do you see the error if you restart your machine before starting Kafka?
> 4. Do you see the error if you delete the log directory and format it
> again before starting Kafka?
> 5. Have you made any changes to the `server.properties`, such as
> changing the log directories? (I see that the default is
> `/tmp/kraft-combined-logs`, I don't know if that is a valid path for
> Windows).
> 
> Thanks,
> Greg
> 
> On Mon, Sep 4, 2023 at 2:21 PM Sumanshu Nankana
>  wrote:
>> 
>> Hi Team,
>> 
>> I am following the steps mentioned here https://kafka.apache.org/quickstart 
>> to Install the Kafka.
>> 
>> Windows 11
>> Kafka Version 
>> https://www.apache.org/dyn/closer.cgi?path=/kafka/3.5.0/kafka_2.13-3.5.0.tgz
>> 64 Bit Operating System
>> 
>> 
>> Step1: Generate the Cluster UUID
>> 
>> $KAFKA_CLUSTER_ID=.\bin\windows\kafka-storage.bat random-uuid
>> 
>> Step2: Format Log Directories
>> 
>> .\bin\windows\kafka-storage.bat format -t $KAFKA_CLUSTER_ID -c 
>> .\config\kraft\server.properties
>> 
>> Step3: Start the Kafka Server
>> 
>> .\bin\windows\kafka-server-start.bat .\config\kraft\server.properties
>> 
>> I am getting the error. Logs are attached
>> 
>> Could you please help me to sort this error.
>> 
>> Kindly let me know, if you need any more information.
>> 
>> -
>> Best
>> Sumanshu Nankana
>> 
>> 

Re: [VOTE] KIP-858: Handle JBOD broker disk failure in KRaft

[ziming deng]

Hi, Igor
I’m +1(binding) for this, looking forward the PR.

--
Best,
Ziming

> On Jul 26, 2023, at 01:13, Igor Soarez  wrote:
> 
> Hi everyone,
> 
> Following a face-to-face discussion with Ron and Colin,
> I have just made further improvements to this KIP:
> 
> 
> 1. Every log directory gets a random UUID assigned, even if just one
>   log dir is configured in the Broker.
> 
> 2. All online log directories are registered, even if just one if configured.
> 
> 3. Partition-to-directory assignments are only performed if more than
>   one log directory is configured/registered.
> 
> 4. A successful reply from the Controller to a AssignReplicasToDirsRequest
>   is taken as a guarantee that the metadata changes are
>   successfully persisted.
> 
> 5. Replica assignments that refer log directories pending a failure
>   notification are prioritized to guarantee the Controller and Broker
>   agree on the assignments before acting on the failure notification.
> 
> 6. The transition from one log directory to multiple log directories
>   relies on a logical update to efficiently update directory assignments
>   to the previously registered single log directory when that's possible.
> 
> I have also introduced a configuration for the maximum time the broker
> will keep trying to send a log directory notification before shutting down.
> 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft
> 
> Best,
> 
> --
> Igor
> 

Re: What are the biggest issues with Apache Kafka?

[ziming deng]

Hi Liam,

The Apache Kafka project has several modules, I think you should firstly select 
a module you are interested in. 

For example, we are currently working on KIP-500 related features, which 
includes
1. KIP-856: KRaft Disk Failure Recovery,  
2. KIP-642: Dynamic quorum reassignment,
3. kafka-metadata-shell.sh, 
4. KIP-866: ZooKeeper to KRaft Migration, 
5. KIP-858: Handle JBOD broker disk failure in KRaft
6. Migrtion test cases to support Kraft mode
7. KRaft transactions

We even have the idea of implementing multi raft and using it to replace kakfa 
replica protocal. Apart from KRaft, you can also explore tired storage, kafka 
streams, kafka connect,  group coordinator, transaction coordinator, which are 
also In rapid iteration.

--,
Best,
Ziming


> On Aug 11, 2023, at 08:16, Liam Hodges  
> wrote:
> 
> I'm working with a small team of engineers looking to contribute to the
> open source tools for Apache Kafka. What is missing in the Kafka community
> right now? Are there any problems an open source project could solve for
> it's developers? Appreciate all feedback.

Re: [VOTE] KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum and add Controller Registration

[ziming deng]

+1 (binding) from me.

Thanks for the KIP!

--
Ziming

> On Jul 26, 2023, at 20:18, Luke Chen  wrote:
> 
> +1 (binding) from me.
> 
> Thanks for the KIP!
> 
> Luke
> 
> On Tue, Jul 25, 2023 at 1:24 AM Colin McCabe  wrote:
> 
>> Hi all,
>> 
>> I'd like to start the vote for KIP-919: Allow AdminClient to Talk Directly
>> with the KRaft Controller Quorum and add Controller Registration.
>> 
>> The KIP is here: https://cwiki.apache.org/confluence/x/Owo0Dw
>> 
>> Thanks to everyone who reviewed the proposal.
>> 
>> best,
>> Colin
>> 

Re: [ANNOUNCE] Apache Kafka 3.5.0

[ziming deng]

Thanks for this work, Mickael!

--
Ziming

> On Jun 15, 2023, at 16:35, Luke Chen  wrote:
> 
> Thanks for running this release, Mickael!
> 
> 
> On Thu, Jun 15, 2023 at 4:27 PM Mickael Maison  wrote:
> 
>> The Apache Kafka community is pleased to announce the release for Apache
>> Kafka 3.5.0.
>> 
>> This is a minor release and it includes fixes and improvements from 201
>> JIRAs.
>> 
>> All of the changes in this release can be found in the release notes:
>> https://downloads.apache.org/kafka/3.5.0/RELEASE_NOTES.html
>> 
>> An overview of the release can be found in our announcement blog post:
>> https://kafka.apache.org/blog
>> 
>> You can download the source and binary release (Scala 2.12 and Scala 2.13)
>> from:
>> https://kafka.apache.org/downloads#3.5.0
>> 
>> 
>> ---
>> 
>> Apache Kafka is a distributed streaming platform with four core APIs:
>> ** The Producer API allows an application to publish a stream records to
>> one or more Kafka topics.
>> ** The Consumer API allows an application to subscribe to one or more
>> topics and process the stream of records produced to them.
>> ** The Streams API allows an application to act as a stream processor,
>> consuming an input stream from one or more topics and producing an output
>> stream to one or more output topics, effectively transforming the input
>> streams to output streams.
>> ** The Connector API allows building and running reusable producers or
>> consumers that connect Kafka topics to existing applications or data
>> systems. For example, a connector to a relational database might capture
>> every change to a table.
>> 
>> With these APIs, Kafka can be used for two broad classes of application:
>> ** Building real-time streaming data pipelines that reliably get data
>> between systems or applications.
>> ** Building real-time streaming applications that transform or react to the
>> streams of data.
>> 
>> Apache Kafka is in use at large and small companies worldwide, including
>> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
>> Target, The New York Times, Uber, Yelp, and Zalando, among others.
>> 
>> A big thank you for the following 103 contributors to this release!
>> A. Sophie Blee-Goldman, Akhilesh Chaganti, Alex Sorokoumov, Alexandre
>> Dupriez, Alyssa Huang, Anastasia Vela, Andreas Maechler, andymg3, Artem
>> Livshits, atu-sharm, bachmanity1, Bill Bejeck, Brendan Ribera, Calvin Liu,
>> Chaitanya Mukka, Cheryl Simmons, Chia-Ping Tsai, Chris Egerton, Christo
>> Lolov, Colin P. McCabe, csolidum, Daniel Scanteianu, David Arthur, David
>> Jacot, David Karlsson, David Mao, Dejan Stojadinović, Divij Vaidya, dorwi,
>> drgnchan, Dániel Urbán, Edoardo Comar, egyedt, emilnkrastev, Eric Haag,
>> Farooq Qaiser, Federico Valeri, Gantigmaa Selenge, Greg Harris, Guozhang
>> Wang, Hao Li, Hector Geraldino, Himani Arora, Hoki Min, hudeqi, iamazy,
>> Iblis Lin, Ismael Juma, Ivan Yurchenko, Jakub Scholz, Jason Gustafson, Jeff
>> Kim, Jim Galasyn, Jorge Esteban Quilcate Otoya, Josep Prat, José Armando
>> García Sancio, Juan José Ramos, Junyang Liu, Justine Olshan, Kamal
>> Chandraprakash, Kirk True, Kowshik Prakasam, littlehorse-eng, liuzc9, Lucas
>> Brutschy, Lucia Cerchie, Luke Chen, Manikumar Reddy, Manyanda Chitimbo,
>> Matthew Wong, Matthias J. Sax, Matthias Seiler, Michael Marshall, Mickael
>> Maison, nicolasguyomar, Nikolay, Paolo Patierno, Philip Nee, Pierangelo Di
>> Pilato, Proven Provenzano, Purshotam Chauhan, Qing, Rajini Sivaram,
>> RivenSun, Robert Young, Rohan, Roman Schmitz, Ron Dagostino, Ruslan
>> Krivoshein, Satish Duggana, Shay Elkin, Shekhar Rajak, Simon Woodman,
>> Spacrocket, stejani-cflt, Terry, Tom Bentley, vamossagar12, Victoria Xia,
>> Viktor Somogyi-Vass, Vladimir Korenev, Yash Mayya, Zheng-Xian Li
>> 
>> We welcome your help and feedback. For more information on how to report
>> problems, and to get involved, visit the project website at
>> https://kafka.apache.org/
>> 
>> Thank you!
>> 
>> Regards,
>> Mickael Maison
>> 

Re: [ANNOUNCE] New committer: Divij Vaidya

[ziming deng]

Congratulations, Divij!
Well deserved!

--
Ziming

> On Jun 14, 2023, at 09:41, Luke Chen  wrote:
> 
> Congratulations, Divij!
> Well deserved!
> 
> Luke
> 
> On Wed, Jun 14, 2023 at 7:01 AM Viktor Somogyi-Vass
>  wrote:
> 
>> Congrats Divij!
>> 
>> On Tue, Jun 13, 2023, 20:27 Philip Nee  wrote:
>> 
>>> Congrats!
>>> 
>>> On Tue, Jun 13, 2023 at 8:17 PM Randall Hauch  wrote:
>>> 
 Congratulations!
 
 On Tue, Jun 13, 2023 at 12:48 PM Matthias J. Sax 
>>> wrote:
 
> Congrats!
> 
> On 6/13/23 10:24 AM, Satish Duggana wrote:
>> Congratulations Divij!!
>> 
>> On Tue, 13 Jun 2023 at 22:41, Manyanda Chitimbo
>>  wrote:
>>> 
>>> Congratulations Divij.
>>> 
>>> On Tue 13 Jun 2023 at 17:50, Bruno Cadonna 
 wrote:
>>> 
 Hi all,
 
 The PMC of Apache Kafka is pleased to announce a new Kafka
>>> committer
 Divij Vaidya.
 
 Divij's major contributions are:
 
 GDPR compliance enforcement of kafka-site -
 https://issues.apache.org/jira/browse/KAFKA-13868
 
 Performance improvements:
 
 Improve performance of VarInt encoding and decoding -
 https://github.com/apache/kafka/pull/13312
 
 Reduce data copy & buffer allocation during decompression -
 https://github.com/apache/kafka/pull/13135
 
 He also was heavily involved in the migration to Mockito.
 
 Furthermore, Divij is very active on the mailing lists as well as
>>> in
 maintaining and reviewing pull requests.
 
 Congratulations, Divij!
 
 Thanks,
 
 Bruno (on behalf of the Apache Kafka PMC)
 
 
 --
>>> Manyanda Chitimbo.
> 
 
>>> 
>> 

Re: [VOTE] KIP-858: Handle JBOD broker disk failure in KRaft

[ziming deng]

Hi Igor, Thanks for this work.

+1(binding) from me

--
Best,
Ziming

> On Jun 12, 2023, at 18:07, Igor Soarez  wrote:
> 
> Hi everyone,
> 
> We're getting closer to dropping ZooKeeper support, and JBOD
> in KRaft mode is one of the outstanding big missing features.
> 
> It's been a while since there was new feedback on KIP-858 [1]
> which aims to address this gap, so I'm calling for a vote.
> 
> A huge thank you to everyone who has reviewed this KIP, and
> participated in the discussion thread! [2]
> 
> I'd also like to thank you in advance for taking the time to vote.
> 
> Best,
> 
> --
> Igor
> 
> [1] 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft
> [2] https://lists.apache.org/thread/8dqvfhzcyy87zyy12837pxx9lgsdhvft
> 

Re: [VOTE] KIP-872: Add Serializer#serializeToByteBuffer() to reduce memory copying

[ziming deng]

Hello ShunKang,

+1(binding) from me

--
Thanks,
Ziming

> On May 30, 2023, at 20:07, ShunKang Lin  wrote:
> 
> Hi all,
> 
> Bump this thread again and see if we could get a few more votes. Currently
> we have +2 non-binding and +1 binding.
> Hoping we can get this approved, reviewed, and merged in time for 3.6.0.
> 
> Best,
> ShunKang
> 
> ShunKang Lin  于2023年5月7日周日 15:24写道：
> 
>> Hi everyone,
>> 
>> I'd like to open the vote for KIP-872, which proposes to add
>> Serializer#serializeToByteBuffer() to reduce memory copying.
>> 
>> The proposal is here:
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=228495828
>> 
>> The pull request is here:
>> https://github.com/apache/kafka/pull/12685
>> 
>> Thanks to all who reviewed the proposal, and thanks in advance for taking
>> the time to vote!
>> 
>> Best,
>> ShunKang
>> 

Re: [VOTE] KIP-927: Improve the kafka-metadata-quorum output

[ziming deng]

Thanks for this improvement, +1 from me(binging)

—
Best,
Ziming

> On May 16, 2023, at 00:43, Federico Valeri  wrote:
> 
> Hi all,
> 
> I'd like to start a vote on KIP-927: Improve the kafka-metadata-quorum output.
> 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-927%3A+Improve+the+kafka-metadata-quorum+output
> 
> Discussion thread:
> https://lists.apache.org/thread/pph59hxvz5jkk709x53p44xrpdqwv8qc
> 
> Thanks
> Fede

Re: Consumer offset value -Apache kafka 3.2.3

[ziming deng]

Hello,
It seems kafka.tools.ConsumerOffsetChecker has been deprecated in 0.9.0, in the 
new version, please use kafka-consumer-groups.sh

--
Best,
Ziming

> On Apr 26, 2023, at 18:21, Kafka Life  wrote:
> 
> Dear Kafka Experts
> 
> How can we check for a particular offset number in Apache kafka 3.2.3
> version.Could you please share some light.
> The kafka_console_consumer tool is throwing class not found error.
> 
> ./kafka-run-class.sh kafka.tools.ConsumerOffsetChecker
>--topic your-topic
>--group your-consumer-group
>--zookeeper localhost:2181

Re: [DISCUSS] KIP-858: Handle JBOD broker disk failure in KRaft

[ziming deng]

Thank you for the continuous work, I have some small problems related to the 
implementation details:

1. We have decided to add a new metadata.version, and you have said that  "We 
should also avoid gating on metadata.version to include the 
`OnlineLogDirectories` field in the broker registration, the existing 
versioning in the message schemas should suffice to guarantee compatibility",  
I think this is a big difference from what we had done in the past design, it's 
better to add this point in  "Compatibility, Deprecation, and Migration Plan " 
section.
2. Related to (1),  an important thing about migration is that we should first 
upgrade controllers before upgrade brokers since broker will only send 
`BrokerRegisterationRequest` once and we should use the new format and the 
controller should recoganize the new format, this is difficult in co-resident 
mode, so you should point out that we only support distributed mode similar to 
KIP-866.
3. What will happen on receiving "AssignReplicasToDirsRequest"? this should be 
described in "Controller" section, for example, generating 
`PartitionChangeRecord` or return error code.

Apart from these problems, this KIP LGTM, 
--
Best,
Ziming


> On Sep 28, 2022, at 04:59, Igor Soarez  wrote:
> 
> Hi Ziming,
> 
> Thank you for having another look at this KIP, and please accept my apologies 
> as to my delay in replying.
> 
> The migration introduces JBOD support, so before the migration there should 
> only be one log directory configured per broker. This assumption simplifies 
> how the controller learns about the initial log directory placement for 
> existing partitions. The first broker registration referencing log 
> directories must declare a single log directory - which must be online as 
> brokers cannot start without any online log dirs. If there are no previous 
> log dirs registered for this  broker, the controller assigns all existing 
> partitions in the broker to this single directory.
> 
> Since we assume a single log dir per broker as a migration starting point and 
> brokers cannot start without any online log directories, the 
> `OfflineLogDirectories` field simply cannot be used in the first registration 
> after the upgrade. However, the issue you described applies to 
> `OnlineLogDirectories`. If use of this field is gated by metadata.version 
> then we do require a double roll for brokers to make their first registration 
> with a log directory.
> 
> As you describe, we could move the list of online log directories from the 
> broker registration request to the heartbea request, but I believe we 
> shouldn't do this because:
>  a) The intention is to address a transient problem — the migration into the 
> feature, which should happen only once — by accepting a long term commitment 
> as to how the online log dirs are signaled.
>  b) While the log directory status can change at any point from online to 
> offline, the list of existing log directories is static for broker's 
> lifetime, so it makes little sense to send this information over and over 
> again to the controller.
> 
> We should also avoid gating on metadata.version to include the 
> `OnlineLogDirectories` field in the broker registration. If the controllers 
> are upgraded first, the existing versioning in the message schemas should 
> suffice to guarantee compatibility. After the controllers are upgraded, the 
> brokers can then be upgraded and each of them can register their single 
> online log directory from the first instantiation.
> 
> Please, let me know if this makes sense and any other thoughts you might have.
> 
> Best,
> 
> --
> Igor
> 
> On Thu, Sep 1, 2022, at 2:04 AM, deng ziming wrote:
>> Hi Igor,
>> 
>> I think this KIP can solve the current problems, I have some problems 
>> relating to the migration section.
>> 
>> Since we have bumped broker RPC version and metadata record version, 
>> there will be some problems between brokers/controllers of different 
>> versions. In ZK mode we use IBP as a flag to help solve this, in KRaft 
>> mode we use a feature flag(metadata.version) as a flag for using new 
>> RPC/metadata or not. 
>> 
>> Assuming that we are upgrading from 3.3 to 3.4, firstly the finalized 
>> version of metadata.version is 3.3, brokers will use version 1 of 
>> `BrokerRegistrationRequest` which contains no `OfflineLogDirectories`, 
>> finally the finalized version of metadata.version is 3.4, but brokers 
>> will no longer send `BrokerRegistrationRequest` unless we restart the 
>> broker, so controllers can’t be aware of the `OfflineLogDirectories` of 
>> each broker, so we should reconsider the suggestion of Jason to use 
>> `BrokerHeartbeatRequest` to communicate `OfflineLogDirectories`.
>> 
>> Of course this problem can be solved through a double roll(restart 
>> broker twice when upgrading), but we should trying to avoid it since we 
>> now have feature flag.
>> 
>> One solution is that we include `OfflineLogDirectories` both in 
>> 

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

[ziming deng]

Hello Colin,
There is a mistake that we use `—bootstrap-server` instead of 
`—bootstrap-server(s)`, so should we also change the new argument 
`—bootstrap-controller` (no s).

--
Ziming

> On Apr 20, 2023, at 05:17, Colin McCabe  wrote:
> 
> Hi all,
> 
> I wrote a short KIP about allowing AdminClient to talk directly with the 
> KRaft controller quorum. Check it out here:
> 
> https://cwiki.apache.org/confluence/x/Owo0Dw
> 
> best,
> Colin

Re: kafka rebalancing problem

[ziming deng]

Thank you for reporting this, if you can stably reproduce it, you can create a 
JIRA ticket to track it. You are also welcomed if you can submit a PR to fix it.

- -
Best,
Ziming.

> On Apr 4, 2023, at 18:38, 大肉丸 <382650...@qq.com.INVALID> wrote:
> 
> Excuse me, I would like to ask you a question:
> 
> I have two topic，every topic only have 1 participation, theirs name are 
> topic-A and topic B, and my broker set auto.create.topics.enable=false (its 
> important!)
> 
> 
> 
> 
> And now I start consumer-A listen topic-A, and start consumer-B listen 
> topic-B, and this perform normal. （consumer-A and consumer-B belong to the 
> same consumer group，lets say the name is group_test）
> 
> 
> When i start a consumer-C to listen a Topic that doesn't exist, and 
> consumer-C is also belong to the "group_test" consumer group,  it will always 
> be rebalancing！！！
> 
> 
> I don't know the reason, I thought one of them would not consume, but the 
> result is that it has been rebalancing, could you please tell me the reason？

Re: [ANNOUNCE] New Kafka PMC Member: David Arthur

[ziming deng]

Congrats David!

Ziming

> On Mar 10, 2023, at 10:02, Luke Chen  wrote:
> 
> Congratulations, David!
> 
> On Fri, Mar 10, 2023 at 9:56 AM Yash Mayya  wrote:
> 
>> Congrats David!
>> 
>> On Thu, Mar 9, 2023, 23:42 Jun Rao  wrote:
>> 
>>> Hi, Everyone,
>>> 
>>> David Arthur has been a Kafka committer since 2013. He has been very
>>> instrumental to the community since becoming a committer. It's my
>> pleasure
>>> to announce that David is now a member of Kafka PMC.
>>> 
>>> Congratulations David!
>>> 
>>> Jun
>>> on behalf of Apache Kafka PMC
>>> 
>> 

Re: Entry point to start reading the code

[ziming deng]

Hello,
I think you can start by reading `clients` module, starting from Producer and 
Consumer and AdminClient, there are many related blogs which can be found by 
google but may be outdated since Kafka are updating very quickly, a good new is 
that Kafka have very comprehensive test cases and you can run them locally. 
Then you can read `core` and other modules.

Here are some of my experience:
1. Build kafka using gradle build tool
2. KafkaProducer
  2.1 KafkaProducer API
  2.2 KafkaProducer class
2.2.1 ProducerInterceptor
2.2.2 ProcuderMetadata
2.2.3 Serializer and Deserializer
2.2.4 Partitioner
  2.3 RecordAccumulator
2.3.1 MemoryRecords
2.3.4 RecordBatch
2.3.3 BufferPool
2.3.4 RecordAccumulator
  2.4 Sender
2.4.1 Request
2.4.2 Selector
2.4.3 InFlightRequests
2.4.4 MetadataUpdater
2.4.5 NetworkClient
3. KafkaConsumer
  3.1 KafkaConsumer API
  3.2 delivery guarantee semantic
  3.3 Consumer group rebalance
  3.4 KafkaConsumer
3.4.1 ConsumerNetworkClient
3.4.2 SubscriptionState
3.4.3 ConsumerCoordinator
3.4.4 PartitionAssignor
3.4.5 Heartbeat Request
3.4.6 Reblance request
3.4.7 offset request
3.4.8 Fetcher
3.4.9 overall
4. Core module
  4.1 network
4.1.1 reactor pattern
4.1.2 SockerServer
4.1.3 AbstractServerThread
4.1.4 Acceptor
4.1.5 Processor
4.1.6 RequestChannel
  4.2 API 
4.2.1 KafkaRequestHandler
4.2.2 KafkaApis
  4.3 Log
4.3.1 document
4.3.2 FileMessageSet
4.3.3 ButeBufferMessageSet
4.3.4 OffsetIndex
4.3.5 LogSegment
4.3.6 Log
4.3.7 LogManager
  4.4. DelayedOperationPugatory 
4.4.1 TimingWheel
4.4.2 SystemTimer
4.4.3 DelayedOperation
4.4.4 DelayedOperationPugatory
4.4.5 DelayedFetch
4.4.6 DelayedProcuce
  4.5 replica
4.5.1 Replica
4.5.2 Partition
4.5.3 ReplicaManager
  4.6 kafkaController (deprecated, using KRaft instead)
4.6.1 ControllerChannelManager
4.6.2 ControllerContext
4.6.3 ControllerBrokerRequestBatch
4.6.4 PartitionStateMachine
4.6.5 PartitionLeaderSelectir
4.6.6 REplicateStateMachine
4.6.7 Zookeeper Listener
4.6.8 KafkaController  startup and failover 
4.6.9 ControlledShutdownRequest
  4.7 GroupCoordinator
4.7.1 GroupMetadataManager
4.7.2 GroupCoordinator
5. Acl and authentication
6. Jmx
7. Kafka tool
8. raft/metadata/stream/connect  modules

Some classes may have deprecated or renamed, you can using git to check their 
change history.
--
Best,
Ziming

> On Jan 29, 2023, at 21:35, Nelson Bighetti  wrote:
> 
> Hello everyone! :)
> 
> I've just started reading the code but it appears a little overwhelming to
> me. Is there any documentation describing the directory structure? Or maybe
> could anybody suggest a good starting point to read the code?
> 
> Thanks in advance.

Re: [ANNOUNCE] New committer: Justine Olshan

[ziming deng]

Congratulations Justine!
—
Best, 
Ziming

> On Dec 30, 2022, at 10:06, Luke Chen  wrote:
> 
> Congratulations, Justine!
> Well deserved!
> 
> Luke
> 
> On Fri, Dec 30, 2022 at 9:15 AM Ron Dagostino  wrote:
> 
>> Congratulations, Justine!Well-deserved., and I’m very happy for you.
>> 
>> Ron
>> 
>>> On Dec 29, 2022, at 6:13 PM, Israel Ekpo  wrote:
>>> 
>>> Congratulations Justine!
>>> 
>>> 
 On Thu, Dec 29, 2022 at 5:05 PM Greg Harris
>> 
 wrote:
 
 Congratulations Justine!
 
> On Thu, Dec 29, 2022 at 1:37 PM Bill Bejeck  wrote:
> 
> Congratulations Justine!
> 
> 
> -Bill
> 
>> On Thu, Dec 29, 2022 at 4:36 PM Philip Nee 
>> wrote:
> 
>> wow congrats!
>> 
>> On Thu, Dec 29, 2022 at 1:05 PM Chris Egerton <
>> fearthecel...@gmail.com
> 
>> wrote:
>> 
>>> Congrats, Justine!
>>> 
>>> On Thu, Dec 29, 2022, 15:58 David Jacot  wrote:
>>> 
 Hi all,
 
 The PMC of Apache Kafka is pleased to announce a new Kafka
 committer
 Justine
 Olshan.
 
 Justine has been contributing to Kafka since June 2019. She
> contributed
>>> 53
 PRs including the following KIPs.
 
 KIP-480: Sticky Partitioner
 KIP-516: Topic Identifiers & Topic Deletion State Improvements
 KIP-854: Separate configuration for producer ID expiry
 KIP-890: Transactions Server-Side Defense (in progress)
 
 Congratulations, Justine!
 
 Thanks,
 
 David (on behalf of the Apache Kafka PMC)
 
>>> 
>> 
> 
 
>> 

Re: [ANNOUNCE] New committer: Ron Dagostino

[ziming deng]

Congratulations, Ron!
Well deserved!

--
Ziming

> On Dec 15, 2022, at 09:16, Luke Chen  wrote:
> 
> Congratulations, Ron!
> Well deserved!
> 
> Luke

Re: [DISCUSS] KIP-879: Multi-level Rack Awareness

[ziming deng]

Hi Viktor,

As far as I know, we haven't make ReplicaPlacer a public interface, and we have 
no plan to make it public. I think you can submit a discussion or create a JIRA 
ticket directly without KIP if you have ideas on improving it, right?

--
Best,
Ziming

> On Nov 29, 2022, at 21:52, Viktor Somogyi-Vass 
>  wrote:
> 
> Hi All,
> 
> I'd like to bump this. I've also updated the KIP to incorporate the new
> KRaft changes (ReplicaPlacer). Luckily my proposals were quite similar to
> that, so mostly I've made some minor rewording, naming changes, etc.
> 
> Again, the brief summary of the KIP:
> - expose replica placement strategies with a new config
> - create an admin API and protocol to expose replica placement
> functionality (mainly for the reassignment tool)
> - create a new multi-level rack awareness strategy which improves
> availability on stretch clusters
> 
> I'm happy for any feedback.
> 
> Best,
> Viktor
> 
> On Fri, Oct 28, 2022 at 4:14 PM Viktor Somogyi-Vass <
> viktor.somo...@cloudera.com> wrote:
> 
>> Hey all,
>> 
>> I'd like to propose a new broker side replica assignment strategy and an
>> interface that generalizes replica assignment on brokers and makes them
>> pluggable.
>> 
>> Briefly, the motivation for the new replica assignment strategy is that
>> more and more of our customers would want to run their clusters in a
>> stretched environment, where for instance a cluster is running over
>> multiple regions (and multiple racks inside a region). Since this seems
>> like a more common need, we'd like to contribute back our implementation
>> and also make a generalized interface, so that new strategies that people
>> may come up with could be served better.
>> 
>> I welcome any feedback on this KIP.
>> 
>> The link:
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-879%3A+Multi-level+Rack+Awareness
>> 
>> Best to all,
>> Viktor
>>