Re: [DISCUSS] Apache Kafka 3.8.0 release

[Ivan Yurchenko]

Hi,

I'll try to do all the fixes and changes for KIP-899 [1] sooner today, but 
please proceed with the release if I don't manage.

Ivan

[1] https://github.com/apache/kafka/pull/13277

On Wed, Jun 12, 2024, at 12:54, Josep Prat wrote:
> Hi Luke,
> I think Jose, also mentioned that it won't be ready for v3.8.0 (but he can
> confirm this). My question now would be, given that it seems we would need
> a v3.9.0, do you think it's important to include
> https://github.com/apache/kafka/pull/16284 in v3.8.0?
> 
> Best,
> 
> On Wed, Jun 12, 2024 at 11:40 AM Luke Chen  wrote:
> 
> > Hi Josep
> >
> > For KIP-966, I think Calvin had mentioned he won't complete in v3.8.0.
> > https://lists.apache.org/thread/fsnr8wy5fznzfso7jgk90skgyo277fmw
> >
> > For unclean leader election, all we need is this PR:
> > https://github.com/apache/kafka/pull/16284
> > For this PR, I think it needs one more week to be completed.
> >
> > Thanks.
> > Luke
> >
> > On Wed, Jun 12, 2024 at 4:51 PM Josep Prat 
> > wrote:
> >
> > > Hi all,
> > >
> > > We are now really close to the planned code freeze deadline (today EOD).
> > > According to KIP-1012 [1] we agreed to stay in the 3.x branch until we
> > > achieve feature parity regarding Zookeeper and KRaft. The two main KIPs
> > > identified that would achieve this are: KIP-853 [2] and KIP-966 [3].
> > > At the moment of writing this email both KIPs are not completed. My
> > > question to the people driving both KIPs would be, how much more time do
> > > you think it's needed to bring these KIPs to completion?
> > >
> > > - If the time needed would be short, we could still include these 2 KIPs
> > in
> > > the release.
> > > - However, if the time needed would be on the scale of weeks, we should
> > > continue with the release plan for 3.8 and after start working on the 3.9
> > > release.
> > >
> > > What are your thoughts?
> > >
> > >
> > > [1]:
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1012%3A+The+need+for+a+Kafka+3.8.x+release
> > > [2]:
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Controller+Membership+Changes
> > > [3]:
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-966%3A+Eligible+Leader+Replicas
> > >
> > > On Wed, Jun 12, 2024 at 10:40 AM Josep Prat  wrote:
> > >
> > > > Hi Rajini,
> > > > Yes, we could backport this one to the 3.8 branch. Would you be able to
> > > do
> > > > this once you merge this PR?
> > > >
> > > > Thanks
> > > >
> > > > On Tue, Jun 11, 2024 at 10:53 PM Rajini Sivaram <
> > rajinisiva...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > >> Hi Josep,
> > > >>
> > > >> The PR https://github.com/apache/kafka/pull/13277 for KIP-899 looks
> > > ready
> > > >> to be merged (waiting for the PR build).The PR changes several files,
> > > but
> > > >> is relatively straightforward and not risky. Also the changes are
> > under
> > > a
> > > >> config that is not enabled by default. Since the KIP was approved
> > before
> > > >> KIP freeze, will it be ok to include in 3.8.0?
> > > >>
> > > >> Thank you,
> > > >>
> > > >> Rajini
> > > >>
> > > >>
> > > >> On Tue, Jun 11, 2024 at 9:35 AM Josep Prat
> >  > > >
> > > >> wrote:
> > > >>
> > > >> > Hi all,
> > > >> >
> > > >> > I just want to remind everybody that the code freeze deadline is
> > > >> > approaching. June 12th EOD is the deadline.
> > > >> >
> > > >> > Please do not automatically backport any commit to the 3.8 branch
> > > after
> > > >> > June 12th EOD. Ping me if you think the commit should be backported
> > > >> (fixes
> > > >> > failures in the branch or critical bug fixes).
> > > >> >
> > > >> > Thanks all!
> > > >> >
> > > >> > On Sat, Jun 1, 2024 at 8:43 PM José Armando García Sancio
> > > >> >  wrote:
> > > >> >
> > > >> > > Hi Josep,
> > > >> > >
> > > >> > > See my comments below.
> > > >> > >
> > > >> > > On Wed, May 29, 2024 at 10:52 AM Josep Prat
> > > >>  > > >> > >
> > > >> > > wrote:
> > > >> > > > So I would propose to leave the deadlines as they are and
> > manually
> > > >> > cherry
> > > >> > > > pick the commits related to KIP-853 and KIP-966.
> > > >> > >
> > > >> > > Your proposal sounds good to me. I suspect that will be doing
> > > feature
> > > >> > > development for KIP-853 past the feature freeze and code freeze
> > > date.
> > > >> > > Maybe feature development will be finished around the end of June.
> > > >> > >
> > > >> > > I'll make sure to cherry pick commits for KIP-853 to the 3.8
> > branch
> > > >> > > once we have one.
> > > >> > >
> > > >> > > Thanks,
> > > >> > > --
> > > >> > > -José
> > > >> > >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > [image: Aiven] 
> > > >> >
> > > >> > *Josep Prat*
> > > >> > Open Source Engineering Director, *Aiven*
> > > >> > josep.p...@aiven.io   |   +491715557497
> > > >> > aiven.io    |   <
> > > >> https://www.facebook.com/aivencloud
> > > >> > >
> > > >> >      <
> > > >> 

Re: [VOTE] KIP-899: Allow producer and consumer clients to rebootstrap

[Ivan Yurchenko]

Hi!

I had a look at the KIP-932, and it seems KafkaShareConsumer is to be 
configured the same way as the normal consumer using key-value props. As I 
understand correctly, no adaptation is needed for it to benefit from KIP-899?

Meanwhile, the PR [1] is open for review. If there are comments that require 
changes, we can address them in the PR or in case it's already merged, 
afterwards.

Best,
Ivan

[1] https://github.com/apache/kafka/pull/13277

On Thu, May 16, 2024, at 01:52, Jun Rao wrote:
> Hi, Ivan,
> 
> You are right. StreamsConfigs can take all existing consumer configs, with
> or without prefixes. So, we don't need to add the new config to
> StreamsConfig explicitly.
> 
> For KIP-932, it says for each new consumer config, we need to determine
> whether it should be added to ShareConsumer config too.
> 
> Thanks,
> 
> Jun
> 
> On Wed, May 15, 2024 at 12:16 PM Ivan Yurchenko  wrote:
> 
> > Hi Jun,
> >
> > Thank you for you comment. I was thinking that this
> > `metadata.recovery.strategy` could be passed to the relevant consumer in
> > streams using the `restore.consumer.` prefix. I that what you meant or I
> > misunderstand?
> > As for the KIP-932, I'll have a closer look.
> >
> > Ivan
> >
> >
> > On Wed, May 15, 2024, at 20:14, Jun Rao wrote:
> > > Hi, Ivan,
> > >
> > > Thanks for the KIP. +1
> > >
> > > Just a minor comment. Should we add metadata.recovery.strategy to the
> > > Streams and the newly introduced ShareConsumer (KIP-932) too?
> > >
> > > Jun
> > >
> > > On Wed, May 8, 2024 at 11:35 AM Manikumar 
> > wrote:
> > >
> > > > Thanks for the KIP.
> > > >
> > > > +1 (binding).
> > > >
> > > > On Wed, Apr 17, 2024 at 7:50 PM Omnia Ibrahim  > >
> > > > wrote:
> > > > >
> > > > > Hi Ivan,
> > > > > Thanks for the KIP this is a very nice feature to have.
> > > > > +1(non-binding)
> > > > > Omnia
> > > > > > On 15 Apr 2024, at 14:33, Andrew Schofield <
> > andrew_schofi...@live.com>
> > > > wrote:
> > > > > >
> > > > > > Thanks for the KIP
> > > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > Andrew
> > > > > >
> > > > > >> On 15 Apr 2024, at 14:16, Chris Egerton 
> > > > wrote:
> > > > > >>
> > > > > >> Hi Ivan,
> > > > > >>
> > > > > >> Thanks for the KIP. After the recent changes, this LGTM. +1
> > (binding)
> > > > > >>
> > > > > >> Cheers,
> > > > > >>
> > > > > >> Chris
> > > > > >>
> > > > > >> On Wed, Aug 2, 2023 at 12:15 AM Ivan Yurchenko <
> > > > ivan0yurche...@gmail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> Hello,
> > > > > >>>
> > > > > >>> The discussion [1] for KIP-899 [2] has been open for quite some
> > > > time. I'd
> > > > > >>> like to put the KIP up for a vote.
> > > > > >>>
> > > > > >>> Best,
> > > > > >>> Ivan
> > > > > >>>
> > > > > >>> [1]
> > https://lists.apache.org/thread/m0ncbmfxs5m87sszby2jbmtjx2bdpcdl
> > > > > >>> [2]
> > > > > >>>
> > > > > >>>
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-899%3A+Allow+producer+and+consumer+clients+to+rebootstrap
> > > > > >>>
> > > > > >
> > > > >
> > > >
> > >
> >
> 

Re: [VOTE] KIP-899: Allow producer and consumer clients to rebootstrap

[Ivan Yurchenko]

Hi Jun,

Thank you for you comment. I was thinking that this 
`metadata.recovery.strategy` could be passed to the relevant consumer in 
streams using the `restore.consumer.` prefix. I that what you meant or I 
misunderstand?
As for the KIP-932, I'll have a closer look.

Ivan


On Wed, May 15, 2024, at 20:14, Jun Rao wrote:
> Hi, Ivan,
> 
> Thanks for the KIP. +1
> 
> Just a minor comment. Should we add metadata.recovery.strategy to the
> Streams and the newly introduced ShareConsumer (KIP-932) too?
> 
> Jun
> 
> On Wed, May 8, 2024 at 11:35 AM Manikumar  wrote:
> 
> > Thanks for the KIP.
> >
> > +1 (binding).
> >
> > On Wed, Apr 17, 2024 at 7:50 PM Omnia Ibrahim 
> > wrote:
> > >
> > > Hi Ivan,
> > > Thanks for the KIP this is a very nice feature to have.
> > > +1(non-binding)
> > > Omnia
> > > > On 15 Apr 2024, at 14:33, Andrew Schofield 
> > wrote:
> > > >
> > > > Thanks for the KIP
> > > >
> > > > +1 (non-binding)
> > > >
> > > > Andrew
> > > >
> > > >> On 15 Apr 2024, at 14:16, Chris Egerton 
> > wrote:
> > > >>
> > > >> Hi Ivan,
> > > >>
> > > >> Thanks for the KIP. After the recent changes, this LGTM. +1 (binding)
> > > >>
> > > >> Cheers,
> > > >>
> > > >> Chris
> > > >>
> > > >> On Wed, Aug 2, 2023 at 12:15 AM Ivan Yurchenko <
> > ivan0yurche...@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> Hello,
> > > >>>
> > > >>> The discussion [1] for KIP-899 [2] has been open for quite some
> > time. I'd
> > > >>> like to put the KIP up for a vote.
> > > >>>
> > > >>> Best,
> > > >>> Ivan
> > > >>>
> > > >>> [1] https://lists.apache.org/thread/m0ncbmfxs5m87sszby2jbmtjx2bdpcdl
> > > >>> [2]
> > > >>>
> > > >>>
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-899%3A+Allow+producer+and+consumer+clients+to+rebootstrap
> > > >>>
> > > >
> > >
> >
> 

Re: [VOTE] KIP-899: Allow producer and consumer clients to rebootstrap

[Ivan Yurchenko]

Hi all,

It seems we can close the vote in favor of the KIP:
- Binding +1s: Chris Egerton, Manikumar Reddy, Jun Rao, Rajini Sivaram.
- Non-binding +1s: Andrew Schofield, Omnia Ibrahim.
- No -1s.

Thank you all for your votes!
I'll do the necessary changes to the KIPs page and proceed with the 
implementation.

Best,
Ivan


On Wed, May 15, 2024, at 21:50, Rajini Sivaram wrote:
> Hi Ivan,
> 
> Thanks for the KIP, +1 (binding)
> 
> Regards,
> 
> Rajini
> 
> 
> On Wed, May 15, 2024 at 6:15 PM Jun Rao  wrote:
> 
> > Hi, Ivan,
> >
> > Thanks for the KIP. +1
> >
> > Just a minor comment. Should we add metadata.recovery.strategy to the
> > Streams and the newly introduced ShareConsumer (KIP-932) too?
> >
> > Jun
> >
> > On Wed, May 8, 2024 at 11:35 AM Manikumar 
> > wrote:
> >
> > > Thanks for the KIP.
> > >
> > > +1 (binding).
> > >
> > > On Wed, Apr 17, 2024 at 7:50 PM Omnia Ibrahim 
> > > wrote:
> > > >
> > > > Hi Ivan,
> > > > Thanks for the KIP this is a very nice feature to have.
> > > > +1(non-binding)
> > > > Omnia
> > > > > On 15 Apr 2024, at 14:33, Andrew Schofield <
> > andrew_schofi...@live.com>
> > > wrote:
> > > > >
> > > > > Thanks for the KIP
> > > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > Andrew
> > > > >
> > > > >> On 15 Apr 2024, at 14:16, Chris Egerton 
> > > wrote:
> > > > >>
> > > > >> Hi Ivan,
> > > > >>
> > > > >> Thanks for the KIP. After the recent changes, this LGTM. +1
> > (binding)
> > > > >>
> > > > >> Cheers,
> > > > >>
> > > > >> Chris
> > > > >>
> > > > >> On Wed, Aug 2, 2023 at 12:15 AM Ivan Yurchenko <
> > > ivan0yurche...@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >>> Hello,
> > > > >>>
> > > > >>> The discussion [1] for KIP-899 [2] has been open for quite some
> > > time. I'd
> > > > >>> like to put the KIP up for a vote.
> > > > >>>
> > > > >>> Best,
> > > > >>> Ivan
> > > > >>>
> > > > >>> [1]
> > https://lists.apache.org/thread/m0ncbmfxs5m87sszby2jbmtjx2bdpcdl
> > > > >>> [2]
> > > > >>>
> > > > >>>
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-899%3A+Allow+producer+and+consumer+clients+to+rebootstrap
> > > > >>>
> > > > >
> > > >
> > >
> >
> 

[jira] [Resolved] (KAFKA-16445) PATCH method for connector configuration

[Ivan Yurchenko (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ivan Yurchenko resolved KAFKA-16445.

Resolution: Fixed

> PATCH method for connector configuration
> 
>
> Key: KAFKA-16445
> URL: https://issues.apache.org/jira/browse/KAFKA-16445
> Project: Kafka
>  Issue Type: Improvement
>  Components: connect
>    Reporter: Ivan Yurchenko
>    Assignee: Ivan Yurchenko
>Priority: Minor
>
> As  [KIP-477: Add PATCH method for connector config in Connect REST 
> API|https://cwiki.apache.org/confluence/display/KAFKA/KIP-477%3A+Add+PATCH+method+for+connector+config+in+Connect+REST+API]
>  suggests, we should introduce the PATCH method for connector configuration.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-477: Add PATCH method for connector config in Connect REST API

[Ivan Yurchenko]

Hello all,

Thank you for your votes!

I think we have reached a decision here to accept the KIP with the following 
votes:

- 3x binding +1s from Chris Egerton, Yash Mayya, and Greg Harris.
- 1x non-binding +1s from Knowles Atchison Jr.
- No -1s.

I will proceed with the implementation. Feel free to track the jira [1] and the 
(now still in draft) PR [2].

Best,
Ivan

[1] https://issues.apache.org/jira/browse/KAFKA-16445
[2] https://github.com/apache/kafka/pull/6934

On Wed, May 1, 2024, at 20:24, Greg Harris wrote:
> Hi Ivan,
> 
> Thank you for the KIP!
> I think PATCH using the same return type as PUT benefits the clients,
> even though the "created" field will always be false.
> 
> +1 (binding)
> 
> Thanks,
> Greg
> 
> On Fri, Apr 12, 2024 at 4:31 AM Yash Mayya  wrote:
> >
> > Hi Ivan,
> >
> > Thanks for reviving this KIP, I think it will be a useful addition to
> > Connect!
> >
> > +1 (binding)
> >
> > Cheers,
> > Yash
> >
> > On Tue, Apr 9, 2024 at 4:23 AM Knowles Atchison Jr 
> > wrote:
> >
> > > +1 (non binding)
> > >
> > > On Mon, Apr 8, 2024, 3:30 PM Chris Egerton 
> > > wrote:
> > >
> > > > Thanks Ivan! +1 (binding) from me.
> > > >
> > > > On Mon, Apr 8, 2024, 06:59 Ivan Yurchenko  wrote:
> > > >
> > > > > Hello!
> > > > >
> > > > > I'd like to put the subj KIP[1] to a vote. Thank you.
> > > > >
> > > > > Best regards,
> > > > > Ivan
> > > > >
> > > > > [1]
> > > > >
> > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-477%3A+Add+PATCH+method+for+connector+config+in+Connect+REST+API
> > > > >
> > > >
> > >
> 

Re: [ANNOUNCE] New Kafka PMC Member: Greg Harris

[Ivan Yurchenko]

Congrats Greg!

On Sun, Apr 14, 2024, at 22:51, Sophie Blee-Goldman wrote:
> Congrats Greg! Happy to have you
> 
> On Sun, Apr 14, 2024 at 9:26 AM Jorge Esteban Quilcate Otoya <
> quilcate.jo...@gmail.com> wrote:
> 
> > Congrats, Greg!!
> >
> > On Sun 14. Apr 2024 at 15.05, Josep Prat 
> > wrote:
> >
> > > Congrats Greg!!!
> > >
> > >
> > > Best,
> > >
> > > Josep Prat
> > > Open Source Engineering Director, aivenjosep.p...@aiven.io   |
> > > +491715557497 | aiven.io
> > > Aiven Deutschland GmbH
> > > Alexanderufer 3-7, 10117 Berlin
> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > >
> > > On Sun, Apr 14, 2024, 12:30 Divij Vaidya 
> > wrote:
> > >
> > > > Congratulations Greg!
> > > >
> > > > --
> > > > Divij Vaidya
> > > >
> > > >
> > > >
> > > > On Sun, Apr 14, 2024 at 6:39 AM Kamal Chandraprakash <
> > > > kamal.chandraprak...@gmail.com> wrote:
> > > >
> > > > > Congratulations, Greg!
> > > > >
> > > > > On Sun, Apr 14, 2024 at 8:57 AM Yash Mayya 
> > > wrote:
> > > > >
> > > > > > Congrats Greg!
> > > > > >
> > > > > > On Sun, 14 Apr, 2024, 05:56 Randall Hauch, 
> > wrote:
> > > > > >
> > > > > > > Congratulations, Greg!
> > > > > > >
> > > > > > > On Sat, Apr 13, 2024 at 6:36 PM Luke Chen 
> > > wrote:
> > > > > > >
> > > > > > > > Congrats, Greg!
> > > > > > > >
> > > > > > > > On Sun, Apr 14, 2024 at 7:05 AM Viktor Somogyi-Vass
> > > > > > > >  wrote:
> > > > > > > >
> > > > > > > > > Congrats Greg! :)
> > > > > > > > >
> > > > > > > > > On Sun, Apr 14, 2024, 00:35 Bill Bejeck 
> > > > wrote:
> > > > > > > > >
> > > > > > > > > > Congrats Greg!
> > > > > > > > > >
> > > > > > > > > > -Bill
> > > > > > > > > >
> > > > > > > > > > On Sat, Apr 13, 2024 at 4:25 PM Boudjelda Mohamed Said <
> > > > > > > > > bmsc...@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Congratulations Greg
> > > > > > > > > > >
> > > > > > > > > > > On Sat 13 Apr 2024 at 20:42, Chris Egerton <
> > > > > ceger...@apache.org>
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi all,
> > > > > > > > > > > >
> > > > > > > > > > > > Greg Harris has been a Kafka committer since July 2023.
> > > He
> > > > > has
> > > > > > > > > remained
> > > > > > > > > > > > very active and instructive in the community since
> > > > becoming a
> > > > > > > > > > committer.
> > > > > > > > > > > > It's my pleasure to announce that Greg is now a member
> > of
> > > > > Kafka
> > > > > > > > PMC.
> > > > > > > > > > > >
> > > > > > > > > > > > Congratulations, Greg!
> > > > > > > > > > > >
> > > > > > > > > > > > Chris, on behalf of the Apache Kafka PMC
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> 

Re: [DISCUSS] KIP-899: Allow clients to rebootstrap

[Ivan Yurchenko]

Hi Chris and all,

Thank you for your feedback. Your proposals seems good to me. I did these 
changed to the KIP, please have a look at the change [1]

Best,
Ivan

[1] 
https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=240881396=14=12

On Thu, Apr 11, 2024, at 10:49, Chris Egerton wrote:
> Hi Ivan,
> 
> I agree with Andrew that we can save cluster ID checking for later. This
> feature is opt-in and if necessary we can add a note to users about only
> enabling it if they can be certain that the same cluster will always be
> resolved by the bootstrap servers. This would apply regardless of whether
> we did client ID checking anyways.
> 
> Thanks for exploring a variety of options and ironing out the details on
> this KIP. I think this is acceptable as-is but have a couple of final
> suggestions we might consider:
> 
> 1. Although the definition of an unavailable broker is useful ("A broker is
> unavailable when the client doesn't have an established connection with it
> and cannot establish a connection (e.g. due to the reconnect backoff)"), I
> think this is a little too restrictive. It's useful to note this as an
> example of what we may consider an unavailable broker, but if we leave some
> more wiggle room, it could save us the trouble of a follow-up KIP when
> tweaking behavior in the future. For example, to reduce discovery time for
> a migrated Kafka cluster, it could be nice to re-bootstrap after a
> connection attempt has failed for every currently-known broker with no
> successful attempts in between, instead of waiting for the reconnection
> backoff interval to kick in. Again, I don't think this needs to happen with
> the initial implementation of the KIP, I just want us to be able to explore
> options like this in the future.
> 
> 2. In a similar vein, I think we can leave more room in our definition of
> re-bootstrapping. Instead of "During the rebootstrap process, the client
> forgets the brokers it knows about and falls back on the bootstrap brokers
> (i.e. provided by bootstrap.servers provided by the client configuration)
> as if it had just been initialized.", we could say something like "During
> the rebootstrap process, the client attempts to re-contact the bootstrap
> servers (i.e. ...) that it contacted during initialization." This could be
> useful if we want to add the bootstrap servers to the previously-known list
> of brokers instead of completely discarding the previously-known set. This
> too can be left out of the initial implementation and just give us a bit
> more room for future options.
> 
> Cheers,
> 
> Chris
> 
> On Tue, Apr 9, 2024 at 11:51 AM Andrew Schofield 
> wrote:
> 
> > Hi Ivan,
> > I think you have to go one way or the other with the cluster ID, so I
> > think removing that from this KIP might
> > be the best. I think there’s another KIP waiting to be written for
> > ensuring consistency of clusters, but
> > I think that wouldn’t conflict at all with this one.
> >
> > Thanks,
> > Andrew
> >
> > > On 9 Apr 2024, at 19:11, Ivan Yurchenko  wrote:
> > >
> > > Hi Andrew and all,
> > >
> > > I looked deeper into the code [1] and it seems the Metadata class is OK
> > with cluster ID changing. So I'm thinking that the rebootstrapping
> > shouldn't introduce a new failure mode here. And I should remove the
> > mention of this cluster ID checks from the KIP.
> > >
> > > Best,
> > > Ivan
> > >
> > > [1]
> > https://github.com/apache/kafka/blob/ff90f78c700c582f9800013faad827c36b45ceb7/clients/src/main/java/org/apache/kafka/clients/Metadata.java#L355
> > >
> > > On Tue, Apr 9, 2024, at 09:28, Andrew Schofield wrote:
> > >> Hi Ivan,
> > >> Thanks for the KIP. I can see situations in which this would be
> > helpful. I have one question.
> > >>
> > >> The KIP says the client checks the cluster ID when it re-bootstraps and
> > that it will fail if the
> > >> cluster ID doesn’t match the previously known one. How does it fail?
> > Which exception does
> > >> it throw and when?
> > >>
> > >> In a similar vein, now that we are checking cluster IDs, I wonder if it
> > could be extended to
> > >> cover all situations in which there are cluster ID mismatches, such as
> > the bootstrap server
> > >> list erroneously pointing at brokers from different clusters and the
> > problem only being
> > >> detectable later on.
> > >>
> > >> Thanks,
> > >> Andrew
> > >>
> > >>> On 8 Apr 2024, a

Re: [DISCUSS] KIP-899: Allow clients to rebootstrap

[Ivan Yurchenko]

Hi Andrew and all,

I did the mentioned change.

As there are no more comments, I'm letting this sit for e.g. a week and then I 
will call for the vote.

Best,
Ivan


On Tue, Apr 9, 2024, at 23:51, Andrew Schofield wrote:
> Hi Ivan,
> I think you have to go one way or the other with the cluster ID, so I think 
> removing that from this KIP might
> be the best. I think there’s another KIP waiting to be written for ensuring 
> consistency of clusters, but
> I think that wouldn’t conflict at all with this one.
> 
> Thanks,
> Andrew
> 
> > On 9 Apr 2024, at 19:11, Ivan Yurchenko  wrote:
> >
> > Hi Andrew and all,
> >
> > I looked deeper into the code [1] and it seems the Metadata class is OK 
> > with cluster ID changing. So I'm thinking that the rebootstrapping 
> > shouldn't introduce a new failure mode here. And I should remove the 
> > mention of this cluster ID checks from the KIP.
> >
> > Best,
> > Ivan
> >
> > [1] 
> > https://github.com/apache/kafka/blob/ff90f78c700c582f9800013faad827c36b45ceb7/clients/src/main/java/org/apache/kafka/clients/Metadata.java#L355
> >
> > On Tue, Apr 9, 2024, at 09:28, Andrew Schofield wrote:
> >> Hi Ivan,
> >> Thanks for the KIP. I can see situations in which this would be helpful. I 
> >> have one question.
> >>
> >> The KIP says the client checks the cluster ID when it re-bootstraps and 
> >> that it will fail if the
> >> cluster ID doesn’t match the previously known one. How does it fail? Which 
> >> exception does
> >> it throw and when?
> >>
> >> In a similar vein, now that we are checking cluster IDs, I wonder if it 
> >> could be extended to
> >> cover all situations in which there are cluster ID mismatches, such as the 
> >> bootstrap server
> >> list erroneously pointing at brokers from different clusters and the 
> >> problem only being
> >> detectable later on.
> >>
> >> Thanks,
> >> Andrew
> >>
> >>> On 8 Apr 2024, at 18:24, Ivan Yurchenko  wrote:
> >>>
> >>> Hello!
> >>>
> >>> I changed the KIP a bit, specifying that the certain benefit goes to 
> >>> consumers not participating in a group, but that other clients can 
> >>> benefit as well in certain situations.
> >>>
> >>> You can see the changes in the history [1]
> >>>
> >>> Thank you!
> >>>
> >>> Ivan
> >>>
> >>> [1] 
> >>> https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=240881396=10=11
> >>>
> >>> On 2023/07/15 16:37:52 Ivan Yurchenko wrote:
> >>>> Hello!
> >>>>
> >>>> I've made several changes to the KIP based on the comments:
> >>>>
> >>>> 1. Reduced the scope to producer and consumer clients only.
> >>>> 2. Added more details to the description of the rebootstrap process.
> >>>> 3. Documented the role of low values of reconnect.backoff.max.ms in
> >>>> preventing rebootstrapping.
> >>>> 4. Some wording changes.
> >>>>
> >>>> You can see the changes in the history [1]
> >>>>
> >>>> I'm planning to put the KIP to a vote in some days if there are no new
> >>>> comments.
> >>>>
> >>>> Thank you!
> >>>>
> >>>> Ivan
> >>>>
> >>>> [1]
> >>>> https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=240881396=9=5
> >>>>
> >>>> On Tue, 30 May 2023 at 08:23, Ivan Yurchenko 
> >>>> wrote:
> >>>>
> >>>>> Hi Chris and all,
> >>>>>
> >>>>>> I believe the logic you've linked is only applicable for the producer 
> >>>>>> and
> >>>>>> consumer clients; the admin client does something different (see [1]).
> >>>>>
> >>>>> I see, thank you for the pointer. It seems the admin client is fairly
> >>>>> different from the producer and consumer. Probably it makes sense to 
> >>>>> reduce
> >>>>> the scope of the KIP to the producer and consumer clients only.
> >>>>>
> >>>>>> it'd be nice to have a definition of when re-bootstrapping
> >>>>>> would occur that doesn't rely on internal implementation details. What
> >>>>>> user-visi

Re: [DISCUSS] KIP-899: Allow clients to rebootstrap

[Ivan Yurchenko]

Hi Andrew and all,

I looked deeper into the code [1] and it seems the Metadata class is OK with 
cluster ID changing. So I'm thinking that the rebootstrapping shouldn't 
introduce a new failure mode here. And I should remove the mention of this 
cluster ID checks from the KIP.

Best,
Ivan

[1] 
https://github.com/apache/kafka/blob/ff90f78c700c582f9800013faad827c36b45ceb7/clients/src/main/java/org/apache/kafka/clients/Metadata.java#L355

On Tue, Apr 9, 2024, at 09:28, Andrew Schofield wrote:
> Hi Ivan,
> Thanks for the KIP. I can see situations in which this would be helpful. I 
> have one question.
> 
> The KIP says the client checks the cluster ID when it re-bootstraps and that 
> it will fail if the
> cluster ID doesn’t match the previously known one. How does it fail? Which 
> exception does
> it throw and when?
> 
> In a similar vein, now that we are checking cluster IDs, I wonder if it could 
> be extended to
> cover all situations in which there are cluster ID mismatches, such as the 
> bootstrap server
> list erroneously pointing at brokers from different clusters and the problem 
> only being
> detectable later on.
> 
> Thanks,
> Andrew
> 
> > On 8 Apr 2024, at 18:24, Ivan Yurchenko  wrote:
> > 
> > Hello!
> > 
> > I changed the KIP a bit, specifying that the certain benefit goes to 
> > consumers not participating in a group, but that other clients can benefit 
> > as well in certain situations.
> > 
> > You can see the changes in the history [1]
> > 
> > Thank you!
> > 
> > Ivan
> > 
> > [1] 
> > https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=240881396=10=11
> > 
> > On 2023/07/15 16:37:52 Ivan Yurchenko wrote:
> >> Hello!
> >> 
> >> I've made several changes to the KIP based on the comments:
> >> 
> >> 1. Reduced the scope to producer and consumer clients only.
> >> 2. Added more details to the description of the rebootstrap process.
> >> 3. Documented the role of low values of reconnect.backoff.max.ms in
> >> preventing rebootstrapping.
> >> 4. Some wording changes.
> >> 
> >> You can see the changes in the history [1]
> >> 
> >> I'm planning to put the KIP to a vote in some days if there are no new
> >> comments.
> >> 
> >> Thank you!
> >> 
> >> Ivan
> >> 
> >> [1]
> >> https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=240881396=9=5
> >> 
> >> On Tue, 30 May 2023 at 08:23, Ivan Yurchenko 
> >> wrote:
> >> 
> >>> Hi Chris and all,
> >>> 
> >>>> I believe the logic you've linked is only applicable for the producer and
> >>>> consumer clients; the admin client does something different (see [1]).
> >>> 
> >>> I see, thank you for the pointer. It seems the admin client is fairly
> >>> different from the producer and consumer. Probably it makes sense to 
> >>> reduce
> >>> the scope of the KIP to the producer and consumer clients only.
> >>> 
> >>>> it'd be nice to have a definition of when re-bootstrapping
> >>>> would occur that doesn't rely on internal implementation details. What
> >>>> user-visible phenomena can we identify that would lead to a
> >>>> re-bootstrapping?
> >>> 
> >>> Let's put it this way: "Re-bootstrapping means that the client forgets
> >>> about nodes it knows about and falls back on the bootstrap nodes as if it
> >>> had just been initialized. Re-bootstrapping happens when, during a 
> >>> metadata
> >>> update (which may be scheduled by `metadata.max.age.ms` or caused by
> >>> certain error responses like NOT_LEADER_OR_FOLLOWER, 
> >>> REPLICA_NOT_AVAILABLE,
> >>> etc.), the client doesn't have a node with an established connection or
> >>> establishable connection."
> >>> Does this sound good?
> >>> 
> >>>> I also believe that if someone has "
> >>>> reconnect.backoff.max.ms" set to a low-enough value,
> >>>> NetworkClient::leastLoadedNode may never return null. In that case,
> >>>> shouldn't we still attempt a re-bootstrap at some point (if the user has
> >>>> enabled this feature)?
> >>> 
> >>> Yes, you're right. Particularly `canConnect` here [1] can always be
> >>> returning `true` if `reconnect.backoff.max.ms` is low enough.
> >>> It seems pretty difficul

RE: Re: [DISCUSS] KIP-899: Allow clients to rebootstrap

[Ivan Yurchenko]

Hello!

I changed the KIP a bit, specifying that the certain benefit goes to consumers 
not participating in a group, but that other clients can benefit as well in 
certain situations.

You can see the changes in the history [1]

Thank you!

Ivan

[1] 
https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=240881396=10=11

On 2023/07/15 16:37:52 Ivan Yurchenko wrote:
> Hello!
> 
> I've made several changes to the KIP based on the comments:
> 
> 1. Reduced the scope to producer and consumer clients only.
> 2. Added more details to the description of the rebootstrap process.
> 3. Documented the role of low values of reconnect.backoff.max.ms in
> preventing rebootstrapping.
> 4. Some wording changes.
> 
> You can see the changes in the history [1]
> 
> I'm planning to put the KIP to a vote in some days if there are no new
> comments.
> 
> Thank you!
> 
> Ivan
> 
> [1]
> https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=240881396=9=5
> 
> On Tue, 30 May 2023 at 08:23, Ivan Yurchenko 
> wrote:
> 
> > Hi Chris and all,
> >
> > > I believe the logic you've linked is only applicable for the producer and
> > > consumer clients; the admin client does something different (see [1]).
> >
> > I see, thank you for the pointer. It seems the admin client is fairly
> > different from the producer and consumer. Probably it makes sense to reduce
> > the scope of the KIP to the producer and consumer clients only.
> >
> > > it'd be nice to have a definition of when re-bootstrapping
> > > would occur that doesn't rely on internal implementation details. What
> > > user-visible phenomena can we identify that would lead to a
> > > re-bootstrapping?
> >
> > Let's put it this way: "Re-bootstrapping means that the client forgets
> > about nodes it knows about and falls back on the bootstrap nodes as if it
> > had just been initialized. Re-bootstrapping happens when, during a metadata
> > update (which may be scheduled by `metadata.max.age.ms` or caused by
> > certain error responses like NOT_LEADER_OR_FOLLOWER, REPLICA_NOT_AVAILABLE,
> > etc.), the client doesn't have a node with an established connection or
> > establishable connection."
> > Does this sound good?
> >
> > > I also believe that if someone has "
> > > reconnect.backoff.max.ms" set to a low-enough value,
> > > NetworkClient::leastLoadedNode may never return null. In that case,
> > > shouldn't we still attempt a re-bootstrap at some point (if the user has
> > > enabled this feature)?
> >
> > Yes, you're right. Particularly `canConnect` here [1] can always be
> > returning `true` if `reconnect.backoff.max.ms` is low enough.
> > It seems pretty difficult to find a good criteria when re-bootstrapping
> > should be forced in this case, so it'd be difficult to configure and reason
> > about. I think it's worth mentioning in the KIP and later in the
> > documentation, but we should not try to do anything special here.
> >
> > > Would it make sense to re-bootstrap only after "
> > > metadata.max.age.ms" has elapsed since the last metadata update, and
> > when
> > > at least one request has been made to contact each known server and been
> > > met with failure?
> >
> > The first condition is satisfied by the check in the beginning of
> > `maybeUpdate` [2].
> > It seems to me, the second one is also satisfied by `leastLoadedNode`.
> > Admittedly, it's more relaxed than you propose: it tracks unavailability of
> > nodes that was detected by all types of requests, not only by metadata
> > requests.
> > What do you think, would this be enough?
> >
> > [1]
> > https://github.com/apache/kafka/blob/c9a42c85e2c903329b3550181d230527e90e3646/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java#L698
> > [2]
> > https://github.com/apache/kafka/blob/c9a42c85e2c903329b3550181d230527e90e3646/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java#L1034-L1041
> >
> > Best,
> > Ivan
> >
> >
> > On Tue, 21 Feb 2023 at 20:07, Chris Egerton 
> > wrote:
> >
> >> Hi Ivan,
> >>
> >> I believe the logic you've linked is only applicable for the producer and
> >> consumer clients; the admin client does something different (see [1]).
> >>
> >> Either way, it'd be nice to have a definition of when re-bootstrapping
> >> would occur that doesn't rely on internal implementation details. What
> >> user-visible phenomena can we identify that would lead

[VOTE] KIP-477: Add PATCH method for connector config in Connect REST API

[Ivan Yurchenko]

Hello!

I'd like to put the subj KIP[1] to a vote. Thank you.

Best regards,
Ivan

[1] 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-477%3A+Add+PATCH+method+for+connector+config+in+Connect+REST+API

Re: [DISCUSS] KIP-477: Add PATCH method for connector config in Connect REST API

[Ivan Yurchenko]

Hello Chris,

Thanks for your feedback. I created the jira and also updated the description a 
bit mentioning that other similar race scenarios exist and the KIP is not 
trying to solve them.

Best,
Ivan

On Wed, Mar 27, 2024, at 17:08, Chris Egerton wrote:
> Hi Ivan,
> 
> Thanks for the updates. LGTM!
> 
> RE atomicity: I think it should be possible and not _too_ invasive to
> detect and handle these kinds of races by tracking the offset in the config
> topic for connector configs and aborting an operation if that offset
> changes between when the request was initiated and when the write to the
> config topic will take place, but since the same kind of issue is also
> possible with other connector operations (concurrent configuration PUTs,
> for example) due to how validation is split out into a separate thread, I
> agree that it's not worth blocking the KIP on fixing this.
> 
> One final nit: Can you update the Jira ticket link in the KIP?
> 
> Cheers,
> 
> Chris
> 
> On Wed, Mar 27, 2024 at 2:56 PM Ivan Yurchenko  wrote:
> 
> > Hi,
> >
> > I updated the KIP with the two following changes:
> > 1. Using `null` values as tombstone value for removing existing fields
> > from configuration.
> > 2. Added a note about the lack of 100% atomicity, which seems very
> > difficult to achieve practically.
> >
> > Ivan
> >
> >
> > On Tue, Mar 26, 2024, at 14:45, Ivan Yurchenko wrote:
> > > Speaking of the Chris' comment
> > >
> > > > One thought that comes to mind is that sometimes it may be useful to
> > > > explicitly remove properties from a connector configuration. We might
> > > > permit this by allowing users to specify null (the JSON literal, not a
> > > > string containing the characters "null") as the value for to-be-removed
> > > > properties.
> > >
> > > This actually makes sense. AFAIU, `null` cannot be in the connector
> > config since https://github.com/apache/kafka/pull/11333, so using it as a
> > tombstone value is a good idea. I can update the KIP.
> > >
> > > Ivan
> > >
> > >
> > > On Tue, Mar 26, 2024, at 14:19, Ivan Yurchenko wrote:
> > > > Hi all,
> > > >
> > > > This KIP is a bit old now :) but I think its context hasn't changed
> > much since then and the KIP is still valid. I would like to finally bring
> > it to some conclusion.
> > > >
> > > > Best,
> > > > Ivan
> > > >
> > > > On 2021/07/12 14:49:47 Chris Egerton wrote:
> > > > > Hi all,
> > > > >
> > > > > Know it's been a while for this KIP but in my personal experience
> > the value
> > > > > of a PATCH method in the REST API has actually increased over time
> > and I'd
> > > > > love to have this option for quick, stop-the-bleeding remediation
> > efforts.
> > > > >
> > > > > One thought that comes to mind is that sometimes it may be useful to
> > > > > explicitly remove properties from a connector configuration. We might
> > > > > permit this by allowing users to specify null (the JSON literal, not
> > a
> > > > > string containing the characters "null") as the value for
> > to-be-removed
> > > > > properties.
> > > > >
> > > > > I'd love to see this change if you're still interested in driving
> > it, Ivan.
> > > > > Hopefully we can give it the attention it deserves in the upcoming
> > months!
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Chris
> > > > >
> > > > > On Fri, Jun 28, 2019 at 4:56 AM Ivan Yurchenko 
> > > > > wrote:
> > > > >
> > > > > > Thank you for your feedback Ryanne!
> > > > > > These are all surely valid concerns and PATCH isn't really
> > necessary or
> > > > > > suitable for normal production configuration management. However,
> > there are
> > > > > > cases where quick patching of the configuration is useful, such as
> > hot
> > > > > > fixes of production or in development.
> > > > > >
> > > > > > Overall, the change itself is really tiny and if the cost-benefit
> > balance
> > > > > > is positive, I'd still like to drive it further.
> > > > > >
> > > > > > Ivan
> > > > > >
> > > > > > On Wed, 26 Jun

[jira] [Created] (KAFKA-16445) PATCH method for connecto configuration

[Ivan Yurchenko (Jira)]

Ivan Yurchenko created KAFKA-16445:
--

 Summary: PATCH method for connecto configuration
 Key: KAFKA-16445
 URL: https://issues.apache.org/jira/browse/KAFKA-16445
 Project: Kafka
  Issue Type: Improvement
  Components: connect
Reporter: Ivan Yurchenko
Assignee: Ivan Yurchenko


As  [KIP-477: Add PATCH method for connector config in Connect REST 
API|https://cwiki.apache.org/confluence/display/KAFKA/KIP-477%3A+Add+PATCH+method+for+connector+config+in+Connect+REST+API]
 suggests, we should introduce the PATCH method for connector configuration.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-477: Add PATCH method for connector config in Connect REST API

[Ivan Yurchenko]

Hi,

I updated the KIP with the two following changes:
1. Using `null` values as tombstone value for removing existing fields from 
configuration.
2. Added a note about the lack of 100% atomicity, which seems very difficult to 
achieve practically.

Ivan


On Tue, Mar 26, 2024, at 14:45, Ivan Yurchenko wrote:
> Speaking of the Chris' comment
> 
> > One thought that comes to mind is that sometimes it may be useful to
> > explicitly remove properties from a connector configuration. We might
> > permit this by allowing users to specify null (the JSON literal, not a
> > string containing the characters "null") as the value for to-be-removed
> > properties.
> 
> This actually makes sense. AFAIU, `null` cannot be in the connector config 
> since https://github.com/apache/kafka/pull/11333, so using it as a tombstone 
> value is a good idea. I can update the KIP.
> 
> Ivan
> 
> 
> On Tue, Mar 26, 2024, at 14:19, Ivan Yurchenko wrote:
> > Hi all,
> > 
> > This KIP is a bit old now :) but I think its context hasn't changed much 
> > since then and the KIP is still valid. I would like to finally bring it to 
> > some conclusion.
> > 
> > Best,
> > Ivan
> > 
> > On 2021/07/12 14:49:47 Chris Egerton wrote:
> > > Hi all,
> > > 
> > > Know it's been a while for this KIP but in my personal experience the 
> > > value
> > > of a PATCH method in the REST API has actually increased over time and I'd
> > > love to have this option for quick, stop-the-bleeding remediation efforts.
> > > 
> > > One thought that comes to mind is that sometimes it may be useful to
> > > explicitly remove properties from a connector configuration. We might
> > > permit this by allowing users to specify null (the JSON literal, not a
> > > string containing the characters "null") as the value for to-be-removed
> > > properties.
> > > 
> > > I'd love to see this change if you're still interested in driving it, 
> > > Ivan.
> > > Hopefully we can give it the attention it deserves in the upcoming months!
> > > 
> > > Cheers,
> > > 
> > > Chris
> > > 
> > > On Fri, Jun 28, 2019 at 4:56 AM Ivan Yurchenko 
> > > wrote:
> > > 
> > > > Thank you for your feedback Ryanne!
> > > > These are all surely valid concerns and PATCH isn't really necessary or
> > > > suitable for normal production configuration management. However, there 
> > > > are
> > > > cases where quick patching of the configuration is useful, such as hot
> > > > fixes of production or in development.
> > > >
> > > > Overall, the change itself is really tiny and if the cost-benefit 
> > > > balance
> > > > is positive, I'd still like to drive it further.
> > > >
> > > > Ivan
> > > >
> > > > On Wed, 26 Jun 2019 at 17:45, Ryanne Dolan  wrote:
> > > >
> > > > > Ivan, I looked at adding PATCH a while ago as well. I decided not to
> > > > pursue
> > > > > the idea for a few reasons:
> > > > >
> > > > > 1) PATCH is still racy. For example, if you want to add a topic to the
> > > > > "topics" property, you still need to read, modify, and write the 
> > > > > existing
> > > > > value. To handle this, you'd need to support atomic sub-document
> > > > > operations, which I don't see happening.
> > > > >
> > > > > 2) A common pattern is to store your configurations in git or 
> > > > > something,
> > > > > and then apply them via PUT. Throw in some triggers or jenkins etc, 
> > > > > and
> > > > you
> > > > > have a more robust solution than PATCH provides.
> > > > >
> > > > > 3) For properties that change a lot, it's possible to use an 
> > > > > out-of-band
> > > > > data source, e.g. Kafka or Zookeeper, and then have your Connector
> > > > > subscribe to changes. I've done something like this to enable dynamic
> > > > > reconfiguration of Connectors from command-line tools and dashboards
> > > > > without involving the Connect REST API at all. Moreover, I've done so 
> > > > > in
> > > > an
> > > > > atomic, non-racy way.
> > > > >
> > > > > So I don't think PATCH is strictly necessary nor sufficient for atomic
> > > > > partial updates. Tha

Re: [ANNOUNCE] New committer: Christo Lolov

[Ivan Yurchenko]

Congrats!

On Tue, Mar 26, 2024, at 14:48, Lucas Brutschy wrote:
> Congrats!
> 
> On Tue, Mar 26, 2024 at 2:44 PM Federico Valeri  wrote:
> >
> > Congrats!
> >
> > On Tue, Mar 26, 2024 at 2:27 PM Mickael Maison  
> > wrote:
> > >
> > > Congratulations Christo!
> > >
> > > On Tue, Mar 26, 2024 at 2:26 PM Chia-Ping Tsai  wrote:
> > > >
> > > > Congrats Christo!
> > > >
> > > > Chia-Ping
> 

Re: [DISCUSS] KIP-477: Add PATCH method for connector config in Connect REST API

[Ivan Yurchenko]

Speaking of the Chris' comment

> One thought that comes to mind is that sometimes it may be useful to
> explicitly remove properties from a connector configuration. We might
> permit this by allowing users to specify null (the JSON literal, not a
> string containing the characters "null") as the value for to-be-removed
> properties.

This actually makes sense. AFAIU, `null` cannot be in the connector config 
since https://github.com/apache/kafka/pull/11333, so using it as a tombstone 
value is a good idea. I can update the KIP.

Ivan


On Tue, Mar 26, 2024, at 14:19, Ivan Yurchenko wrote:
> Hi all,
> 
> This KIP is a bit old now :) but I think its context hasn't changed much 
> since then and the KIP is still valid. I would like to finally bring it to 
> some conclusion.
> 
> Best,
> Ivan
> 
> On 2021/07/12 14:49:47 Chris Egerton wrote:
> > Hi all,
> > 
> > Know it's been a while for this KIP but in my personal experience the value
> > of a PATCH method in the REST API has actually increased over time and I'd
> > love to have this option for quick, stop-the-bleeding remediation efforts.
> > 
> > One thought that comes to mind is that sometimes it may be useful to
> > explicitly remove properties from a connector configuration. We might
> > permit this by allowing users to specify null (the JSON literal, not a
> > string containing the characters "null") as the value for to-be-removed
> > properties.
> > 
> > I'd love to see this change if you're still interested in driving it, Ivan.
> > Hopefully we can give it the attention it deserves in the upcoming months!
> > 
> > Cheers,
> > 
> > Chris
> > 
> > On Fri, Jun 28, 2019 at 4:56 AM Ivan Yurchenko 
> > wrote:
> > 
> > > Thank you for your feedback Ryanne!
> > > These are all surely valid concerns and PATCH isn't really necessary or
> > > suitable for normal production configuration management. However, there 
> > > are
> > > cases where quick patching of the configuration is useful, such as hot
> > > fixes of production or in development.
> > >
> > > Overall, the change itself is really tiny and if the cost-benefit balance
> > > is positive, I'd still like to drive it further.
> > >
> > > Ivan
> > >
> > > On Wed, 26 Jun 2019 at 17:45, Ryanne Dolan  wrote:
> > >
> > > > Ivan, I looked at adding PATCH a while ago as well. I decided not to
> > > pursue
> > > > the idea for a few reasons:
> > > >
> > > > 1) PATCH is still racy. For example, if you want to add a topic to the
> > > > "topics" property, you still need to read, modify, and write the 
> > > > existing
> > > > value. To handle this, you'd need to support atomic sub-document
> > > > operations, which I don't see happening.
> > > >
> > > > 2) A common pattern is to store your configurations in git or something,
> > > > and then apply them via PUT. Throw in some triggers or jenkins etc, and
> > > you
> > > > have a more robust solution than PATCH provides.
> > > >
> > > > 3) For properties that change a lot, it's possible to use an out-of-band
> > > > data source, e.g. Kafka or Zookeeper, and then have your Connector
> > > > subscribe to changes. I've done something like this to enable dynamic
> > > > reconfiguration of Connectors from command-line tools and dashboards
> > > > without involving the Connect REST API at all. Moreover, I've done so in
> > > an
> > > > atomic, non-racy way.
> > > >
> > > > So I don't think PATCH is strictly necessary nor sufficient for atomic
> > > > partial updates. That said, it doesn't hurt and I'm happy to support the
> > > > KIP.
> > > >
> > > > Ryanne
> > > >
> > > > On Tue, Jun 25, 2019 at 12:15 PM Ivan Yurchenko <
> > > ivan0yurche...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Since Kafka 2.3 has just been release and more people may have time to
> > > > look
> > > > > at this now, I'd like to bump this discussion.
> > > > > Thanks.
> > > > >
> > > > > Ivan
> > > > >
> > > > >
> > > > > On Thu, 13 Jun 2019 at 17:20, Ivan Yurchenko  > > >
> > > > > wrote:
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > I'd like to start the discussion of KIP-477: Add PATCH method for
> > > > > > connector config in Connect REST API.
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-477%3A+Add+PATCH+method+for+connector+config+in+Connect+REST+API
> > > > > >
> > > > > > There is also a draft PR: https://github.com/apache/kafka/pull/6934.
> > > > > >
> > > > > > Thank you.
> > > > > >
> > > > > > Ivan
> > > > > >
> > > > >
> > > >
> > >
> > 

RE: Re: [DISCUSS] KIP-477: Add PATCH method for connector config in Connect REST API

[Ivan Yurchenko]

Hi all,

This KIP is a bit old now :) but I think its context hasn't changed much since 
then and the KIP is still valid. I would like to finally bring it to some 
conclusion.

Best,
Ivan

On 2021/07/12 14:49:47 Chris Egerton wrote:
> Hi all,
> 
> Know it's been a while for this KIP but in my personal experience the value
> of a PATCH method in the REST API has actually increased over time and I'd
> love to have this option for quick, stop-the-bleeding remediation efforts.
> 
> One thought that comes to mind is that sometimes it may be useful to
> explicitly remove properties from a connector configuration. We might
> permit this by allowing users to specify null (the JSON literal, not a
> string containing the characters "null") as the value for to-be-removed
> properties.
> 
> I'd love to see this change if you're still interested in driving it, Ivan.
> Hopefully we can give it the attention it deserves in the upcoming months!
> 
> Cheers,
> 
> Chris
> 
> On Fri, Jun 28, 2019 at 4:56 AM Ivan Yurchenko 
> wrote:
> 
> > Thank you for your feedback Ryanne!
> > These are all surely valid concerns and PATCH isn't really necessary or
> > suitable for normal production configuration management. However, there are
> > cases where quick patching of the configuration is useful, such as hot
> > fixes of production or in development.
> >
> > Overall, the change itself is really tiny and if the cost-benefit balance
> > is positive, I'd still like to drive it further.
> >
> > Ivan
> >
> > On Wed, 26 Jun 2019 at 17:45, Ryanne Dolan  wrote:
> >
> > > Ivan, I looked at adding PATCH a while ago as well. I decided not to
> > pursue
> > > the idea for a few reasons:
> > >
> > > 1) PATCH is still racy. For example, if you want to add a topic to the
> > > "topics" property, you still need to read, modify, and write the existing
> > > value. To handle this, you'd need to support atomic sub-document
> > > operations, which I don't see happening.
> > >
> > > 2) A common pattern is to store your configurations in git or something,
> > > and then apply them via PUT. Throw in some triggers or jenkins etc, and
> > you
> > > have a more robust solution than PATCH provides.
> > >
> > > 3) For properties that change a lot, it's possible to use an out-of-band
> > > data source, e.g. Kafka or Zookeeper, and then have your Connector
> > > subscribe to changes. I've done something like this to enable dynamic
> > > reconfiguration of Connectors from command-line tools and dashboards
> > > without involving the Connect REST API at all. Moreover, I've done so in
> > an
> > > atomic, non-racy way.
> > >
> > > So I don't think PATCH is strictly necessary nor sufficient for atomic
> > > partial updates. That said, it doesn't hurt and I'm happy to support the
> > > KIP.
> > >
> > > Ryanne
> > >
> > > On Tue, Jun 25, 2019 at 12:15 PM Ivan Yurchenko <
> > ivan0yurche...@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > Since Kafka 2.3 has just been release and more people may have time to
> > > look
> > > > at this now, I'd like to bump this discussion.
> > > > Thanks.
> > > >
> > > > Ivan
> > > >
> > > >
> > > > On Thu, 13 Jun 2019 at 17:20, Ivan Yurchenko  > >
> > > > wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > I'd like to start the discussion of KIP-477: Add PATCH method for
> > > > > connector config in Connect REST API.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-477%3A+Add+PATCH+method+for+connector+config+in+Connect+REST+API
> > > > >
> > > > > There is also a draft PR: https://github.com/apache/kafka/pull/6934.
> > > > >
> > > > > Thank you.
> > > > >
> > > > > Ivan
> > > > >
> > > >
> > >
> >
> 

[jira] [Created] (KAFKA-15931) Cached transaction index gets closed if tiered storage read is interrupted

[Ivan Yurchenko (Jira)]

Ivan Yurchenko created KAFKA-15931:
--

 Summary: Cached transaction index gets closed if tiered storage 
read is interrupted
 Key: KAFKA-15931
 URL: https://issues.apache.org/jira/browse/KAFKA-15931
 Project: Kafka
  Issue Type: Bug
  Components: Tiered-Storage
Affects Versions: 3.6.0
Reporter: Ivan Yurchenko


This reproduces when reading from remote storage with the default 
{{fetch.max.wait.ms}} (500) or lower. This error is logged

 
{noformat}
[2023-11-29 14:01:01,166] ERROR Error occurred while reading the remote data 
for topic1-0 (kafka.log.remote.RemoteLogReader)
org.apache.kafka.common.KafkaException: Failed read position from the 
transaction index 
    at 
org.apache.kafka.storage.internals.log.TransactionIndex$1.hasNext(TransactionIndex.java:235)
    at 
org.apache.kafka.storage.internals.log.TransactionIndex.collectAbortedTxns(TransactionIndex.java:171)
    at 
kafka.log.remote.RemoteLogManager.collectAbortedTransactions(RemoteLogManager.java:1359)
    at 
kafka.log.remote.RemoteLogManager.addAbortedTransactions(RemoteLogManager.java:1341)
    at kafka.log.remote.RemoteLogManager.read(RemoteLogManager.java:1310)
    at kafka.log.remote.RemoteLogReader.call(RemoteLogReader.java:62)
    at kafka.log.remote.RemoteLogReader.call(RemoteLogReader.java:31)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.nio.channels.ClosedChannelException
    at java.base/sun.nio.ch.FileChannelImpl.ensureOpen(FileChannelImpl.java:150)
    at java.base/sun.nio.ch.FileChannelImpl.position(FileChannelImpl.java:325)
    at 
org.apache.kafka.storage.internals.log.TransactionIndex$1.hasNext(TransactionIndex.java:233)
    ... 10 more
{noformat}
and after that this txn index becomes unusable until the process is restarted.

I suspect, it's caused by the reading thread being interrupted due to the fetch 
timeout. At least [this 
code|https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/19fb8f93c59dfd791f62d41f332db9e306bc1422/src/java.base/share/classes/java/nio/channels/spi/AbstractInterruptibleChannel.java#L159-L160]
 in {{AbstractInterruptibleChannel}} is called.

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

RemoteStorageManager.fetchLogSegment - how to deal with InterruptedException?

[Ivan Yurchenko]

Hello!

`RemoteStorageManager.fetchLogSegment` is called in a background thread by the 
broker [1]. When a fetch request times out, the associated Future is cancelled 
[2] and the thread is interrupted. If the InterruptedException is propagated 
from the `RemoteStorageManager`, it pollutes the broker logs with not very 
useful exceptions and also metrics with false errors [3]. It would be good to 
deal with this somehow. I'm see two options:

1. Add a `catch` to handle `InterruptedException` here [4] in a less noisy way. 
Maybe log on the debug level + not increase the error metrics.
2. Catch `InterruptedException` in the `RemoteStorageManager` implementation 
and return an empty `InputStream`. It seems safe because it goes this way [5]. 
In this case, this needs to be documented in the Javadoc.

Personally, the first seems a better way. What does the community think about 
this? Regardless of the decision, I volunteer to make a PR (and a KIP if 
needed).
Thank you!

Best,
Ivan

[1] 
https://github.com/apache/kafka/blob/61fb83e20280ed3ac83de8290dd9815bdb7efcea/core/src/main/java/kafka/log/remote/RemoteLogManager.java#L1457-L1459
[2] 
https://github.com/apache/kafka/blob/6f197301646135e0bb39a461ca0a07c09c3185fb/core/src/main/scala/kafka/server/DelayedRemoteFetch.scala#L79-L83
[3] 
https://github.com/apache/kafka/blob/8aaf7daff393f2b26438fb7fe28016e06e23558c/core/src/main/java/kafka/log/remote/RemoteLogReader.java#L68-L72
[4] 
https://github.com/apache/kafka/blob/8aaf7daff393f2b26438fb7fe28016e06e23558c/core/src/main/java/kafka/log/remote/RemoteLogReader.java#L68
[5] 
https://github.com/apache/kafka/blob/61fb83e20280ed3ac83de8290dd9815bdb7efcea/core/src/main/java/kafka/log/remote/RemoteLogManager.java#L1276-L1278

Re: [ANNOUNCE] New Kafka PMC Member: Satish Duggana

[Ivan Yurchenko]

Congrats Satish!

Ivan

On Fri, Oct 27, 2023, at 19:02, Kamal Chandraprakash wrote:
> Congratulations Satish!
> 
> On Fri, Oct 27, 2023, 21:10 Jorge Esteban Quilcate Otoya <
> quilcate.jo...@gmail.com> wrote:
> 
> > Congratulations Satish!!
> >
> > On Fri, 27 Oct 2023 at 18:38, Mickael Maison 
> > wrote:
> >
> > > Congratulations Satish!
> > >
> > > On Fri, Oct 27, 2023 at 5:18 PM Lucas Brutschy
> > >  wrote:
> > > >
> > > > Congrats!
> > > >
> > > > On Fri, Oct 27, 2023 at 5:06 PM Manikumar 
> > > wrote:
> > > > >
> > > > > Congrats!
> > > > >
> > > > > On Fri, Oct 27, 2023 at 8:35 PM Jun Rao 
> > > wrote:
> > > > >
> > > > > > Hi, Everyone,
> > > > > >
> > > > > > Satish Duggana has been a Kafka committer since 2022. He has been
> > > very
> > > > > > instrumental to the community since becoming a committer. It's my
> > > pleasure
> > > > > > to announce that Satish is now a member of Kafka PMC.
> > > > > >
> > > > > > Congratulations Satish!
> > > > > >
> > > > > > Jun
> > > > > > on behalf of Apache Kafka PMC
> > > > > >
> > >
> >
> 

RE: [DISCUSS] KIP-905: Broker interceptors

[Ivan Yurchenko]

Hi David,

I wonder if you considered interceptors that work not only on produce request, 
but on any request type the broker is receiving (apart from maybe internal 
request types). For example, being able to rewrite topic names in all the 
request types is essential to create a virtual clusters solutions (many virtual 
clusters, invisible for each other, inside one physical).

Best,
Ivan


On 2023/02/09 19:28:21 David Mariassy wrote:
> Hi everyone,
> 
> I'd like to get a discussion going for KIP-905
> ,
> which proposes the addition of broker interceptors to the stack.
> 
> The KIP contains the motivation, and lists the new public interfaces that
> this change would entail. Since my company had its quarterly hack days this
> week, I also took the liberty to throw together a first prototype of the
> proposed new feature here: https://github.com/apache/kafka/pull/13224.
> 
> Looking forward to the group's feedback!
> 
> Thanks,
> David
> 

RE: Re: [DISCUSS] KIP-905: Broker interceptors

[Ivan Yurchenko]

Hi David and Ahmed,

First, thank you David for the KIP. It would be very valuable for multiple use 
cases. Products like Conduktor Gateway [1] validate the demand and offer many 
potential use cases [2].

Now, I understand Ahmed's concerns about possible in-band interruptions, the 
are valid. However, certain use cases cannot be handled without intercepting 
the request flow to Kafka brokers (for example, the broker-side schema 
validation.) A number of open source and proprietary proxy solutions exist and 
they have their user base, for which the benefits outweigh the risks. In the 
current state, the broker itself already has injection points for custom code 
executed in the hot path of message handling, namely the Authorizer.

One benefit of pluggable interceptors is that they don't affect users who don't 
need and don't use them, so the Kafka robustness remains at the baseline. Those 
who need this functionality, can make their conscious decision. So to me it 
seems this will be positive to Kafka community and ecosystem.

Best regards,
Ivan

[1] https://docs.conduktor.io/gateway/
[2] https://marketplace.conduktor.io/

On 2023/02/10 16:41:01 David Mariassy wrote:
> Hi Ahmed,
> 
> Thanks for taking a look at the KIP, and for your insightful feedback!
> 
> I don't disagree with the sentiment that in-band interceptors could be a
> potential source of bugs in a cluster.
> 
> Having said that, I don't necessarily think that an in-band interceptor is
> significantly riskier than an out-of-band pre-processor. Let's take the
> example of platform-wide privacy scrubbing. In my opinion it doesn't really
> matter if this feature is deployed as an out-of-band stream processor app
> that consumes from all topics OR if the logic is implemented as an in-ban
> interceptor. Either way, a faulty release of the scrubber will result in
> the platform-wide disruption of data flows. Thus, I'd argue that from the
> perspective of the platform's overall health, the level of risk is very
> comparable in both cases. However in-band interceptors have a couple of
> advantages in my opinion:
> 1. They are significantly cheaper (don't require duplicating data between
> raw and sanitized topics. There are also a lot of potential savings in
> network costs)
> 2. They are easier to maintain (no need to set up additional infrastructure
> for out-of-band processing)
> 3. They can provide accurate produce responses to clients (since there is
> no downstream processing that could render a client's messages invalid
> async)
> 
> Also, in-band interceptors could be as safe or risky as their authors
> design them to be. There's nothing stopping someone from catching all
> exceptions in a `processRecord` method, and letting all unprocessed
> messages go through or sending them to a DLQ. Once the interceptor is
> fixed, those unprocessed messages could get re-ingested into Kafka to
> re-attempt pre-processing.
> 
> Thanks and happy Friday,
> David
> 
> 
> 
> 
> 
> On Fri, Feb 10, 2023 at 8:23 AM Ahmed Abdalla 
> wrote:
> 
> > Hi David,
> >
> > That's a very interesting KIP and I wanted to share my two cents. I believe
> > there's a lot of value and use cases for the ability to intercept, mutate
> > and filter Kafka's messages, however I'm not sure if trying to achieve that
> > via in-band interceptors is the best approach for this.
> >
> >- My mental model around one of Kafka's core values is the brokers'
> >focus on a single functionality (more or less): highly available and
> > fault
> >tolerant commit log. I see this in many design decisions such as
> >off-loading responsibilities to the clients (partitioner, assignor,
> >consumer groups coordination etc).
> >- And the impact of this KIP on the Kafka server would be adding another
> >moving part to their "state of the world" that they try to maintain.
> > What
> >if an interceptor goes bad? What if there're version-mismatch? etc, a
> > lot
> >of responsibilities that can be managed very efficiently out-of-band
> > IMHO.
> >- The comparison to NginX and Kubernetes is IMHO comparing apples to
> >oranges
> >   - NginX
> >  - Doesn't maintain persisted data.
> >  - It's designed as a middleware, it's an interceptor by nature.
> >   - Kubernetes
> >  - CRDs extend the API surface, they don't "augment" existing APIs.
> >  I think admission webhooks
> >  <
> > https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/
> > >
> > is
> >  Kubernetes' solution for providing interceptors.
> >  - The admission webhooks are out-of-band, and in fact they're a
> >  great example of "opening up your cluster for extensibility"
> > going wrong.
> >  Installing a misbehaving webhook can brick the whole cluster.
> >
> > As I mentioned, I see a value for users being able to intercept and
> > transform Kafka's messages. But I'm worried that having this as a 

Re: Apache Kafka 3.6.0 release

[Ivan Yurchenko]

Hi Satish and all,

I wonder if https://issues.apache.org/jira/browse/KAFKA-14993 should be 
included in the 3.6 release plan. I'm thinking that when implemented, it would 
be a small, but still a change in the RSM contract: throw an exception instead 
of returning an empty InputStream. Maybe it should be included right away to 
save the migration later? What do you think?

Best,
Ivan

On Fri, Sep 8, 2023, at 02:52, Satish Duggana wrote:
> Hi Jose,
> Thanks for looking into this issue and resolving it with a quick fix.
> 
> ~Satish.
> 
> On Thu, 7 Sept 2023 at 21:40, José Armando García Sancio
>  wrote:
> >
> > Hi Satish,
> >
> > On Wed, Sep 6, 2023 at 4:58 PM Satish Duggana  
> > wrote:
> > >
> > > Hi Greg,
> > > It seems https://issues.apache.org/jira/browse/KAFKA-14273 has been
> > > there in 3.5.x too.
> >
> > I also agree that it should be a blocker for 3.6.0. It should have
> > been a blocker for those previous releases. I didn't fix it because,
> > unfortunately, I wasn't aware of the issue and jira.
> > I'll create a PR with a fix in case the original author doesn't respond in 
> > time.
> >
> > Satish, do you agree?
> >
> > Thanks!
> > --
> > -José
> 

[VOTE] KIP-899: Allow producer and consumer clients to rebootstrap

[Ivan Yurchenko]

Hello,

The discussion [1] for KIP-899 [2] has been open for quite some time. I'd
like to put the KIP up for a vote.

Best,
Ivan

[1] https://lists.apache.org/thread/m0ncbmfxs5m87sszby2jbmtjx2bdpcdl
[2]
https://cwiki.apache.org/confluence/display/KAFKA/KIP-899%3A+Allow+producer+and+consumer+clients+to+rebootstrap

Re: [DISCUSS] KIP-899: Allow clients to rebootstrap

[Ivan Yurchenko]

Hello!

I've made several changes to the KIP based on the comments:

1. Reduced the scope to producer and consumer clients only.
2. Added more details to the description of the rebootstrap process.
3. Documented the role of low values of reconnect.backoff.max.ms in
preventing rebootstrapping.
4. Some wording changes.

You can see the changes in the history [1]

I'm planning to put the KIP to a vote in some days if there are no new
comments.

Thank you!

Ivan

[1]
https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=240881396=9=5

On Tue, 30 May 2023 at 08:23, Ivan Yurchenko 
wrote:

> Hi Chris and all,
>
> > I believe the logic you've linked is only applicable for the producer and
> > consumer clients; the admin client does something different (see [1]).
>
> I see, thank you for the pointer. It seems the admin client is fairly
> different from the producer and consumer. Probably it makes sense to reduce
> the scope of the KIP to the producer and consumer clients only.
>
> > it'd be nice to have a definition of when re-bootstrapping
> > would occur that doesn't rely on internal implementation details. What
> > user-visible phenomena can we identify that would lead to a
> > re-bootstrapping?
>
> Let's put it this way: "Re-bootstrapping means that the client forgets
> about nodes it knows about and falls back on the bootstrap nodes as if it
> had just been initialized. Re-bootstrapping happens when, during a metadata
> update (which may be scheduled by `metadata.max.age.ms` or caused by
> certain error responses like NOT_LEADER_OR_FOLLOWER, REPLICA_NOT_AVAILABLE,
> etc.), the client doesn't have a node with an established connection or
> establishable connection."
> Does this sound good?
>
> > I also believe that if someone has "
> > reconnect.backoff.max.ms" set to a low-enough value,
> > NetworkClient::leastLoadedNode may never return null. In that case,
> > shouldn't we still attempt a re-bootstrap at some point (if the user has
> > enabled this feature)?
>
> Yes, you're right. Particularly `canConnect` here [1] can always be
> returning `true` if `reconnect.backoff.max.ms` is low enough.
> It seems pretty difficult to find a good criteria when re-bootstrapping
> should be forced in this case, so it'd be difficult to configure and reason
> about. I think it's worth mentioning in the KIP and later in the
> documentation, but we should not try to do anything special here.
>
> > Would it make sense to re-bootstrap only after "
> > metadata.max.age.ms" has elapsed since the last metadata update, and
> when
> > at least one request has been made to contact each known server and been
> > met with failure?
>
> The first condition is satisfied by the check in the beginning of
> `maybeUpdate` [2].
> It seems to me, the second one is also satisfied by `leastLoadedNode`.
> Admittedly, it's more relaxed than you propose: it tracks unavailability of
> nodes that was detected by all types of requests, not only by metadata
> requests.
> What do you think, would this be enough?
>
> [1]
> https://github.com/apache/kafka/blob/c9a42c85e2c903329b3550181d230527e90e3646/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java#L698
> [2]
> https://github.com/apache/kafka/blob/c9a42c85e2c903329b3550181d230527e90e3646/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java#L1034-L1041
>
> Best,
> Ivan
>
>
> On Tue, 21 Feb 2023 at 20:07, Chris Egerton 
> wrote:
>
>> Hi Ivan,
>>
>> I believe the logic you've linked is only applicable for the producer and
>> consumer clients; the admin client does something different (see [1]).
>>
>> Either way, it'd be nice to have a definition of when re-bootstrapping
>> would occur that doesn't rely on internal implementation details. What
>> user-visible phenomena can we identify that would lead to a
>> re-bootstrapping? I also believe that if someone has "
>> reconnect.backoff.max.ms" set to a low-enough value,
>> NetworkClient::leastLoadedNode may never return null. In that case,
>> shouldn't we still attempt a re-bootstrap at some point (if the user has
>> enabled this feature)? Would it make sense to re-bootstrap only after "
>> metadata.max.age.ms" has elapsed since the last metadata update, and when
>> at least one request has been made to contact each known server and been
>> met with failure?
>>
>> [1] -
>>
>> https://github.com/apache/kafka/blob/c9a42c85e2c903329b3550181d230527e90e3646/clients/src/main/java/org/apache/kafka/clients/admin/internals/AdminMetadataManager.java#L100
>>
>> Cheers,
>>
>> Chris
>>
>> On Sun, 

Re: [DISCUSS] KIP-917: Additional custom metadata for remote log segment

[Ivan Yurchenko]

Hi!

For those who are interested, I submitted the PR with the implementation:
https://github.com/apache/kafka/pull/13984
Thanks!

Ivan


On Tue, 13 Jun 2023 at 15:05, Ivan Yurchenko 
wrote:

> Hi Satish,
>
> I understand your point and I agree with it. TBH, I didn't take into
> account the in-memory cache.
> Surely, 10 KiB is a relatively arbitrary number. Lowering it to the
> proposed 128 bytes won't probably change anything for most of the
> implementations. Others could ask for an increase.
> I've updated the KIP.
> Thanks!
>
> Ivan
>
>
> On Tue, 13 Jun 2023 at 09:47, Satish Duggana 
> wrote:
>
>> Thanks Ivan for taking the feedback/suggestions into the KIP.
>>
>> 10kb limit as default is huge for each segment metadata. This should
>> only be used for a few keys pointing to more values if needed from a
>> different store. This is intended to be used for purposes like
>> pointing to a specific cluster or a specific bucket etc. This is
>> generally required for identifiers like uuid or other formats that can
>> amount to a few 10s of bytes. So, I suggest having a default value of
>> 128 bytes instead of 10kb. If it requires metadata with a few KBs then
>> it is better to avoid keeping all of them inmemory with default topic
>> based RLMM and have them in a cache with separate storage. In that
>> case, it is better to think about why we need that large metadata with
>> each segment.
>>
>> ~Satish.
>>
>>
>> On Wed, 7 Jun 2023 at 23:38, Ivan Yurchenko 
>> wrote:
>> >
>> > Hi Satish,
>> >
>> > Thank you for your feedback.
>> >
>> > I've nothing against going from Map to byte[].
>> > Serialization should not be a problem for RSM implementations: `Struct`,
>> > `Schema` and other useful serde classes are distributed as a part of the
>> > kafka-clients library.
>> >
>> > Also a good idea to add the size limiting setting, some
>> > `remote.log.metadata.custom.metadata.max.size`. A sensible default may
>> be
>> > 10 KB, which is enough to host a struct with 10 long (almost) 1K symbol
>> > ASCII strings.
>> >
>> > If a piece of custom metadata exceeds the limit, the execution of
>> > RLMTask.copyLogSegmentsToRemote should be interrupted with an error
>> message.
>> >
>> > Does this sound good?
>> > If so, I'll update the KIP accordingly. And I think it may be time for
>> the
>> > vote after that.
>> >
>> > Best,
>> > Ivan
>> >
>> >
>> >
>> > On Sat, 3 Jun 2023 at 17:13, Satish Duggana 
>> > wrote:
>> >
>> > > Hi Ivan,
>> > > Thanks for the KIP.
>> > >
>> > > The motivation of the KIP looks reasonable to me. It requires a way
>> > > for RSM providers to add custom metadata with the existing
>> > > RemoteLogSegmentMetadata. I remember we wanted to introduce a very
>> > > similar change in the earlier proposals called
>> > > RemoteLogMetadataContext. But we dropped that as we did not feel a
>> > > strong need at that time and we wanted to revisit it if needed. But I
>> > > see there is a clear need for this kind of custom metadata to keep
>> > > with RemoteLogSegmentMetadata.
>> > >
>> > > It is better to introduce a new class for this custom metadata in
>> > > RemoteLogSegmentMetadata like below for any changes in the future.
>> > > RemoteLogSegmentMetadata will have this as an optional value.
>> > >
>> > > public class RemoteLogSegmentMetadata {
>> > > ...
>> > > public static class CustomMetadata {
>> > >  private final byte[] value;
>> > > ...
>> > > }
>> > > ...
>> > > }
>> > >
>> > > This is completely opaque and it is the RSM implementation provider's
>> > > responsibility in serializing and deserializing the bytes. We can
>> > > introduce a property to guard the size with a configurable property
>> > > with a default value to avoid any unwanted large size values.
>> > >
>> > > Thanks,
>> > > Satish.
>> > >
>> > > On Tue, 30 May 2023 at 10:59, Ivan Yurchenko <
>> ivan0yurche...@gmail.com>
>> > > wrote:
>> > > >
>> > > > Hi all,
>> > > >
>> > > > I want to bring this to a conclusion (positive or negative), so if
>> there
>> > > > are no more questions

Re: [VOTE] KIP-917: Additional custom metadata for remote log segment

[Ivan Yurchenko]

Hi all,

For your information: while developing this, I made a couple of minor
changes to the KIP:
1. `nullableVersions` was added to the `CustomMetadata` schema.
2. `CustomMetadata` was made `null` by default.
3. The same changes as discussed for `RemoteLogSegmentMetadataRecord` were
added for `RemoteLogSegmentMetadataUpdateRecord`.
4. The documentation for `remote.log.metadata.custom.metadata.max.size` was
slightly reworded for clarity.

You can check the diff [1].

I hope this doesn't affect the consensus on this KIP.

Thanks!

Ivan

[1]
https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=252513269=10=8


On Mon, 19 Jun 2023 at 13:29, Josep Prat 
wrote:

> Hi Ivan,
>
> Thanks and congrats for your first accepted KIP.
> Regarding the process, yes you are right. You should create an issue (this
> can be done pre-KIP as well), it would be good if you could link this vote
> thread in the JIRA. Then you need to update your KIP's wiki page with the
> status (approved now), in your case adding the Jira ticket link as well in
> the right section. Then you should go to the general KIP overview page and
> move your KIP from "under discussion" to "approved" and 3.6.0 as the
> version.
>
> Best,
>
> On Mon, Jun 19, 2023 at 12:20 PM Ivan Yurchenko 
> wrote:
>
> > Hi all,
> >
> > Thank you for your votes.
> >
> > The voting has been open for 7 days and we have:
> > - 4 binding +1: Divij Vaidya, Luke Chen, Josep Prat, and Satish Duggana
> > (based on today's state of https://kafka.apache.org/committers).
> > - 2 non-binding +1: Kamal Chandraprakash and Matthew Benedict de Detrich.
> > - No -1.
> >
> > AFAIU, we can declare the KIP as accepted. I will create a ticket and
> also
> > volunteer to implement it.
> >
> > Please correct me if I'm wrong, that's the first time I'm bringing a KIP
> to
> > this stage :)
> >
> > Thanks!
> >
> > Best,
> > Ivan
> >
> >
> > On Wed, 14 Jun 2023 at 18:28, Satish Duggana 
> > wrote:
> >
> > > Thanks Ivan for addressing the comments in the KIP, LGTM.
> > >
> > > +1
> > >
> > > On Tue, 13 Jun 2023 at 18:31, Luke Chen  wrote:
> > > >
> > > > Looks good. Thanks for the update.
> > > >
> > > > On Tue, Jun 13, 2023 at 8:08 PM Ivan Yurchenko <
> > ivan0yurche...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi all!
> > > > >
> > > > > Thank you for your votes.
> > > > >
> > > > > Based on the proposal of Satish in the discussion thread, I
> modified
> > > the
> > > > > KIP a little bit by lowering the default value of
> > > > > `remote.log.metadata.custom.metadata.max.size` from 10 KiB to 128
> > > bytes. I
> > > > > hope this doesn't change your vote, but feel free to raise your
> > > concerns.
> > > > > Thank you!
> > > > >
> > > > > Best,
> > > > > Ivan
> > > > >
> > > > >
> > > > > On Tue, 13 Jun 2023 at 11:09, Josep Prat
>  > >
> > > > > wrote:
> > > > >
> > > > > > Hi Ivan,
> > > > > >
> > > > > > Thank you very much for this KIP. +1 (binding) from me.
> > > > > >
> > > > > > Best,
> > > > > >
> > > > > >
> > > > > > On Tue, Jun 13, 2023 at 10:03 AM Luke Chen 
> > > wrote:
> > > > > >
> > > > > > > +1 (binding) from me.
> > > > > > >
> > > > > > > Thanks.
> > > > > > > Luke
> > > > > > >
> > > > > > > On Tue, Jun 13, 2023 at 3:44 PM Matthew Benedict de Detrich
> > > > > > >  wrote:
> > > > > > > >
> > > > > > > > +1 (non binding). Thanks for KIP
> > > > > > > >
> > > > > > > > On Tue, Jun 13, 2023 at 3:38 AM Kamal Chandraprakash <
> > > > > > > > kamal.chandraprak...@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > +1 (non-binding). Thanks for the KIP!
> > > > > > > > >
> > > > > > > > > On Mon, Jun 12, 2023, 21:23 Divij Vaidya <
> > > divijvaidy...@gmail.com>
> > > > > > > wrote:
> > > > > > > > >

Re: Apache Kafka 3.6.0 release

[Ivan Yurchenko]

Thank you. If by closing you mean summing up and announcing the result,
then already did.

Ivan


On Mon, 19 Jun 2023 at 15:28, Satish Duggana 
wrote:

> Hi Ivan,
> Sure, KIP freeze date is 26th July 23 for 3.6.0. Please close the
> voting for KIP acceptance before that.
>
> Thanks,
> Satish.
>
> On Mon, 19 Jun 2023 at 16:03, Ivan Yurchenko 
> wrote:
> >
> > Hi,
> >
> > I would like to propose to include the newly accepted "KIP-917:
> Additional
> > custom metadata for remote log segment" [1] in the release plan. Would it
> > be possible?
> > Thanks!
> >
> > Best,
> > Ivan
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-917%3A+Additional+custom+metadata+for+remote+log+segment
> >
> > On Mon, 12 Jun 2023 at 13:22, Satish Duggana 
> > wrote:
> >
> > > Hi,
> > > I have created a release plan for Apache Kafka version 3.6.0 on the
> > > wiki. You can access the release plan and all related information by
> > > following this link:
> > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.6.0
> > >
> > > The release plan outlines the key milestones and important dates for
> > > version 3.6.0. Currently, the following dates have been set for the
> > > release:
> > >
> > > KIP Freeze: 26th July 23
> > > Feature Freeze : 16th Aug 23
> > > Code Freeze : 30th Aug 23
> > >
> > > Please review the release plan and provide any additional information
> > > or updates regarding KIPs targeting version 3.6.0. If you have
> > > authored any KIPs that are missing a status or if there are incorrect
> > > status details, please make the necessary updates and inform me so
> > > that I can keep the plan accurate and up to date.
> > >
> > > Thanks,
> > > Satish.
> > >
> > > On Mon, 17 Apr 2023 at 21:17, Luke Chen  wrote:
> > > >
> > > > Thanks for volunteering!
> > > >
> > > > +1
> > > >
> > > > Luke
> > > >
> > > > On Mon, Apr 17, 2023 at 2:03 AM Ismael Juma 
> wrote:
> > > >
> > > > > Thanks for volunteering Satish. +1.
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Sun, Apr 16, 2023 at 10:08 AM Satish Duggana <
> > > satish.dugg...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > > I would like to volunteer as release manager for the next
> release,
> > > > > > which will be Apache Kafka 3.6.0.
> > > > > >
> > > > > > If there are no objections, I will start a release plan a week
> after
> > > > > > 3.5.0 release(around early May).
> > > > > >
> > > > > > Thanks,
> > > > > > Satish.
> > > > > >
> > > > >
> > >
>

[jira] [Created] (KAFKA-15107) Additional custom metadata for remote log segment

[Ivan Yurchenko (Jira)]

Ivan Yurchenko created KAFKA-15107:
--

 Summary: Additional custom metadata for remote log segment
 Key: KAFKA-15107
 URL: https://issues.apache.org/jira/browse/KAFKA-15107
 Project: Kafka
  Issue Type: Improvement
  Components: core
Reporter: Ivan Yurchenko
Assignee: Ivan Yurchenko
 Fix For: 3.6.0


Based on the [KIP-917: Additional custom metadata for remote log 
segment|https://cwiki.apache.org/confluence/display/KAFKA/KIP-917%3A+Additional+custom+metadata+for+remote+log+segment],
 the following needs to be implemented:
 # {{{}{{RemoteLogSegmentMetadata}}{}}}{{{}.{{{}CustomMetadata{}}}{}}}.
 # {{RemoteStorageManager.copyLogSegmentData}} needs to be updated to the new 
return type (+ javadoc).
 # {{RemoteLogSegmentMetadata.customMetadata}} and 
{{RemoteLogSegmentMetadata.{{{}createWithCustomMetadata{} methods. Same in 
{{{}RemoteLogSegmentMetadataSnapshot{}}}.
 # {{RemoteLogSegmentMetadataRecord}} and 
{{RemoteLogSegmentMetadataSnapshotRecord}} definitions need to be updated.
 # Custom metadata should be persisted by {{RemoteLogManager}} if provided.
 # The new config {{remote.log.metadata.custom.metadata.max.size}} needs to be 
introduced.
 # The custom metadata size limit must be applied according to the KIP.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Apache Kafka 3.6.0 release

[Ivan Yurchenko]

Hi,

I would like to propose to include the newly accepted "KIP-917: Additional
custom metadata for remote log segment" [1] in the release plan. Would it
be possible?
Thanks!

Best,
Ivan

[1]
https://cwiki.apache.org/confluence/display/KAFKA/KIP-917%3A+Additional+custom+metadata+for+remote+log+segment

On Mon, 12 Jun 2023 at 13:22, Satish Duggana 
wrote:

> Hi,
> I have created a release plan for Apache Kafka version 3.6.0 on the
> wiki. You can access the release plan and all related information by
> following this link:
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.6.0
>
> The release plan outlines the key milestones and important dates for
> version 3.6.0. Currently, the following dates have been set for the
> release:
>
> KIP Freeze: 26th July 23
> Feature Freeze : 16th Aug 23
> Code Freeze : 30th Aug 23
>
> Please review the release plan and provide any additional information
> or updates regarding KIPs targeting version 3.6.0. If you have
> authored any KIPs that are missing a status or if there are incorrect
> status details, please make the necessary updates and inform me so
> that I can keep the plan accurate and up to date.
>
> Thanks,
> Satish.
>
> On Mon, 17 Apr 2023 at 21:17, Luke Chen  wrote:
> >
> > Thanks for volunteering!
> >
> > +1
> >
> > Luke
> >
> > On Mon, Apr 17, 2023 at 2:03 AM Ismael Juma  wrote:
> >
> > > Thanks for volunteering Satish. +1.
> > >
> > > Ismael
> > >
> > > On Sun, Apr 16, 2023 at 10:08 AM Satish Duggana <
> satish.dugg...@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > > I would like to volunteer as release manager for the next release,
> > > > which will be Apache Kafka 3.6.0.
> > > >
> > > > If there are no objections, I will start a release plan a week after
> > > > 3.5.0 release(around early May).
> > > >
> > > > Thanks,
> > > > Satish.
> > > >
> > >
>

Re: [VOTE] KIP-917: Additional custom metadata for remote log segment

[Ivan Yurchenko]

Hi all,

Thank you for your votes.

The voting has been open for 7 days and we have:
- 4 binding +1: Divij Vaidya, Luke Chen, Josep Prat, and Satish Duggana
(based on today's state of https://kafka.apache.org/committers).
- 2 non-binding +1: Kamal Chandraprakash and Matthew Benedict de Detrich.
- No -1.

AFAIU, we can declare the KIP as accepted. I will create a ticket and also
volunteer to implement it.

Please correct me if I'm wrong, that's the first time I'm bringing a KIP to
this stage :)

Thanks!

Best,
Ivan


On Wed, 14 Jun 2023 at 18:28, Satish Duggana 
wrote:

> Thanks Ivan for addressing the comments in the KIP, LGTM.
>
> +1
>
> On Tue, 13 Jun 2023 at 18:31, Luke Chen  wrote:
> >
> > Looks good. Thanks for the update.
> >
> > On Tue, Jun 13, 2023 at 8:08 PM Ivan Yurchenko  >
> > wrote:
> >
> > > Hi all!
> > >
> > > Thank you for your votes.
> > >
> > > Based on the proposal of Satish in the discussion thread, I modified
> the
> > > KIP a little bit by lowering the default value of
> > > `remote.log.metadata.custom.metadata.max.size` from 10 KiB to 128
> bytes. I
> > > hope this doesn't change your vote, but feel free to raise your
> concerns.
> > > Thank you!
> > >
> > > Best,
> > > Ivan
> > >
> > >
> > > On Tue, 13 Jun 2023 at 11:09, Josep Prat 
> > > wrote:
> > >
> > > > Hi Ivan,
> > > >
> > > > Thank you very much for this KIP. +1 (binding) from me.
> > > >
> > > > Best,
> > > >
> > > >
> > > > On Tue, Jun 13, 2023 at 10:03 AM Luke Chen 
> wrote:
> > > >
> > > > > +1 (binding) from me.
> > > > >
> > > > > Thanks.
> > > > > Luke
> > > > >
> > > > > On Tue, Jun 13, 2023 at 3:44 PM Matthew Benedict de Detrich
> > > > >  wrote:
> > > > > >
> > > > > > +1 (non binding). Thanks for KIP
> > > > > >
> > > > > > On Tue, Jun 13, 2023 at 3:38 AM Kamal Chandraprakash <
> > > > > > kamal.chandraprak...@gmail.com> wrote:
> > > > > >
> > > > > > > +1 (non-binding). Thanks for the KIP!
> > > > > > >
> > > > > > > On Mon, Jun 12, 2023, 21:23 Divij Vaidya <
> divijvaidy...@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > I have reviewed the proposal and feel that it would be
> beneficial
> > > > to
> > > > > > > > implement.
> > > > > > > >
> > > > > > > > Vote +1 (non-binding)
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Divij Vaidya
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Mon, Jun 12, 2023 at 4:39 PM Ivan Yurchenko <
> > > > > ivan0yurche...@gmail.com
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello,
> > > > > > > > >
> > > > > > > > > Some interest in KIP-917 was expressed in the discussion
> thread
> > > > > [1].
> > > > > > > > After
> > > > > > > > > addressing all the comments there, I'm putting it up to a
> vote.
> > > > > > > > > Thanks!
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Ivan
> > > > > > > > >
> > > > > > > > > [1]
> > > > > https://lists.apache.org/thread/qpccqd3jy5rzvbt5ngtzo3dg9pzp722y
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Matthew de Detrich
> > > > > >
> > > > > > *Aiven Deutschland GmbH*
> > > > > >
> > > > > > Immanuelkirchstraße 26, 10405 Berlin
> > > > > >
> > > > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > > > >
> > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > > >
> > > > > > *m:* +491603708037
> > > > > >
> > > > > > *w:* aiven.io *e:* matthew.dedetr...@aiven.io
> > > > >
> > > >
> > > >
> > > > --
> > > > [image: Aiven] <https://www.aiven.io>
> > > >
> > > > *Josep Prat*
> > > > Open Source Engineering Director, *Aiven*
> > > > josep.p...@aiven.io   |   +491715557497
> > > > aiven.io <https://www.aiven.io>   |   <
> > > https://www.facebook.com/aivencloud
> > > > >
> > > >   <https://www.linkedin.com/company/aiven/>   <
> > > > https://twitter.com/aiven_io>
> > > > *Aiven Deutschland GmbH*
> > > > Alexanderufer 3-7, 10117 Berlin
> > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > >
> > >
>

Re: [VOTE] KIP-917: Additional custom metadata for remote log segment

[Ivan Yurchenko]

Hi all!

Thank you for your votes.

Based on the proposal of Satish in the discussion thread, I modified the
KIP a little bit by lowering the default value of
`remote.log.metadata.custom.metadata.max.size` from 10 KiB to 128 bytes. I
hope this doesn't change your vote, but feel free to raise your concerns.
Thank you!

Best,
Ivan


On Tue, 13 Jun 2023 at 11:09, Josep Prat 
wrote:

> Hi Ivan,
>
> Thank you very much for this KIP. +1 (binding) from me.
>
> Best,
>
>
> On Tue, Jun 13, 2023 at 10:03 AM Luke Chen  wrote:
>
> > +1 (binding) from me.
> >
> > Thanks.
> > Luke
> >
> > On Tue, Jun 13, 2023 at 3:44 PM Matthew Benedict de Detrich
> >  wrote:
> > >
> > > +1 (non binding). Thanks for KIP
> > >
> > > On Tue, Jun 13, 2023 at 3:38 AM Kamal Chandraprakash <
> > > kamal.chandraprak...@gmail.com> wrote:
> > >
> > > > +1 (non-binding). Thanks for the KIP!
> > > >
> > > > On Mon, Jun 12, 2023, 21:23 Divij Vaidya 
> > wrote:
> > > >
> > > > > I have reviewed the proposal and feel that it would be beneficial
> to
> > > > > implement.
> > > > >
> > > > > Vote +1 (non-binding)
> > > > >
> > > > >
> > > > > --
> > > > > Divij Vaidya
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Jun 12, 2023 at 4:39 PM Ivan Yurchenko <
> > ivan0yurche...@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > Some interest in KIP-917 was expressed in the discussion thread
> > [1].
> > > > > After
> > > > > > addressing all the comments there, I'm putting it up to a vote.
> > > > > > Thanks!
> > > > > >
> > > > > > Best,
> > > > > > Ivan
> > > > > >
> > > > > > [1]
> > https://lists.apache.org/thread/qpccqd3jy5rzvbt5ngtzo3dg9pzp722y
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > >
> > > Matthew de Detrich
> > >
> > > *Aiven Deutschland GmbH*
> > >
> > > Immanuelkirchstraße 26, 10405 Berlin
> > >
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > >
> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >
> > > *m:* +491603708037
> > >
> > > *w:* aiven.io *e:* matthew.dedetr...@aiven.io
> >
>
>
> --
> [image: Aiven] <https://www.aiven.io>
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io <https://www.aiven.io>   |   <https://www.facebook.com/aivencloud
> >
>   <https://www.linkedin.com/company/aiven/>   <
> https://twitter.com/aiven_io>
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B
>

Re: [DISCUSS] KIP-917: Additional custom metadata for remote log segment

[Ivan Yurchenko]

Hi Satish,

I understand your point and I agree with it. TBH, I didn't take into
account the in-memory cache.
Surely, 10 KiB is a relatively arbitrary number. Lowering it to the
proposed 128 bytes won't probably change anything for most of the
implementations. Others could ask for an increase.
I've updated the KIP.
Thanks!

Ivan


On Tue, 13 Jun 2023 at 09:47, Satish Duggana 
wrote:

> Thanks Ivan for taking the feedback/suggestions into the KIP.
>
> 10kb limit as default is huge for each segment metadata. This should
> only be used for a few keys pointing to more values if needed from a
> different store. This is intended to be used for purposes like
> pointing to a specific cluster or a specific bucket etc. This is
> generally required for identifiers like uuid or other formats that can
> amount to a few 10s of bytes. So, I suggest having a default value of
> 128 bytes instead of 10kb. If it requires metadata with a few KBs then
> it is better to avoid keeping all of them inmemory with default topic
> based RLMM and have them in a cache with separate storage. In that
> case, it is better to think about why we need that large metadata with
> each segment.
>
> ~Satish.
>
>
> On Wed, 7 Jun 2023 at 23:38, Ivan Yurchenko 
> wrote:
> >
> > Hi Satish,
> >
> > Thank you for your feedback.
> >
> > I've nothing against going from Map to byte[].
> > Serialization should not be a problem for RSM implementations: `Struct`,
> > `Schema` and other useful serde classes are distributed as a part of the
> > kafka-clients library.
> >
> > Also a good idea to add the size limiting setting, some
> > `remote.log.metadata.custom.metadata.max.size`. A sensible default may be
> > 10 KB, which is enough to host a struct with 10 long (almost) 1K symbol
> > ASCII strings.
> >
> > If a piece of custom metadata exceeds the limit, the execution of
> > RLMTask.copyLogSegmentsToRemote should be interrupted with an error
> message.
> >
> > Does this sound good?
> > If so, I'll update the KIP accordingly. And I think it may be time for
> the
> > vote after that.
> >
> > Best,
> > Ivan
> >
> >
> >
> > On Sat, 3 Jun 2023 at 17:13, Satish Duggana 
> > wrote:
> >
> > > Hi Ivan,
> > > Thanks for the KIP.
> > >
> > > The motivation of the KIP looks reasonable to me. It requires a way
> > > for RSM providers to add custom metadata with the existing
> > > RemoteLogSegmentMetadata. I remember we wanted to introduce a very
> > > similar change in the earlier proposals called
> > > RemoteLogMetadataContext. But we dropped that as we did not feel a
> > > strong need at that time and we wanted to revisit it if needed. But I
> > > see there is a clear need for this kind of custom metadata to keep
> > > with RemoteLogSegmentMetadata.
> > >
> > > It is better to introduce a new class for this custom metadata in
> > > RemoteLogSegmentMetadata like below for any changes in the future.
> > > RemoteLogSegmentMetadata will have this as an optional value.
> > >
> > > public class RemoteLogSegmentMetadata {
> > > ...
> > > public static class CustomMetadata {
> > >  private final byte[] value;
> > > ...
> > > }
> > > ...
> > > }
> > >
> > > This is completely opaque and it is the RSM implementation provider's
> > > responsibility in serializing and deserializing the bytes. We can
> > > introduce a property to guard the size with a configurable property
> > > with a default value to avoid any unwanted large size values.
> > >
> > > Thanks,
> > > Satish.
> > >
> > > On Tue, 30 May 2023 at 10:59, Ivan Yurchenko  >
> > > wrote:
> > > >
> > > > Hi all,
> > > >
> > > > I want to bring this to a conclusion (positive or negative), so if
> there
> > > > are no more questions in a couple of days, I'll put the KIP to the
> vote.
> > > >
> > > > Best,
> > > > Ivan
> > > >
> > > >
> > > > On Fri, 5 May 2023 at 18:42, Ivan Yurchenko <
> ivan0yurche...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Alexandre,
> > > > >
> > > > > > combining custom
> > > > > > metadata with rlmMetadata increases coupling between Kafka and
> the
> > > > > > plugin.
> > > > >
> > > > > This is true. However, (if I understand your concern correctly,)
&

Re: [DISCUSS] KIP-917: Additional custom metadata for remote log segment

[Ivan Yurchenko]

Hi all
FYI, I opened this for a vote:
https://lists.apache.org/thread/pnrkgomomwb03h8pfgs0k7dzwgpbtloz

On Tue, 13 Jun 2023 at 08:01, Ivan Yurchenko 
wrote:

> Hi Luke,
>
> > I saw you add the `RemoteLogSegmentMetadataRecord` and
> > `RemoteLogSegmentMetadataSnapshotRecord`, I'd like to confirm we're
> > not adding these records into any new API request/response, right?
>
> Not really, the KIP only extends these records, they were added before.
> And to your question: no, they remain internal and used for the serde
> purpose only. That's basically a convenience to piggyback on the existing
> code generation mechanism we already have to the API.
>
> > I think the serialization/deserialization should be the remote storage
> > plugin provider's responsibility. And in Kafka's point of view, this
> > customMetadata is just a bunch of byte array, we don't have to parse
> > it.
> > Is my understanding correct?
> > If so, maybe you can add some comments into the KIP to mention it, to
> > avoid confusion.
>
> Yes, your understanding is correct. I updated the wording a bit to
> emphasize it's the plugin's responsibility to serialize and deserialize
> these bytes.
>
> Best,
> Ivan
>
>
>
> On Tue, 13 Jun 2023 at 05:30, Luke Chen  wrote:
>
>> Hi Ivan,
>>
>> I agree this is a good improvement for remote storage plugin.
>> One question from me,
>> I saw you add the `RemoteLogSegmentMetadataRecord` and
>> `RemoteLogSegmentMetadataSnapshotRecord`, I'd like to confirm we're
>> not adding these records into any new API request/response, right?
>> I think the serialization/deserialization should be the remote storage
>> plugin provider's responsibility. And in Kafka's point of view, this
>> customMetadata is just a bunch of byte array, we don't have to parse
>> it.
>> Is my understanding correct?
>> If so, maybe you can add some comments into the KIP to mention it, to
>> avoid confusion.
>>
>> Usually the "record" being added in the KIP will be the one affecting
>> the public interface, ex: API request/response.
>> So I'd like to confirm it.
>>
>>
>> Thanks.
>> Luke
>>
>> On Thu, Jun 8, 2023 at 2:09 AM Ivan Yurchenko 
>> wrote:
>> >
>> > Hi Satish,
>> >
>> > Thank you for your feedback.
>> >
>> > I've nothing against going from Map to byte[].
>> > Serialization should not be a problem for RSM implementations: `Struct`,
>> > `Schema` and other useful serde classes are distributed as a part of the
>> > kafka-clients library.
>> >
>> > Also a good idea to add the size limiting setting, some
>> > `remote.log.metadata.custom.metadata.max.size`. A sensible default may
>> be
>> > 10 KB, which is enough to host a struct with 10 long (almost) 1K symbol
>> > ASCII strings.
>> >
>> > If a piece of custom metadata exceeds the limit, the execution of
>> > RLMTask.copyLogSegmentsToRemote should be interrupted with an error
>> message.
>> >
>> > Does this sound good?
>> > If so, I'll update the KIP accordingly. And I think it may be time for
>> the
>> > vote after that.
>> >
>> > Best,
>> > Ivan
>> >
>> >
>> >
>> > On Sat, 3 Jun 2023 at 17:13, Satish Duggana 
>> > wrote:
>> >
>> > > Hi Ivan,
>> > > Thanks for the KIP.
>> > >
>> > > The motivation of the KIP looks reasonable to me. It requires a way
>> > > for RSM providers to add custom metadata with the existing
>> > > RemoteLogSegmentMetadata. I remember we wanted to introduce a very
>> > > similar change in the earlier proposals called
>> > > RemoteLogMetadataContext. But we dropped that as we did not feel a
>> > > strong need at that time and we wanted to revisit it if needed. But I
>> > > see there is a clear need for this kind of custom metadata to keep
>> > > with RemoteLogSegmentMetadata.
>> > >
>> > > It is better to introduce a new class for this custom metadata in
>> > > RemoteLogSegmentMetadata like below for any changes in the future.
>> > > RemoteLogSegmentMetadata will have this as an optional value.
>> > >
>> > > public class RemoteLogSegmentMetadata {
>> > > ...
>> > > public static class CustomMetadata {
>> > >  private final byte[] value;
>> > > ...
>> > > }
>> > > ...
>> > > }
>> > >
>> > > This 

Re: [DISCUSS] KIP-917: Additional custom metadata for remote log segment

[Ivan Yurchenko]

Hi Luke,

> I saw you add the `RemoteLogSegmentMetadataRecord` and
> `RemoteLogSegmentMetadataSnapshotRecord`, I'd like to confirm we're
> not adding these records into any new API request/response, right?

Not really, the KIP only extends these records, they were added before.
And to your question: no, they remain internal and used for the serde
purpose only. That's basically a convenience to piggyback on the existing
code generation mechanism we already have to the API.

> I think the serialization/deserialization should be the remote storage
> plugin provider's responsibility. And in Kafka's point of view, this
> customMetadata is just a bunch of byte array, we don't have to parse
> it.
> Is my understanding correct?
> If so, maybe you can add some comments into the KIP to mention it, to
> avoid confusion.

Yes, your understanding is correct. I updated the wording a bit to
emphasize it's the plugin's responsibility to serialize and deserialize
these bytes.

Best,
Ivan



On Tue, 13 Jun 2023 at 05:30, Luke Chen  wrote:

> Hi Ivan,
>
> I agree this is a good improvement for remote storage plugin.
> One question from me,
> I saw you add the `RemoteLogSegmentMetadataRecord` and
> `RemoteLogSegmentMetadataSnapshotRecord`, I'd like to confirm we're
> not adding these records into any new API request/response, right?
> I think the serialization/deserialization should be the remote storage
> plugin provider's responsibility. And in Kafka's point of view, this
> customMetadata is just a bunch of byte array, we don't have to parse
> it.
> Is my understanding correct?
> If so, maybe you can add some comments into the KIP to mention it, to
> avoid confusion.
>
> Usually the "record" being added in the KIP will be the one affecting
> the public interface, ex: API request/response.
> So I'd like to confirm it.
>
>
> Thanks.
> Luke
>
> On Thu, Jun 8, 2023 at 2:09 AM Ivan Yurchenko 
> wrote:
> >
> > Hi Satish,
> >
> > Thank you for your feedback.
> >
> > I've nothing against going from Map to byte[].
> > Serialization should not be a problem for RSM implementations: `Struct`,
> > `Schema` and other useful serde classes are distributed as a part of the
> > kafka-clients library.
> >
> > Also a good idea to add the size limiting setting, some
> > `remote.log.metadata.custom.metadata.max.size`. A sensible default may be
> > 10 KB, which is enough to host a struct with 10 long (almost) 1K symbol
> > ASCII strings.
> >
> > If a piece of custom metadata exceeds the limit, the execution of
> > RLMTask.copyLogSegmentsToRemote should be interrupted with an error
> message.
> >
> > Does this sound good?
> > If so, I'll update the KIP accordingly. And I think it may be time for
> the
> > vote after that.
> >
> > Best,
> > Ivan
> >
> >
> >
> > On Sat, 3 Jun 2023 at 17:13, Satish Duggana 
> > wrote:
> >
> > > Hi Ivan,
> > > Thanks for the KIP.
> > >
> > > The motivation of the KIP looks reasonable to me. It requires a way
> > > for RSM providers to add custom metadata with the existing
> > > RemoteLogSegmentMetadata. I remember we wanted to introduce a very
> > > similar change in the earlier proposals called
> > > RemoteLogMetadataContext. But we dropped that as we did not feel a
> > > strong need at that time and we wanted to revisit it if needed. But I
> > > see there is a clear need for this kind of custom metadata to keep
> > > with RemoteLogSegmentMetadata.
> > >
> > > It is better to introduce a new class for this custom metadata in
> > > RemoteLogSegmentMetadata like below for any changes in the future.
> > > RemoteLogSegmentMetadata will have this as an optional value.
> > >
> > > public class RemoteLogSegmentMetadata {
> > > ...
> > > public static class CustomMetadata {
> > >  private final byte[] value;
> > > ...
> > > }
> > > ...
> > > }
> > >
> > > This is completely opaque and it is the RSM implementation provider's
> > > responsibility in serializing and deserializing the bytes. We can
> > > introduce a property to guard the size with a configurable property
> > > with a default value to avoid any unwanted large size values.
> > >
> > > Thanks,
> > > Satish.
> > >
> > > On Tue, 30 May 2023 at 10:59, Ivan Yurchenko  >
> > > wrote:
> > > >
> > > > Hi all,
> > > >
> > > > I want to bring this to a conclusion (positive or negative), so if
> there

[VOTE] KIP-917: Additional custom metadata for remote log segment

[Ivan Yurchenko]

Hello,

Some interest in KIP-917 was expressed in the discussion thread [1]. After
addressing all the comments there, I'm putting it up to a vote.
Thanks!

Best,
Ivan

[1] https://lists.apache.org/thread/qpccqd3jy5rzvbt5ngtzo3dg9pzp722y

Re: [DISCUSS] KIP-917: Additional custom metadata for remote log segment

[Ivan Yurchenko]

Hi Satish,

Thank you for your feedback.

I've nothing against going from Map to byte[].
Serialization should not be a problem for RSM implementations: `Struct`,
`Schema` and other useful serde classes are distributed as a part of the
kafka-clients library.

Also a good idea to add the size limiting setting, some
`remote.log.metadata.custom.metadata.max.size`. A sensible default may be
10 KB, which is enough to host a struct with 10 long (almost) 1K symbol
ASCII strings.

If a piece of custom metadata exceeds the limit, the execution of
RLMTask.copyLogSegmentsToRemote should be interrupted with an error message.

Does this sound good?
If so, I'll update the KIP accordingly. And I think it may be time for the
vote after that.

Best,
Ivan



On Sat, 3 Jun 2023 at 17:13, Satish Duggana 
wrote:

> Hi Ivan,
> Thanks for the KIP.
>
> The motivation of the KIP looks reasonable to me. It requires a way
> for RSM providers to add custom metadata with the existing
> RemoteLogSegmentMetadata. I remember we wanted to introduce a very
> similar change in the earlier proposals called
> RemoteLogMetadataContext. But we dropped that as we did not feel a
> strong need at that time and we wanted to revisit it if needed. But I
> see there is a clear need for this kind of custom metadata to keep
> with RemoteLogSegmentMetadata.
>
> It is better to introduce a new class for this custom metadata in
> RemoteLogSegmentMetadata like below for any changes in the future.
> RemoteLogSegmentMetadata will have this as an optional value.
>
> public class RemoteLogSegmentMetadata {
> ...
> public static class CustomMetadata {
>  private final byte[] value;
> ...
> }
> ...
> }
>
> This is completely opaque and it is the RSM implementation provider's
> responsibility in serializing and deserializing the bytes. We can
> introduce a property to guard the size with a configurable property
> with a default value to avoid any unwanted large size values.
>
> Thanks,
> Satish.
>
> On Tue, 30 May 2023 at 10:59, Ivan Yurchenko 
> wrote:
> >
> > Hi all,
> >
> > I want to bring this to a conclusion (positive or negative), so if there
> > are no more questions in a couple of days, I'll put the KIP to the vote.
> >
> > Best,
> > Ivan
> >
> >
> > On Fri, 5 May 2023 at 18:42, Ivan Yurchenko 
> > wrote:
> >
> > > Hi Alexandre,
> > >
> > > > combining custom
> > > > metadata with rlmMetadata increases coupling between Kafka and the
> > > > plugin.
> > >
> > > This is true. However, (if I understand your concern correctly,)
> > > rlmMetadata in the current form may be independent from RSM plugins,
> but
> > > data they point to are accessible only via the particular plugin (the
> one
> > > that wrote the data or a compatible one). It seems, this coupling
> already
> > > exists, but it is implicit. To make my point more concrete, imagine an
> S3
> > > RSM which maps RemoteLogSegmentMetadata objects to S3 object keys. This
> > > mapping logic is a part of the RSM plugin and without it the metadata
> is
> > > useless. I think it will not get worse if (to follow the example) the
> > > plugin makes the said S3 object keys explicit by adding them to the
> > > metadata. From the high level point of view, moving the custom
> metadata to
> > > a separate topic doesn't change the picture: it's still the plugin that
> > > binds the standard and custom metadata together.
> > >
> > >
> > > > For instance, the custom metadata may need to be modified
> > > > outside of Kafka, but the rlmMetadata would still be cached on
> brokers
> > > > independently of any update of custom metadata. Since both types of
> > > > metadata are authored by different systems, and are cached in
> > > > different layers, this may become a problem, or make plugin migration
> > > > more difficult. What do you think?
> > >
> > > This is indeed a problem. I think a solution to this would be to
> clearly
> > > state that metadata being modified outside of Kafka is out of scope and
> > > instruct the plugin authors that custom metadata could be provided only
> > > reactively from the copyLogSegmentData method and must remain immutable
> > > after that. Does it make sense?
> > >
> > >
> > > > Yes, you are right that the suggested alternative is to let the
> plugin
> > > > store its own metadata separately with a solution chosen by the admin
> > > > or plugin provider. For instance, it could be using a dedicated topic
> &g

Re: [DISCUSS] KIP-917: Additional custom metadata for remote log segment

[Ivan Yurchenko]

Hi all,

I want to bring this to a conclusion (positive or negative), so if there
are no more questions in a couple of days, I'll put the KIP to the vote.

Best,
Ivan


On Fri, 5 May 2023 at 18:42, Ivan Yurchenko 
wrote:

> Hi Alexandre,
>
> > combining custom
> > metadata with rlmMetadata increases coupling between Kafka and the
> > plugin.
>
> This is true. However, (if I understand your concern correctly,)
> rlmMetadata in the current form may be independent from RSM plugins, but
> data they point to are accessible only via the particular plugin (the one
> that wrote the data or a compatible one). It seems, this coupling already
> exists, but it is implicit. To make my point more concrete, imagine an S3
> RSM which maps RemoteLogSegmentMetadata objects to S3 object keys. This
> mapping logic is a part of the RSM plugin and without it the metadata is
> useless. I think it will not get worse if (to follow the example) the
> plugin makes the said S3 object keys explicit by adding them to the
> metadata. From the high level point of view, moving the custom metadata to
> a separate topic doesn't change the picture: it's still the plugin that
> binds the standard and custom metadata together.
>
>
> > For instance, the custom metadata may need to be modified
> > outside of Kafka, but the rlmMetadata would still be cached on brokers
> > independently of any update of custom metadata. Since both types of
> > metadata are authored by different systems, and are cached in
> > different layers, this may become a problem, or make plugin migration
> > more difficult. What do you think?
>
> This is indeed a problem. I think a solution to this would be to clearly
> state that metadata being modified outside of Kafka is out of scope and
> instruct the plugin authors that custom metadata could be provided only
> reactively from the copyLogSegmentData method and must remain immutable
> after that. Does it make sense?
>
>
> > Yes, you are right that the suggested alternative is to let the plugin
> > store its own metadata separately with a solution chosen by the admin
> > or plugin provider. For instance, it could be using a dedicated topic
> > if chosen to, or relying on an external key-value store.
>
> I see. Yes, this option always exists and doesn't even require a KIP. The
> biggest drawback I see is that a plugin will need to reimplement the
> consumer/producer + caching mechanics that will exist on the broker side
> for the standard remote metadata. I'd like to avoid this and this KIP is
> the best solution I see.
>
> Best,
> Ivan
>
>
>
> On Tue, 18 Apr 2023 at 13:02, Alexandre Dupriez <
> alexandre.dupr...@gmail.com> wrote:
>
>> Hi Ivan,
>>
>> Thanks for the follow-up.
>>
>> Yes, you are right that the suggested alternative is to let the plugin
>> store its own metadata separately with a solution chosen by the admin
>> or plugin provider. For instance, it could be using a dedicated topic
>> if chosen to, or relying on an external key-value store.
>>
>> I agree with you on the existing risks associated with running
>> third-party code inside Apache Kafka. That said, combining custom
>> metadata with rlmMetadata increases coupling between Kafka and the
>> plugin. For instance, the custom metadata may need to be modified
>> outside of Kafka, but the rlmMetadata would still be cached on brokers
>> independently of any update of custom metadata. Since both types of
>> metadata are authored by different systems, and are cached in
>> different layers, this may become a problem, or make plugin migration
>> more difficult. What do you think?
>>
>> I have a vague memory of this being discussed back when the tiered
>> storage KIP was started. Maybe Satish has more background on this.
>>
>> Thanks,
>> Alexandre
>>
>> Le lun. 17 avr. 2023 à 16:50, Ivan Yurchenko
>>  a écrit :
>> >
>> > Hi Alexandre,
>> >
>> > Thank you for your feedback!
>> >
>> > > One question I would have is, what is the benefit of adding these
>> > > custom metadata in the rlmMetadata rather than letting the plugin
>> > > manage access and persistence to them?
>> >
>> > Could you please elaborate? Do I understand correctly that the idea is
>> that
>> > the plugin will have its own storage for those custom metadata, for
>> example
>> > a special topic?
>> >
>> > > It would be possible for a user
>> > > to use custom metadata large enough to adversely impact access to and
>> > > caching of the rlmMetadata by Kafka.
>> >

Re: [DISCUSS] KIP-899: Allow clients to rebootstrap

[Ivan Yurchenko]

Hi Chris and all,

> I believe the logic you've linked is only applicable for the producer and
> consumer clients; the admin client does something different (see [1]).

I see, thank you for the pointer. It seems the admin client is fairly
different from the producer and consumer. Probably it makes sense to reduce
the scope of the KIP to the producer and consumer clients only.

> it'd be nice to have a definition of when re-bootstrapping
> would occur that doesn't rely on internal implementation details. What
> user-visible phenomena can we identify that would lead to a
> re-bootstrapping?

Let's put it this way: "Re-bootstrapping means that the client forgets
about nodes it knows about and falls back on the bootstrap nodes as if it
had just been initialized. Re-bootstrapping happens when, during a metadata
update (which may be scheduled by `metadata.max.age.ms` or caused by
certain error responses like NOT_LEADER_OR_FOLLOWER, REPLICA_NOT_AVAILABLE,
etc.), the client doesn't have a node with an established connection or
establishable connection."
Does this sound good?

> I also believe that if someone has "
> reconnect.backoff.max.ms" set to a low-enough value,
> NetworkClient::leastLoadedNode may never return null. In that case,
> shouldn't we still attempt a re-bootstrap at some point (if the user has
> enabled this feature)?

Yes, you're right. Particularly `canConnect` here [1] can always be
returning `true` if `reconnect.backoff.max.ms` is low enough.
It seems pretty difficult to find a good criteria when re-bootstrapping
should be forced in this case, so it'd be difficult to configure and reason
about. I think it's worth mentioning in the KIP and later in the
documentation, but we should not try to do anything special here.

> Would it make sense to re-bootstrap only after "
> metadata.max.age.ms" has elapsed since the last metadata update, and when
> at least one request has been made to contact each known server and been
> met with failure?

The first condition is satisfied by the check in the beginning of
`maybeUpdate` [2].
It seems to me, the second one is also satisfied by `leastLoadedNode`.
Admittedly, it's more relaxed than you propose: it tracks unavailability of
nodes that was detected by all types of requests, not only by metadata
requests.
What do you think, would this be enough?

[1]
https://github.com/apache/kafka/blob/c9a42c85e2c903329b3550181d230527e90e3646/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java#L698
[2]
https://github.com/apache/kafka/blob/c9a42c85e2c903329b3550181d230527e90e3646/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java#L1034-L1041

Best,
Ivan


On Tue, 21 Feb 2023 at 20:07, Chris Egerton  wrote:

> Hi Ivan,
>
> I believe the logic you've linked is only applicable for the producer and
> consumer clients; the admin client does something different (see [1]).
>
> Either way, it'd be nice to have a definition of when re-bootstrapping
> would occur that doesn't rely on internal implementation details. What
> user-visible phenomena can we identify that would lead to a
> re-bootstrapping? I also believe that if someone has "
> reconnect.backoff.max.ms" set to a low-enough value,
> NetworkClient::leastLoadedNode may never return null. In that case,
> shouldn't we still attempt a re-bootstrap at some point (if the user has
> enabled this feature)? Would it make sense to re-bootstrap only after "
> metadata.max.age.ms" has elapsed since the last metadata update, and when
> at least one request has been made to contact each known server and been
> met with failure?
>
> [1] -
>
> https://github.com/apache/kafka/blob/c9a42c85e2c903329b3550181d230527e90e3646/clients/src/main/java/org/apache/kafka/clients/admin/internals/AdminMetadataManager.java#L100
>
> Cheers,
>
> Chris
>
> On Sun, Feb 19, 2023 at 3:39 PM Ivan Yurchenko 
> wrote:
>
> > Hi Chris,
> >
> > Thank you for your question. As a part of various lifecycle phases
> > (including node disconnect), NetworkClient can request metadata update
> > eagerly (the `Metadata.requestUpdate` method), which results in
> > `MetadataUpdater.maybeUpdate` being called during next poll. Inside, it
> has
> > a way to find a known node it can connect to for the fresh metadata. If
> no
> > such node is found, it backs off. (Code [1]). I'm thinking of
> piggybacking
> > on this logic and injecting the rebootstrap attempt before the backoff.
> >
> > As of the second part of you question: the re-bootstrapping means
> replacing
> > the node addresses in the client with the original bootstrap addresses,
> so
> > if the first bootstrap attempt fails, the client will continue using the
> > bootstrap addresses until success -- pretty much as if 

Re: [DISCUSS] KIP-917: Additional custom metadata for remote log segment

[Ivan Yurchenko]

Hi Alexandre,

> combining custom
> metadata with rlmMetadata increases coupling between Kafka and the
> plugin.

This is true. However, (if I understand your concern correctly,)
rlmMetadata in the current form may be independent from RSM plugins, but
data they point to are accessible only via the particular plugin (the one
that wrote the data or a compatible one). It seems, this coupling already
exists, but it is implicit. To make my point more concrete, imagine an S3
RSM which maps RemoteLogSegmentMetadata objects to S3 object keys. This
mapping logic is a part of the RSM plugin and without it the metadata is
useless. I think it will not get worse if (to follow the example) the
plugin makes the said S3 object keys explicit by adding them to the
metadata. From the high level point of view, moving the custom metadata to
a separate topic doesn't change the picture: it's still the plugin that
binds the standard and custom metadata together.


> For instance, the custom metadata may need to be modified
> outside of Kafka, but the rlmMetadata would still be cached on brokers
> independently of any update of custom metadata. Since both types of
> metadata are authored by different systems, and are cached in
> different layers, this may become a problem, or make plugin migration
> more difficult. What do you think?

This is indeed a problem. I think a solution to this would be to clearly
state that metadata being modified outside of Kafka is out of scope and
instruct the plugin authors that custom metadata could be provided only
reactively from the copyLogSegmentData method and must remain immutable
after that. Does it make sense?


> Yes, you are right that the suggested alternative is to let the plugin
> store its own metadata separately with a solution chosen by the admin
> or plugin provider. For instance, it could be using a dedicated topic
> if chosen to, or relying on an external key-value store.

I see. Yes, this option always exists and doesn't even require a KIP. The
biggest drawback I see is that a plugin will need to reimplement the
consumer/producer + caching mechanics that will exist on the broker side
for the standard remote metadata. I'd like to avoid this and this KIP is
the best solution I see.

Best,
Ivan



On Tue, 18 Apr 2023 at 13:02, Alexandre Dupriez 
wrote:

> Hi Ivan,
>
> Thanks for the follow-up.
>
> Yes, you are right that the suggested alternative is to let the plugin
> store its own metadata separately with a solution chosen by the admin
> or plugin provider. For instance, it could be using a dedicated topic
> if chosen to, or relying on an external key-value store.
>
> I agree with you on the existing risks associated with running
> third-party code inside Apache Kafka. That said, combining custom
> metadata with rlmMetadata increases coupling between Kafka and the
> plugin. For instance, the custom metadata may need to be modified
> outside of Kafka, but the rlmMetadata would still be cached on brokers
> independently of any update of custom metadata. Since both types of
> metadata are authored by different systems, and are cached in
> different layers, this may become a problem, or make plugin migration
> more difficult. What do you think?
>
> I have a vague memory of this being discussed back when the tiered
> storage KIP was started. Maybe Satish has more background on this.
>
> Thanks,
> Alexandre
>
> Le lun. 17 avr. 2023 à 16:50, Ivan Yurchenko
>  a écrit :
> >
> > Hi Alexandre,
> >
> > Thank you for your feedback!
> >
> > > One question I would have is, what is the benefit of adding these
> > > custom metadata in the rlmMetadata rather than letting the plugin
> > > manage access and persistence to them?
> >
> > Could you please elaborate? Do I understand correctly that the idea is
> that
> > the plugin will have its own storage for those custom metadata, for
> example
> > a special topic?
> >
> > > It would be possible for a user
> > > to use custom metadata large enough to adversely impact access to and
> > > caching of the rlmMetadata by Kafka.
> >
> > Since the custom metadata is 100% under control of the RSM plugin, the
> risk
> > is as big as the risk of running a third-party code (i.e. the RSM
> plugin).
> > The cluster admin must make the decision if they trust it.
> > To mitigate this risk and put it under control, the RSM plugin
> > implementations could document what custom metadata they use and estimate
> > their size.
> >
> > Best,
> > Ivan
> >
> >
> > On Mon, 17 Apr 2023 at 18:14, Alexandre Dupriez <
> alexandre.dupr...@gmail.com>
> > wrote:
> >
> > > Hi Ivan,
> > >
> > > Thank you for th

Re: [DISCUSS] KIP-917: Additional custom metadata for remote log segment

[Ivan Yurchenko]

Hi Alexandre,

Thank you for your feedback!

> One question I would have is, what is the benefit of adding these
> custom metadata in the rlmMetadata rather than letting the plugin
> manage access and persistence to them?

Could you please elaborate? Do I understand correctly that the idea is that
the plugin will have its own storage for those custom metadata, for example
a special topic?

> It would be possible for a user
> to use custom metadata large enough to adversely impact access to and
> caching of the rlmMetadata by Kafka.

Since the custom metadata is 100% under control of the RSM plugin, the risk
is as big as the risk of running a third-party code (i.e. the RSM plugin).
The cluster admin must make the decision if they trust it.
To mitigate this risk and put it under control, the RSM plugin
implementations could document what custom metadata they use and estimate
their size.

Best,
Ivan


On Mon, 17 Apr 2023 at 18:14, Alexandre Dupriez 
wrote:

> Hi Ivan,
>
> Thank you for the KIP.
>
> I think the KIP clearly explains the need for out-of-band metadata
> authored and used by an implementation of the remote storage manager.
> One question I would have is, what is the benefit of adding these
> custom metadata in the rlmMetadata rather than letting the plugin
> manage access and persistence to them?
>
> Maybe one disadvantage and potential risk with the approach proposed
> in the KIP is that the rlmMetadata is not of a predefined, relatively
> constant size (although corner cases with thousands of leader epochs
> in the leader epoch map are possible). It would be possible for a user
> to use custom metadata large enough to adversely impact access to and
> caching of the rlmMetadata by Kafka.
>
> Thanks,
> Alexandre
>
> Le jeu. 6 avr. 2023 à 16:03, hzh0425  a écrit :
> >
> > I think it's a good idea as we may want to store remote segments in
> different buckets
> >
> >
> >
> > | |
> > hzhka...@163.com
> > |
> > |
> > 邮箱：hzhka...@163.com
> > |
> >
> >
> >
> >
> >  回复的原邮件 
> > | 发件人 | Ivan Yurchenko |
> > | 日期 | 2023年04月06日 22:37 |
> > | 收件人 | dev@kafka.apache.org |
> > | 抄送至 | |
> > | 主题 | [DISCUSS] KIP-917: Additional custom metadata for remote log
> segment |
> > Hello!
> >
> > I would like to start the discussion thread on KIP-917: Additional custom
> > metadata for remote log segment [1]
> > This KIP is fairly small and proposes to add a new field to the remote
> > segment metadata.
> >
> > Thank you!
> >
> > Best,
> > Ivan
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-917%3A+Additional+custom+metadata+for+remote+log+segment
>

[DISCUSS] KIP-917: Additional custom metadata for remote log segment

[Ivan Yurchenko]

Hello!

I would like to start the discussion thread on KIP-917: Additional custom
metadata for remote log segment [1]
This KIP is fairly small and proposes to add a new field to the remote
segment metadata.

Thank you!

Best,
Ivan

[1]
https://cwiki.apache.org/confluence/display/KAFKA/KIP-917%3A+Additional+custom+metadata+for+remote+log+segment

[jira] [Created] (KAFKA-14795) Provide message formatter for RemoteLogMetadata

[Ivan Yurchenko (Jira)]

Ivan Yurchenko created KAFKA-14795:
--

 Summary: Provide message formatter for RemoteLogMetadata
 Key: KAFKA-14795
 URL: https://issues.apache.org/jira/browse/KAFKA-14795
 Project: Kafka
  Issue Type: Improvement
  Components: tools
Reporter: Ivan Yurchenko
Assignee: Ivan Yurchenko


For the debugging purpose, it'd be convenient to have a message formatter that 
can display the content of the {{__remote_log_metadata}} topic.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-899: Allow clients to rebootstrap

[Ivan Yurchenko]

Hi Chris,

Thank you for your question. As a part of various lifecycle phases
(including node disconnect), NetworkClient can request metadata update
eagerly (the `Metadata.requestUpdate` method), which results in
`MetadataUpdater.maybeUpdate` being called during next poll. Inside, it has
a way to find a known node it can connect to for the fresh metadata. If no
such node is found, it backs off. (Code [1]). I'm thinking of piggybacking
on this logic and injecting the rebootstrap attempt before the backoff.

As of the second part of you question: the re-bootstrapping means replacing
the node addresses in the client with the original bootstrap addresses, so
if the first bootstrap attempt fails, the client will continue using the
bootstrap addresses until success -- pretty much as if it were recreated
from scratch.

Best,
Ivan

[1]
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java#L1045-L1049

On Thu, 16 Feb 2023 at 17:18, Chris Egerton  wrote:

> Hi Ivan,
>
> I'm not very familiar with the clients side of things but the proposal
> seems reasonable.
>
> I like the flexibility of the "metadata.recovery.strategy" property as a
> string instead of, e.g., a "rebootstrap.enabled" boolean. We may want to
> adapt a different approach in the future, like the background thread
> mentioned in the rejected alternatives section.
>
> I also like handling this via configuration property instead of adding a
> Java-level API or suggesting that users re-instantiate their clients since
> we may want to enable this new behavior by default in the future, and it
> also reduces the level of effort required for users to benefit from this
> improvement.
>
> One question I have--that may have an obvious answer to anyone more
> familiar with client internals--is under which conditions we will determine
> a rebootstrap is appropriate. Taking the admin client as an example, the "
> default.api.timeout.ms" property gives us a limit on the time an operation
> will be allowed to take before it completes or fails (with optional
> per-request overrides in the various *Options classes), and the "
> request.timeout.ms" property gives us a limit on the time each request
> issued for that operation will be allowed to take before it completes, is
> retried, or causes the operation to fail (if no more retries can be
> performed). If all of the known servers (i.e., bootstrap servers for the
> first operation, or discovered brokers if bootstrapping has already been
> completed) are unavailable, the admin client will keep (re)trying to fetch
> metadata until the API timeout is exhausted, issuing multiple requests to
> the same server if necessary. When would a re-bootstrapping occur here?
> Ideally we could find some approach that minimizes false positives (where a
> re-bootstrapping is performed even though the current set of known brokers
> is only temporarily unavailable, as opposed to permanently moved). Of
> course, given the opt-in nature of the re-bootstrapping feature, we can
> always shoot for "good enough" on that front, but, it'd be nice to
> understand some of the potential pitfalls of enabling it.
>
> Following up on the above, would we cache the need to perform a
> re-bootstrap across separate operations? For example, if I try to describe
> a cluster, then a re-bootstrapping takes place and fails, and then I try to
> describe the cluster a second time. With that second attempt, would we
> immediately resort to the bootstrap servers for any initial metadata
> updates, or would we still try to go through the last-known set of brokers
> first?
>
> Cheers,
>
> Chris
>
> On Mon, Feb 6, 2023 at 4:32 AM Ivan Yurchenko 
> wrote:
>
> > Hi!
> >
> > There seems to be not much more discussion going, so I'm planning to
> start
> > the vote in a couple of days.
> >
> > Thanks,
> >
> > Ivan
> >
> > On Wed, 18 Jan 2023 at 12:06, Ivan Yurchenko 
> > wrote:
> >
> > > Hello!
> > > I would like to start the discussion thread on KIP-899: Allow clients
> to
> > > rebootstrap.
> > > This KIP proposes to allow Kafka clients to repeat the bootstrap
> process
> > > when fetching metadata if none of the known nodes are available.
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-899%3A+Allow+clients+to+rebootstrap
> > >
> > > A question right away: should we eventually change the default behavior
> > or
> > > it can remain configurable "forever"? The latter is proposed in the
> KIP.
> > >
> > > Thank you!
> > >
> > > Ivan
> > >
> > >
> > >
> >
>

Re: [DISCUSS] KIP-899: Allow clients to rebootstrap

[Ivan Yurchenko]

Hi!

There seems to be not much more discussion going, so I'm planning to start
the vote in a couple of days.

Thanks,

Ivan

On Wed, 18 Jan 2023 at 12:06, Ivan Yurchenko 
wrote:

> Hello!
> I would like to start the discussion thread on KIP-899: Allow clients to
> rebootstrap.
> This KIP proposes to allow Kafka clients to repeat the bootstrap process
> when fetching metadata if none of the known nodes are available.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-899%3A+Allow+clients+to+rebootstrap
>
> A question right away: should we eventually change the default behavior or
> it can remain configurable "forever"? The latter is proposed in the KIP.
>
> Thank you!
>
> Ivan
>
>
>

Re: [DISCUSS] KIP-899: Allow clients to rebootstrap

[Ivan Yurchenko]

Hi Philip and all,

> if you don't use the client for a long time, why can't you just close the
client and re-instantiate a new one when needed? I'm not familiar with the
stream thread, so I don't know if that's possible.

Yes, it's always possible to recreate the client (I think, it's the main
workaround for this issue).
Streams don't do this currently. If we do the fix on the Streams level
(catch and recreate the client), it solves the problem there, but it
remains unsolved fundamentally and it will be still necessary to do the
same fix in other applications.

> Another idea here is, would it make sense to expose a
maybeUpdateMetadata() API to serve such a purpose?

Could you please elaborate, what exactly this function should do?

Best,
Ivan

On Tue, 24 Jan 2023 at 00:33, Philip Nee  wrote:

> Hey Ivan,
>
> Thanks for the KIP. Some questions for clarification: It seems like the
> main problem is that if we don't poll frequently enough, the cluster
> topology can change entirely before the metadata is refreshed and thus
> causing staled clients. My question is: if you don't use the client for a
> long time, why can't you just close the client and re-instantiate a new one
> when needed? I'm not familiar with the stream thread, so I don't know if
> that's possible.  Another idea here is, would it make sense to expose a
> maybeUpdateMetadata() API to serve such a purpose?
>
> Thanks,
> P
>
>
> On Wed, Jan 18, 2023 at 4:07 AM Ivan Yurchenko 
> wrote:
>
> > Hello!
> > I would like to start the discussion thread on KIP-899: Allow clients to
> > rebootstrap.
> > This KIP proposes to allow Kafka clients to repeat the bootstrap process
> > when fetching metadata if none of the known nodes are available.
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-899%3A+Allow+clients+to+rebootstrap
> >
> > A question right away: should we eventually change the default behavior
> or
> > it can remain configurable "forever"? The latter is proposed in the KIP.
> >
> > Thank you!
> >
> > Ivan
> >
>

Re: [DISCUSS] KIP-899: Allow clients to rebootstrap

[Ivan Yurchenko]

Hi Christo and all,

> Currently a long-running client refreshes their metadata from a set of
brokers obtained when first contacting the cluster. If they have been
“away” for too long those brokers might have all changed and upon trying to
refresh the metadata the client will fail because it cannot find an
available broker. What you propose is that whenever such a situation is
encountered the client should try to get the new set of brokers by
communicating with the bootstrap-servers again.

Yes, your understanding is correct.

> To answer your question, in my opinion this behaviour should not be
guarded by a configuration and should be the default once implemented. As a
customer of Kafka, I cannot think of a situation where I would prefer my
clients to give up if they have stale data without even trying to get the
latest information. As far as I understand, the new behaviour will be
entirely constrained to the client code which makes this change easier.

This sounds reasonable. If we consider the current behavior as a bug, then
the config is probably not needed for the "fix".
It's a slight change of the client behavior and may be considered as
incompatibility. However, I agree with you, it's hard to come up with a
case where the current behavior is preferred over the proposed.
I don't know what's the current approach to this in the community, would be
great to hear some more opinions.

> As a starting point can we confirm that this is indeed the current
behaviour either by a reproducible manual test or by a branch with a
failing unit/integration test?

Yes, the reproduction is very easy and consistent. I reproduced this
manually and I also have tests for both consumers and producers that fail
without the fix [1]

Best,
Ivan.

[1]
https://github.com/ivanyu/kafka/blob/bd769d3eb76bf72f8aaf37553d6a1507dd7f8c55/core/src/test/scala/integration/kafka/api/RebootstrapTest.scala


On Mon, 23 Jan 2023 at 12:26, Christo Lolov  wrote:

> Hello!
>
> Thank you for the KIP. I would like to summarise my understanding of the
> problem in case I am wrong.
>
> Currently a long-running client refreshes their metadata from a set of
> brokers obtained when first contacting the cluster. If they have been
> “away” for too long those brokers might have all changed and upon trying to
> refresh the metadata the client will fail because it cannot find an
> available broker. What you propose is that whenever such a situation is
> encountered the client should try to get the new set of brokers by
> communicating with the bootstrap-servers again.
>
> If I have understood this correctly, then I agree with what is proposed as
> a solution in this KIP. To answer your question, in my opinion this
> behaviour should not be guarded by a configuration and should be the
> default once implemented. As a customer of Kafka, I cannot think of a
> situation where I would prefer my clients to give up if they have stale
> data without even trying to get the latest information. As far as I
> understand, the new behaviour will be entirely constrained to the client
> code which makes this change easier.
>
> As a starting point can we confirm that this is indeed the current
> behaviour either by a reproducible manual test or by a branch with a
> failing unit/integration test?
>
> Best,
> Christo
>
> > On 18 Jan 2023, at 12:07, Ivan Yurchenko 
> wrote:
> >
> > Hello!
> > I would like to start the discussion thread on KIP-899: Allow clients to
> > rebootstrap.
> > This KIP proposes to allow Kafka clients to repeat the bootstrap process
> > when fetching metadata if none of the known nodes are available.
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-899%3A+Allow+clients+to+rebootstrap
> >
> > A question right away: should we eventually change the default behavior
> or
> > it can remain configurable "forever"? The latter is proposed in the KIP.
> >
> > Thank you!
> >
> > Ivan
>

[DISCUSS] KIP-899: Allow clients to rebootstrap

[Ivan Yurchenko]

Hello!
I would like to start the discussion thread on KIP-899: Allow clients to
rebootstrap.
This KIP proposes to allow Kafka clients to repeat the bootstrap process
when fetching metadata if none of the known nodes are available.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-899%3A+Allow+clients+to+rebootstrap

A question right away: should we eventually change the default behavior or
it can remain configurable "forever"? The latter is proposed in the KIP.

Thank you!

Ivan

Re: [VOTE] KIP-797: Accept duplicate listener on port for IPv4/IPv6

[Ivan Yurchenko]

Hi,

Thank you for the KIP.

+1 (non-binding)

Ivan


On Tue, 23 Nov 2021 at 04:18, Luke Chen  wrote:

> Hi Matthew,
> Thanks for the KIP.
> It makes sense to allow IPv4 and IPv6 listening on the same port for the
> listener config.
>
> +1 (non-binding)
>
> Thank you.
> Luke
>
> On Mon, Nov 22, 2021 at 6:28 PM Matthew de Detrich
>  wrote:
>
> > Hello everyone,
> >
> > I would like to start a vote for KIP-797: Accept duplicate listener on
> port
> > for IPv4/IPv6
> >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=195726330
> >
> > The purpose of the KIP is to loosen current validation for non advertised
> > listeners so that you can have an IPv4 address and an IPv6 address on the
> > same port. All other behaviour remains the same as before (since these
> are
> > disparate IP stacks there are no technical reasons not to allow this).
> >
> > PR is located at https://github.com/apache/kafka/pull/11478
> >
> > Comments and feedback are welcome!
> >
> > Regards
> >
> > --
> >
> > Matthew de Detrich
> >
> > *Aiven Deutschland GmbH*
> >
> > Immanuelkirchstraße 26, 10405 Berlin
> >
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >
> > *m:* +491603708037
> >
> > *w:* aiven.io *e:* matthew.dedetr...@aiven.io
> >
>

[jira] [Created] (KAFKA-13376) Allow MirrorMaker producer and consumer customization per replication flow

[Ivan Yurchenko (Jira)]

Ivan Yurchenko created KAFKA-13376:
--

 Summary: Allow MirrorMaker producer and consumer customization per 
replication flow
 Key: KAFKA-13376
 URL: https://issues.apache.org/jira/browse/KAFKA-13376
 Project: Kafka
  Issue Type: Bug
  Components: mirrormaker
Reporter: Ivan Yurchenko


Currently, it's possible to set producer and consumer configurations for a 
cluster in MirrorMaker, like this:

{noformat}
{source}.consumer.{consumer_config_name}
{target}.producer.{producer_config_name}
{noformat}

However, in some cases it makes sense to set these configs differently for 
different replication flows (e.g. when they have different latency/throughput 
trade-offs), something like:

{noformat}
{source}->{target}.{source}.consumer.{consumer_config_name}
{source}->{target}.{target}.producer.{producer_config_name}
{noformat}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: CVE-2021-38153: Timing Attack Vulnerability for Apache Kafka Connect and Clients

[Ivan Yurchenko]

Hi Randall,

Could you please share the JIRA ticket or the fixing commit? It might help
to evaluate the impact better.
Thank you!

Ivan


On Tue, 21 Sept 2021 at 19:37, Randall Hauch  wrote:

> Severity: moderate
>
> Description:
>
> Some components in Apache Kafka use `Arrays.equals` to validate a
> password or key, which is vulnerable to timing attacks that make brute
> force attacks for such credentials more likely to be successful. Users
> should upgrade to 2.8.1 or higher, or 3.0.0 or higher where this
> vulnerability has been fixed. The affected versions include Apache
> Kafka 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1,
> 2.4.0, 2.4.1, 2.5.0, 2.5.1, 2.6.0, 2.6.1, 2.6.2, 2.7.0, 2.7.1, and
> 2.8.0.
>
> Credit:
>
> Apache Kafka would like to thank J. Santilli for reporting this issue.
>
> References:
> https://kafka.apache.org/cve-list
>

Re: [idea] Kafka topic metadata

[Ivan Yurchenko]

Hi Garmes,

I think it's worth writing a separate email for this as described in
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#KafkaImprovementProposals-WhoshouldinitiatetheKIP?
(2nd paragraph).

Best,
Ivan


On Tue, 15 Jun 2021 at 12:04, Garmes Amine 
wrote:

> Hello Israel, Ivan
>
> thank you for your feedback.
>
> Can I have the rights to create the KIP page please.
>
> Bests
> Garmes
>
> > On 14 Jun 2021, at 19:27, Israel Ekpo  wrote:
> >
> > Sorry Ivan and Garmes,
> >
> > I misunderstood the suggestion earlier. I think this will be a great idea
> > for a KIP.
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
> >
> > You were referring to metadata for the actual topic and not its contents.
> >
> > Sorry about that confusion.
> >
> >
> >
> > On Mon, Jun 14, 2021 at 1:05 PM Ivan Yurchenko  >
> > wrote:
> >
> >> Hi,
> >>
> >> Having metadata for topics seems pretty useful. Currently, one has to
> use
> >> external storage for this (e.g. a database) and the question of keeping
> >> topic and metadata in sync exists: A topic is deleted, how to delete its
> >> metadata? How to deal with delete-then-recreate scenarios (well, we have
> >> topic IDs now)? Making Kafka self-sufficient in this aspect sounds good.
> >>
> >> Ivan
> >>
> >>
> >> On Mon, 14 Jun 2021 at 19:55, Israel Ekpo  wrote:
> >>
> >>> Garmes,
> >>>
> >>> I had similar questions in the past but @Matthias J. Sax
> >>>  pointed
> >>> me to this
> >>>
> >>>
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-244%3A+Add+Record+Header+support+to+Kafka+Streams+Processor+API
> >>>
> >>> With the headers, you can filter based on the header content and not
> just
> >>> the contents of the topic.
> >>>
> >>>
> >>>
> >>>
> >>> On Fri, Jun 11, 2021 at 9:04 AM Garmes Amine
> >>  >>>>
> >>> wrote:
> >>>
> >>>> Hello Apache Kafka community,
> >>>>
> >>>> In the driection to make Kafka more cloud friendly, do you think it
> >> make
> >>>> sense to have key-velue metadata in the topic definition,
> >>>> this will allow to search, filter, select ... topics not only by name
> >> but
> >>>> also by metadata
> >>>>
> >>>> Ex:
> >>>> topic name: test
> >>>> topic metadata:
> >>>>   - namespace: space1 (will allow to have multi tenant Kafka cluster)
> >>>>   - project: test
> >>>>   - phase: dev
> >>>>   - type: json
> >>>>   - area: logging
> >>>>
> >>>>
> >>>> Best regards,
> >>>> Garmes
> >>>
> >>
>
>

Re: [idea] Kafka topic metadata

[Ivan Yurchenko]

Hi,

Having metadata for topics seems pretty useful. Currently, one has to use
external storage for this (e.g. a database) and the question of keeping
topic and metadata in sync exists: A topic is deleted, how to delete its
metadata? How to deal with delete-then-recreate scenarios (well, we have
topic IDs now)? Making Kafka self-sufficient in this aspect sounds good.

Ivan


On Mon, 14 Jun 2021 at 19:55, Israel Ekpo  wrote:

> Garmes,
>
> I had similar questions in the past but @Matthias J. Sax
>  pointed
> me to this
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-244%3A+Add+Record+Header+support+to+Kafka+Streams+Processor+API
>
> With the headers, you can filter based on the header content and not just
> the contents of the topic.
>
>
>
>
> On Fri, Jun 11, 2021 at 9:04 AM Garmes Amine  >
> wrote:
>
> > Hello Apache Kafka community,
> >
> > In the driection to make Kafka more cloud friendly, do you think it make
> > sense to have key-velue metadata in the topic definition,
> > this will allow to search, filter, select ... topics not only by name but
> > also by metadata
> >
> > Ex:
> > topic name: test
> > topic metadata:
> >- namespace: space1 (will allow to have multi tenant Kafka cluster)
> >- project: test
> >- phase: dev
> >- type: json
> >- area: logging
> >
> >
> > Best regards,
> > Garmes
>

[jira] [Created] (KAFKA-12835) Topic IDs can mismatch on brokers (after interbroker protocol version update)

[Ivan Yurchenko (Jira)]

Ivan Yurchenko created KAFKA-12835:
--

 Summary: Topic IDs can mismatch on brokers (after interbroker 
protocol version update)
 Key: KAFKA-12835
 URL: https://issues.apache.org/jira/browse/KAFKA-12835
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 2.8.0
Reporter: Ivan Yurchenko


We had a Kafka cluster running 2.8 version with interbroker protocol set to 
2.7. It had a number of topics and everything was fine.
Then we decided to update the interbroker protocol to 2.8 by the following 
procedure:
1. Run new brokers with the interbroker protocol set to 2.8.
2. Move the data from the old brokers to the new ones (normal partition 
reassignment API).
3. Decommission the old brokers.

At the stage 2 we had the problem: old brokers started failing on 
{{LeaderAndIsrRequest}} handling with
{code:java}
ERROR [Broker id=<...>] Topic Id in memory: <...> does not match the topic Id 
for partition <...> provided in the request: <...>. (state.change.logger)
{code}
for multiple topics. Topics were not recreated.

We checked {{partition.metadata}} files and IDs there were indeed different 
from the values in ZooKeeper. It was fixed by deleting the metadata files (and 
letting them be recreated).

 


The logs, unfortunately, didn't show anything that might point to the cause of 
the issue (or it happened longer ago than we store the logs).

We tried to reproduce this also, but no success.

If the community can point out what to check or beware of in future, it will be 
great. We'll be happy to provide additional information if needed. Thank you! 

Sorry for the ticket that might be not very actionable. We hope to at least 
rise awareness of this issue.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-12430) emit.heartbeats.enabled = false should disable heartbeats topic creation

[Ivan Yurchenko (Jira)]

Ivan Yurchenko created KAFKA-12430:
--

 Summary: emit.heartbeats.enabled = false should disable heartbeats 
topic creation
 Key: KAFKA-12430
 URL: https://issues.apache.org/jira/browse/KAFKA-12430
 Project: Kafka
  Issue Type: Bug
  Components: mirrormaker
Reporter: Ivan Yurchenko


Currently, MirrorMaker 2's {{MirrorHeartbeatConnector}} emits heartbeats or not 
based on {{emit.heartbeats.enabled}} setting. However, {{heartbeats}} topic is 
created unconditionally. It seems that the same setting should really disable 
the topic creation as well.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP - 405: Kafka Tiered Storage.

[Ivan Yurchenko]

Hi,

Great!
+1 (non-binding)

Best,
Ivan

On Mon, 15 Feb 2021 at 22:47, Kowshik Prakasam  wrote:

> +1 (non-binding). Thanks for the excellent KIP!
>
>
> Cheers,
> Kowshik
>
>
>
>
>
> On Mon, Feb 15, 2021 at 2:50 AM Manikumar 
> wrote:
>
> > Hi Satish,
> >
> > Thanks for driving this KIP. I’m sure there will be a few tweaks as we
> > implement the KIP, but I
> > think KIP is in good shape.
> >
> > I'm  +1 (binding).
> >
> > Thanks,
> > Manikumar
> >
> > On Thu, Feb 11, 2021 at 10:57 PM Harsha Chintalapani 
> > wrote:
> >
> > > +1 (binding).
> > >
> > > Thanks,
> > > Harsha
> > >
> > > On Thu, Feb 11, 2021 at 6:21 AM Satish Duggana 
> > wrote:
> > >
> > > > Hi All,
> > > > We would like to start voting on “KIP-405: Kafka Tiered Storage”.
> > > >
> > > > For reference here is the KIP:
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage
> > > >
> > > > Thanks,
> > > > Satish.
> > > >
> > >
> >
>

[jira] [Created] (KAFKA-12235) ZkAdminManager.describeConfigs returns no config when 2+ configuration keys are specified

[Ivan Yurchenko (Jira)]

Ivan Yurchenko created KAFKA-12235:
--

 Summary: ZkAdminManager.describeConfigs returns no config when 2+ 
configuration keys are specified
 Key: KAFKA-12235
 URL: https://issues.apache.org/jira/browse/KAFKA-12235
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 2.7.0
Reporter: Ivan Yurchenko
Assignee: Ivan Yurchenko


When {{ZkAdminManager.describeConfigs}} receives {{DescribeConfigsResource}} 
with 2 or more {{configurationKeys}} specified, it returns an empty 
configuration.

Here's a test for {{ZkAdminManagerTest}} that reproduces this issue:
  
{code:scala}
@Test
def testDescribeConfigsWithConfigurationKeys(): Unit = {
  EasyMock.expect(zkClient.getEntityConfigs(ConfigType.Topic, 
topic)).andReturn(TestUtils.createBrokerConfig(brokerId, "zk"))
  EasyMock.expect(metadataCache.contains(topic)).andReturn(true)

  EasyMock.replay(zkClient, metadataCache)

  val resources = List(new DescribeConfigsRequestData.DescribeConfigsResource()
.setResourceName(topic)
.setResourceType(ConfigResource.Type.TOPIC.id)
.setConfigurationKeys(List("retention.ms", "retention.bytes", 
"segment.bytes").asJava)
  )

  val adminManager = createAdminManager()
  val results: List[DescribeConfigsResponseData.DescribeConfigsResult] = 
adminManager.describeConfigs(resources, true, true)
  assertEquals(Errors.NONE.code, results.head.errorCode())
  val resultConfigKeys = results.head.configs().asScala.map(r => r.name()).toSet
  assertEquals(Set("retention.ms", "retention.bytes", "segment.bytes"), 
resultConfigKeys)
}

{code}
Works fine with one configuration key, though.

The patch is following shortly.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9672) Dead broker in ISR cause isr-expiration to fail with exception

[Ivan Yurchenko (Jira)]

Ivan Yurchenko created KAFKA-9672:
-

 Summary: Dead broker in ISR cause isr-expiration to fail with 
exception
 Key: KAFKA-9672
 URL: https://issues.apache.org/jira/browse/KAFKA-9672
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 2.4.0, 2.4.1
Reporter: Ivan Yurchenko


We're running Kafka 2.4 and facing a pretty strange situation.
 Let's say there were three brokers in the cluster 0, 1, and 2. Then:
 1. Broker 3 was added.
 2. Partitions were reassigned from broker 0 to broker 3.
 3. Broker 0 was shut down (not gracefully) and removed from the cluster.
 4. We see the following state in ZooKeeper:
{code:java}
ls /brokers/ids
[1, 2, 3]

get /brokers/topics/foo
{"version":2,"partitions":{"0":[2,1,3]},"adding_replicas":{},"removing_replicas":{}}

get /brokers/topics/foo/partitions/0/state
{"controller_epoch":123,"leader":1,"version":1,"leader_epoch":42,"isr":[0,2,3,1]}
{code}
It means, the dead broker 0 remains in the partitions's ISR. A big share of the 
partitions in the cluster have this issue.

This is actually causing an errors:
{code:java}
Uncaught exception in scheduled task 'isr-expiration' 
(kafka.utils.KafkaScheduler)
org.apache.kafka.common.errors.ReplicaNotAvailableException: Replica with id 12 
is not available on broker 17
{code}
It means that effectively {{isr-expiration}} task is not working any more.

I have a suspicion that this was introduced by [this commit (line 
selected)|https://github.com/apache/kafka/commit/57baa4079d9fc14103411f790b9a025c9f2146a4#diff-5450baca03f57b9f2030f93a480e6969R856]

Unfortunately, I haven't been able to reproduce this in isolation.

Any hints about how to reproduce (so I can write a patch) or mitigate the issue 
on a running cluster are welcome.

Generally, I assume that not throwing {{ReplicaNotAvailableException}} on a 
dead (i.e. non-existent) broker, considering them out-of-sync and removing from 
the ISR should fix the problem.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9478) Controller may stop react on partition reassignment command in ZooKeeper

[Ivan Yurchenko (Jira)]

Ivan Yurchenko created KAFKA-9478:
-

 Summary: Controller may stop react on partition reassignment 
command in ZooKeeper
 Key: KAFKA-9478
 URL: https://issues.apache.org/jira/browse/KAFKA-9478
 Project: Kafka
  Issue Type: Bug
  Components: controller, core
Affects Versions: 2.4.0, 2.4.1
Reporter: Ivan Yurchenko
Assignee: Ivan Yurchenko


Seemingly after 
[bdf2446ccce592f3c000290f11de88520327aa19|https://github.com/apache/kafka/commit/bdf2446ccce592f3c000290f11de88520327aa19],
 the controller may stop watching {{/admin/reassign_partitions}} node in 
ZooKeeper and consequently accept partition reassignment commands via ZooKeeper.

I'm not 100% sure that bdf2446ccce592f3c000290f11de88520327aa19 causes this, 
but it doesn't reproduce on 
[3fe6b5e951db8f7184a4098f8ad8a1afb2b2c1a0|https://github.com/apache/kafka/commit/3fe6b5e951db8f7184a4098f8ad8a1afb2b2c1a0]
 - the one right before it.

Also, reproduces on the trunk HEAD 
[a87decb9e4df5bfa092c26ae4346f65c426f1321|https://github.com/apache/kafka/commit/a87decb9e4df5bfa092c26ae4346f65c426f1321].
h1. How to reproduce

1. Run ZooKeeper and two Kafka brokers.

2. Create a topic with 100 partitions and place them on Broker 0:
{code:bash}
distro/bin/kafka-topics.sh --bootstrap-server localhost:9092,localhost:9093 
--create \
--topic foo \
--replica-assignment $(for i in {0..99}; do echo -n "0,"; done | sed 
's/.$$//')
{code}
3. Add some data:
{code:bash}
seq 1 100 | bin/kafka-console-producer.sh --broker-list 
localhost:9092,localhost:9093 --topic foo
{code}
4. Create the partition reassignment node {{/admin/reassign_partitions}} in Zoo 
and shortly after that update the data in the node (even the same value will 
do). I made a simple Python script for this:
{code:python}
import time
import json
from kazoo.client import KazooClient

zk = KazooClient(hosts='127.0.0.1:2181')
zk.start()

reassign = {
"version": 1,
"partitions":[]
}
for p in range(100):
reassign["partitions"].append({"topic": "foo", "partition": p, 
"replicas": [1]})

zk.create("/admin/reassign_partitions", json.dumps(reassign).encode())

time.sleep(0.05)

zk.set("/admin/reassign_partitions", json.dumps(reassign).encode())
{code}
4. Observe that the controller doesn't react on further updates to 
{{/admin/reassign_partitions}} and doesn't delete the node.

Also, it can be confirmed with
{code:bash}
echo wchc | nc 127.0.0.1 2181
{code}
that there is no watch on the node in ZooKeeper (for this, you should run 
ZooKeeper with {{4lw.commands.whitelist=*}}).

Since it's about timing, it might not work on first attempt, so you might need 
to do 4 a couple of times. However, the reproducibility rate is pretty high.

The data in the topic and the big amount of partitions are not needed per se, 
only to make the timing more favourable.

Controller re-election will solve the issue, but a new controller can be put in 
this state the same way.
h1. Proposed solution

TBD, suggestions are welcome.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

[Ivan Yurchenko]

Hi all,


Jun:
> (a) Cost: S3 list object requests cost $0.005 per 1000 requests. If you
> have 100,000 partitions and want to pull the metadata for each partition
at
> the rate of 1/sec. It can cost $0.5/sec, which is roughly $40K per day.

I want to note here, that no reasonably durable storage will be cheap
at 100k RPS. For example, DynamoDB might give the same ballpark figures.
If we want to keep the pull-based approach, we can try to reduce this number
in several ways: doing listings less frequently (as Satish mentioned,
with the current defaults it's ~3.33k RPS for your example),
batching listing operations in some way (depending on the storage;
it might require the change of RSM's interface).


> There are different ways for doing push based metadata propagation. Some
> object stores may support that already. For example, S3 supports events
> notification
This sounds interesting. However, I see a couple of issues using it:
  1. As I understand the documentation, notification delivery is not
guaranteed
and it's recommended to periodically do LIST to fill the gaps.
Which brings us back to the same LIST consistency guarantees issue.
  2. The same goes for the broker start: to get the current state, we need
to LIST.
  3. The dynamic set of multiple consumers (RSMs): AFAIK SQS and SNS aren't
designed for such a case.


Alexandre:
> A.1 As commented on PR 7561, S3 consistency model [1][2] implies RSM
cannot
> relies solely on S3 APIs to guarantee the expected strong consistency. The
> proposed implementation [3] would need to be updated to take this into
> account. Let’s talk more about this.

Thank you for the feedback. I clearly see the need for changing the S3
implementation
to provide stronger consistency guarantees. As it see from this thread,
there are
several possible approaches to this. Let's discuss RemoteLogManager's
contract and
behavior (like pull vs push model) further before picking one (or several -
?) of them.
I'm going to do some evaluation of DynamoDB for the pull-based approach,
if it's possible to apply it paying a reasonable bill. Also, of the
push-based approach
with a Kafka topic as the medium.


> A.2.3 Atomicity – what does an implementation of RSM need to provide with
> respect to atomicity of the APIs copyLogSegment, cleanupLogUntil and
> deleteTopicPartition? If a partial failure happens in any of those (e.g.
in
> the S3 implementation, if one of the multiple uploads fails [4]),

The S3 implementation is going to change, but it's worth clarifying anyway.
The segment log file is being uploaded after S3 has acked uploading of
all other files associated with the segment and only after this the whole
segment file set becomes visible remotely for operations like
listRemoteSegments [1].
In case of upload failure, the files that has been successfully uploaded
stays
as invisible garbage that is collected by cleanupLogUntil (or overwritten
successfully later).
And the opposite happens during the deletion: log files are deleted first.
This approach should generally work when we solve consistency issues
by adding a strongly consistent storage: a segment's uploaded files remain
invisible garbage until some metadata about them is written.


> A.3 Caching – storing locally the segments retrieved from the remote
> storage is excluded as it does not align with the original intent and even
> defeat some of its purposes (save disk space etc.). That said, could there
> be other types of use cases where the pattern of access to the remotely
> stored segments would benefit from local caching (and potentially
> read-ahead)? Consider the use case of a large pool of consumers which
start
> a backfill at the same time for one day worth of data from one year ago
> stored remotely. Caching the segments locally would allow to uncouple the
> load on the remote storage from the load on the Kafka cluster. Maybe the
> RLM could expose a configuration parameter to switch that feature on/off?

I tend to agree here, caching remote segments locally and making
this configurable sounds pretty practical to me. We should implement this,
maybe not in the first iteration.


Br,
Ivan

[1]
https://github.com/harshach/kafka/pull/18/files#diff-4d73d01c16caed6f2548fc3063550ef0R152

On Thu, 19 Dec 2019 at 19:49, Alexandre Dupriez 
wrote:

> Hi Jun,
>
> Thank you for the feedback. I am trying to understand how a push-based
> approach would work.
> In order for the metadata to be propagated (under the assumption you
> stated), would you plan to add a new API in Kafka to allow the
> metadata store to send them directly to the brokers?
>
> Thanks,
> Alexandre
>
>
> Le mer. 18 déc. 2019 à 20:14, Jun Rao  a écrit :
> >
> > Hi, Satish and Ivan,
> >
> > There are different ways for doing push based metadata propagation. Some
> > object stores may support that already. For example, S3 supports events
> > notification (
> > https://docs.aws.amazon.com/AmazonS3/latest/dev/NotificationHowTo.html).
> > Otherwise one could 

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

[Ivan Yurchenko]

Hi Jun and others.

Jun,
All these are really valid concerns.
Probably we should think about backing implementations like S3 with a
metadata storage whose consistency model and pricing is better that pure
S3, maybe even a Kafka topic (I guess this might be something you refer to
as push-based approach?)

Br,
Ivan

On Thu, 5 Dec 2019 at 00:53, Jun Rao  wrote:

> Hi, Harsha,
>
> Thanks for the reply.
>
> 40/41. I am curious which block storages you have tested. S3 seems to be
> one of the popular block stores. The concerns that I have with pull based
> approach are the following.
> (a) Cost: S3 list object requests cost $0.005 per 1000 requests. If you
> have 100,000 partitions and want to pull the metadata for each partition at
> the rate of 1/sec. It can cost $0.5/sec, which is roughly $40K per day.
> (b) Semantics: S3 list objects are eventually consistent. So, when you do a
> list object request, there is no guarantee that you can see all uploaded
> objects. This could impact the correctness of subsequent logics.
> (c) Efficiency: Blindly pulling metadata when there is no change adds
> unnecessary overhead in the broker as well as in the block store.
>
> So, have you guys tested S3? If so, could you share your experience in
> terms of cost, semantics and efficiency?
>
> Jun
>
>
> On Tue, Dec 3, 2019 at 10:11 PM Harsha Chintalapani 
> wrote:
>
> > Hi Jun,
> >   Thanks for the reply.
> >
> >
> >
> > On Tue, Nov 26, 2019 at 3:46 PM, Jun Rao  wrote:
> >
> > > Hi, Satish and Ying,
> > >
> > > Thanks for the reply.
> > >
> > > 40/41. There are two different ways that we can approach this. One is
> > what
> > > you said. We can have an opinionated way of storing and populating the
> > tier
> > > metadata that we think is good enough for everyone. I am not sure if
> this
> > > is the case based on what's currently proposed in the KIP. For
> example, I
> > > am not sure that (1) everyone always needs local metadata; (2) the
> > current
> > > local storage format is general enough and (3) everyone wants to use
> the
> > > pull based approach to propagate the metadata. Another approach is to
> > make
> > > this pluggable and let the implementor implements the best approach
> for a
> > > particular block storage. I haven't seen any comments from Slack/AirBnb
> > in
> > > the mailing list on this topic. It would be great if they can provide
> > > feedback directly here.
> > >
> >
> > The current interfaces are designed with most popular block storages
> > available today  and we did 2 implementations with these interfaces and
> > they both are yielding good results as we going through the testing of
> it.
> > If there is ever a need for pull based approach  we can definitely evolve
> > the interface.
> > In the past we did mark interfaces to be evolving to make room for
> unknowns
> > in the future.
> > If you have any suggestions around the current interfaces please propose
> we
> > are happy to see if we can work them into it.
> >
> >
> > 43. To offer tier storage as a general feature, ideally all existing
> > > capabilities should still be supported. It's fine if the uber
> > > implementation doesn't support all capabilities for internal usage.
> > > However, the framework should be general enough.
> > >
> >
> > We agree on that as a principle. But all of these major features mostly
> > coming right now and to have a new big feature such as tiered storage to
> > support all the new features will be a big ask. We can document on how do
> > we approach solving these in future iterations.
> > Our goal is to make this tiered storage feature work for everyone.
> >
> > 43.3 This is more than just serving the tier-ed data from block storage.
> > > With KIP-392, the consumer now can resolve the conflicts with the
> replica
> > > based on leader epoch. So, we need to make sure that leader epoch can
> be
> > > recovered properly from tier storage.
> > >
> >
> > We are working on testing our approach and we will update the KIP with
> > design details.
> >
> > 43.4 For JBOD, if tier storage stores the tier metadata locally, we need
> to
> > > support moving such metadata across disk directories since JBOD
> supports
> > > moving data across disks.
> > >
> >
> > KIP is updated with JBOD details. Having said that JBOD tooling needs to
> > evolve to support production loads. Most of the users will be interested
> in
> > using tiered storage without JBOD support support on day 1.
> >
> > Thanks,
> > Harsha
> >
> > As for meeting, we could have a KIP e-meeting on this if needed, but it
> > > will be open to everyone and will be recorded and shared. Often, the
> > > details are still resolved through the mailing list.
> > >
> > > Jun
> > >
> > > On Tue, Nov 19, 2019 at 6:48 PM Ying Zheng 
> > > wrote:
> > >
> > >
> > > Please ignore my previous email
> > > I didn't know Apache requires all the discussions to be "open"
> > >
> > >
> > > On Tue, Nov 19, 2019, 5:40 PM Ying Zheng  wrote:
> > >
> > > Hi Jun,
> > >
> > > Thank you 

Re: [VOTE] KIP-531: Drop support for Scala 2.11 in Kafka 2.5

[Ivan Yurchenko]

Do I understand correctly, that non-commiters can also vote, despite their
votes don't decide?

If so, then +1 from me.

Ivan


On Mon, 18 Nov 2019 at 15:19, Ismael Juma  wrote:

> Hi all,
>
> People seemed supportive in general, so I'd like to start a vote on
> KIP-531:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-531%3A+Drop+support+for+Scala+2.11+in+Kafka+2.5
>
> Ismael
>

[jira] [Created] (KAFKA-9143) DistributedHerder misleadingly log error on connector task reconfiguration

[Ivan Yurchenko (Jira)]

Ivan Yurchenko created KAFKA-9143:
-

 Summary: DistributedHerder misleadingly log error on connector 
task reconfiguration
 Key: KAFKA-9143
 URL: https://issues.apache.org/jira/browse/KAFKA-9143
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Reporter: Ivan Yurchenko
Assignee: Ivan Yurchenko


In {{DistributedHerder}} in {{reconfigureConnectorTasksWithRetry}} method 
there's a 
[callback|https://github.com/apache/kafka/blob/c552c06aed50b4d4d9a85f73ccc89bc06fa7e094/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1247]:

{code:java}
@Override
public void onCompletion(Throwable error, Void result) {
log.error("Unexpected error during connector task reconfiguration: ", 
error);
log.error("Task reconfiguration for {} failed unexpectedly, this connector 
will not be properly reconfigured unless manually triggered.", connName);
}
{code}

It an error message even when the operation succeeded (i.e., {{error}} is 
{{null}}).

It should include {{if (error != null)}} condition, like in the same class [in 
another 
method|https://github.com/apache/kafka/blob/c552c06aed50b4d4d9a85f73ccc89bc06fa7e094/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L792].



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9035) Improve

[Ivan Yurchenko (Jira)]

Ivan Yurchenko created KAFKA-9035:
-

 Summary: Improve 
 Key: KAFKA-9035
 URL: https://issues.apache.org/jira/browse/KAFKA-9035
 Project: Kafka
  Issue Type: Improvement
  Components: KafkaConnect
Affects Versions: 2.2.1, 2.3.0, 2.1.1, 2.2.0, 2.1.0, 2.0.1, 2.0.0
Reporter: Ivan Yurchenko
Assignee: Ivan Yurchenko


Kafka Connect workers in the distributed mode can be set up so that they have 
the same advertised URL (e.g. 
{{[http://127.0.0.1:8083|http://127.0.0.1:8083/]}}). When a request (e.g., for 
connector creation) lands on a worker that is not the leader, it will be 
forwarded to the leader's advertised URL. However, if advertised URLs are all 
the same, it might never reach the leader (due to the limited number of 
forwards). (See https://issues.apache.org/jira/browse/KAFKA-7121 for an 
example.)

I propose to address this by detecting such a situation and warning the user on 
the log.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Need some clarification of Kafka and MQTT

[Ivan Yurchenko]

Hi Sarvesh,

That's true that Kafka by itself can't work with MQTT directly. Kafka
doesn't support any interfaces (like MQTT) apart from its own
general-purpose producer interface. Also Kafka is quite passive in
the terms of getting data in or out, i.e., you need some application to
read data from it or write them to it.

On the other hand, there is a component in Kafka ecosystem called Kafka
Connect [1]. You can find it in the standard Apache Kafka distribution and
run it along with your Kafka cluster. It allows to run specialized pieces
of code called connectors that read data from some sources and put them
into Kafka or vice versa. There are many of them around and most are open
source.

Your use case sounds like a perfect fit for using Kafka Connect. You will
need Lenses MQTT connectors [2] [3] to run in Kafka Connect, but they are
open source and run where you run your Kafka Connect.

An alternative would be to just write your own application that reads from
Kafka and writes to MQTT or vice versa using Kafka's producers and
consumers.

I hope this helps!

Best,
Ivan Yurchenko

[1] https://kafka.apache.org/documentation/#connect
[2] https://docs.lenses.io/connectors/sink/mqtt.html
[3] https://docs.lenses.io/connectors/source/mqtt.html

On Tue, 8 Oct 2019 at 19:01, Sarvesh Gupta 
wrote:

> Hi,
>
> We are building some IoT driven solution for industries, I am very new to
> Kafka and I need some clarifications so please help me out with below given
> doubts.
> We have use case where we wanted to connect Kafka with MQTT as source of
> receiving data, But I am little bit confused on this. I saw lots of blog
> and videos where people are connecting Kafka with MQTT using confluent,
> lenses and some another third party libraries. So my question is can’t we
> directly connect Kafka with MQTT without using any third party dependencies.
> And I also wanted to know what are ways to connect Kafka and MQTT. If
> there is any way to connect apache Kafka and MQTT directly without using
> confluent or other platforms then please give me some idea about that
> method. And also we are using python language for out product so if there
> is any pythonic way to do this then please let me know.
>
>
> Thanks and Regards,
> Sarvesh Gupta
>
>

Re: Are segment files identical in different replicas?

[Ivan Yurchenko]

Hi Lisheng,

Thank you.

By "byte level" I literally mean bytes stored on disk.

I'm looking into scenarios like this: in a cluster of Kafka brokers, a
topic is replicated in N replicas. Messages are being produced to the
topic. At some point, a segment e.g. 0123 overflows and
becomes inactive, the log file 0123.log is closed on all
brokers. Let's assume the same range of offsets is written to this segment
on all the brokers. So I wonder if these files are identical on the byte
level, that is literally byte-by-byte comparison of files.

Let's say we stick to magic number 2 or higher, Kafka version 2.3 or higher
and as I mention previously, ignore compaction.

I looked into the replication code and DefaultRecordBatch and DefaultRecord
implementations and it seems they should be, since the format is rather
straightforward. I also did some tests and it seems to be true at least for
a normally operating cluster. But I'd like to tap into the community
knowledge on this.

Br,
Ivan

On Mon, 2 Sep 2019 at 05:52, Lisheng Wang  wrote:

> Hi Ivan
>
> your assumption about question 1 is true,  they will have same records in
> same order.
>
> about question 2,  i'm not following what did you mean about "byte level",
> could you plz to make more explanations?
>
> Best,
> Lisheng
>
>
> Ivan Yurchenko  于2019年8月30日周五 下午8:39写道：
>
> > Hi,
> >
> > Let's say I have a topic-partition replicated to several replicas. On
> each
> > replica there is a segment of this topic-partition containing records
> with
> > offsets N..M. I'm trying to figure out:
> > 1. Will the content of these segment files be identical on the logical
> > level? I.e., will they contain the same records in the same order,
> nothing
> > skipped, nothing extra? (I assume this is true, sounds pretty logical.)
> > 2. Will the content of these segment files be identical on the byte
> level?
> > 3. If 2 is no, will at least record layout (positions) be identical?
> >
> > Let's keep topic compaction out of consideration here.
> >
> > Could you please help me figure this out?
> >
> > Br,
> > Ivan
> >
>

Are segment files identical in different replicas?

[Ivan Yurchenko]

Hi,

Let's say I have a topic-partition replicated to several replicas. On each
replica there is a segment of this topic-partition containing records with
offsets N..M. I'm trying to figure out:
1. Will the content of these segment files be identical on the logical
level? I.e., will they contain the same records in the same order, nothing
skipped, nothing extra? (I assume this is true, sounds pretty logical.)
2. Will the content of these segment files be identical on the byte level?
3. If 2 is no, will at least record layout (positions) be identical?

Let's keep topic compaction out of consideration here.

Could you please help me figure this out?

Br,
Ivan

Re: [DISCUSS] KIP-477: Add PATCH method for connector config in Connect REST API

[Ivan Yurchenko]

Thank you for your feedback Ryanne!
These are all surely valid concerns and PATCH isn't really necessary or
suitable for normal production configuration management. However, there are
cases where quick patching of the configuration is useful, such as hot
fixes of production or in development.

Overall, the change itself is really tiny and if the cost-benefit balance
is positive, I'd still like to drive it further.

Ivan

On Wed, 26 Jun 2019 at 17:45, Ryanne Dolan  wrote:

> Ivan, I looked at adding PATCH a while ago as well. I decided not to pursue
> the idea for a few reasons:
>
> 1) PATCH is still racy. For example, if you want to add a topic to the
> "topics" property, you still need to read, modify, and write the existing
> value. To handle this, you'd need to support atomic sub-document
> operations, which I don't see happening.
>
> 2) A common pattern is to store your configurations in git or something,
> and then apply them via PUT. Throw in some triggers or jenkins etc, and you
> have a more robust solution than PATCH provides.
>
> 3) For properties that change a lot, it's possible to use an out-of-band
> data source, e.g. Kafka or Zookeeper, and then have your Connector
> subscribe to changes. I've done something like this to enable dynamic
> reconfiguration of Connectors from command-line tools and dashboards
> without involving the Connect REST API at all. Moreover, I've done so in an
> atomic, non-racy way.
>
> So I don't think PATCH is strictly necessary nor sufficient for atomic
> partial updates. That said, it doesn't hurt and I'm happy to support the
> KIP.
>
> Ryanne
>
> On Tue, Jun 25, 2019 at 12:15 PM Ivan Yurchenko 
> wrote:
>
> > Hi,
> >
> > Since Kafka 2.3 has just been release and more people may have time to
> look
> > at this now, I'd like to bump this discussion.
> > Thanks.
> >
> > Ivan
> >
> >
> > On Thu, 13 Jun 2019 at 17:20, Ivan Yurchenko 
> > wrote:
> >
> > > Hello,
> > >
> > > I'd like to start the discussion of KIP-477: Add PATCH method for
> > > connector config in Connect REST API.
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-477%3A+Add+PATCH+method+for+connector+config+in+Connect+REST+API
> > >
> > > There is also a draft PR: https://github.com/apache/kafka/pull/6934.
> > >
> > > Thank you.
> > >
> > > Ivan
> > >
> >
>

Re: [DISCUSS] KIP-477: Add PATCH method for connector config in Connect REST API

[Ivan Yurchenko]

Hi,

Since Kafka 2.3 has just been release and more people may have time to look
at this now, I'd like to bump this discussion.
Thanks.

Ivan


On Thu, 13 Jun 2019 at 17:20, Ivan Yurchenko 
wrote:

> Hello,
>
> I'd like to start the discussion of KIP-477: Add PATCH method for
> connector config in Connect REST API.
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-477%3A+Add+PATCH+method+for+connector+config+in+Connect+REST+API
>
> There is also a draft PR: https://github.com/apache/kafka/pull/6934.
>
> Thank you.
>
> Ivan
>

[DISCUSS] KIP-477: Add PATCH method for connector config in Connect REST API

[Ivan Yurchenko]

Hello,

I'd like to start the discussion of KIP-477: Add PATCH method for connector
config in Connect REST API.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-477%3A+Add+PATCH+method+for+connector+config+in+Connect+REST+API

There is also a draft PR: https://github.com/apache/kafka/pull/6934.

Thank you.

Ivan

Permission to create KIP

[Ivan Yurchenko]

Hi,

I'd like to be able to create KIPs. Could someone please set up the
permissions for me? The username is ivanyu .
Thanks!

Ivan

Add to contributors list

[Ivan Yurchenko]

Hello,

Could somebody add me to the Kafka contributor list so I can assign myself
tickets in JIRA? My ID/username is ivanyu.
Thank you.

Best,
Ivan

Add to contributors list

[Ivan Yurchenko]

Hello,

Could somebody add me to the Kafka contributor list so I can assign myself
tickets in JIRA? My ID/username is ivanyu.
Thank you.

Best,
Ivan