date:20220509

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #917

2022-05-09 Thread Apache Jenkins Server

See

[jira] [Created] (KAFKA-13890) Improve documentation of ssl.keystore.type and ssl.truststore.type

2022-05-09 Thread RivenSun (Jira)

RivenSun created KAFKA-13890:


 Summary: Improve documentation of ssl.keystore.type and 
ssl.truststore.type
 Key: KAFKA-13890
 URL: https://issues.apache.org/jira/browse/KAFKA-13890
 Project: Kafka
  Issue Type: Improvement
  Components: documentation
Reporter: RivenSun


As mentioned in 
[KIP-651|https://cwiki.apache.org/confluence/display/KAFKA/KIP-651+-+Support+PEM+format+for+SSL+certificates+and+private+key]
{quote}A new key store type `PEM` will be added for key and trust stores. Both 
ssl.keystore.type and ssl.truststore.type may specify PEM in addition to JKS 
and PKCS12.
{quote}
 

Now the documentation of these two parameters does not indicate which optional 
values are available, only a default value is shown.
Take the 
[ssl.truststore.type|https://kafka.apache.org/documentation/#brokerconfigs_ssl.truststore.type]
 parameter as an example,
The revised document is:
{code:java}
The file format of the trust store file. Valid values are: JKS , PKCS12, 
PEM.{code}
At the same time, we can also add corresponding validators for these two 
parameters.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] KIP-834: Pause / Resume KafkaStreams Topologies

2022-05-09 Thread Sophie Blee-Goldman

Thanks Jim, just one note/question on the standby tasks:

At the minute, my moderately held position is that standby tasks ought to
> continue reading and remain caught up.  If standby tasks would run out of
> space, there are probably bigger problems.

For a single node application, or when the #pause API is invoked on all
instances,
then there won't be any further active processing and thus nothing to keep
up with,
right? So for that case, it's just a matter of whether any standbys that
are lagging
will have the chance to catch up to the (paused) active task state before
they stop
as well, in which case having them continue feels fine to me. However this
is a
relatively trivial benefit and I would only consider it as a deciding
factor when all
things are equal otherwise.

My concern is the more interesting case: when this feature is used to pause
only
one nodes, or some subset of the overall application. In this case, yes,
the standby
tasks will indeed fall out of sync. But the only reason I can imagine
someone using
the pause feature in such a way is because there is something going wrong,
or about
to go wrong, on that particular node. For example as mentioned above, if
the user
wants to cut down on costs without stopping everything, or if the node is
about to
run out of disk or needs to be debugged or so on. And in this case,
continuing to
process the standby tasks while other instances continue to run would
pretty much
defeat the purpose of pausing it entirely, and might have unpleasant
consequences
for the unsuspecting developer.

All that said, I don't want to block this KIP so if you have strong
feelings about the
standby behavior I'm happy to back down. I'm only pushing back now because
it
felt like there wasn't any particular motivation for the standbys to
continue processing
or not, and I figured I'd try to fill in this gap with my thoughts on the
matter :)
Either way we should just make sure that this behavior is documented
clearly,
since it may be surprising if we decide to only pause active processing
(another option
is to rename the method something like #pauseProcessing or
#pauseActiveProcessing
so that it's hard to miss).

Thanks! Sorry for the lengthy response, but hopefully we won't need to
debate this any
further. Beyond this I'm satisfied with the latest proposal

On Mon, May 9, 2022 at 5:16 PM John Roesler  wrote:

> Thanks for the updates, Jim!
>
> After this discussion and your updates, this KIP looks good to me.
>
> Thanks,
> John
>
> On Mon, May 9, 2022, at 17:52, Jim Hughes wrote:
> > Hi Sophie, all,
> >
> > I've updated the KIP with feedback from the discussion so far:
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211882832
> >
> > As a terse summary of my current position:
> > Pausing will only stop processing and punctuation (respecting modular
> > topologies).
> > Paused topologies will still a) consume from input topics, b) call the
> > usual commit pathways (commits will happen basically as they would have),
> > and c) standBy tasks will still be processed.
> >
> > Shout if the KIP or those details still need some TLC.  Responding to
> > Sophie inline below.
> >
> >
> > On Mon, May 9, 2022 at 6:06 PM Sophie Blee-Goldman
> >  wrote:
> >
> >> Don't worry, I'm going to be adding the APIs for topology-level pausing
> as
> >> part of the modular topologies KIP,
> >> so we don't need to worry about that for now. That said, I don't think
> we
> >> should brush it off entirely and design
> >> this feature in a way that's going to be incompatible or hugely raise
> the
> >> LOE on bringing the (mostly already
> >> implemented) modular topologies feature into the public API, just
> >> because it "won the race to write a KIP" :)
> >>
> >
> > Yes, I'm hoping that this is all compatible with modular topologies.  I
> > haven't seen anything so far which seems to be a problem; this KIP is
> just
> > in a weird state to discuss details of acting on modular topologies.:)
> >
> >
> >> I may be biased (ok, I definitely am), but I'm not in favor of adding
> this
> >> as a state regardless of the modular topologies.
> >> First of all any change to the KafkaStreams state machine is a breaking
> >> change, no? So we would have to wait until
> >> the next major release which seems like an unnecessary thing to block
> on.
> >> (Whether to add this as a state to the
> >> StreamThread's FSM is an implementation detail).
> >>
> >
> > +1.  I am sold on skipping out on new states.  I had that as a rejected
> > alternative in the KIP and have added a few more words to that bit.
> >
> >
> >> Also, the semantics of using an `isPaused` method to distinguish a
> paused
> >> instance (or topology) make more sense
> >> to me -- this is a user-specified status, whereas the KafkaStreams
> state is
> >> intended to relay the status of the system
> >> itself. For example, if we are going to continue to poll during pause,
> then
> >> shouldn't the client transition to REBALANCING?
> >> I

Re: [DISCUSS] KIP-836: Addition of Information in DescribeQuorumResponse about Voter Lag

2022-05-09 Thread deng ziming

Hello Niket, currently DescribeQuorumResponse is not a public API, we don’t 
have a Admin api or shell script to get DescribeQuorumResponse, so it’s 
unnecessary to submit a KIP to change it, you can just submit a PR to 
accomplish this.

--
Thanks
Ziming

> On May 10, 2022, at 1:33 AM, Niket Goel  wrote:
> 
> Hi all,
> 
> I created a KIP to add some more information to `DesscribeQuorumResponse` to 
> enable ascertaining voter lag in the quorum a little better.
> Please see KIP -- 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-836%3A+Additional+Information+in+DescribeQuorumResponse+about+Voter+Lag
> 
> Thanks for your feedback,
> Niket Goel

Re: [DISCUSS] KIP-834: Pause / Resume KafkaStreams Topologies

2022-05-09 Thread John Roesler

Thanks for the updates, Jim!

After this discussion and your updates, this KIP looks good to me. 

Thanks,
John

On Mon, May 9, 2022, at 17:52, Jim Hughes wrote:
> Hi Sophie, all,
>
> I've updated the KIP with feedback from the discussion so far:
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211882832
>
> As a terse summary of my current position:
> Pausing will only stop processing and punctuation (respecting modular
> topologies).
> Paused topologies will still a) consume from input topics, b) call the
> usual commit pathways (commits will happen basically as they would have),
> and c) standBy tasks will still be processed.
>
> Shout if the KIP or those details still need some TLC.  Responding to
> Sophie inline below.
>
>
> On Mon, May 9, 2022 at 6:06 PM Sophie Blee-Goldman
>  wrote:
>
>> Don't worry, I'm going to be adding the APIs for topology-level pausing as
>> part of the modular topologies KIP,
>> so we don't need to worry about that for now. That said, I don't think we
>> should brush it off entirely and design
>> this feature in a way that's going to be incompatible or hugely raise the
>> LOE on bringing the (mostly already
>> implemented) modular topologies feature into the public API, just
>> because it "won the race to write a KIP" :)
>>
>
> Yes, I'm hoping that this is all compatible with modular topologies.  I
> haven't seen anything so far which seems to be a problem; this KIP is just
> in a weird state to discuss details of acting on modular topologies.:)
>
>
>> I may be biased (ok, I definitely am), but I'm not in favor of adding this
>> as a state regardless of the modular topologies.
>> First of all any change to the KafkaStreams state machine is a breaking
>> change, no? So we would have to wait until
>> the next major release which seems like an unnecessary thing to block on.
>> (Whether to add this as a state to the
>> StreamThread's FSM is an implementation detail).
>>
>
> +1.  I am sold on skipping out on new states.  I had that as a rejected
> alternative in the KIP and have added a few more words to that bit.
>
>
>> Also, the semantics of using an `isPaused` method to distinguish a paused
>> instance (or topology) make more sense
>> to me -- this is a user-specified status, whereas the KafkaStreams state is
>> intended to relay the status of the system
>> itself. For example, if we are going to continue to poll during pause, then
>> shouldn't the client transition to REBALANCING?
>> I believe it makes sense to still allow distinguishing these states while a
>> client is paused, whereas making PAUSED its
>> own state means you can't tell when the client is rebalancing vs running,
>> or whether it is paused or dead: presumably
>> the NOT_RUNNING/ERROR state would trump the PAUSED state, which means you
>> would not be able to rely on
>> checking the state to see if you had called PAUSED on that instance.
>> Obviously you can work around this by just
>> maintaining a flag in the usercode, but all this feels very unnatural to me
>> vs just checking the `#isPaused` API.
>>
>> On that note, I had one question -- at what point would the `#isPaused`
>> check return true? Would it do so immediately
>> after pausing the instance, or only once it has finished committing offsets
>> and stopped returning records?
>>
>
> Immediately, `#isPaused` tells you about metadata.
>
>
>> Finally, on the note of punctuators I think it would make most sense to
>> either pause these as well or else add this an
>> an explicit option for the user. If this feature is used to, for example,
>> help save on processing costs while an app is
>> not in use, then it would probably be surprising and perhaps alarming to
>> see certain kinds of processing still continue.
>>
>
> From other parts of the discussion, I'm sold on pausing punctuation.
>
>
>> The question of whether to continue fetching for standby tasks is maybe a
>> bit more debatable, as it would certainly be
>> nice to find your clients all caught up when you go to resume the instance
>> again, but I would still strongly suggest
>> pausing these as well. To use a similar example, imagine if you paused an
>> app because it was about to run out of
>> disk. If the standbys kept processing and filled up the remaining space,
>> you'd probably feel a bit betrayed by this API.
>>
>> WDYT?
>>
>
> At the minute, my moderately held position is that standby tasks ought to
> continue reading and remain caught up.  If standby tasks would run out of
> space, there are probably bigger problems.
>
> If later it is desirable to manage punctuation or standby tasks, then it
> should be easy for future folks to modify things.
>
> Overall, I'd frame this KIP as "pause processing resulting in outputs".
>
> Cheers,
>
> Jim
>
>
>
>> On Mon, May 9, 2022 at 10:33 AM Guozhang Wang  wrote:
>>
>> > I think for named topology we can leave the scope of this KIP as "all or
>> > nothing", i.e. when you pause an instance you pause all of its
>> topologies.
>> > I

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #916

2022-05-09 Thread Apache Jenkins Server

See

Re: [DISCUSS] KIP-834: Pause / Resume KafkaStreams Topologies

2022-05-09 Thread Jim Hughes

Hi Sophie, all,

I've updated the KIP with feedback from the discussion so far:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211882832

As a terse summary of my current position:
Pausing will only stop processing and punctuation (respecting modular
topologies).
Paused topologies will still a) consume from input topics, b) call the
usual commit pathways (commits will happen basically as they would have),
and c) standBy tasks will still be processed.

Shout if the KIP or those details still need some TLC.  Responding to
Sophie inline below.


On Mon, May 9, 2022 at 6:06 PM Sophie Blee-Goldman
 wrote:

> Don't worry, I'm going to be adding the APIs for topology-level pausing as
> part of the modular topologies KIP,
> so we don't need to worry about that for now. That said, I don't think we
> should brush it off entirely and design
> this feature in a way that's going to be incompatible or hugely raise the
> LOE on bringing the (mostly already
> implemented) modular topologies feature into the public API, just
> because it "won the race to write a KIP" :)
>

Yes, I'm hoping that this is all compatible with modular topologies.  I
haven't seen anything so far which seems to be a problem; this KIP is just
in a weird state to discuss details of acting on modular topologies.:)


> I may be biased (ok, I definitely am), but I'm not in favor of adding this
> as a state regardless of the modular topologies.
> First of all any change to the KafkaStreams state machine is a breaking
> change, no? So we would have to wait until
> the next major release which seems like an unnecessary thing to block on.
> (Whether to add this as a state to the
> StreamThread's FSM is an implementation detail).
>

+1.  I am sold on skipping out on new states.  I had that as a rejected
alternative in the KIP and have added a few more words to that bit.


> Also, the semantics of using an `isPaused` method to distinguish a paused
> instance (or topology) make more sense
> to me -- this is a user-specified status, whereas the KafkaStreams state is
> intended to relay the status of the system
> itself. For example, if we are going to continue to poll during pause, then
> shouldn't the client transition to REBALANCING?
> I believe it makes sense to still allow distinguishing these states while a
> client is paused, whereas making PAUSED its
> own state means you can't tell when the client is rebalancing vs running,
> or whether it is paused or dead: presumably
> the NOT_RUNNING/ERROR state would trump the PAUSED state, which means you
> would not be able to rely on
> checking the state to see if you had called PAUSED on that instance.
> Obviously you can work around this by just
> maintaining a flag in the usercode, but all this feels very unnatural to me
> vs just checking the `#isPaused` API.
>
> On that note, I had one question -- at what point would the `#isPaused`
> check return true? Would it do so immediately
> after pausing the instance, or only once it has finished committing offsets
> and stopped returning records?
>

Immediately, `#isPaused` tells you about metadata.


> Finally, on the note of punctuators I think it would make most sense to
> either pause these as well or else add this an
> an explicit option for the user. If this feature is used to, for example,
> help save on processing costs while an app is
> not in use, then it would probably be surprising and perhaps alarming to
> see certain kinds of processing still continue.
>

>From other parts of the discussion, I'm sold on pausing punctuation.


> The question of whether to continue fetching for standby tasks is maybe a
> bit more debatable, as it would certainly be
> nice to find your clients all caught up when you go to resume the instance
> again, but I would still strongly suggest
> pausing these as well. To use a similar example, imagine if you paused an
> app because it was about to run out of
> disk. If the standbys kept processing and filled up the remaining space,
> you'd probably feel a bit betrayed by this API.
>
> WDYT?
>

At the minute, my moderately held position is that standby tasks ought to
continue reading and remain caught up.  If standby tasks would run out of
space, there are probably bigger problems.

If later it is desirable to manage punctuation or standby tasks, then it
should be easy for future folks to modify things.

Overall, I'd frame this KIP as "pause processing resulting in outputs".

Cheers,

Jim



> On Mon, May 9, 2022 at 10:33 AM Guozhang Wang  wrote:
>
> > I think for named topology we can leave the scope of this KIP as "all or
> > nothing", i.e. when you pause an instance you pause all of its
> topologies.
> > I raised this question in my previous email just trying to clarify if
> this
> > is what you have in mind. We can leave the question of finer controlled
> > pausing behavior for later when we have named topology being exposed via
> > another KIP.
> >
> >
> > Guozhang
> >
> > On Mon, May 9, 2022 at 7:50 AM

Contributor access

2022-05-09 Thread Jakub Miroś

Hello,

I want to offer a help as a new contributor. Please add me to the Jira, my
ID is SpacRocket

Regards,

Jakub Miroś

Re: [DISCUSS] KIP-834: Pause / Resume KafkaStreams Topologies

2022-05-09 Thread Sophie Blee-Goldman

Don't worry, I'm going to be adding the APIs for topology-level pausing as
part of the modular topologies KIP,
so we don't need to worry about that for now. That said, I don't think we
should brush it off entirely and design
this feature in a way that's going to be incompatible or hugely raise the
LOE on bringing the (mostly already
implemented) modular topologies feature into the public API, just
because it "won the race to write a KIP" :)

I may be biased (ok, I definitely am), but I'm not in favor of adding this
as a state regardless of the modular topologies.
First of all any change to the KafkaStreams state machine is a breaking
change, no? So we would have to wait until
the next major release which seems like an unnecessary thing to block on.
(Whether to add this as a state to the
StreamThread's FSM is an implementation detail).

Also, the semantics of using an `isPaused` method to distinguish a paused
instance (or topology) make more sense
to me -- this is a user-specified status, whereas the KafkaStreams state is
intended to relay the status of the system
itself. For example, if we are going to continue to poll during pause, then
shouldn't the client transition to REBALANCING?
I believe it makes sense to still allow distinguishing these states while a
client is paused, whereas making PAUSED its
own state means you can't tell when the client is rebalancing vs running,
or whether it is paused or dead: presumably
the NOT_RUNNING/ERROR state would trump the PAUSED state, which means you
would not be able to rely on
checking the state to see if you had called PAUSED on that instance.
Obviously you can work around this by just
maintaining a flag in the usercode, but all this feels very unnatural to me
vs just checking the `#isPaused` API.

On that note, I had one question -- at what point would the `#isPaused`
check return true? Would it do so immediately
after pausing the instance, or only once it has finished committing offsets
and stopped returning records?

Finally, on the note of punctuators I think it would make most sense to
either pause these as well or else add this an
an explicit option for the user. If this feature is used to, for example,
help save on processing costs while an app is
not in use, then it would probably be surprising and perhaps alarming to
see certain kinds of processing still continue.

The question of whether to continue fetching for standby tasks is maybe a
bit more debatable, as it would certainly be
nice to find your clients all caught up when you go to resume the instance
again, but I would still strongly suggest
pausing these as well. To use a similar example, imagine if you paused an
app because it was about to run out of
disk. If the standbys kept processing and filled up the remaining space,
you'd probably feel a bit betrayed by this API.

WDYT?

On Mon, May 9, 2022 at 10:33 AM Guozhang Wang  wrote:

> I think for named topology we can leave the scope of this KIP as "all or
> nothing", i.e. when you pause an instance you pause all of its topologies.
> I raised this question in my previous email just trying to clarify if this
> is what you have in mind. We can leave the question of finer controlled
> pausing behavior for later when we have named topology being exposed via
> another KIP.
>
>
> Guozhang
>
> On Mon, May 9, 2022 at 7:50 AM John Roesler  wrote:
>
> > Hi Jim,
> >
> > Thanks for the replies. This all sounds good to me. Just two further
> > comments:
> >
> > 3. It seems like you should aim for the simplest semantics. If the intent
> > is to “pause” the instance, then you’d better pause the whole instance.
> If
> > you leave punctuations and standbys running, I expect we’d see bug
> reports
> > come in that the instance isn’t really paused.
> >
> > 5. Since you won the race to write a KIP, I don’t think it makes too much
> > sense to worry too much about modular topologies. When they propose their
> > KIP, they will have to specify a lot of state management behavior, and
> > pause/resume will have to be part of it. If they have some concern about
> > your KIP, they’ll chime in. It doesn’t make sense for you to try and
> guess
> > what that proposal will look like.
> >
> > To be honest, you’re proposing a KafkaStreams runtime-level pause/resume
> > function, not a topology-level one anyway, so it seems pretty clear that
> it
> > would pause the whole runtime (of a single instance) regardless of any
> > modular topologies. If the intent is to pause individual topologies in
> the
> > future, you’d need a different API anyway.
> >
> > Thanks!
> > -John
> >
> > On Mon, May 9, 2022, at 08:10, Jim Hughes wrote:
> > > Hi John,
> > >
> > > Long emails are great; responding inline!
> > >
> > > On Sat, May 7, 2022 at 4:54 PM John Roesler 
> wrote:
> > >
> > >> Thanks for the KIP, Jim!
> > >>
> > >> This conversation seems to highlight that the KIP needs to specify
> > >> some of its behavior as well as its APIs, where the behavior is
> > >> observable and significant to

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.2 #47

2022-05-09 Thread Apache Jenkins Server

See

[VOTE] KIP-832 Allow creating a producer/consumer using a producer/consumer

2022-05-09 Thread François Rosière

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #915

2022-05-09 Thread Apache Jenkins Server

See

Re: [DISCUSS] KIP-832 Allow creating a producer/consumer using a producer/consumer config

2022-05-09 Thread Bruno Cadonna

Hi Francois,

I think you can go ahead and call for votes.

Could you please also clean up a little bit the KIP since it has still
parts that refer to its first version? For example, "Compatibility,
Deprecation, and Migration Plan" still mentions only two constructors.
IMO you can also remove section "Public Interfaces" since it does not
contain much information.

Best,
Bruno

On 09.05.22 17:45, Bruno Cadonna wrote:

Hi Francois,

You can open a PR and people can review it, but it must not be merged
until the KIP is approved.

Best,
Bruno

On 09.05.22 16:07, François Rosière wrote:

Can a PR be dropped on Github or do we still need some approval first?

Le dim. 8 mai 2022 à 06:08, John Roesler a écrit :

Thanks, François!

Those changes look good to me.

Thanks,
-John

On Fri, May 6, 2022, at 13:51, François Rosière wrote:

The KIP has been updated to reflect the last discussion

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211882578#KIP832:Allowcreatingaproducer/consumerusingaproducer/consumerconfig-ProposedChanges

Le ven. 6 mai 2022 à 20:44, François Rosière

écrit :

Hello,

No problem to also add a constructor taking the StreamsConfig in the
TopologyTestDriver.

Summary about the changes to apply:

- Create 2 new constructors in KafkaProducer
- Create a new constructor in KafkaConsumer and increase de

visibility

of an existing one
- Create a new constructor in TopologyTestDriver

Kr,

Le ven. 6 mai 2022 à 16:57, John Roesler a écrit

Thanks for the KIP, François!

I'm generally in favor of your KIP, since you're
proposing to follow the existing pattern of the
constructors for both Producer and Consumer,
but with the config object instead of Properties
or Map configs. Also, because we already have
this pattern in Streams, and we are just
extending it to Producer and Consumer.

Following on the KIP-378 discussion, I do still think
this is somewhat of an abuse of the Config objects,
and it would be better to have a formal dependency
injection interface, but I also don't want to let perfect
be the enemy of good. Since it looks like this approach
works, and there is also some precedent for it already,
I'd be inclined to approve it.

Since KIP-378 didn't make it over the finish line, and it
seems like a small expansion to your proposal, do you
mind also adding the StreamsConfig to the
TopologyTestDriver constructors? That way, we can go
ahead and resolve both KIPs at once.

Thank you,
-John

On Fri, May 6, 2022, at 06:06, François Rosière wrote:

To stay consistent with existing code, we should simply add 2

constructors.

One with ser/deser and one without.
So that, users have the choice to use one or the other.
I updated the KIP accordingly.

Le ven. 6 mai 2022 à 12:55, François Rosière <

francois.rosi...@gmail.com> a

écrit :

On the other hand, the KafkaConsumer constructor with a config +
serializer and deserializer already exists but is not public.
It would also complexify a bit the caller to not have the
serializer/deserializer exposed at constructor level.

Once the KIP would have been implemented, for streams, instead of

having a

custom config (already possible), I may simply define a custom
KafkaClientSupplier reusing the custom configs of both the producer

and the

consumer.
This supplier currently creates producers and consumers using the
constructors with a map of config + serializer/deserializer.

So, it seems it's easier to have the constructor with 3 parameters.

But in

any case, it will work if the config can be accessed...

Le ven. 6 mai 2022 à 12:14, François Rosière <

francois.rosi...@gmail.com>

a écrit :

Hello,

We may create a constructor with a single parameter which is the

config

but then, I would need to give the serializer/deserializer by also
overriding the config.
Like I would do for the interceptors.
So, no real opinion on that, both solutions are ok for me.
Maybe easier to take the approach of the single parameter.

Hope it respond to the question.

Kr,

Le ven. 6 mai 2022 à 11:59, Bruno Cadonna a

écrit :

Hi Francois,

Thank you for updating the KIP!

Now the motivation of the KIP is much clearer.

I would still be interested in:

>> 2. Why do you only want to change/add the constructors that

take

the

>> properties objects and de/serializers and you do not also

want to

>> add/change the constructors that take only the properties?

Best,
Bruno

On 05.05.22 23:15, François Rosière wrote:

Hello Bruno,

The KIP as been updated. Feel free to give more feedbacks and I

will

complete accordingly.

Kr,

Le jeu. 5 mai 2022 à 22:22, Bruno Cadonna

écrit :

Hi Francois,

Thanks for the KIP!

Here my first feedback:

1. Could you please extend the motivation section, so that it

clear

for a non-Spring dev why the change is needed? Usually, a

motivation

section benefits a lot from an actual example.
Extending the motivation section would also make the KIP more

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

2022-05-09 Thread Omnia Ibrahim

Hi, I gave the KIP another look after talking to some people at the Kafka
Summit in London. And I would like to clear up the motivation of this KIP.


At the moment, MM2 has some opinionated decisions that are creating issues
for teams that use IaC, federated solutions or have a capacity/budget
planning system for Kafka destination clusters. To explain it better, let's
assume we have MM2 with the following configurations to highlight these
problems.

```

topics = .*

refresh.topics.enabled = true

sync.topic.configs.enabled = true

sync.topic.acls.enabled = true

// Maybe in futrue we can have sync.group.acls.enabled = true

```


These configurations allow us to run MM2 with the value of its full
features. However, there are two main concerns when we run on a scale with
these configs:

1. *Capacity/Budgeting Planning:*

Functionality or features that impact capacity planning using MM2 are:

   1. MM2 automatically creates topics (breaking the rule of
   `auto.create.topics.enable=false`) and creates topic partitions on
   destination clusters if the number of partitions increases on the source.
   In the previous example, this functionality will apply to any topic that
   matches the regex of the `topics` config.
   2. Sync topic configs include configurations that impact capacity like `
   retention.ms` and `retention.bytes`.

These 2 points lead to adding new untracked capacity to destination
clusters without a way to count for them up-front or safeguard the cluster.
The team that runs the cluster will only see the capacity issue when their
disk usage hits the threshold for their alerts. The desk capacity issue can
be avoided if MM2 is flexible enough to

   - have a way for teams that run their ecosystem to have MM2 behave
   within their system.
   - disable the auto-creation and avoid syncing configs that impact
   capacity


2. *Provisioning conflict:*

In the previous MM2 configurations; we ended up with conflict as MM2 used
`AdminClient` directly to perform the following functionality

   -  Create a Kafka topic (no way to disable this at the moment)
   -  Add new Kafka partitions (no way to disable this at the moment)
   -  Sync Kafka Topic configurations (can be disabled, but then this
   reduces the value of MM2 potential for users)
   -  Sync Kafka topic's ACLs (can be disabled, but this reduces the users'
   value). Disabling this feature also means that users must ensure they have
   the right ACLs to the mirrored topics on the destination cluster before
   switching their consumers, especially when MM2 is used for disaster
   recovery. It may lead to extra downtime for them.


All these functionalities are using AdminClient; which causes an issue with
teams that

   - Manage their Kafka resources using tools like Strimizi or custom
   federated solutions. For example, Strimizi's UserOperator doesn't sync the
   topic ACLs when MM2 is enabled. Strimzi documentation mentions that users
   must to disable MM2 `sync.topic.acls.enabled` if they use `UserOperator`.
   On the other hand, Strimizi's TopicOperator doesn't have the same issue
   because it has a bi-directional reconciliation process that watches the
   topics state on the Kafka cluster and updates KafkaTopic resources for
   Strimzi. This design works fine with Kafka MM2 for Topics but not for
   syncing ACLs. Strimizi TopicOperator also doesn't have a way to stop
   syncing config that impact capacity for example retention configs.


   - Teams that run MM2 but don't own the destination cluster. In this
   case, these teams don't have Admin access, but they may have Kafka
   management solutions, such as yahoo/CMAK or an in-house solution. For such
   a tool as CMAK, these teams can update/create resources using CMAK REST API.


The Proposed KIP gives users the flexibility to integrate MM2 within their
ecosystem without disabling any MM2 features. We can achieve this
flexibility with one of the following solutions.

   1. Introduce a new interface that hides Admin interactions in one place.
   Then users can provide their way of resource management. As well as clean
   up the MM2 code by having one place that manages the resources, as at the
   moment, MM2 usage of AdminClient is all over the code.
   2. The second solution could be to add only a new config that allows the
   users to override AdminClient with another implementation of the
   AdminClient interface, as Ryanne suggested before. The downside is that
   AdminClient is enormous and constantly under development, so any users who
   opt-in for custom implementation will need to carry this burden.

I favour the first solution as it will make it either later to add any new
feature related to resource management. But don't mind if others think that
the second solution is easier for MM2 design.


*Note*: There are two possible future KIPs following this KIP to

   1. Add config to disable MM2 from auto creating or adding new topic
   partitions.
   2. Add a way to exclude a

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.1 #117

2022-05-09 Thread Apache Jenkins Server

See

[jira] [Resolved] (KAFKA-13790) ReplicaManager should be robust to all partition updates from kraft metadata log

2022-05-09 Thread David Jacot (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-13790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-13790.
-
Fix Version/s: 3.3.0
 Reviewer: Jason Gustafson
   Resolution: Fixed

> ReplicaManager should be robust to all partition updates from kraft metadata 
> log
> 
>
> Key: KAFKA-13790
> URL: https://issues.apache.org/jira/browse/KAFKA-13790
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: David Jacot
>Priority: Major
> Fix For: 3.3.0
>
>
> There are two ways that partition state can be updated in the zk world: one 
> is through `LeaderAndIsr` requests and one is through `AlterPartition` 
> responses. All changes made to partition state result in new LeaderAndIsr 
> requests, but replicas will ignore them if the leader epoch is less than or 
> equal to the current known leader epoch. Basically it works like this:
>  * Changes made by the leader are done through AlterPartition requests. These 
> changes bump the partition epoch (or zk version), but leave the leader epoch 
> unchanged. LeaderAndIsr requests are sent by the controller, but replicas 
> ignore them. Partition state is instead only updated when the AlterIsr 
> response is received.
>  * Changes made by the controller are made directly by the controller and 
> always result in a leader epoch bump. These changes are sent to replicas 
> through LeaderAndIsr requests and are applied by replicas.
> The code in `kafka.server.ReplicaManager` and `kafka.cluster.Partition` are 
> built on top of these assumptions. The logic in `makeLeader`, for example, 
> assumes that the leader epoch has indeed been bumped. Specifically, follower 
> state gets reset and a new entry is written to the leader epoch cache.
> In KRaft, we also have two paths to update partition state. One is 
> AlterPartition, just like in the zk world. The second is updates received 
> from the metadata log. These follow the same path as LeaderAndIsr requests 
> for the most part, but a big difference is that all changes are sent down to 
> `kafka.cluster.Partition`, even those which do not have a bumped leader 
> epoch. This breaks the assumptions mentioned above in `makeLeader`, which 
> could result in leader epoch cache inconsistency. Another side effect of this 
> on the follower side is that replica fetchers for updated partitions get 
> unnecessarily restarted. There may be others as well.
> We need to either replicate the same logic on the zookeeper side or make the 
> logic robust to all updates including those without a leader epoch bump.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (KAFKA-13889) Broker can't handle ACCESS_CONTROL_ENTRY_RECORD quickly followed by REMOVE_ACCESS_CONTROL_ENTRY_RECORD for same ACL

2022-05-09 Thread Andrew Grant (Jira)

Andrew Grant created KAFKA-13889:


 Summary: Broker can't handle ACCESS_CONTROL_ENTRY_RECORD quickly 
followed by REMOVE_ACCESS_CONTROL_ENTRY_RECORD for same ACL
 Key: KAFKA-13889
 URL: https://issues.apache.org/jira/browse/KAFKA-13889
 Project: Kafka
  Issue Type: Bug
Reporter: Andrew Grant


In 
https://github.com/apache/kafka/blob/trunk/metadata/src/main/java/org/apache/kafka/image/AclsDelta.java#L64
 we store the pending deletion in the changes map. This could override a 
creation that might have just happened. This is an issue because in 
BrokerMetadataPublisher this results in us making a removeAcl call which 
finally results in 
[https://github.com/apache/kafka/blob/trunk/metadata/src/main/java/org/apache/kafka/metadata/authorizer/StandardAuthorizerData.java#L203]
 being executed and this code throws an exception if the ACL isnt in the Map 
yet. If the ACCESS_CONTROL_ENTRY_RECORD event never got processed by 
BrokerMetadataPublisher then the ACL wont be in the Map yet.

My feeling is we might want to make removeAcl idempotent in that it returns 
success if the ACL doesn't exist: no matter how many times removeAcl is called 
it returns success if the ACL is deleted. Maybe we’d just log a warning or 
something?

 

Note, I dont think the AclControlManager has this issue because it doesn't 
batch the events like AclsDelta does. However, we still do throw a 
RuntimeException here 
[https://github.com/apache/kafka/blob/trunk/metadata/src/main/java/org/apache/kafka/controller/AclControlManager.java#L197]
 - maybe we should still follow the same logic (if we make the fix suggested 
above) and just log a warning if the ACL doesnt exist in the Map?



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Resolved] (KAFKA-13793) Add validators for serialization and deserialization related configuration

2022-05-09 Thread Mickael Maison (Jira)



 [ 
https://issues.apache.org/jira/browse/KAFKA-13793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-13793.

Fix Version/s: 3.3.0
   Resolution: Fixed

> Add validators for serialization and deserialization related configuration
> --
>
> Key: KAFKA-13793
> URL: https://issues.apache.org/jira/browse/KAFKA-13793
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, config
>Reporter: RivenSun
>Assignee: RivenSun
>Priority: Major
> Fix For: 3.3.0
>
>
> These configurations of producer and consumer have the same problem.
> {code:java}
> key.serializer, value.serializer, key.deserializer, value.deserializer{code}
>  
> Take the `key.serializer` configuration as an example:
> {code:java}
> Map props = new HashMap<>(); 
> props.put("key.serializer", null);{code}
> It is expected that this abnormal configuration can be verified during the 
> startup process of kafkaProducer, but the actual startup result:
> {code:java}
> Exception in thread "main" org.apache.kafka.common.KafkaException: Failed to 
> construct kafka producer
>     at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:440)
>     at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:291)
>     at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:274)
>     at 
> us.zoom.mq.server.adapter.kafka.ProducerTest.main(ProducerTest.java:139)
> Caused by: java.lang.NullPointerException
>     at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:368)
>     ... 3 more {code}
> There was a line of code that threw a null pointer, causing KafkaProducer 
> initialization to fail.
> I think we should be able to find this bad configuration during the 
> validation of all the configuration i.e. execute the 
> *ConfigDef.parseValue(ConfigKey key, Object value, boolean isSet) method* and 
> throw a *ConfigException* instead of NullPointerException.
> Solution:
> Add *NonNullValidator* to these configurations. 
> For example, when ProducerConfig defines `key.serializer` configuration, add 
> Validator:
> {code:java}
> .define(KEY_SERIALIZER_CLASS_CONFIG,
> Type.CLASS,
> ConfigDef.NO_DEFAULT_VALUE,
> new ConfigDef.NonNullValidator(),
> Importance.HIGH,
> KEY_SERIALIZER_CLASS_DOC) {code}
>  



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

Re: [DISCUSS] KIP-660: Pluggable ReplicaPlacer

2022-05-09 Thread Vikas Singh

Hi Mickael,

It's a nice proposal. It's appealing to have a pluggable way to override
default kafka placement decisions, and the motivation section lists some of
them. Here are few comments:

* The motivation section has "When adding brokers to a cluster, Kafka
currently does not necessarily place new partitions on new brokers". I am
not sure how valuable doing this will be. A newly created kafka topic takes
time to reach the same usage level as existing topics, say because the
topic created by a new workload that is getting onboarded, or the expansion
was done to relieve disk pressure on existing nodes etc. While new topics
catch up to existing workload, the new brokers are not sharing equal load
in the cluster, which probably defeats the purpose of adding new brokers.
In addition to that clustering new topics like this on new brokers have
implications from fault domain perspective. A reasonable way to approach it
is to indeed use CruiseControl to move things around so that the newly
added nodes become immediately involved and share cluster load.
* Regarding "When administrators want to remove brokers from a cluster,
there is no way to prevent Kafka from placing partitions on them", this is
indeed an issue. I would argue that this is needed by everyone and should
be part of Kafka, instead of being implemented as part of a plugin
interface by multiple teams.
* For "When some brokers are near their storage/throughput limit, Kafka
could avoid putting new partitions on them", while this can help relieve
short term overload I think again the correct solution here is something
like CruiseControl where the system is monitored and things moved around to
maintain a balanced cluster. A new topic will not take any disk space, so
placing them anywhere normally isn't going to add to the storage overload.
Similar to the previous case, maybe a mechanism in Kafka to put nodes in a
quarantine state is a better way to approach this.

In terms of the proposed api, I have a couple of comments:

* It is not clear if the proposal applies to partitions of new topics or
addition on partitions to an existing topic. Explicitly stating that will
be helpful.
* Regarding part "To address the use cases identified in the motivation
section, some knowledge about the current state of the cluster is
necessary. Details whether a new broker has just been added or is being
decommissioned are not part of the cluster metadata. Therefore such
knowledge has to be provided via an external means to the ReplicaPlacer,
for example via the configuration". It's not clear how this will be done.
If I have to implement this interface, it will be helpful to have clear
guidance/examples here which hopefully ties to the use cases in the
motivation section. It also allows us to figure out if the proposed
interface is complete and helps future implementers of the interface.

Couple of minor comments:
* The KIP is not listed in the main KIP page (
https://cwiki-test.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals).
Can you please add it there.
* The page has "This is especially true for the 4 scenarios listed in the
Motivation section", but there are only 3 scenarios listed.

Regards,
Vikas

On Tue, May 3, 2022 at 5:51 PM Colin McCabe  wrote:

> Hi Mickael,
>
> We did discuss this earlier, and I remember not being too enthusiastic
> about a pluggable policy here :)
>
> There have been several changes to the placement code in the last few
> weeks. (These are examples of the kind of changes that are impossible to do
> once an API is established, by the way.) Can you please revise the KIP to
> take these into account?
>
> I'd also like to understand a little bit better why we need this API when
> we have the explicit placement API for createTopics and createPartitions.
> Can you give me a few scenarios where the manual placement API would be
> insufficient?
>
> best,
> Colin
>
>
> On Mon, May 2, 2022, at 09:28, Mickael Maison wrote:
> > Hi,
> >
> > If there are no further comments, I'll start a vote in the next few days.
> >
> > Thanks,
> > Mickael
> >
> > On Wed, Mar 30, 2022 at 3:51 AM Luke Chen  wrote:
> >>
> >> Hi Mickael,
> >>
> >> Thanks for the update.
> >> It answered my questions!
> >>
> >> Thank you.
> >> Luke
> >>
> >> On Wed, Mar 30, 2022 at 12:09 AM Mickael Maison <
> mickael.mai...@gmail.com>
> >> wrote:
> >>
> >> > Hi Luke,
> >> >
> >> > Thanks for the feedback.
> >> >
> >> > 1. Thanks, fixed!
> >> > 2. Yes that's right. It's the same behavior for topic policies
> >> > 3. I've added details about how the mentioned scenarios could be
> >> > addressed. The information required to make such decisions is not part
> >> > of the Kafka cluster metadata so an external input is necessary. This
> >> > KIP does not propose a specific mechanism for doing it.
> >> >
> >> > I hope this answers your questions.
> >> >
> >> > Thanks,
> >> > Mickael
> >> >
> >> >
> >> > On Tue, Mar 29, 2022 at 5:42 PM Mickael Maison <
> mickael.mai...@gmail.com>
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.1 #116

2022-05-09 Thread Apache Jenkins Server

See 


Changes:


--
[...truncated 505364 lines...]
[2022-05-09T18:04:26.889Z] > Task :metadata:testClasses UP-TO-DATE
[2022-05-09T18:04:26.889Z] > Task :core:compileScala UP-TO-DATE
[2022-05-09T18:04:26.889Z] > Task :core:classes UP-TO-DATE
[2022-05-09T18:04:26.889Z] > Task :core:compileTestJava NO-SOURCE
[2022-05-09T18:04:26.889Z] 
[2022-05-09T18:04:26.889Z] > Task :streams:processMessages
[2022-05-09T18:04:26.889Z] Execution optimizations have been disabled for task 
':streams:processMessages' to ensure correctness due to the following reasons:
[2022-05-09T18:04:26.889Z]   - Gradle detected a problem with the following 
location: 
'/home/jenkins/workspace/Kafka_kafka_3.1/streams/src/generated/java/org/apache/kafka/streams/internals/generated'.
 Reason: Task ':streams:srcJar' uses this output of task 
':streams:processMessages' without declaring an explicit or implicit 
dependency. This can lead to incorrect results being produced, depending on 
what order the tasks are executed. Please refer to 
https://docs.gradle.org/7.2/userguide/validation_problems.html#implicit_dependency
 for more details about this problem.
[2022-05-09T18:04:26.889Z] MessageGenerator: processed 1 Kafka message JSON 
files(s).
[2022-05-09T18:04:26.889Z] 
[2022-05-09T18:04:26.889Z] > Task 
:clients:generateMetadataFileForMavenJavaPublication
[2022-05-09T18:04:26.889Z] > Task 
:clients:generatePomFileForMavenJavaPublication
[2022-05-09T18:04:26.889Z] > Task :streams:compileJava UP-TO-DATE
[2022-05-09T18:04:26.889Z] > Task :streams:classes UP-TO-DATE
[2022-05-09T18:04:26.889Z] > Task :streams:copyDependantLibs UP-TO-DATE
[2022-05-09T18:04:26.889Z] > Task :streams:test-utils:compileJava UP-TO-DATE
[2022-05-09T18:04:26.889Z] > Task :streams:jar UP-TO-DATE
[2022-05-09T18:04:26.889Z] > Task :core:compileTestScala UP-TO-DATE
[2022-05-09T18:04:26.889Z] > Task :core:testClasses UP-TO-DATE
[2022-05-09T18:04:26.889Z] > Task 
:streams:generateMetadataFileForMavenJavaPublication
[2022-05-09T18:04:31.922Z] > Task :connect:api:javadoc
[2022-05-09T18:04:31.922Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2022-05-09T18:04:31.922Z] > Task :connect:api:jar UP-TO-DATE
[2022-05-09T18:04:31.922Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2022-05-09T18:04:31.922Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2022-05-09T18:04:31.922Z] > Task :connect:json:jar UP-TO-DATE
[2022-05-09T18:04:31.922Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2022-05-09T18:04:31.922Z] > Task :connect:api:javadocJar
[2022-05-09T18:04:31.922Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2022-05-09T18:04:31.922Z] > Task :connect:json:publishToMavenLocal
[2022-05-09T18:04:31.922Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2022-05-09T18:04:31.922Z] > Task :connect:api:testClasses UP-TO-DATE
[2022-05-09T18:04:31.922Z] > Task :connect:api:testJar
[2022-05-09T18:04:31.922Z] > Task :connect:api:testSrcJar
[2022-05-09T18:04:31.922Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2022-05-09T18:04:31.922Z] > Task :connect:api:publishToMavenLocal
[2022-05-09T18:04:35.076Z] > Task :streams:javadoc
[2022-05-09T18:04:36.074Z] > Task :streams:javadocJar
[2022-05-09T18:04:36.074Z] > Task :streams:compileTestJava UP-TO-DATE
[2022-05-09T18:04:36.074Z] > Task :streams:testClasses UP-TO-DATE
[2022-05-09T18:04:37.013Z] > Task :streams:testJar
[2022-05-09T18:04:37.013Z] > Task :streams:testSrcJar
[2022-05-09T18:04:37.013Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2022-05-09T18:04:37.013Z] > Task :streams:publishToMavenLocal
[2022-05-09T18:04:37.013Z] 
[2022-05-09T18:04:37.013Z] > Task :clients:javadoc
[2022-05-09T18:04:37.013Z] 
/home/jenkins/workspace/Kafka_kafka_3.1/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/OAuthBearerLoginCallbackHandler.java:147:
 warning - Tag @link: reference not found: 
[2022-05-09T18:04:38.044Z] 1 warning
[2022-05-09T18:04:39.075Z] 
[2022-05-09T18:04:39.075Z] > Task :clients:javadocJar
[2022-05-09T18:04:40.282Z] 
[2022-05-09T18:04:40.282Z] > Task :clients:srcJar
[2022-05-09T18:04:40.282Z] Execution optimizations have been disabled for task 
':clients:srcJar' to ensure correctness due to the following reasons:
[2022-05-09T18:04:40.282Z]   - Gradle detected a problem with the following 
location: '/home/jenkins/workspace/Kafka_kafka_3.1/clients/src/generated/java'. 
Reason: Task ':clients:srcJar' uses this output of task 
':clients:processMessages' without declaring an explicit or implicit 
dependency. This can lead to incorrect results being produced, depending on 
what order the tasks are executed. Please refer to 
https://docs.gradle.org/7.2/userguide/validation_problems.html#implicit_dependency
 for more details about this problem.
[2022-05-09T18:04:41.422Z] 
[2022-05-09T18:04:41.422Z] > Task

[DISCUSS] KIP-836: Addition of Information in DescribeQuorumResponse about Voter Lag

2022-05-09 Thread Niket Goel

Hi all,

I created a KIP to add some more information to `DesscribeQuorumResponse` to 
enable ascertaining voter lag in the quorum a little better.
Please see KIP -- 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-836%3A+Additional+Information+in+DescribeQuorumResponse+about+Voter+Lag

Thanks for your feedback,
Niket Goel

Re: [DISCUSS] KIP-834: Pause / Resume KafkaStreams Topologies

2022-05-09 Thread Guozhang Wang

I think for named topology we can leave the scope of this KIP as "all or
nothing", i.e. when you pause an instance you pause all of its topologies.
I raised this question in my previous email just trying to clarify if this
is what you have in mind. We can leave the question of finer controlled
pausing behavior for later when we have named topology being exposed via
another KIP.


Guozhang

On Mon, May 9, 2022 at 7:50 AM John Roesler  wrote:

> Hi Jim,
>
> Thanks for the replies. This all sounds good to me. Just two further
> comments:
>
> 3. It seems like you should aim for the simplest semantics. If the intent
> is to “pause” the instance, then you’d better pause the whole instance. If
> you leave punctuations and standbys running, I expect we’d see bug reports
> come in that the instance isn’t really paused.
>
> 5. Since you won the race to write a KIP, I don’t think it makes too much
> sense to worry too much about modular topologies. When they propose their
> KIP, they will have to specify a lot of state management behavior, and
> pause/resume will have to be part of it. If they have some concern about
> your KIP, they’ll chime in. It doesn’t make sense for you to try and guess
> what that proposal will look like.
>
> To be honest, you’re proposing a KafkaStreams runtime-level pause/resume
> function, not a topology-level one anyway, so it seems pretty clear that it
> would pause the whole runtime (of a single instance) regardless of any
> modular topologies. If the intent is to pause individual topologies in the
> future, you’d need a different API anyway.
>
> Thanks!
> -John
>
> On Mon, May 9, 2022, at 08:10, Jim Hughes wrote:
> > Hi John,
> >
> > Long emails are great; responding inline!
> >
> > On Sat, May 7, 2022 at 4:54 PM John Roesler  wrote:
> >
> >> Thanks for the KIP, Jim!
> >>
> >> This conversation seems to highlight that the KIP needs to specify
> >> some of its behavior as well as its APIs, where the behavior is
> >> observable and significant to users.
> >>
> >> For example:
> >>
> >> 1. Do you plan to have a guarantee that immediately after
> >> calling KafkaStreams.pause(), users should observe that the instance
> >> stops processing new records? Or should they expect that the threads
> >> will continue to process some records and pause asynchronously
> >> (you already answered this in the thread earlier)?
> >>
> >
> > I'm happy to build up to a guarantee of sorts.  My current idea is that
> > pause() does not do anything "exceptional" to get control back from a
> > running topology.  A currently running topology would get to complete its
> > loop.
> >
> > Separately, I'm still piecing together how commits work.  By some
> > mechanism, after a pause, I do agree that the topology needs to commit
> its
> > work in some manner.
> >
> >
> >> 2. Will the threads continue to poll new records until they naturally
> fill
> >> up the task buffers, or will they immediately pause their Consumers
> >> as well?
> >>
> >
> > Presently, I'm suggesting that consumers would fill up their buffers.
> >
> >
> >> 3. Will threads continue to call (system time) punctuators, or would
> >> punctuations also be paused?
> >>
> >
> > In my first pass at thinking through this, I left the punctuators
> running.
> > To be honest, I'm not sure what they do, so my approach is either lucky
> and
> > correct or it could be Very Clearly Wrong.;)
> >
> >
> >> I realize that some of those questions simply may not have occurred to
> >> you, so this is not a criticism for leaving them off; I'm just pointing
> out
> >> that although we don't tend to mention implementation details in KIPs,
> >> we also can't be too high level, since there are a lot of operational
> >> details that users rely on to achieve various behaviors in Streams.
> >>
> >
> > Ayup, I will add some details as we iron out the guarantees,
> implementation
> > details that are at the API level.  This one is tough since internal
> > features like NamedTopologies are part of the discussion.
> >
> >
> >
> >> A couple more comments:
> >>
> >> 4. +1 to what Guozhang said. It seems like we should we also do a commit
> >> before entering the paused state. That way, any open transactions would
> >> be closed and not have to worry about timing out. Even under ALOS, it
> >> seems best to go ahead and complete the processing of in-flight records
> >> by committing. That way, if anything happens to die while it's paused,
> >> existing
> >> work won't have to be repeated. Plus, if there are any processors with
> side
> >> effects, users won't have to tolerate weird edge cases where a pause
> occurs
> >> after a processor sees a record, but before the result is sent to its
> >> outputs.
> >>
> >> 5. I noticed that you proposed not to add a PAUSED state, but I didn't
> >> follow
> >> the rationale. Adding a state seems beneficial for a number of reasons:
> >> StreamThreads already use the thread state to determine whether to
> process
> >> or not, so avoiding a new State

[jira] [Created] (KAFKA-13888) KIP-836: Addition of Information in DescribeQuorumResponse about Voter Lag

2022-05-09 Thread Niket Goel (Jira)

Niket Goel created KAFKA-13888:
--

 Summary: KIP-836: Addition of Information in 
DescribeQuorumResponse about Voter Lag
 Key: KAFKA-13888
 URL: https://issues.apache.org/jira/browse/KAFKA-13888
 Project: Kafka
  Issue Type: Improvement
  Components: kraft
Reporter: Niket Goel


Tracking issue for the implementation of KIP:836



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[RESULTS] [VOTE] Release Kafka version 3.2.0

2022-05-09 Thread Bruno Cadonna


This vote passes with 5 +1 votes (3 bindings) and no 0 or -1 votes.

+1 votes
PMC Members:
* Bill Bejeck
* Mickael Maison
* David Jacot

Committers:
* Luke Chen

Community:
* Jakub Scholz

0 votes
* No votes

-1 votes
* No votes

Vote thread:
https://lists.apache.org/thread/3b05b74hw7cf4p2o1dn5n33bltxmvvwm

I'll continue with the release process and the release announcement will 
follow in the next few days.


Best,
Bruno

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #914

2022-05-09 Thread Apache Jenkins Server

See

Re: [DISCUSS] KIP-832 Allow creating a producer/consumer using a producer/consumer config

2022-05-09 Thread Bruno Cadonna

Hi Francois,

You can open a PR and people can review it, but it must not be merged
until the KIP is approved.

Best,
Bruno

On 09.05.22 16:07, François Rosière wrote:

Can a PR be dropped on Github or do we still need some approval first?

Le dim. 8 mai 2022 à 06:08, John Roesler a écrit :

Thanks, François!

Those changes look good to me.

Thanks,
-John

On Fri, May 6, 2022, at 13:51, François Rosière wrote:

The KIP has been updated to reflect the last discussion

https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211882578#KIP832:Allowcreatingaproducer/consumerusingaproducer/consumerconfig-ProposedChanges

Le ven. 6 mai 2022 à 20:44, François Rosière

écrit :

Hello,

No problem to also add a constructor taking the StreamsConfig in the
TopologyTestDriver.

Summary about the changes to apply:

- Create 2 new constructors in KafkaProducer
- Create a new constructor in KafkaConsumer and increase de

visibility

of an existing one
- Create a new constructor in TopologyTestDriver

Kr,

Le ven. 6 mai 2022 à 16:57, John Roesler a écrit

Thanks for the KIP, François!

Thank you,
-John

On Fri, May 6, 2022, at 06:06, François Rosière wrote:

To stay consistent with existing code, we should simply add 2

constructors.

One with ser/deser and one without.
So that, users have the choice to use one or the other.
I updated the KIP accordingly.

Le ven. 6 mai 2022 à 12:55, François Rosière <

francois.rosi...@gmail.com> a

écrit :

Once the KIP would have been implemented, for streams, instead of

having a

custom config (already possible), I may simply define a custom
KafkaClientSupplier reusing the custom configs of both the producer

and the

consumer.
This supplier currently creates producers and consumers using the
constructors with a map of config + serializer/deserializer.

So, it seems it's easier to have the constructor with 3 parameters.

But in

any case, it will work if the config can be accessed...

Le ven. 6 mai 2022 à 12:14, François Rosière <

francois.rosi...@gmail.com>

a écrit :

Hello,

We may create a constructor with a single parameter which is the

config

Hope it respond to the question.

Kr,

Le ven. 6 mai 2022 à 11:59, Bruno Cadonna a

écrit :

Hi Francois,

Thank you for updating the KIP!

Now the motivation of the KIP is much clearer.

I would still be interested in:

>> 2. Why do you only want to change/add the constructors that

take

the

>> properties objects and de/serializers and you do not also

want to

>> add/change the constructors that take only the properties?

Best,
Bruno

On 05.05.22 23:15, François Rosière wrote:

Hello Bruno,

The KIP as been updated. Feel free to give more feedbacks and I

will

complete accordingly.

Kr,

Le jeu. 5 mai 2022 à 22:22, Bruno Cadonna

écrit :

Hi Francois,

Thanks for the KIP!

Here my first feedback:

1. Could you please extend the motivation section, so that it

clear

for a non-Spring dev why the change is needed? Usually, a

motivation

section benefits a lot from an actual example.
Extending the motivation section would also make the KIP more
self-contained which is important IMO since this is kind of a

log

the

major changes to Kafka. Descriptions of major changes should

not

completely depend on external links (which may become dead in

future).

Referencing external resources to point to more details or give

context

is useful, though.

2. Why do you only want to change/add the constructors that

take

the

properties objects and de/serializers and you do not

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.1 #115

2022-05-09 Thread Apache Jenkins Server

See

Re: [DISCUSS] KIP-834: Pause / Resume KafkaStreams Topologies

2022-05-09 Thread John Roesler

Hi Jim,

Thanks for the replies. This all sounds good to me. Just two further comments:

3. It seems like you should aim for the simplest semantics. If the intent is to 
“pause” the instance, then you’d better pause the whole instance. If you leave 
punctuations and standbys running, I expect we’d see bug reports come in that 
the instance isn’t really paused.

5. Since you won the race to write a KIP, I don’t think it makes too much sense 
to worry too much about modular topologies. When they propose their KIP, they 
will have to specify a lot of state management behavior, and pause/resume will 
have to be part of it. If they have some concern about your KIP, they’ll chime 
in. It doesn’t make sense for you to try and guess what that proposal will look 
like.

To be honest, you’re proposing a KafkaStreams runtime-level pause/resume 
function, not a topology-level one anyway, so it seems pretty clear that it 
would pause the whole runtime (of a single instance) regardless of any modular 
topologies. If the intent is to pause individual topologies in the future, 
you’d need a different API anyway. 

Thanks!
-John

On Mon, May 9, 2022, at 08:10, Jim Hughes wrote:
> Hi John,
>
> Long emails are great; responding inline!
>
> On Sat, May 7, 2022 at 4:54 PM John Roesler  wrote:
>
>> Thanks for the KIP, Jim!
>>
>> This conversation seems to highlight that the KIP needs to specify
>> some of its behavior as well as its APIs, where the behavior is
>> observable and significant to users.
>>
>> For example:
>>
>> 1. Do you plan to have a guarantee that immediately after
>> calling KafkaStreams.pause(), users should observe that the instance
>> stops processing new records? Or should they expect that the threads
>> will continue to process some records and pause asynchronously
>> (you already answered this in the thread earlier)?
>>
>
> I'm happy to build up to a guarantee of sorts.  My current idea is that
> pause() does not do anything "exceptional" to get control back from a
> running topology.  A currently running topology would get to complete its
> loop.
>
> Separately, I'm still piecing together how commits work.  By some
> mechanism, after a pause, I do agree that the topology needs to commit its
> work in some manner.
>
>
>> 2. Will the threads continue to poll new records until they naturally fill
>> up the task buffers, or will they immediately pause their Consumers
>> as well?
>>
>
> Presently, I'm suggesting that consumers would fill up their buffers.
>
>
>> 3. Will threads continue to call (system time) punctuators, or would
>> punctuations also be paused?
>>
>
> In my first pass at thinking through this, I left the punctuators running.
> To be honest, I'm not sure what they do, so my approach is either lucky and
> correct or it could be Very Clearly Wrong.;)
>
>
>> I realize that some of those questions simply may not have occurred to
>> you, so this is not a criticism for leaving them off; I'm just pointing out
>> that although we don't tend to mention implementation details in KIPs,
>> we also can't be too high level, since there are a lot of operational
>> details that users rely on to achieve various behaviors in Streams.
>>
>
> Ayup, I will add some details as we iron out the guarantees, implementation
> details that are at the API level.  This one is tough since internal
> features like NamedTopologies are part of the discussion.
>
>
>
>> A couple more comments:
>>
>> 4. +1 to what Guozhang said. It seems like we should we also do a commit
>> before entering the paused state. That way, any open transactions would
>> be closed and not have to worry about timing out. Even under ALOS, it
>> seems best to go ahead and complete the processing of in-flight records
>> by committing. That way, if anything happens to die while it's paused,
>> existing
>> work won't have to be repeated. Plus, if there are any processors with side
>> effects, users won't have to tolerate weird edge cases where a pause occurs
>> after a processor sees a record, but before the result is sent to its
>> outputs.
>>
>> 5. I noticed that you proposed not to add a PAUSED state, but I didn't
>> follow
>> the rationale. Adding a state seems beneficial for a number of reasons:
>> StreamThreads already use the thread state to determine whether to process
>> or not, so avoiding a new State would just mean adding a separate flag to
>> track
>> and then checking your new flag in addition to the State in the thread.
>> Also,
>> operating Streams applications is a non-trivial task, and users rely on
>> the State
>> (and transitions) to understand Streams's behavior. Adding a PAUSED state
>> is an elegant way to communicate to operators what is happening with the
>> application. Note that the person digging though logs and metrics, trying
>> to understand why the application isn't doing anything is probably not
>> going
>> to be the same person who is calling pause() and resume(). Also, if you add
>> a state, you don't need `isPaused()`.
>>

Re: [DISCUSS] KIP-832 Allow creating a producer/consumer using a producer/consumer config

2022-05-09 Thread François Rosière

Can a PR be dropped on Github or do we still need some approval first?

Le dim. 8 mai 2022 à 06:08, John Roesler  a écrit :

> Thanks, François!
>
> Those changes look good to me.
>
> Thanks,
> -John
>
> On Fri, May 6, 2022, at 13:51, François Rosière wrote:
> > The KIP has been updated to reflect the last discussion
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=211882578#KIP832:Allowcreatingaproducer/consumerusingaproducer/consumerconfig-ProposedChanges
> >
> >
> > Le ven. 6 mai 2022 à 20:44, François Rosière 
> a
> > écrit :
> >
> >> Hello,
> >>
> >> No problem to also add a constructor taking the StreamsConfig in the
> >> TopologyTestDriver.
> >>
> >> Summary about the changes to apply:
> >>
> >>- Create 2 new constructors in KafkaProducer
> >>- Create a new constructor in KafkaConsumer and increase de
> visibility
> >>of an existing one
> >>- Create a new constructor in TopologyTestDriver
> >>
> >> Kr,
> >>
> >> F.
> >>
> >> Le ven. 6 mai 2022 à 16:57, John Roesler  a écrit
> :
> >>
> >>> Thanks for the KIP, François!
> >>>
> >>> I'm generally in favor of your KIP, since you're
> >>> proposing to follow the existing pattern of the
> >>> constructors for both Producer and Consumer,
> >>> but with the config object instead of Properties
> >>> or Map configs. Also, because we already have
> >>> this pattern in Streams, and we are just
> >>> extending it to Producer and Consumer.
> >>>
> >>> Following on the KIP-378 discussion, I do still think
> >>> this is somewhat of an abuse of the Config objects,
> >>> and it would be better to have a formal dependency
> >>> injection interface, but I also don't want to let perfect
> >>> be the enemy of good. Since it looks like this approach
> >>> works, and there is also some precedent for it already,
> >>> I'd be inclined to approve it.
> >>>
> >>> Since KIP-378 didn't make it over the finish line, and it
> >>> seems like a small expansion to your proposal, do you
> >>> mind also adding the StreamsConfig to the
> >>> TopologyTestDriver constructors? That way, we can go
> >>> ahead and resolve both KIPs at once.
> >>>
> >>> Thank you,
> >>> -John
> >>>
> >>>
> >>> On Fri, May 6, 2022, at 06:06, François Rosière wrote:
> >>> > To stay consistent with existing code, we should simply add 2
> >>> constructors.
> >>> > One with ser/deser and one without.
> >>> > So that, users have the choice to use one or the other.
> >>> > I updated the KIP accordingly.
> >>> >
> >>> > Le ven. 6 mai 2022 à 12:55, François Rosière <
> >>> francois.rosi...@gmail.com> a
> >>> > écrit :
> >>> >
> >>> >> On the other hand, the KafkaConsumer constructor with a config +
> >>> >> serializer and deserializer already exists but is not public.
> >>> >> It would also complexify a bit the caller to not have the
> >>> >> serializer/deserializer exposed at constructor level.
> >>> >>
> >>> >> Once the KIP would have been implemented, for streams, instead of
> >>> having a
> >>> >> custom config (already possible), I may simply define a custom
> >>> >> KafkaClientSupplier reusing the custom configs of both the producer
> >>> and the
> >>> >> consumer.
> >>> >> This supplier currently creates producers and consumers using the
> >>> >> constructors with a map of config + serializer/deserializer.
> >>> >>
> >>> >> So, it seems it's easier to have the constructor with 3 parameters.
> >>> But in
> >>> >> any case, it will work if the config can be accessed...
> >>> >>
> >>> >> Le ven. 6 mai 2022 à 12:14, François Rosière <
> >>> francois.rosi...@gmail.com>
> >>> >> a écrit :
> >>> >>
> >>> >>> Hello,
> >>> >>>
> >>> >>> We may create a constructor with a single parameter which is the
> >>> config
> >>> >>> but then, I would need to give the serializer/deserializer by also
> >>> >>> overriding the config.
> >>> >>> Like I would do for the interceptors.
> >>> >>> So, no real opinion on that, both solutions are ok for me.
> >>> >>> Maybe easier to take the approach of the single parameter.
> >>> >>>
> >>> >>> Hope it respond to the question.
> >>> >>>
> >>> >>> Kr,
> >>> >>>
> >>> >>> F.
> >>> >>>
> >>> >>> Le ven. 6 mai 2022 à 11:59, Bruno Cadonna  a
> >>> écrit :
> >>> >>>
> >>>  Hi Francois,
> >>> 
> >>>  Thank you for updating the KIP!
> >>> 
> >>>  Now the motivation of the KIP is much clearer.
> >>> 
> >>>  I would still be interested in:
> >>> 
> >>>   >> 2. Why do you only want to change/add the constructors that
> take
> >>> the
> >>>   >> properties objects and de/serializers and you do not also
> want to
> >>>   >> add/change the constructors that take only the properties?
> >>> 
> >>> 
> >>>  Best,
> >>>  Bruno
> >>> 
> >>>  On 05.05.22 23:15, François Rosière wrote:
> >>>  > Hello Bruno,
> >>>  >
> >>>  > The KIP as been updated. Feel free to give more feedbacks and I
> >>> will
> >>>  > complete accordingly.
> >>>  >
> >>>  > Kr,
> >>>  >
> >>>  > F.
> >>>

Re: [DISCUSS] KIP-834: Pause / Resume KafkaStreams Topologies

2022-05-09 Thread Jim Hughes

Hi John,

Long emails are great; responding inline!

On Sat, May 7, 2022 at 4:54 PM John Roesler  wrote:

> Thanks for the KIP, Jim!
>
> This conversation seems to highlight that the KIP needs to specify
> some of its behavior as well as its APIs, where the behavior is
> observable and significant to users.
>
> For example:
>
> 1. Do you plan to have a guarantee that immediately after
> calling KafkaStreams.pause(), users should observe that the instance
> stops processing new records? Or should they expect that the threads
> will continue to process some records and pause asynchronously
> (you already answered this in the thread earlier)?
>

I'm happy to build up to a guarantee of sorts.  My current idea is that
pause() does not do anything "exceptional" to get control back from a
running topology.  A currently running topology would get to complete its
loop.

Separately, I'm still piecing together how commits work.  By some
mechanism, after a pause, I do agree that the topology needs to commit its
work in some manner.

> 2. Will the threads continue to poll new records until they naturally fill
> up the task buffers, or will they immediately pause their Consumers
> as well?
>

Presently, I'm suggesting that consumers would fill up their buffers.

> 3. Will threads continue to call (system time) punctuators, or would
> punctuations also be paused?
>

In my first pass at thinking through this, I left the punctuators running.
To be honest, I'm not sure what they do, so my approach is either lucky and
correct or it could be Very Clearly Wrong.;)

> I realize that some of those questions simply may not have occurred to
> you, so this is not a criticism for leaving them off; I'm just pointing out
> that although we don't tend to mention implementation details in KIPs,
> we also can't be too high level, since there are a lot of operational
> details that users rely on to achieve various behaviors in Streams.
>

Ayup, I will add some details as we iron out the guarantees, implementation
details that are at the API level.  This one is tough since internal
features like NamedTopologies are part of the discussion.

> A couple more comments:
>
> 4. +1 to what Guozhang said. It seems like we should we also do a commit
> before entering the paused state. That way, any open transactions would
> be closed and not have to worry about timing out. Even under ALOS, it
> seems best to go ahead and complete the processing of in-flight records
> by committing. That way, if anything happens to die while it's paused,
> existing
> work won't have to be repeated. Plus, if there are any processors with side
> effects, users won't have to tolerate weird edge cases where a pause occurs
> after a processor sees a record, but before the result is sent to its
> outputs.
>
> 5. I noticed that you proposed not to add a PAUSED state, but I didn't
> follow
> the rationale. Adding a state seems beneficial for a number of reasons:
> StreamThreads already use the thread state to determine whether to process
> or not, so avoiding a new State would just mean adding a separate flag to
> track
> and then checking your new flag in addition to the State in the thread.
> Also,
> operating Streams applications is a non-trivial task, and users rely on
> the State
> (and transitions) to understand Streams's behavior. Adding a PAUSED state
> is an elegant way to communicate to operators what is happening with the
> application. Note that the person digging though logs and metrics, trying
> to understand why the application isn't doing anything is probably not
> going
> to be the same person who is calling pause() and resume(). Also, if you add
> a state, you don't need `isPaused()`.
>
> 5b. If you buy the arguments to go ahead and commit as well as the
> argument to add a State, then I'd also suggest to follow the existing
> patterns
> for the shutdown states by also adding PAUSING. That
> way, you'll also expose a way to understand that Streams received the
> signal
> to pause, and that it's still processing and committing some records in
> preparation to enter a PAUSED state. I'm not sure if a RESUMING state would
> also make sense.
>

I hit a tricky bit when thinking through having a PAUSED state...  If one
is using Named Topologies, and some of them are paused, what state is the
Streams instance in?  If we can agree on that, things may become clear
I can see two quick ideas:

1.  The state is RUNNING and NamedTopologies have some other way to
indicate state.

2.  The state is something messy like PARTIALLY_PAUSED to reflect that the
instance has something interesting going on.

When I poked at things initially, I did try out having different states,
and I readily agree that a PAUSING state may make sense.  (Especially if
there's a need to run commits before transitioning all the way to PAUSED.)

> And that's all I have to say about that. I hope you don't find my
> long message offputting. I'm fundamentally in favor of your KIP,
> and I

[GitHub] [kafka-site] rbowen opened a new pull request, #408: Move IRC pointer to libera.chat

2022-05-09 Thread GitBox



rbowen opened a new pull request, #408:
URL: https://github.com/apache/kafka-site/pull/408

   1) It looks like you have moved your primary IRC presence to libera.chat, as 
you should. Update the website accordingly.
   
   2) The link to botbot.me is no longer accurate, and it references freenode 
anyways. Remove link to defunct irc log site.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready

2022-05-09 Thread Andrew Otto

> Deprecating ZK Mode and
> Removing Zookeeper Mode

I'm excited about KRaft, but quick Q.  I'm researching Kafka 'stretch'
cluster deployments, and as far as I can tell stretch clusters require
Zookeeper to function properly, is this correct?  If so, we might want to
solve that before Deprecating ZK Mode.

Thanks!



On Thu, May 5, 2022 at 6:57 PM Israel Ekpo  wrote:

> Thanks Colin.
>
> I think we may need to update the KIP name to reflect the intent of the KIP
> and convey everything it’s about if all the 3 action items will be covered
> by the same KIP
>
> It contains three parts:
>
> Marking KRraft as Production Ready
> Deprecating ZK Mode and
> Removing Zookeeper Mode
>
> Should this be broken up in three separate KIPs since it will be done in
> multiple releases?
>
> The current name of the KIP only conveys the first item and not the next
> two
>
> It is just a thought. I will like to get your perspective
>
>
>
> On Thu, May 5, 2022 at 1:19 PM Colin McCabe  wrote:
>
> > Hi all,
> >
> > Thanks for the comments. I agree that we should split out declaring KRaft
> > going production for new clusters from deprecating ZK. We can do the
> former
> > in the next release, 3.3, and the latter in the release after that, 3.4.
> >
> > I also talked offline with some of the people working on upgrade from ZK
> > and it seems like 3.4 is a more realistic target for that work. Partly
> this
> > is because 3.3 will be a bit earlier than I originally thought (for some
> > reason I thought it might be October, but Ismael pointed out it's planned
> > for August)
> >
> > I also agree that it will probably be useful to have a 3.5 release
> > following the 3.4 one, which will also support ZK. Hopefully we will not
> > need a 3.6, but we don't have to decide that now.
> >
> > I added a timeline section to the KIP to make this all clearer. To be
> > clear, it is a preliminary timeline, which may change. It's difficult to
> > fully plan out the next 1.5 years of Apache Kafka releases right now --
> and
> > obviously, there are things which may come up to change our plans.
> However,
> > I think it is still helpful to have the section to give us a feeling for
> > the general roadmap.
> >
> > When you read the KIP, please consider it all speculative except for the
> > three proposed changes at the top:
> >
> > 1. Mark KRaft as production-ready for new clusters in the upcoming Kafka
> > 3.3 release.
> > 2. Deprecate ZooKeeper mode in the upcoming Kafka 3.4 release
> > 3. Plan to remove ZooKeeper mode entirely in Kafka 4.0.
> >
> > best,
> > Colin
> >
> >
> > On Wed, May 4, 2022, at 19:31, Ismael Juma wrote:
> > > Yes, all features supported by zk mode will be available in kraft mode
> in
> > > the 3.x series.
> > >
> > > Ismael
> > >
> > > On Wed, May 4, 2022, 5:28 PM Israel Ekpo  wrote:
> > >
> > >> Ismael,
> > >>
> > >> I like the timeline. However, does this or will this also account for
> > >> features  users rely on today in Zookeeper mode being available when
> > >> Zookeeper is dropped?
> > >>
> > >> That’s my main concern
> > >>
> > >> On Wed, May 4, 2022 at 8:12 PM Ismael Juma  wrote:
> > >>
> > >> > Hi Colin,
> > >> >
> > >> > Thanks for the KIP, this is exciting. Trying to balance progress and
> > >> > compatibility, how about the following?
> > >> >
> > >> > 1. 3.3 (~August 2022): kraft is production ready for new clusters
> > >> > 2. 3.4 (~December 2022/January 2023): migration from zk to kraft is
> > >> > production ready and zk mode is deprecated
> > >> > 3. 3.5 (~April 2023): buffer release
> > >> > 4. 4.0 (~August 2023): kraft mode is on by default and zk mode is
> > removed
> > >> >
> > >> > This would mean about 1 year from kraft being production ready to zk
> > >> > removal and 8 months from zk deprecation to zk removal.
> > >> >
> > >> > If necessary (due to important bugs or security issues), we can do a
> > >> couple
> > >> > of additional bug fix releases in the 3.5 series after 4.0 is
> > released.
> > >> >
> > >> > Thoughts?
> > >> >
> > >> > Ismael
> > >> >
> > >> > On Tue, May 3, 2022, 6:03 PM Colin McCabe 
> wrote:
> > >> >
> > >> > > Hi all,
> > >> > >
> > >> > > I've written a KIP for marking KRaft as production ready. Please
> > take a
> > >> > > look if you have a chance:
> > >> > >
> > >> > > https://cwiki.apache.org/confluence/x/8xKhD
> > >> > >
> > >> > > thanks,
> > >> > > Colin
> > >> > >
> > >> >
> > >> --
> > >> Israel Ekpo
> > >> Lead Instructor, IzzyAcademy.com
> > >> https://www.youtube.com/c/izzyacademy
> > >> https://izzyacademy.com/
> > >>
> >
> --
> Israel Ekpo
> Lead Instructor, IzzyAcademy.com
> https://www.youtube.com/c/izzyacademy
> https://izzyacademy.com/
>

Re: Permission to contribute

2022-05-09 Thread Luke Chen

Hi Tom,

You're all done.
Thanks for joining us! :)

Luke

On Mon, May 9, 2022 at 4:47 PM Thomas Cooper  wrote:

> Hi,
>
> I would like to contribute to the project. My usernames are:
>
> wiki: tomncooper
> jira: tomncooper
>
> If someone could approve access for me I would be grateful.
>
> Cheers,
>
> Tom Cooper
>
> [@tomncooper](https://twitter.com/tomncooper) | tomcooper.dev

Permission to contribute

2022-05-09 Thread Thomas Cooper

Hi,

I would like to contribute to the project. My usernames are:

wiki: tomncooper
jira: tomncooper

If someone could approve access for me I would be grateful.

Cheers,

Tom Cooper

[@tomncooper](https://twitter.com/tomncooper) | tomcooper.dev

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #917

[jira] [Created] (KAFKA-13890) Improve documentation of ssl.keystore.type and ssl.truststore.type

Re: [DISCUSS] KIP-834: Pause / Resume KafkaStreams Topologies

Re: [DISCUSS] KIP-836: Addition of Information in DescribeQuorumResponse about Voter Lag

Re: [DISCUSS] KIP-834: Pause / Resume KafkaStreams Topologies

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #916

Re: [DISCUSS] KIP-834: Pause / Resume KafkaStreams Topologies

Contributor access

Re: [DISCUSS] KIP-834: Pause / Resume KafkaStreams Topologies

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.2 #47

[VOTE] KIP-832 Allow creating a producer/consumer using a producer/consumer

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #915

Re: [DISCUSS] KIP-832 Allow creating a producer/consumer using a producer/consumer config

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.1 #117

[jira] [Resolved] (KAFKA-13790) ReplicaManager should be robust to all partition updates from kraft metadata log

[jira] [Created] (KAFKA-13889) Broker can't handle ACCESS_CONTROL_ENTRY_RECORD quickly followed by REMOVE_ACCESS_CONTROL_ENTRY_RECORD for same ACL

[jira] [Resolved] (KAFKA-13793) Add validators for serialization and deserialization related configuration

Re: [DISCUSS] KIP-660: Pluggable ReplicaPlacer

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.1 #116

[DISCUSS] KIP-836: Addition of Information in DescribeQuorumResponse about Voter Lag

Re: [DISCUSS] KIP-834: Pause / Resume KafkaStreams Topologies

[jira] [Created] (KAFKA-13888) KIP-836: Addition of Information in DescribeQuorumResponse about Voter Lag

[RESULTS] [VOTE] Release Kafka version 3.2.0

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #914

Re: [DISCUSS] KIP-832 Allow creating a producer/consumer using a producer/consumer config

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.1 #115

Re: [DISCUSS] KIP-834: Pause / Resume KafkaStreams Topologies

Re: [DISCUSS] KIP-832 Allow creating a producer/consumer using a producer/consumer config

Re: [DISCUSS] KIP-834: Pause / Resume KafkaStreams Topologies

[GitHub] [kafka-site] rbowen opened a new pull request, #408: Move IRC pointer to libera.chat

Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready

Re: Permission to contribute

Permission to contribute

34 matches

Site Navigation

Mail list logo

Footer information