Re: [DISCUSS] KIP-591: Add Kafka Streams config to set default state store

[Matthias J. Sax]

Thanks for the KIP!

I think it's a good step forward for the DSL and it makes sense to 
exclude the PAPI and custom stores for now.


About the config name, it seems to be overly complicated. Guozhang's 
argument about "store type" that is used to refer to kv, windowed, 
session stores make sense. But having "type" and "impl" in the name 
sounds clumsy to me. What about a simplified name:


  default.dsl.store.impl

or maybe even better

  default.dsl.store

for the config?

Btw: the KIP still contains a view case of the originally proposed 
config name `default.dsl.store.type` which should be updated.



About `TopologyConfig`: we should add all public methods of this class, 
including constructors.



-Matthias

On 12/22/21 4:54 AM, Luke Chen wrote:

Hi Guozhang,

Thanks for the comments.
And I agree it's better to rename it to `default.dsl.store.impl.type` for
differentiation.
I've updated the KIP.

Thank you.
Luke


On Wed, Dec 22, 2021 at 3:12 AM Guozhang Wang  wrote:


Thanks Luke, I do not have any major comments on the wiki any more. BTW
thanks for making the "public StreamsBuilder(final TopologyConfig
topologyConfigs)" API public now, I think it will benefit a lot of future
works!

I think with the new API, we can deprecate the `build(props)` function in
StreamsBuilder now, and will file a separate JIRA for it.

Just a few nits:

1) There seems to be a typo at the end: "ROCK_DB"
2) Sometimes people refer to "store type" as kv-store, window-store etc;
maybe we can differentiate them a bit by calling the new API names
`StoreImplType`,
`default.dsl.store.impl.type` and `The default store implementation type
used by DSL operators`.

On Thu, Dec 16, 2021 at 2:29 AM Luke Chen  wrote:


Hi Guozhang,

I've updated the KIP to use `enum`, instead of store implementation, and
some content accordingly.
Please let me know if there's other comments.


Thank you.
Luke

On Sun, Dec 12, 2021 at 3:55 PM Luke Chen  wrote:


Hi Guozhang,

Thanks for your comments.
I agree that in the KIP, there's a trade-off regarding the API

complexity.

With the store impl, we can support default custom stores, but

introduce

more complexity for users, while with the enum types, users can

configure

default built-in store types easily, but it can't work for custom

stores.


For me, I'm OK to narrow down the scope and introduce the default

built-in

enum store types first.
And if there's further request, we can consider a better way to support
default store impl.

I'll update the KIP next week, unless there are other opinions from

other

members.

Thank you.
Luke

On Fri, Dec 10, 2021 at 6:33 AM Guozhang Wang 

wrote:



Thanks Luke for the updated KIP.

One major change the new proposal has it to change the original enum

store

type with a new interface. Where in the enum approach our internal
implementations would be something like:

"
Stores#keyValueBytesStoreSupplier(storeImplTypes, storeName, ...)
Stores#windowBytesStoreSupplier(storeImplTypes, storeName, ...)
Stores#sessionBytesStoreSupplier(storeImplTypes, storeName, ...)
"

And inside the impl classes like here we would could directly do:

"
if ((supplier = materialized.storeSupplier) == null) {
 supplier =
Stores.windowBytesStoreSupplier(materialized.storeImplType())
}
"

While I understand the benefit of having an interface such that user
customized stores could be used as the default store types as well,
there's
a trade-off I feel regarding the API complexity. Since with this

approach,

our API complexity granularity is in the order of "number of impl

types" *

"number of store types". This means that whenever we add new store

types

in
the future, this API needs to be augmented and customized impl needs

to

be

updated to support the new store types, in addition, not all custom

impl

types may support all store types, but with this interface they are

forced

to either support all or explicitly throw un-supported exceptions.

The way I see a default impl type is that, they would be safe to use

for

any store types, and since store types are evolved by the library

itself,

the default impls would better be controlled by the library as well.
Custom
impl classes can choose to replace some of the stores explicitly, but

may

not be a best fit as the default impl classes --- if there are in the
future, one way we can consider is to make Stores class extensible

along

with the enum so that advanced users can add more default impls,

assuming

such scenarios are not very common.

So I'm personally still a bit learning towards the enum approach with

a

narrower scope, for its simplicity as an API and also its low

maintenance

cost in the future. Let me know what do you think?


Guozhang


On Wed, Dec 1, 2021 at 6:48 PM Luke Chen  wrote:


Hi devs,

I'd like to propose a KIP to allow users to set default store
implementation class (built-in RocksDB/InMemory, or custom one), and
default to RocksDB state store, to keep backward compatibility.

Detailed 

Re: [VOTE] KIP-810: Allow producing records with null values in Kafka Console Producer

[Matthias J. Sax]

Thanks for the KIP. Addresses a long standing gap!

+1 (binding)

On 1/12/22 2:53 AM, Tom Bentley wrote:

Hi Mickael,

Thanks! +1 (binding).

Tom



On Mon, 10 Jan 2022 at 15:31, Chris Egerton 
wrote:


Hi Mickael,

Happy new year! +1 non-binding.

Cheers,

Chris

On Wed, Jan 5, 2022 at 3:08 PM John Roesler  wrote:


Thanks Mickael!

I’m +1 (binding)

Thanks,
John

On Tue, Jan 4, 2022, at 08:53, Israel Ekpo wrote:

Thanks for the KIP, Mickael

Looks great!

+1 (non-binding)



On Tue, Jan 4, 2022 at 5:24 AM Mickael Maison <

mickael.mai...@gmail.com>

wrote:


Hi,

I'd like to start a vote on KIP-810 that adds the option to produce
records with a null value





https://cwiki.apache.org/confluence/display/KAFKA/KIP-810%3A+Allow+producing+records+with+null+values+in+Kafka+Console+Producer


Let me know if you have any feedback.

Thanks,
Mickael


--
Israel Ekpo
Lead Instructor, IzzyAcademy.com
https://www.youtube.com/c/izzyacademy
https://izzyacademy.com/

Re: [DISCUSS] KIP-811 Add separate delete.interval.ms to Kafka Streams

[Matthias J. Sax]

Thanks for the KIP! Very good discussion.

I want to raise one point: I am not a fan of the `min.` prefix of the 
config, because all configs like this are _minimums_. We also use 
`commit.interval.ms` and not `min.commit.interval.ms` for example. I 
would suggest to strip the `min.` prefix.



For the config description, I would remove


The default value is the same as the default for commit.interval.ms (3).


It seems unnecessary to me.


-Matthias



On 1/12/22 7:21 AM, Nick Telford wrote:

Thanks Bruno, I've made those changes.

I'll call a vote on the KIP later today.

Regards,

Nick Telford

On Wed, 12 Jan 2022 at 12:13, Bruno Cadonna  wrote:


Hi Nick,

Great!

I think the KIP is ready for voting. I have just a couple of minor
comments.

a.
In the config description, I would replace

"Purging will occur after at least this value since the last purge, but
may be delayed until later in order to meet the processing guarantee."

with

"Purging will occur after at least this value since the last purge, but
may be delayed until later."

I do not really understand what you mean with "meet the processing
guarantee" and I think it suffices to say that the purge might be delayed.


b.
I would change the title to

"Add config min.repartition.purge.interval.ms to Kafka Streams"


c.
The current rejected alternative leaks implementation details since it
refers to the coupling of purge to commit and we agreed to leave that
out of the KIP.


d.
Could you add my proposal to specify the config as a multiple of the
commit interval to the rejected alternatives with the reason why we
discarded it?

For the rest, I am +1.

Best,
Bruno



On 11.01.22 16:47, Nick Telford wrote:

Hi Bruno,

Thanks for your recommendations.

I've made those changes, and also updated the description for the new
config to read: "The minimum frequency in milliseconds with which to

delete

fully consumed records from repartition topics. *Purging will occur after
at least this value since the last purge, but may be delayed until later

in

order to meet the processing guarantee.* The default value is the same as
the default for commit.interval.ms (3).  (Note, unlike
commit.interval.ms, the default for this value remains unchanged when
processing.guarantee is set to exactly_once_v2)."

This should make it clear that this is just a minimum interval, without
leaking too much detail in to the specification.

If there are no other issues, I'll put this to a vote.

Regards,

Nick Telford

On Tue, 11 Jan 2022 at 15:34, Bruno Cadonna  wrote:


Hi Nick,

Sorry for the delay!

Regarding point 7, I am fine with keeping the config as an interval and
keeping it independently of the commit interval. However, I would then
remove the following paragraph from the KIP:

"We will still wait for a commit before explicitly deleting repartition
records, but we will only do so if the time since the last record
deletion is at least repartition.purge.interval.ms. This means the
lower-bound for repartition.purge.interval.ms  is effectively capped by
the value of commit.interval.ms."

The reason is that in the previous paragraph you say that the configs
can be modified separately and then in this paragraph you bind the purge
interval to the commit interval. This seems a contradiction and
indicates that you are leaking too much implementation details into the
KIP. I think, just saying that the purge interval is a minimum and name
it accordingly without talking about the actual implementation makes the
KIP more robust against future implementation changes.

My proposal for the config name is min.repartition.purge.interval.ms or
even min.purge.interval.ms with a preference for the former.

Best,
Bruno



On 04.01.22 17:21, Nick Telford wrote:


You missed one "delete.interval.ms" in the last paragraph in section
"Proposed Changes".



I've fixed these now, including the title that I somehow missed!

I am afraid, I again need to comment on point 7. IMO, it does not make

sense to be able to tune repartition.purge.interval.ms and

commit.interval.ms separately when the purge can only happen during a

commit. For example, if I set commit.interval.ms to 3 ms and

repartition.purge.interval.ms to 35000 ms, the records will be purged

at


every second commit, i.e., every 6 ms. What benefit do users have

to


set repartition.purge.interval.ms separately from commit.interval.ms?

The rate of purging will never be 1 / 35000, the rate will be 1 /

2*commit.interval.ms..


Could we address this by chaning the name of the configuration to

something

like "repartition.purge.min.interval.ms", to indicate that the

repartition

purge interval will be *at least* this value?

If that's still not suitable, are there any other existing

configurations

that behave in a similar way, i.e. dictate a multiple of another

interval,

that we could use as a basis for a new name for this configuration?

Additionally, I have a new point.

8. If user code has access to the 

Re: Pls give me kafka jira and confluence permission

[Matthias J. Sax]

Done.

On 1/12/22 3:05 PM, Yu Kvicii wrote:

Hi,
I’d like to request permissions to contribute to Kafka to propose a KIP

Wiki Id:Kvicii.Yu
Jira Id: Kvicii.Yu

Thx

[jira] [Created] (KAFKA-13593) ThrottledChannelReaper slows broker shutdown by multiple seconds

[Steven Schlansker (Jira)]

Steven Schlansker created KAFKA-13593:
-

 Summary: ThrottledChannelReaper slows broker shutdown by multiple 
seconds
 Key: KAFKA-13593
 URL: https://issues.apache.org/jira/browse/KAFKA-13593
 Project: Kafka
  Issue Type: Improvement
  Components: core
Affects Versions: 3.0.0
Reporter: Steven Schlansker


We run an embedded KRaft broker in integration tests, to test that our Producer 
/ Consumers are all hooked up and verify our overall pipeline.

While trying to deliver CI speed improvements, we noticed that the majority of 
time for a small test is actually spent in Kafka broker shutdown. From the 
logs, it looks like the ClientQuotaManager has multiple ThrottledChannelReaper 
threads and each of them takes up to a second to shutdown.
{code:java}
2022-01-12T15:26:31.932Z [main] INFO  kafka.log.LogManager - Shutdown complete.
2022-01-12T15:26:31.934Z [main] INFO  
k.s.ClientQuotaManager$ThrottledChannelReaper - [ThrottledChannelReaper-Fetch]: 
Shutting down
2022-01-12T15:26:32.311Z [main] INFO  
k.s.ClientQuotaManager$ThrottledChannelReaper - [ThrottledChannelReaper-Fetch]: 
Shutdown completed
2022-01-12T15:26:32.311Z [ThrottledChannelReaper-Fetch] INFO  
k.s.ClientQuotaManager$ThrottledChannelReaper - [ThrottledChannelReaper-Fetch]: 
Stopped
2022-01-12T15:26:32.311Z [main] INFO  
k.s.ClientQuotaManager$ThrottledChannelReaper - 
[ThrottledChannelReaper-Produce]: Shutting down
2022-01-12T15:26:33.312Z [ThrottledChannelReaper-Produce] INFO  
k.s.ClientQuotaManager$ThrottledChannelReaper - 
[ThrottledChannelReaper-Produce]: Stopped
2022-01-12T15:26:33.312Z [main] INFO  
k.s.ClientQuotaManager$ThrottledChannelReaper - 
[ThrottledChannelReaper-Produce]: Shutdown completed
2022-01-12T15:26:33.312Z [main] INFO  
k.s.ClientQuotaManager$ThrottledChannelReaper - 
[ThrottledChannelReaper-Request]: Shutting down
2022-01-12T15:26:34.315Z [ThrottledChannelReaper-Request] INFO  
k.s.ClientQuotaManager$ThrottledChannelReaper - 
[ThrottledChannelReaper-Request]: Stopped
2022-01-12T15:26:34.315Z [main] INFO  
k.s.ClientQuotaManager$ThrottledChannelReaper - 
[ThrottledChannelReaper-Request]: Shutdown completed
2022-01-12T15:26:34.315Z [main] INFO  
k.s.ClientQuotaManager$ThrottledChannelReaper - 
[ThrottledChannelReaper-ControllerMutation]: Shutting down
2022-01-12T15:26:35.317Z [main] INFO  
k.s.ClientQuotaManager$ThrottledChannelReaper - 
[ThrottledChannelReaper-ControllerMutation]: Shutdown completed
2022-01-12T15:26:35.317Z [ThrottledChannelReaper-ControllerMutation] INFO  
k.s.ClientQuotaManager$ThrottledChannelReaper - 
[ThrottledChannelReaper-ControllerMutation]: Stopped{code}
Inspecting the code, the ThrottledChannelReaper threads are marked as not 
interruptible, so ShutdownableThread does not interrupt the worker on shutdown. 
Unfortunately, the doWork method polls with a 1 second timeout. So you wait up 
to 1s for every type of quota, in this case for a total of almost 4s.

 

While this is not a problem for production Kafka brokers, where instances are 
expected to stay up for months, in tests that expect to spin up and down it is 
easily noticed and adds real overhead to CI.

 

Suggested possible remediations:
 * Allow ThrottledChannelReaper to be interrupted by ShutdownableThread
 * ThrottledChannelReaper overrides initiateShutdown and after calling 
{{super.initiateShutdown}} then enqueues a {{null}} element on the delayQueue 
to force worker thread to notice shutdown state
 * Reduce 1s poll timeout to something small (carries overhead penalty for all 
users though, so this is less desirable), or make it configurable so we can set 
it to e.g. 10ms in unit tests



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Pls give me kafka jira and confluence permission

[Yu Kvicii]

Hi,
I’d like to request permissions to contribute to Kafka to propose a KIP

Wiki Id:Kvicii.Yu
Jira Id: Kvicii.Yu

Thx 

Re: [VOTE] KIP-801: Implement an Authorizer that stores metadata in __cluster_metadata

[David Arthur]

+1 binding, thanks Colin!

On Mon, Dec 13, 2021 at 7:47 PM Colin McCabe  wrote:

> Hi all,
>
> I'd like to start the vote on KIP-801: Implement an Authorizer that stores
> metadata in __cluster_metadata
>
> The KIP is here: https://cwiki.apache.org/confluence/x/h5KqCw
>
> The original DISCUSS thread is here:
>
> https://lists.apache.org/thread/3d5o7h17ztjztjhblx4fln0wbbs1rmdq
>
> Please take a look and vote if you can.
>
> best,
> Colin
>


-- 
-David

Re: [DISCUSS] KIP-801: Implement an Authorizer that stores metadata in __cluster_metadata

[David Arthur]

Sounds good on the ordering, and yea I agree we can look at atomic ACL
modifications in the future. Thanks!

On Wed, Jan 12, 2022 at 3:53 PM Colin McCabe  wrote:

> Hi David,
>
> On Wed, Dec 15, 2021, at 07:28, David Arthur wrote:
> > 5) Ok, gotcha. So will the StandardAuthorizer be replaying records
> > directly, or will it get an *Image like other metadata consumers on the
> > broker?
> >
>
> It reads the information out of the MetadataDelta, being careful to
> preserve the ordering of the changes.
>
> The current implementation uses a LinkedHashMap to preserve that ordering.
> You can take a look at the PR here:
> https://github.com/apache/kafka/pull/11649/files
>
> > 6) I was thinking since the CreateAcl and DeleteAcl requests can modify
> > multiple ACL in one request, that we should reflect that by committing
> the
> > resulting records as an atomic batch. I think from an operators
> > perspective, they would expect the ACLs sent in their request to be
> > enacted together atomically.
> >
>
> That's never been guaranteed, though. Creating multiple ACLs in ZK
> requires changing multiple znodes, which is not atomic. Given that users
> haven't asked for this and it would add substantial complexity, can be
> discuss it later once we have feature parity with the ZK version?
>
> best,
> Colin
>
>
> >
> >
> > On Tue, Dec 14, 2021 at 4:20 PM Colin McCabe  wrote:
> >
> >> On Tue, Dec 14, 2021, at 08:27, José Armando García Sancio wrote:
> >> > Thanks for the additional information Colin.
> >> >
> >> ...
> >> >
> >> > It sounds to me like KIP-801 is assuming that this "bootstrapping KIP"
> >> > will at least generate a snapshot with this information in all of the
> >> > controllers. I would like to understand this a bit better. Do you
> >> > think that we need to write this "bootstrapping KIP" as soon as
> >> > possible?
> >> >
> >>
> >> Hi José,
> >>
> >> I don't know about "as soon as possible." The authorizer is useful even
> >> without the bootstrapping KIP, as I mentioned (just using super.users).
> But
> >> I do think we'll need the bootstrapping KIP before KRaft goes GA.
> >>
> >> best,
> >> Colin
> >>
> >
> >
> > --
> > -David
>


-- 
-David

Re: [DISCUSS] KIP-801: Implement an Authorizer that stores metadata in __cluster_metadata

[Colin McCabe]

Hi David,

On Wed, Dec 15, 2021, at 07:28, David Arthur wrote:
> 5) Ok, gotcha. So will the StandardAuthorizer be replaying records
> directly, or will it get an *Image like other metadata consumers on the
> broker?
>

It reads the information out of the MetadataDelta, being careful to preserve 
the ordering of the changes.

The current implementation uses a LinkedHashMap to preserve that ordering. You 
can take a look at the PR here: https://github.com/apache/kafka/pull/11649/files

> 6) I was thinking since the CreateAcl and DeleteAcl requests can modify
> multiple ACL in one request, that we should reflect that by committing the
> resulting records as an atomic batch. I think from an operators
> perspective, they would expect the ACLs sent in their request to be
> enacted together atomically.
>

That's never been guaranteed, though. Creating multiple ACLs in ZK requires 
changing multiple znodes, which is not atomic. Given that users haven't asked 
for this and it would add substantial complexity, can be discuss it later once 
we have feature parity with the ZK version?

best,
Colin


>
>
> On Tue, Dec 14, 2021 at 4:20 PM Colin McCabe  wrote:
>
>> On Tue, Dec 14, 2021, at 08:27, José Armando García Sancio wrote:
>> > Thanks for the additional information Colin.
>> >
>> ...
>> >
>> > It sounds to me like KIP-801 is assuming that this "bootstrapping KIP"
>> > will at least generate a snapshot with this information in all of the
>> > controllers. I would like to understand this a bit better. Do you
>> > think that we need to write this "bootstrapping KIP" as soon as
>> > possible?
>> >
>>
>> Hi José,
>>
>> I don't know about "as soon as possible." The authorizer is useful even
>> without the bootstrapping KIP, as I mentioned (just using super.users). But
>> I do think we'll need the bootstrapping KIP before KRaft goes GA.
>>
>> best,
>> Colin
>>
>
>
> -- 
> -David

Re: [VOTE] KIP-801: Implement an Authorizer that stores metadata in __cluster_metadata

[Colin McCabe]

bumping this thread

On Mon, Dec 13, 2021, at 16:46, Colin McCabe wrote:
> Hi all,
>
> I'd like to start the vote on KIP-801: Implement an Authorizer that 
> stores metadata in __cluster_metadata
>
> The KIP is here: https://cwiki.apache.org/confluence/x/h5KqCw
>
> The original DISCUSS thread is here:
>
> https://lists.apache.org/thread/3d5o7h17ztjztjhblx4fln0wbbs1rmdq
>
> Please take a look and vote if you can.
>
> best,
> Colin

Re: [DISCUSS] KIP-704: Send a hint to broker if it is an unclean leader

[David Arthur]

Jose, thanks for the KIP!

1) Does recovering from an unclean state bump the leader epoch?

2) The name of "NewIsUnclean" field in AlterIsrRequest is a little strange.
>From the description, it sounds like this will be used to by the broker to
indicate to the controller that it has recovered from unclean leader
election. If that's the case, maybe something like "RecoverFromUnclean"
would be better?

3) Will followers try to fetch from an unclean leader? Or will they wait
for a LISR with unclean=false (or a PartitionChangeRecord with
unclean=false)?

4) Is there any other way for a partition to recover from an unclean state
other than the leader sending AISR with NewIsUnclean=false? Is it possible
for a leader to fail the recovery process? Are further unclean election
attempts by a user blocked until we recover?

General comment on naming, maybe we can go with something like
"UncleanElection" or "UncleanEpoch" instead of "IsUnclean".

Thanks!
David

On Tue, Jan 11, 2022 at 4:31 PM Jason Gustafson 
wrote:

> Hi Jose,
>
> Thanks for the KIP. Just a minor question about this:
>
> > This means that the leader will not allow followers to join the ISR until
> it has recovered from the unclean leader election.
>
> If I understand correctly, the main reason for this is to avoid the need to
> propagate the "IsUnclean" flag between elections. It ensures that we cannot
> have a "clean" election until the recovery has completed. On the other
> hand, if we need to do another unclean election because the recovering
> leader failed, then we would get the "IsUnclean" flag naturally. Are there
> any additional limitations we should consider while the unclean leader is
> recovering? For example, should we not allow consumers to read from the
> partition until the recovery has completed as well?
>
> By the way, I do find the naming of the "IsUnclean" field a tad awkward.
> The naming suggests that it reflects upon the election, but then it is
> strange that the election becomes clean through recovery (which obviously
> cannot restore the lost data). An alternative name might be
> "UncleanRecoveryRequired." Another option might be to consider it more of a
> partition state. After an unclean election, then the state might be
> UNCLEAN_ELECTED. After recovery, it might transition to UNCLEAN_RECOVERED.
> Then at least we keep track of the fact that the current leader was
> uncleanly elected. Not sure how important that is, just a thought..
>
> Best,
> Jason
>
>
>
>
> On Mon, Jan 10, 2022 at 11:47 AM José Armando García Sancio
>  wrote:
>
> > Hi all,
> >
> > I would like to open the discussion on implementing "KIP-704: Send a
> > hint to broker if it is an unclean leader." See this wiki page for
> > details: https://cwiki.apache.org/confluence/x/kAZRCg
> >
> > Thanks!
> > --
> > -Jose
> >
>


-- 
-David

Re: [DISCUSS] KIP-808: Add support for unix epoch precision in TimestampConverter SMT

[Mickael Maison]

Hi Julien,

Thanks for the KIP. I looks like a useful improvement to the
TimestampConverter SMT.

I'd suggest adding the documentation for the new setting to the KIP.
I've had to go check your PR to fully understand how you want to use
it, both for input and output.
Apart from that, if you don't get any further feedback, feel free to
start a vote.

Thanks,
Mickael

On Tue, Dec 21, 2021 at 2:19 PM Julien Chanaud  wrote:
>
> Hi everyone,
>
> Bumping this KIP discussion.
> It's a small change, entirely backward compatible and I'd love your
> feedback on it.
> Thanks,
> Julien
>
>
> Le jeu. 9 déc. 2021 à 21:56, Julien Chanaud  a 
> écrit :
> >
> > Hi everyone,
> >
> > I would like to start a discussion for KIP-808
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-808%3A+Add+support+for+unix+epoch+precision+in+TimestampConverter+SMT
> >
> > This seems like a simple change but I suspect there are several things to 
> > consider, most notably regarding the java.util.Date object, which is at the 
> > heart of the conversions.
> >
> > Let me know what you think.
> >
> > Julien
> >

[VOTE] KIP-811: Add config min.repartition.purge.interval.ms to Kafka Streams

[Nick Telford]

Hi everyone,

I'd like to call a vote to adopt KIP-811: Add config
min.repartition.purge.interval.ms to Kafka Streams

.

Regards

Nick Telford

Re: [DISCUSS] KIP-811 Add separate delete.interval.ms to Kafka Streams

[Nick Telford]

Thanks Bruno, I've made those changes.

I'll call a vote on the KIP later today.

Regards,

Nick Telford

On Wed, 12 Jan 2022 at 12:13, Bruno Cadonna  wrote:

> Hi Nick,
>
> Great!
>
> I think the KIP is ready for voting. I have just a couple of minor
> comments.
>
> a.
> In the config description, I would replace
>
> "Purging will occur after at least this value since the last purge, but
> may be delayed until later in order to meet the processing guarantee."
>
> with
>
> "Purging will occur after at least this value since the last purge, but
> may be delayed until later."
>
> I do not really understand what you mean with "meet the processing
> guarantee" and I think it suffices to say that the purge might be delayed.
>
>
> b.
> I would change the title to
>
> "Add config min.repartition.purge.interval.ms to Kafka Streams"
>
>
> c.
> The current rejected alternative leaks implementation details since it
> refers to the coupling of purge to commit and we agreed to leave that
> out of the KIP.
>
>
> d.
> Could you add my proposal to specify the config as a multiple of the
> commit interval to the rejected alternatives with the reason why we
> discarded it?
>
> For the rest, I am +1.
>
> Best,
> Bruno
>
>
>
> On 11.01.22 16:47, Nick Telford wrote:
> > Hi Bruno,
> >
> > Thanks for your recommendations.
> >
> > I've made those changes, and also updated the description for the new
> > config to read: "The minimum frequency in milliseconds with which to
> delete
> > fully consumed records from repartition topics. *Purging will occur after
> > at least this value since the last purge, but may be delayed until later
> in
> > order to meet the processing guarantee.* The default value is the same as
> > the default for commit.interval.ms (3).  (Note, unlike
> > commit.interval.ms, the default for this value remains unchanged when
> > processing.guarantee is set to exactly_once_v2)."
> >
> > This should make it clear that this is just a minimum interval, without
> > leaking too much detail in to the specification.
> >
> > If there are no other issues, I'll put this to a vote.
> >
> > Regards,
> >
> > Nick Telford
> >
> > On Tue, 11 Jan 2022 at 15:34, Bruno Cadonna  wrote:
> >
> >> Hi Nick,
> >>
> >> Sorry for the delay!
> >>
> >> Regarding point 7, I am fine with keeping the config as an interval and
> >> keeping it independently of the commit interval. However, I would then
> >> remove the following paragraph from the KIP:
> >>
> >> "We will still wait for a commit before explicitly deleting repartition
> >> records, but we will only do so if the time since the last record
> >> deletion is at least repartition.purge.interval.ms. This means the
> >> lower-bound for repartition.purge.interval.ms  is effectively capped by
> >> the value of commit.interval.ms."
> >>
> >> The reason is that in the previous paragraph you say that the configs
> >> can be modified separately and then in this paragraph you bind the purge
> >> interval to the commit interval. This seems a contradiction and
> >> indicates that you are leaking too much implementation details into the
> >> KIP. I think, just saying that the purge interval is a minimum and name
> >> it accordingly without talking about the actual implementation makes the
> >> KIP more robust against future implementation changes.
> >>
> >> My proposal for the config name is min.repartition.purge.interval.ms or
> >> even min.purge.interval.ms with a preference for the former.
> >>
> >> Best,
> >> Bruno
> >>
> >>
> >>
> >> On 04.01.22 17:21, Nick Telford wrote:
> 
>  You missed one "delete.interval.ms" in the last paragraph in section
>  "Proposed Changes".
> 
> >>>
> >>> I've fixed these now, including the title that I somehow missed!
> >>>
> >>> I am afraid, I again need to comment on point 7. IMO, it does not make
> >>>
> >>> sense to be able to tune repartition.purge.interval.ms and
> >>>
> >>> commit.interval.ms separately when the purge can only happen during a
> >>>
> >>> commit. For example, if I set commit.interval.ms to 3 ms and
> >>>
> >>> repartition.purge.interval.ms to 35000 ms, the records will be purged
> at
> >>>
> >>> every second commit, i.e., every 6 ms. What benefit do users have
> to
> >>>
> >>> set repartition.purge.interval.ms separately from commit.interval.ms?
> >>>
> >>> The rate of purging will never be 1 / 35000, the rate will be 1 /
> >>>
> >>> 2*commit.interval.ms..
> >>>
> >>>
> >>> Could we address this by chaning the name of the configuration to
> >> something
> >>> like "repartition.purge.min.interval.ms", to indicate that the
> >> repartition
> >>> purge interval will be *at least* this value?
> >>>
> >>> If that's still not suitable, are there any other existing
> configurations
> >>> that behave in a similar way, i.e. dictate a multiple of another
> >> interval,
> >>> that we could use as a basis for a new name for this configuration?
> >>>
> >>> Additionally, I have a new point.
> >>>
> >>> 8. If user code has 

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.1 #65

[Apache Jenkins Server]

See 

[DISCUSS] KIP-812: Introduce another form of the `KafkaStreams.close()` API that forces the member to leave the consumer group

[Seung-chan Ahn]

Hi team,

Here's the new KIP

 for this issue .

The title says pretty much what this KIP is for. Even though it's my first
draft, as A. Sophie Blee-Goldman has written rich descriptions and already
the solutions in the issue thread, I've enjoyed following up on the idea.

Please feel free to review on any point!

[VOTE] 3.1.0 RC1

[David Jacot]

Hello Kafka users, developers and client-developers,

This is the second candidate for release of Apache Kafka 3.1.0.

* Apache Kafka supports Java 17
* The FetchRequest supports Topic IDs (KIP-516)
* Extend SASL/OAUTHBEARER with support for OIDC (KIP-768)
* Add broker count metrics (KIP-748)
* Differentiate consistently metric latency measured in millis and
nanos (KIP-773)
* The eager rebalance protocol is deprecated (KAFKA-13439)
* Add TaskId field to StreamsException (KIP-783)
* Custom partitioners in foreign-key joins (KIP-775)
* Fetch/findSessions queries with open endpoints for
SessionStore/WindowStore (KIP-766)
* Range queries with open endpoints (KIP-763)
* Add total blocked time metric to Streams (KIP-761)
* Add additional configuration to control MirrorMaker2 internal topics
naming convention (KIP-690)

Release notes for the 3.1.0 release:
https://home.apache.org/~dajac/kafka-3.1.0-rc1/RELEASE_NOTES.html

*** Please download, test and vote by Monday, January 17, 9am PT

Kafka's KEYS file containing PGP keys we use to sign the release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~dajac/kafka-3.1.0-rc1/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~dajac/kafka-3.1.0-rc1/javadoc/

* Tag to be voted upon (off 3.1 branch) is the 3.1.0 tag:
https://github.com/apache/kafka/releases/tag/3.1.0-rc1

* Documentation:
https://kafka.apache.org/31/documentation.html

* Protocol:
https://kafka.apache.org/31/protocol.html

* Successful Jenkins builds for the 3.1 branch:
Unit/integration tests:
https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.1/60/
System tests:
https://jenkins.confluent.io/job/system-test-kafka/job/3.1/66/

/**

Thanks,
David

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.1 #64

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 501908 lines...]
[2022-01-12T12:18:38.643Z] > Task :connect:api:javadoc
[2022-01-12T12:18:38.643Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2022-01-12T12:18:38.643Z] > Task :connect:api:jar UP-TO-DATE
[2022-01-12T12:18:38.643Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2022-01-12T12:18:38.643Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2022-01-12T12:18:38.643Z] > Task :connect:json:jar UP-TO-DATE
[2022-01-12T12:18:38.643Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2022-01-12T12:18:38.643Z] > Task :connect:api:javadocJar
[2022-01-12T12:18:38.643Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2022-01-12T12:18:38.643Z] > Task :connect:json:publishToMavenLocal
[2022-01-12T12:18:38.643Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2022-01-12T12:18:38.643Z] > Task :connect:api:testClasses UP-TO-DATE
[2022-01-12T12:18:38.643Z] > Task :connect:api:testJar
[2022-01-12T12:18:38.643Z] > Task :connect:api:testSrcJar
[2022-01-12T12:18:38.643Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2022-01-12T12:18:38.643Z] > Task :connect:api:publishToMavenLocal
[2022-01-12T12:18:38.714Z] Recording test results
[2022-01-12T12:18:42.217Z] > Task :streams:javadoc
[2022-01-12T12:18:42.217Z] > Task :streams:javadocJar
[2022-01-12T12:18:43.152Z] > Task :clients:javadoc
[2022-01-12T12:18:43.152Z] > Task :clients:javadocJar
[2022-01-12T12:18:44.087Z] 
[2022-01-12T12:18:44.087Z] > Task :clients:srcJar
[2022-01-12T12:18:44.087Z] Execution optimizations have been disabled for task 
':clients:srcJar' to ensure correctness due to the following reasons:
[2022-01-12T12:18:44.087Z]   - Gradle detected a problem with the following 
location: 
'/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.1/clients/src/generated/java'.
 Reason: Task ':clients:srcJar' uses this output of task 
':clients:processMessages' without declaring an explicit or implicit 
dependency. This can lead to incorrect results being produced, depending on 
what order the tasks are executed. Please refer to 
https://docs.gradle.org/7.2/userguide/validation_problems.html#implicit_dependency
 for more details about this problem.
[2022-01-12T12:18:45.835Z] 
[2022-01-12T12:18:45.835Z] > Task :clients:testJar
[2022-01-12T12:18:45.835Z] > Task :clients:testSrcJar
[2022-01-12T12:18:45.835Z] > Task 
:clients:publishMavenJavaPublicationToMavenLocal
[2022-01-12T12:18:45.835Z] > Task :clients:publishToMavenLocal
[2022-01-12T12:18:47.960Z] [Checks API] No suitable checks publisher found.
[Pipeline] echo
[2022-01-12T12:18:47.961Z] Skipping Kafka Streams archetype test for Java 17
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[2022-01-12T12:19:04.882Z] > Task :core:compileScala
[2022-01-12T12:20:11.865Z] > Task :core:classes
[2022-01-12T12:20:11.865Z] > Task :core:compileTestJava NO-SOURCE
[2022-01-12T12:20:34.115Z] > Task :core:compileTestScala
[2022-01-12T12:21:15.357Z] > Task :core:testClasses
[2022-01-12T12:21:27.132Z] > Task :streams:compileTestJava
[2022-01-12T12:21:27.132Z] > Task :streams:testClasses
[2022-01-12T12:21:28.064Z] > Task :streams:testJar
[2022-01-12T12:21:28.064Z] > Task :streams:testSrcJar
[2022-01-12T12:21:28.064Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2022-01-12T12:21:28.064Z] > Task :streams:publishToMavenLocal
[2022-01-12T12:21:28.064Z] 
[2022-01-12T12:21:28.064Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 8.0.
[2022-01-12T12:21:28.064Z] 
[2022-01-12T12:21:28.064Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2022-01-12T12:21:28.064Z] 
[2022-01-12T12:21:28.064Z] See 
https://docs.gradle.org/7.2/userguide/command_line_interface.html#sec:command_line_warnings
[2022-01-12T12:21:28.064Z] 
[2022-01-12T12:21:28.064Z] Execution optimizations have been disabled for 3 
invalid unit(s) of work during this build to ensure correctness.
[2022-01-12T12:21:28.064Z] Please consult deprecation warnings for more details.
[2022-01-12T12:21:28.064Z] 
[2022-01-12T12:21:28.064Z] BUILD SUCCESSFUL in 3m 9s
[2022-01-12T12:21:28.064Z] 77 actionable tasks: 38 executed, 39 up-to-date
[Pipeline] sh
[2022-01-12T12:21:31.476Z] + grep ^version= gradle.properties
[2022-01-12T12:21:31.477Z] + cut -d= -f 2
[Pipeline] dir
[2022-01-12T12:21:32.160Z] Running in 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.1/streams/quickstart
[Pipeline] {
[Pipeline] sh
[2022-01-12T12:21:34.296Z] + mvn clean install -Dgpg.skip
[2022-01-12T12:21:35.230Z] [INFO] Scanning for projects...

Re: [DISCUSS] KIP-811 Add separate delete.interval.ms to Kafka Streams

[Bruno Cadonna]

Hi Nick,

Great!

I think the KIP is ready for voting. I have just a couple of minor comments.

a.
In the config description, I would replace

"Purging will occur after at least this value since the last purge, but 
may be delayed until later in order to meet the processing guarantee."


with

"Purging will occur after at least this value since the last purge, but 
may be delayed until later."


I do not really understand what you mean with "meet the processing 
guarantee" and I think it suffices to say that the purge might be delayed.



b.
I would change the title to

"Add config min.repartition.purge.interval.ms to Kafka Streams"


c.
The current rejected alternative leaks implementation details since it 
refers to the coupling of purge to commit and we agreed to leave that 
out of the KIP.



d.
Could you add my proposal to specify the config as a multiple of the 
commit interval to the rejected alternatives with the reason why we 
discarded it?


For the rest, I am +1.

Best,
Bruno



On 11.01.22 16:47, Nick Telford wrote:

Hi Bruno,

Thanks for your recommendations.

I've made those changes, and also updated the description for the new
config to read: "The minimum frequency in milliseconds with which to delete
fully consumed records from repartition topics. *Purging will occur after
at least this value since the last purge, but may be delayed until later in
order to meet the processing guarantee.* The default value is the same as
the default for commit.interval.ms (3).  (Note, unlike
commit.interval.ms, the default for this value remains unchanged when
processing.guarantee is set to exactly_once_v2)."

This should make it clear that this is just a minimum interval, without
leaking too much detail in to the specification.

If there are no other issues, I'll put this to a vote.

Regards,

Nick Telford

On Tue, 11 Jan 2022 at 15:34, Bruno Cadonna  wrote:


Hi Nick,

Sorry for the delay!

Regarding point 7, I am fine with keeping the config as an interval and
keeping it independently of the commit interval. However, I would then
remove the following paragraph from the KIP:

"We will still wait for a commit before explicitly deleting repartition
records, but we will only do so if the time since the last record
deletion is at least repartition.purge.interval.ms. This means the
lower-bound for repartition.purge.interval.ms  is effectively capped by
the value of commit.interval.ms."

The reason is that in the previous paragraph you say that the configs
can be modified separately and then in this paragraph you bind the purge
interval to the commit interval. This seems a contradiction and
indicates that you are leaking too much implementation details into the
KIP. I think, just saying that the purge interval is a minimum and name
it accordingly without talking about the actual implementation makes the
KIP more robust against future implementation changes.

My proposal for the config name is min.repartition.purge.interval.ms or
even min.purge.interval.ms with a preference for the former.

Best,
Bruno



On 04.01.22 17:21, Nick Telford wrote:


You missed one "delete.interval.ms" in the last paragraph in section
"Proposed Changes".



I've fixed these now, including the title that I somehow missed!

I am afraid, I again need to comment on point 7. IMO, it does not make

sense to be able to tune repartition.purge.interval.ms and

commit.interval.ms separately when the purge can only happen during a

commit. For example, if I set commit.interval.ms to 3 ms and

repartition.purge.interval.ms to 35000 ms, the records will be purged at

every second commit, i.e., every 6 ms. What benefit do users have to

set repartition.purge.interval.ms separately from commit.interval.ms?

The rate of purging will never be 1 / 35000, the rate will be 1 /

2*commit.interval.ms..


Could we address this by chaning the name of the configuration to

something

like "repartition.purge.min.interval.ms", to indicate that the

repartition

purge interval will be *at least* this value?

If that's still not suitable, are there any other existing configurations
that behave in a similar way, i.e. dictate a multiple of another

interval,

that we could use as a basis for a new name for this configuration?

Additionally, I have a new point.

8. If user code has access to the processor context (e.g. in the

processor API), a commit can also be requested on demand by user code.

The KIP should clarify if purges might also happen during requested

commits or if purges only happen during automatic commits.


Good point. I'm holding off on amending this for now until we agree on an
outcome for point 7 above, because I suspect it may at least influence

the

wording of this section.

Thanks for the feedback, and sorry for the delay. Now that the holidays

are

over I'm able to focus on this again.

Regards,

Nick Telford

On Wed, 22 Dec 2021 at 09:28, Bruno Cadonna  wrote:


Hi Nick,

Thanks for the updates!

The motivation section and 

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.0 #164

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-810: Allow producing records with null values in Kafka Console Producer

[Tom Bentley]

Hi Mickael,

Thanks! +1 (binding).

Tom



On Mon, 10 Jan 2022 at 15:31, Chris Egerton 
wrote:

> Hi Mickael,
>
> Happy new year! +1 non-binding.
>
> Cheers,
>
> Chris
>
> On Wed, Jan 5, 2022 at 3:08 PM John Roesler  wrote:
>
> > Thanks Mickael!
> >
> > I’m +1 (binding)
> >
> > Thanks,
> > John
> >
> > On Tue, Jan 4, 2022, at 08:53, Israel Ekpo wrote:
> > > Thanks for the KIP, Mickael
> > >
> > > Looks great!
> > >
> > > +1 (non-binding)
> > >
> > >
> > >
> > > On Tue, Jan 4, 2022 at 5:24 AM Mickael Maison <
> mickael.mai...@gmail.com>
> > > wrote:
> > >
> > >> Hi,
> > >>
> > >> I'd like to start a vote on KIP-810 that adds the option to produce
> > >> records with a null value
> > >>
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-810%3A+Allow+producing+records+with+null+values+in+Kafka+Console+Producer
> > >>
> > >> Let me know if you have any feedback.
> > >>
> > >> Thanks,
> > >> Mickael
> > >>
> > > --
> > > Israel Ekpo
> > > Lead Instructor, IzzyAcademy.com
> > > https://www.youtube.com/c/izzyacademy
> > > https://izzyacademy.com/
> >
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #611

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-719: Add Log4J2 Appender

[Viktor Somogyi-Vass]

Hi Dongjin,

We're also looking into this matter as our stack was also affected by all
the log4j hell and users increasingly pushing us to upgrade to log4j2 or
logback because of the existing vulnerabilities in log4j1.
Regarding the points raised by Haruki:

a) In my opinion the best would be to make the dynamic logger support
(Log4jController and LoggingResource) pluggable for log4j2 and logback (so
an interface could be used to define the dynamic logging control methods
and a config to specify the implementation). That way we're not bound to
either logback or log4j and seems like a low-effort thing to do.
Additionally this could be used in Connect too in LoggingResource.

b) I think testing dependencies aren't that important from the user
perspective, it's fine to either use log4j2 or logback, whichever is
easier. Kafka is either used from the distribution (tgz) or pulled in
through maven, but test dependencies shouldn't be exposed to the world.

c) I would support deprecating the appender in favor of the log4j2 Kafka
appender. VerifiableLog4jAppender is intended as a testing tool anyway, so
I think it's less important to change this to logback.

Future vulnerabilities will always be found in either logback or log4j2 or
any other logging framework, so I think the safest approach is to allow
users to choose their implementation, while in tests I think we're free to
use whatever we want as that shouldn't be constrained by vulnerabilities.

Viktor

On Thu, Dec 23, 2021 at 9:37 AM Haruki Okada  wrote:

> Thanks for the clarification.
>
> About 2, I wan't aware of those concerns.
> Let me check them first.
>
>
> Thanks,
>
> 2021年12月23日(木) 13:37 Dongjin Lee :
>
> > Hi Haruki,
> >
> >
> > Thanks for organizing the issue.
> >
> >
> > If the community prefers logback, I will gladly change the dependency and
> > update the PR. However, it has the following issues:
> >
> >
> > 1. The log4j2 vulnerabilities seem mostly fixed, and KIP-653 + KIP-719
> are
> > not released yet. So, using log4j2 (whose recent update pace is so high)
> > will not affect the users.
> >
> >
> > 2. To switch to logback, the following features should be reworked:
> >
> >
> >   a. Dynamic logger level configuration (core, connect)
> >
> >   b. Logging tests (streams)
> >
> >   c. Kafka Appender (tools)
> >
> >
> > a and b are the most challenging ones since there is little documentation
> > on how to do this, so it requires analyzing the implementation itself.
> > (what I actually did with log4j2) About c, logback does not provide a
> Kafka
> > Appender so we have to provide an equivalent.
> >
> >
> > It is why I prefer to use log4j2. How do you think?
> >
> >
> > Thanks,
> >
> > Dongjin
> >
> >
> > On Thu, Dec 23, 2021 at 9:01 AM Haruki Okada 
> wrote:
> >
> > > Hi, Dongjin,
> > >
> > > Sorry for interrupting the discussion.
> > > And thank you for your hard work about KIP-653, KIP-719.
> > >
> > > I understand that KIP-653 is already accepted so log4j2 is the choice
> of
> > > the Kafka community though, I'm now feeling that logback is a better
> > choice
> > > here.
> > >
> > > Reasons:
> > >
> > > - even after "log4shell", several vulnerabilities found on log4j2 so
> new
> > > versions are released and users have to update in high-pace
> > > * actually, a CVE was also reported for logback (CVE-2021-42550)
> but
> > it
> > > requires edit-permission of the config file for an attacker so it's
> much
> > > less threatening
> > > - log4j1.x and logback are made by same developer (ceki), so
> > substantially
> > > the successor of log4j1 is logback rather than log4j2
> > > - in Hadoop project, seems similar suggestion was made from a PMC
> > > * https://issues.apache.org/jira/browse/HADOOP-12956
> > >
> > >
> > > What do you think about adopting logback instead?
> > >
> > >
> > > Thanks,
> > >
> > > 2021年12月21日(火) 18:02 Dongjin Lee :
> > >
> > > > Hi Mickael,
> > > >
> > > > > In the meantime, you may want to bump the VOTE thread too.
> > > >
> > > > Sure, I just reset the voting thread with a brief context.
> > > >
> > > > Thanks,
> > > > Dongjin
> > > >
> > > > On Tue, Dec 21, 2021 at 2:13 AM Mickael Maison <
> > mickael.mai...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Thanks Dongjin!
> > > > >
> > > > > I'll take a look soon.
> > > > > In the meantime, you may want to bump the VOTE thread too.
> > > > >
> > > > > Best,
> > > > > Mickael
> > > > >
> > > > >
> > > > > On Sat, Dec 18, 2021 at 10:00 AM Dongjin Lee 
> > > wrote:
> > > > > >
> > > > > > Hi Mickael,
> > > > > >
> > > > > > Finally, I did it! As you can see at the PR
> > > > > > , KIP-719 now uses
> > > > log4j2's
> > > > > > Kafka appender, and log4j-appender is not used by the other
> modules
> > > > > > anymore. You can see how it will work with KIP-653 at this
> preview
> > > > > > <
> http://home.apache.org/~dongjin/post/apache-kafka-log4j2-support/
> > >,
> > > > > based
> > > > > > on Apache Kafka 3.0.0. The 

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.1 #63

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 503432 lines...]
[2022-01-12T10:16:33.644Z] 
[2022-01-12T10:16:33.644Z] > Task :streams:compileJava UP-TO-DATE
[2022-01-12T10:16:33.644Z] > Task :streams:classes UP-TO-DATE
[2022-01-12T10:16:33.644Z] > Task :streams:test-utils:compileJava UP-TO-DATE
[2022-01-12T10:16:33.644Z] > Task :streams:jar UP-TO-DATE
[2022-01-12T10:16:33.644Z] > Task 
:streams:generateMetadataFileForMavenJavaPublication
[2022-01-12T10:16:33.998Z] [Checks API] No suitable checks publisher found.
[Pipeline] echo
[2022-01-12T10:16:33.999Z] Skipping Kafka Streams archetype test for Java 17
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[2022-01-12T10:16:34.851Z] > Task :connect:api:javadoc
[2022-01-12T10:16:34.851Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2022-01-12T10:16:34.851Z] > Task :connect:api:jar UP-TO-DATE
[2022-01-12T10:16:34.851Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2022-01-12T10:16:34.851Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2022-01-12T10:16:34.851Z] > Task :connect:api:javadocJar
[2022-01-12T10:16:34.851Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2022-01-12T10:16:34.851Z] > Task :connect:api:testClasses UP-TO-DATE
[2022-01-12T10:16:34.851Z] > Task :connect:json:jar UP-TO-DATE
[2022-01-12T10:16:34.851Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2022-01-12T10:16:35.987Z] > Task :connect:api:testJar
[2022-01-12T10:16:35.987Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2022-01-12T10:16:35.987Z] > Task :connect:json:publishToMavenLocal
[2022-01-12T10:16:35.987Z] > Task :connect:api:testSrcJar
[2022-01-12T10:16:35.988Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2022-01-12T10:16:35.988Z] > Task :connect:api:publishToMavenLocal
[2022-01-12T10:16:39.477Z] > Task :streams:javadoc
[2022-01-12T10:16:39.477Z] > Task :streams:javadocJar
[2022-01-12T10:16:39.477Z] > Task :streams:compileTestJava UP-TO-DATE
[2022-01-12T10:16:39.477Z] > Task :streams:testClasses UP-TO-DATE
[2022-01-12T10:16:41.868Z] > Task :streams:testJar
[2022-01-12T10:16:41.868Z] > Task :clients:javadoc
[2022-01-12T10:16:42.901Z] > Task :clients:javadocJar
[2022-01-12T10:16:42.901Z] > Task :streams:testSrcJar
[2022-01-12T10:16:42.901Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2022-01-12T10:16:42.901Z] > Task :streams:publishToMavenLocal
[2022-01-12T10:16:44.963Z] 
[2022-01-12T10:16:44.963Z] > Task :clients:srcJar
[2022-01-12T10:16:44.963Z] Execution optimizations have been disabled for task 
':clients:srcJar' to ensure correctness due to the following reasons:
[2022-01-12T10:16:44.963Z]   - Gradle detected a problem with the following 
location: '/home/jenkins/workspace/Kafka_kafka_3.1/clients/src/generated/java'. 
Reason: Task ':clients:srcJar' uses this output of task 
':clients:processMessages' without declaring an explicit or implicit 
dependency. This can lead to incorrect results being produced, depending on 
what order the tasks are executed. Please refer to 
https://docs.gradle.org/7.2/userguide/validation_problems.html#implicit_dependency
 for more details about this problem.
[2022-01-12T10:16:48.149Z] 
[2022-01-12T10:16:48.149Z] > Task :clients:testJar
[2022-01-12T10:16:48.149Z] > Task :clients:testSrcJar
[2022-01-12T10:16:48.149Z] > Task 
:clients:publishMavenJavaPublicationToMavenLocal
[2022-01-12T10:16:48.149Z] > Task :clients:publishToMavenLocal
[2022-01-12T10:16:49.181Z] 
[2022-01-12T10:16:49.181Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 8.0.
[2022-01-12T10:16:49.181Z] 
[2022-01-12T10:16:49.181Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2022-01-12T10:16:49.181Z] 
[2022-01-12T10:16:49.181Z] See 
https://docs.gradle.org/7.2/userguide/command_line_interface.html#sec:command_line_warnings
[2022-01-12T10:16:49.181Z] 
[2022-01-12T10:16:49.181Z] Execution optimizations have been disabled for 3 
invalid unit(s) of work during this build to ensure correctness.
[2022-01-12T10:16:49.181Z] Please consult deprecation warnings for more details.
[2022-01-12T10:16:49.181Z] 
[2022-01-12T10:16:49.181Z] BUILD SUCCESSFUL in 34s
[2022-01-12T10:16:49.181Z] 77 actionable tasks: 34 executed, 43 up-to-date
[Pipeline] sh
[2022-01-12T10:16:52.698Z] + grep ^version= gradle.properties
[2022-01-12T10:16:52.698Z] + cut -d= -f 2
[Pipeline] dir
[2022-01-12T10:16:53.391Z] Running in 
/home/jenkins/workspace/Kafka_kafka_3.1/streams/quickstart
[Pipeline] {
[Pipeline] sh
[2022-01-12T10:16:56.047Z] + mvn clean install -Dgpg.skip

Re: [DISCUSS] KIP-810: Allow producing records with null values in Kafka Console Producer

[Mickael Maison]

Hi Tom,

That's a good point. I agree, in case 5) it makes more sense to return an error.
I've updated the KIP accordingly.

Thanks,
Mickael

On Mon, Jan 10, 2022 at 5:19 PM Tom Bentley  wrote:
>
> Hi Mickael,
>
> Thanks for the KIP.
>
> Case 5 in the KIP seems a bit weird to me:
>
> > > NULL:v0mykeymyvalue   (5)
> > This will result in a record with key="mykey", value="myvalue" and
> > headers={"NULL": "v0"} This is because headers cannot have a null key.
> >
>
> To me this feels like it's more suited to an error message (like the case
> of collision between the null marker and other flags). With the currently
> defined semantics this is the only one of the 5 examples where the null
> marker doesn't result in a null being written in the expected place. And
> it's not like the user couldn't easily select a different null marker to
> avoid a collision with a header key. Was there a reason you went for these
> semantics?
>
> Kind regards,
>
> Tom
>
> On Mon, 20 Dec 2021 at 17:06, Mickael Maison 
> wrote:
>
> > Thanks for the feedback!
> >
> > Chris,
> > I've updated the KIP and added examples to show how the null marker
> > will be applied.
> >
> > Luke,
> > Good point, I've updating the KIP to mention the user will get an
> > error message if the marker collides with other flags.
> >
> > Thanks,
> > Mickael
> >
> >
> > On Thu, Dec 16, 2021 at 2:00 PM Luke Chen  wrote:
> > >
> > > Hi Mickael,
> > >
> > > Thanks for the KIP!
> > > This will be a helpful feature for debugging, for sure!
> > >
> > > I have one question:
> > > Will we have some safe net for the collision of `key.separator` and the
> > new
> > > introduced `null.marker`.
> > > That is, what if user set the same or overlapped  `key.separator` and
> > > `null.marker`, how would we handle it?
> > > Ex: key.separator="-", null.marker="--".
> > > Maybe it's corner case, but I think it'd be better we handle it
> > gracefully.
> > >
> > > Thank you.
> > > Luke
> > >
> > >
> > >
> > > On Wed, Dec 15, 2021 at 11:08 PM Chris Egerton
> > 
> > > wrote:
> > >
> > > > Hi Mickael,
> > > >
> > > > Thanks for the KIP. Given how important tombstone records are it's
> > hard to
> > > > believe that the console producer doesn't already support them!
> > > >
> > > > I wanted to clarify the intended behavior and how it will play with the
> > > > parse.key and the newly-introduced (as of KIP-798 [1]) parse.headers
> > > > properties. Is the intention that the null.marker should match the
> > entire
> > > > line read by the console producer, or that it can match individual
> > portions
> > > > of a line that correspond to the record's key, value, header key, or
> > header
> > > > value? I imagine so but think it may be worth calling out (and possibly
> > > > illustrating with an example or two) in the KIP.
> > > >
> > > > [1] -
> > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-798%3A+Add+possibility+to+write+kafka+headers+in+Kafka+Console+Producer
> > > >
> > > > Cheers,
> > > >
> > > > Chris
> > > >
> > > > On Wed, Dec 15, 2021 at 6:08 AM Mickael Maison <
> > mickael.mai...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I opened a KIP to add the option to produce records with a null value
> > > > > using the Console Producer:
> > > > >
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-810%3A+Allow+producing+records+with+null+values+in+Kafka+Console+Producer
> > > > >
> > > > > Let me know if you have any feedback.
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > > >
> >
> >

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #610

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-13592) Fix flaky test ControllerIntegrationTest.testTopicIdUpgradeAfterReassigningPartitions

[David Jacot (Jira)]

David Jacot created KAFKA-13592:
---

 Summary: Fix flaky test 
ControllerIntegrationTest.testTopicIdUpgradeAfterReassigningPartitions
 Key: KAFKA-13592
 URL: https://issues.apache.org/jira/browse/KAFKA-13592
 Project: Kafka
  Issue Type: Bug
Reporter: David Jacot


{noformat}
java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:529)
at scala.None$.get(Option.scala:527)
at 
kafka.controller.ControllerIntegrationTest.testTopicIdUpgradeAfterReassigningPartitions(ControllerIntegrationTest.scala:1239){noformat}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)