Re: [DISCUSS] KIP-917: Additional custom metadata for remote log segment

[Ivan Yurchenko]

Hi all
FYI, I opened this for a vote:
https://lists.apache.org/thread/pnrkgomomwb03h8pfgs0k7dzwgpbtloz

On Tue, 13 Jun 2023 at 08:01, Ivan Yurchenko 
wrote:

> Hi Luke,
>
> > I saw you add the `RemoteLogSegmentMetadataRecord` and
> > `RemoteLogSegmentMetadataSnapshotRecord`, I'd like to confirm we're
> > not adding these records into any new API request/response, right?
>
> Not really, the KIP only extends these records, they were added before.
> And to your question: no, they remain internal and used for the serde
> purpose only. That's basically a convenience to piggyback on the existing
> code generation mechanism we already have to the API.
>
> > I think the serialization/deserialization should be the remote storage
> > plugin provider's responsibility. And in Kafka's point of view, this
> > customMetadata is just a bunch of byte array, we don't have to parse
> > it.
> > Is my understanding correct?
> > If so, maybe you can add some comments into the KIP to mention it, to
> > avoid confusion.
>
> Yes, your understanding is correct. I updated the wording a bit to
> emphasize it's the plugin's responsibility to serialize and deserialize
> these bytes.
>
> Best,
> Ivan
>
>
>
> On Tue, 13 Jun 2023 at 05:30, Luke Chen  wrote:
>
>> Hi Ivan,
>>
>> I agree this is a good improvement for remote storage plugin.
>> One question from me,
>> I saw you add the `RemoteLogSegmentMetadataRecord` and
>> `RemoteLogSegmentMetadataSnapshotRecord`, I'd like to confirm we're
>> not adding these records into any new API request/response, right?
>> I think the serialization/deserialization should be the remote storage
>> plugin provider's responsibility. And in Kafka's point of view, this
>> customMetadata is just a bunch of byte array, we don't have to parse
>> it.
>> Is my understanding correct?
>> If so, maybe you can add some comments into the KIP to mention it, to
>> avoid confusion.
>>
>> Usually the "record" being added in the KIP will be the one affecting
>> the public interface, ex: API request/response.
>> So I'd like to confirm it.
>>
>>
>> Thanks.
>> Luke
>>
>> On Thu, Jun 8, 2023 at 2:09 AM Ivan Yurchenko 
>> wrote:
>> >
>> > Hi Satish,
>> >
>> > Thank you for your feedback.
>> >
>> > I've nothing against going from Map to byte[].
>> > Serialization should not be a problem for RSM implementations: `Struct`,
>> > `Schema` and other useful serde classes are distributed as a part of the
>> > kafka-clients library.
>> >
>> > Also a good idea to add the size limiting setting, some
>> > `remote.log.metadata.custom.metadata.max.size`. A sensible default may
>> be
>> > 10 KB, which is enough to host a struct with 10 long (almost) 1K symbol
>> > ASCII strings.
>> >
>> > If a piece of custom metadata exceeds the limit, the execution of
>> > RLMTask.copyLogSegmentsToRemote should be interrupted with an error
>> message.
>> >
>> > Does this sound good?
>> > If so, I'll update the KIP accordingly. And I think it may be time for
>> the
>> > vote after that.
>> >
>> > Best,
>> > Ivan
>> >
>> >
>> >
>> > On Sat, 3 Jun 2023 at 17:13, Satish Duggana 
>> > wrote:
>> >
>> > > Hi Ivan,
>> > > Thanks for the KIP.
>> > >
>> > > The motivation of the KIP looks reasonable to me. It requires a way
>> > > for RSM providers to add custom metadata with the existing
>> > > RemoteLogSegmentMetadata. I remember we wanted to introduce a very
>> > > similar change in the earlier proposals called
>> > > RemoteLogMetadataContext. But we dropped that as we did not feel a
>> > > strong need at that time and we wanted to revisit it if needed. But I
>> > > see there is a clear need for this kind of custom metadata to keep
>> > > with RemoteLogSegmentMetadata.
>> > >
>> > > It is better to introduce a new class for this custom metadata in
>> > > RemoteLogSegmentMetadata like below for any changes in the future.
>> > > RemoteLogSegmentMetadata will have this as an optional value.
>> > >
>> > > public class RemoteLogSegmentMetadata {
>> > > ...
>> > > public static class CustomMetadata {
>> > >  private final byte[] value;
>> > > ...
>> > > }
>> > > ...
>> > > }
>> > >
>> > > This is completely opaque and it is the RSM implementation provider's
>> > > responsibility in serializing and deserializing the bytes. We can
>> > > introduce a property to guard the size with a configurable property
>> > > with a default value to avoid any unwanted large size values.
>> > >
>> > > Thanks,
>> > > Satish.
>> > >
>> > > On Tue, 30 May 2023 at 10:59, Ivan Yurchenko <
>> ivan0yurche...@gmail.com>
>> > > wrote:
>> > > >
>> > > > Hi all,
>> > > >
>> > > > I want to bring this to a conclusion (positive or negative), so if
>> there
>> > > > are no more questions in a couple of days, I'll put the KIP to the
>> vote.
>> > > >
>> > > > Best,
>> > > > Ivan
>> > > >
>> > > >
>> > > > On Fri, 5 May 2023 at 18:42, Ivan Yurchenko <
>> ivan0yurche...@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Hi Alexandre,
>> > > > >
>> > > > > > combining custom
>> > > > > > metadata 

Re: [DISCUSS] KIP-917: Additional custom metadata for remote log segment

[Ivan Yurchenko]

Hi Luke,

> I saw you add the `RemoteLogSegmentMetadataRecord` and
> `RemoteLogSegmentMetadataSnapshotRecord`, I'd like to confirm we're
> not adding these records into any new API request/response, right?

Not really, the KIP only extends these records, they were added before.
And to your question: no, they remain internal and used for the serde
purpose only. That's basically a convenience to piggyback on the existing
code generation mechanism we already have to the API.

> I think the serialization/deserialization should be the remote storage
> plugin provider's responsibility. And in Kafka's point of view, this
> customMetadata is just a bunch of byte array, we don't have to parse
> it.
> Is my understanding correct?
> If so, maybe you can add some comments into the KIP to mention it, to
> avoid confusion.

Yes, your understanding is correct. I updated the wording a bit to
emphasize it's the plugin's responsibility to serialize and deserialize
these bytes.

Best,
Ivan



On Tue, 13 Jun 2023 at 05:30, Luke Chen  wrote:

> Hi Ivan,
>
> I agree this is a good improvement for remote storage plugin.
> One question from me,
> I saw you add the `RemoteLogSegmentMetadataRecord` and
> `RemoteLogSegmentMetadataSnapshotRecord`, I'd like to confirm we're
> not adding these records into any new API request/response, right?
> I think the serialization/deserialization should be the remote storage
> plugin provider's responsibility. And in Kafka's point of view, this
> customMetadata is just a bunch of byte array, we don't have to parse
> it.
> Is my understanding correct?
> If so, maybe you can add some comments into the KIP to mention it, to
> avoid confusion.
>
> Usually the "record" being added in the KIP will be the one affecting
> the public interface, ex: API request/response.
> So I'd like to confirm it.
>
>
> Thanks.
> Luke
>
> On Thu, Jun 8, 2023 at 2:09 AM Ivan Yurchenko 
> wrote:
> >
> > Hi Satish,
> >
> > Thank you for your feedback.
> >
> > I've nothing against going from Map to byte[].
> > Serialization should not be a problem for RSM implementations: `Struct`,
> > `Schema` and other useful serde classes are distributed as a part of the
> > kafka-clients library.
> >
> > Also a good idea to add the size limiting setting, some
> > `remote.log.metadata.custom.metadata.max.size`. A sensible default may be
> > 10 KB, which is enough to host a struct with 10 long (almost) 1K symbol
> > ASCII strings.
> >
> > If a piece of custom metadata exceeds the limit, the execution of
> > RLMTask.copyLogSegmentsToRemote should be interrupted with an error
> message.
> >
> > Does this sound good?
> > If so, I'll update the KIP accordingly. And I think it may be time for
> the
> > vote after that.
> >
> > Best,
> > Ivan
> >
> >
> >
> > On Sat, 3 Jun 2023 at 17:13, Satish Duggana 
> > wrote:
> >
> > > Hi Ivan,
> > > Thanks for the KIP.
> > >
> > > The motivation of the KIP looks reasonable to me. It requires a way
> > > for RSM providers to add custom metadata with the existing
> > > RemoteLogSegmentMetadata. I remember we wanted to introduce a very
> > > similar change in the earlier proposals called
> > > RemoteLogMetadataContext. But we dropped that as we did not feel a
> > > strong need at that time and we wanted to revisit it if needed. But I
> > > see there is a clear need for this kind of custom metadata to keep
> > > with RemoteLogSegmentMetadata.
> > >
> > > It is better to introduce a new class for this custom metadata in
> > > RemoteLogSegmentMetadata like below for any changes in the future.
> > > RemoteLogSegmentMetadata will have this as an optional value.
> > >
> > > public class RemoteLogSegmentMetadata {
> > > ...
> > > public static class CustomMetadata {
> > >  private final byte[] value;
> > > ...
> > > }
> > > ...
> > > }
> > >
> > > This is completely opaque and it is the RSM implementation provider's
> > > responsibility in serializing and deserializing the bytes. We can
> > > introduce a property to guard the size with a configurable property
> > > with a default value to avoid any unwanted large size values.
> > >
> > > Thanks,
> > > Satish.
> > >
> > > On Tue, 30 May 2023 at 10:59, Ivan Yurchenko  >
> > > wrote:
> > > >
> > > > Hi all,
> > > >
> > > > I want to bring this to a conclusion (positive or negative), so if
> there
> > > > are no more questions in a couple of days, I'll put the KIP to the
> vote.
> > > >
> > > > Best,
> > > > Ivan
> > > >
> > > >
> > > > On Fri, 5 May 2023 at 18:42, Ivan Yurchenko <
> ivan0yurche...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Alexandre,
> > > > >
> > > > > > combining custom
> > > > > > metadata with rlmMetadata increases coupling between Kafka and
> the
> > > > > > plugin.
> > > > >
> > > > > This is true. However, (if I understand your concern correctly,)
> > > > > rlmMetadata in the current form may be independent from RSM
> plugins,
> > > but
> > > > > data they point to are accessible only via the particular plugin
> (the
> > > one
> > > 

[jira] [Created] (KAFKA-15082) The log retention policy doesn't take effect after altering log dir

[Drawxy (Jira)]

Drawxy created KAFKA-15082:
--

 Summary: The log retention policy doesn't take effect after 
altering log dir
 Key: KAFKA-15082
 URL: https://issues.apache.org/jira/browse/KAFKA-15082
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.3.1
Reporter: Drawxy


There are two scenarios where the log retention policy doesn't take effect:
 # During altering log dir, if a LeaderAndISR request of partition being moved 
is sent to the broker, the broker will never resume the cleaning of log segment.
 # Cancel altering log dir.

In those scenarios, the stale log segment files will never be deleted, which 
can cause the disk full issue. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-917: Additional custom metadata for remote log segment

[Luke Chen]

Hi Ivan,

I agree this is a good improvement for remote storage plugin.
One question from me,
I saw you add the `RemoteLogSegmentMetadataRecord` and
`RemoteLogSegmentMetadataSnapshotRecord`, I'd like to confirm we're
not adding these records into any new API request/response, right?
I think the serialization/deserialization should be the remote storage
plugin provider's responsibility. And in Kafka's point of view, this
customMetadata is just a bunch of byte array, we don't have to parse
it.
Is my understanding correct?
If so, maybe you can add some comments into the KIP to mention it, to
avoid confusion.

Usually the "record" being added in the KIP will be the one affecting
the public interface, ex: API request/response.
So I'd like to confirm it.


Thanks.
Luke

On Thu, Jun 8, 2023 at 2:09 AM Ivan Yurchenko  wrote:
>
> Hi Satish,
>
> Thank you for your feedback.
>
> I've nothing against going from Map to byte[].
> Serialization should not be a problem for RSM implementations: `Struct`,
> `Schema` and other useful serde classes are distributed as a part of the
> kafka-clients library.
>
> Also a good idea to add the size limiting setting, some
> `remote.log.metadata.custom.metadata.max.size`. A sensible default may be
> 10 KB, which is enough to host a struct with 10 long (almost) 1K symbol
> ASCII strings.
>
> If a piece of custom metadata exceeds the limit, the execution of
> RLMTask.copyLogSegmentsToRemote should be interrupted with an error message.
>
> Does this sound good?
> If so, I'll update the KIP accordingly. And I think it may be time for the
> vote after that.
>
> Best,
> Ivan
>
>
>
> On Sat, 3 Jun 2023 at 17:13, Satish Duggana 
> wrote:
>
> > Hi Ivan,
> > Thanks for the KIP.
> >
> > The motivation of the KIP looks reasonable to me. It requires a way
> > for RSM providers to add custom metadata with the existing
> > RemoteLogSegmentMetadata. I remember we wanted to introduce a very
> > similar change in the earlier proposals called
> > RemoteLogMetadataContext. But we dropped that as we did not feel a
> > strong need at that time and we wanted to revisit it if needed. But I
> > see there is a clear need for this kind of custom metadata to keep
> > with RemoteLogSegmentMetadata.
> >
> > It is better to introduce a new class for this custom metadata in
> > RemoteLogSegmentMetadata like below for any changes in the future.
> > RemoteLogSegmentMetadata will have this as an optional value.
> >
> > public class RemoteLogSegmentMetadata {
> > ...
> > public static class CustomMetadata {
> >  private final byte[] value;
> > ...
> > }
> > ...
> > }
> >
> > This is completely opaque and it is the RSM implementation provider's
> > responsibility in serializing and deserializing the bytes. We can
> > introduce a property to guard the size with a configurable property
> > with a default value to avoid any unwanted large size values.
> >
> > Thanks,
> > Satish.
> >
> > On Tue, 30 May 2023 at 10:59, Ivan Yurchenko 
> > wrote:
> > >
> > > Hi all,
> > >
> > > I want to bring this to a conclusion (positive or negative), so if there
> > > are no more questions in a couple of days, I'll put the KIP to the vote.
> > >
> > > Best,
> > > Ivan
> > >
> > >
> > > On Fri, 5 May 2023 at 18:42, Ivan Yurchenko 
> > > wrote:
> > >
> > > > Hi Alexandre,
> > > >
> > > > > combining custom
> > > > > metadata with rlmMetadata increases coupling between Kafka and the
> > > > > plugin.
> > > >
> > > > This is true. However, (if I understand your concern correctly,)
> > > > rlmMetadata in the current form may be independent from RSM plugins,
> > but
> > > > data they point to are accessible only via the particular plugin (the
> > one
> > > > that wrote the data or a compatible one). It seems, this coupling
> > already
> > > > exists, but it is implicit. To make my point more concrete, imagine an
> > S3
> > > > RSM which maps RemoteLogSegmentMetadata objects to S3 object keys. This
> > > > mapping logic is a part of the RSM plugin and without it the metadata
> > is
> > > > useless. I think it will not get worse if (to follow the example) the
> > > > plugin makes the said S3 object keys explicit by adding them to the
> > > > metadata. From the high level point of view, moving the custom
> > metadata to
> > > > a separate topic doesn't change the picture: it's still the plugin that
> > > > binds the standard and custom metadata together.
> > > >
> > > >
> > > > > For instance, the custom metadata may need to be modified
> > > > > outside of Kafka, but the rlmMetadata would still be cached on
> > brokers
> > > > > independently of any update of custom metadata. Since both types of
> > > > > metadata are authored by different systems, and are cached in
> > > > > different layers, this may become a problem, or make plugin migration
> > > > > more difficult. What do you think?
> > > >
> > > > This is indeed a problem. I think a solution to this would be to
> > clearly
> > > > state that metadata being modified outside of Kafka 

Re: [VOTE] KIP-917: Additional custom metadata for remote log segment

[Kamal Chandraprakash]

+1 (non-binding). Thanks for the KIP!

On Mon, Jun 12, 2023, 21:23 Divij Vaidya  wrote:

> I have reviewed the proposal and feel that it would be beneficial to
> implement.
>
> Vote +1 (non-binding)
>
>
> --
> Divij Vaidya
>
>
>
> On Mon, Jun 12, 2023 at 4:39 PM Ivan Yurchenko 
> wrote:
>
> > Hello,
> >
> > Some interest in KIP-917 was expressed in the discussion thread [1].
> After
> > addressing all the comments there, I'm putting it up to a vote.
> > Thanks!
> >
> > Best,
> > Ivan
> >
> > [1] https://lists.apache.org/thread/qpccqd3jy5rzvbt5ngtzo3dg9pzp722y
> >
>

Re: Contributing to Kafka

[Luke Chen]

Hi Joobi,

Thanks for the interest in Apache Kafka.
You can read the contributor guide here first:
https://kafka.apache.org/contributing
And find tickets you're interested from JIRA:
https://issues.apache.org/jira/projects/KAFKA/

Let us know if you have any questions.
Thank you.
Luke

On Tue, Jun 13, 2023 at 1:26 AM Joobi S.B  wrote:
>
> Dear all,
>
> My name is Joobi and I am a software engineer with experience in working
> with Kafka. I am reaching out to express my keen interest in contributing
> to Kafka. Currently, I'm going through the source code and the official
> documentation to better understand the project.
>
> I would greatly appreciate any guidance you can provide on how to get
> started with contributing to the project.
>
> Best Regards,
> Joobi

[jira] [Created] (KAFKA-15081) Implement new consumer offsetsForTimes

[Lianet Magrans (Jira)]

Lianet Magrans created KAFKA-15081:
--

 Summary: Implement new consumer offsetsForTimes
 Key: KAFKA-15081
 URL: https://issues.apache.org/jira/browse/KAFKA-15081
 Project: Kafka
  Issue Type: Task
  Components: clients, consumer
Reporter: Lianet Magrans
Assignee: Lianet Magrans


Implement offsetForTimes for the kafka consumer based on the new threading 
model, using the ListOffsetsRequestManager



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-925: rack aware task assignment in Kafka Streams

[Hao Li]

Hi all,

The vote passed with 3 binding votes and 1 non-binding vote. Thanks all for
voting!

Thanks,
Hao

On Mon, Jun 5, 2023 at 4:29 PM John Roesler  wrote:

> Thanks, Hao!
>
> I'm +1 (binding)
> -John
>
> On 2023/06/05 18:07:56 Walker Carlson wrote:
> > +1 (binding)
> >
> > On Mon, Jun 5, 2023 at 3:14 AM Bruno Cadonna  wrote:
> >
> > > Hi Hao,
> > >
> > > +1 (binding)
> > >
> > > Thanks!
> > > Bruno
> > >
> > > On 30.05.23 21:16, Colt McNealy wrote:
> > > > +1 (non-binding)
> > > >
> > > > Thank you Hao!
> > > >
> > > > Colt McNealy
> > > >
> > > > *Founder, LittleHorse.dev*
> > > >
> > > >
> > > > On Tue, May 30, 2023 at 9:50 AM Hao Li 
> wrote:
> > > >
> > > >> Hi all,
> > > >>
> > > >> I'd like to open the vote for KIP-925: rack aware task assignment in
> > > Kafka
> > > >> Streams. The link for the KIP is
> > > >>
> > > >>
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-925%3A+Rack+aware+task+assignment+in+Kafka+Streams
> > > >> .
> > > >>
> > > >> --
> > > >> Thanks,
> > > >> Hao
> > > >>
> > > >
> > >
> >
>


-- 
Thanks,
Hao

Re: [VOTE] 3.5.0 RC1

[David Arthur]

Hey Mickael, thanks for running the release!

I downloaded the 2.12 binary and did the following:

* Verified signatures
* Performed KRaft quickstart
* Manually ran through a ZK to KRaft migration

I notice we're missing the KRaft migration docs from
https://kafka.apache.org/35/documentation.html#kraft, but I think we can
fix this without building a new RC. I'll try to get that done this week.

+1 binding

Cheers,
David

On Mon, Jun 12, 2023 at 12:15 PM Bill Bejeck  wrote:

> Hi Mickael,
>
> I verified the following:
>
>1. All signatures and checksums
>2. Built from source
>3. Ran all the unit tests
>4. Ran the Kafka-Zookeeper, Kafka-KRaft, and the Kafka Streams
>quickstarts
>5. Spot-checked the docs
>6. Spot-checked the JavaDocs
>
> It's a +1(binding) for me.
> Thanks for running the release!
>
> Regards,
> Bill
>
> On Sun, Jun 11, 2023 at 5:38 AM Mickael Maison 
> wrote:
>
> > Hi,
> >
> > We've passed the deadline but we are still missing votes.
> > So far we have have 2 binding votes (Luke and John) and 2 non-binding
> > votes (Federico and Jakub).
> >
> > Please take time to test this release candidate and vote.
> >
> > Thanks,
> > Mickael
> >
> > On Wed, Jun 7, 2023 at 6:26 PM John Roesler  wrote:
> > >
> > > Thanks for running this release, Mickael!
> > >
> > > I've verified:
> > > * the signature
> > > * that I can compile the project
> > > * that I can run the tests. I saw one flaky test failure, but I don't
> > think it should block us. Reported as
> >
> https://issues.apache.org/jira/browse/KAFKA-13531?focusedCommentId=17730190
> > > * the Kafka, Consumer, and Streams quickstarts with ZK and KRaft
> > >
> > > I'm +1 (binding)
> > >
> > > Thanks,
> > > -John
> > >
> > > On Wed, Jun 7, 2023, at 06:16, Josep Prat wrote:
> > > > Hi MIckael,
> > > >
> > > > Apparently you did it in this PR already :) :
> > > > https://github.com/apache/kafka/pull/13749 (this PR among other
> things
> > > > removes classgraph.
> > > >
> > > > Without being a lawyer, I think I agree with you as stating we depend
> > on
> > > > something we don't would be less problematic than the other way
> around.
> > > >
> > > > Best,
> > > >
> > > > On Wed, Jun 7, 2023 at 12:14 PM Mickael Maison <
> > mickael.mai...@gmail.com>
> > > > wrote:
> > > >
> > > >> Hi Josep,
> > > >>
> > > >> Thanks for spotting this. If not already done, can you open a
> > > >> ticket/PR to fix this on trunk? It looks like the last couple of
> > > >> releases already had that issue. Since we're including a license
> for a
> > > >> dependency we don't ship, I think we can consider this non blocking.
> > > >> The other way around (shipping a dependency without its license)
> would
> > > >> be blocking.
> > > >>
> > > >> Thanks,
> > > >> Mickael
> > > >>
> > > >> On Tue, Jun 6, 2023 at 10:10 PM Jakub Scholz 
> wrote:
> > > >> >
> > > >> > +1 (non-binding) ... I used the staged binaries with Scala 2.13
> and
> > > >> staged
> > > >> > artifacts to run my tests. All seems to work fine.
> > > >> >
> > > >> > Thanks for running the release Mickael!
> > > >> >
> > > >> > Jakub
> > > >> >
> > > >> > On Mon, Jun 5, 2023 at 3:39 PM Mickael Maison <
> mimai...@apache.org>
> > > >> wrote:
> > > >> >
> > > >> > > Hello Kafka users, developers and client-developers,
> > > >> > >
> > > >> > > This is the second candidate for release of Apache Kafka 3.5.0.
> > Some
> > > >> > > of the major features include:
> > > >> > > - KIP-710: Full support for distributed mode in dedicated
> > MirrorMaker
> > > >> > > 2.0 clusters
> > > >> > > - KIP-881: Rack-aware Partition Assignment for Kafka Consumers
> > > >> > > - KIP-887: Add ConfigProvider to make use of environment
> variables
> > > >> > > - KIP-889: Versioned State Stores
> > > >> > > - KIP-894: Use incrementalAlterConfig for syncing topic
> > configurations
> > > >> > > - KIP-900: KRaft kafka-storage.sh API additions to support SCRAM
> > for
> > > >> > > Kafka Brokers
> > > >> > >
> > > >> > > Release notes for the 3.5.0 release:
> > > >> > >
> > https://home.apache.org/~mimaison/kafka-3.5.0-rc1/RELEASE_NOTES.html
> > > >> > >
> > > >> > > *** Please download, test and vote by Friday June 9, 5pm PT
> > > >> > >
> > > >> > > Kafka's KEYS file containing PGP keys we use to sign the
> release:
> > > >> > > https://kafka.apache.org/KEYS
> > > >> > >
> > > >> > > * Release artifacts to be voted upon (source and binary):
> > > >> > > https://home.apache.org/~mimaison/kafka-3.5.0-rc1/
> > > >> > >
> > > >> > > * Maven artifacts to be voted upon:
> > > >> > >
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > >> > >
> > > >> > > * Javadoc:
> > > >> > > https://home.apache.org/~mimaison/kafka-3.5.0-rc1/javadoc/
> > > >> > >
> > > >> > > * Tag to be voted upon (off 3.5 branch) is the 3.5.0 tag:
> > > >> > > https://github.com/apache/kafka/releases/tag/3.5.0-rc1
> > > >> > >
> > > >> > > * Documentation:
> > > >> > > https://kafka.apache.org/35/documentation.html
> > > >> > >

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1919

[Apache Jenkins Server]

See 

Contributing to Kafka

[Joobi S.B]

Dear all,

My name is Joobi and I am a software engineer with experience in working
with Kafka. I am reaching out to express my keen interest in contributing
to Kafka. Currently, I'm going through the source code and the official
documentation to better understand the project.

I would greatly appreciate any guidance you can provide on how to get
started with contributing to the project.

Best Regards,
Joobi

Re: [DISCUSS] KIP-937 Improve Message Timestamp Validation

[Beyene, Mehari]

Hi Luke, Divij and All,

I have updated the KIP to incorporate Luke's comment. We now have two new 
configurations for validating the message timestamp difference.  The existing 
configuration, "log.message.timestamp.difference.max.ms", will be deprecated. 
The "before" validation uses the same default value as 
"log.message.timestamp.difference.max.ms" to maintain backward compatibility. 
The "after" validation uses a one-hour default value.

Thanks,
Mehari

Re: [VOTE] 3.5.0 RC1

[Bill Bejeck]

Hi Mickael,

I verified the following:

   1. All signatures and checksums
   2. Built from source
   3. Ran all the unit tests
   4. Ran the Kafka-Zookeeper, Kafka-KRaft, and the Kafka Streams
   quickstarts
   5. Spot-checked the docs
   6. Spot-checked the JavaDocs

It's a +1(binding) for me.
Thanks for running the release!

Regards,
Bill

On Sun, Jun 11, 2023 at 5:38 AM Mickael Maison 
wrote:

> Hi,
>
> We've passed the deadline but we are still missing votes.
> So far we have have 2 binding votes (Luke and John) and 2 non-binding
> votes (Federico and Jakub).
>
> Please take time to test this release candidate and vote.
>
> Thanks,
> Mickael
>
> On Wed, Jun 7, 2023 at 6:26 PM John Roesler  wrote:
> >
> > Thanks for running this release, Mickael!
> >
> > I've verified:
> > * the signature
> > * that I can compile the project
> > * that I can run the tests. I saw one flaky test failure, but I don't
> think it should block us. Reported as
> https://issues.apache.org/jira/browse/KAFKA-13531?focusedCommentId=17730190
> > * the Kafka, Consumer, and Streams quickstarts with ZK and KRaft
> >
> > I'm +1 (binding)
> >
> > Thanks,
> > -John
> >
> > On Wed, Jun 7, 2023, at 06:16, Josep Prat wrote:
> > > Hi MIckael,
> > >
> > > Apparently you did it in this PR already :) :
> > > https://github.com/apache/kafka/pull/13749 (this PR among other things
> > > removes classgraph.
> > >
> > > Without being a lawyer, I think I agree with you as stating we depend
> on
> > > something we don't would be less problematic than the other way around.
> > >
> > > Best,
> > >
> > > On Wed, Jun 7, 2023 at 12:14 PM Mickael Maison <
> mickael.mai...@gmail.com>
> > > wrote:
> > >
> > >> Hi Josep,
> > >>
> > >> Thanks for spotting this. If not already done, can you open a
> > >> ticket/PR to fix this on trunk? It looks like the last couple of
> > >> releases already had that issue. Since we're including a license for a
> > >> dependency we don't ship, I think we can consider this non blocking.
> > >> The other way around (shipping a dependency without its license) would
> > >> be blocking.
> > >>
> > >> Thanks,
> > >> Mickael
> > >>
> > >> On Tue, Jun 6, 2023 at 10:10 PM Jakub Scholz  wrote:
> > >> >
> > >> > +1 (non-binding) ... I used the staged binaries with Scala 2.13 and
> > >> staged
> > >> > artifacts to run my tests. All seems to work fine.
> > >> >
> > >> > Thanks for running the release Mickael!
> > >> >
> > >> > Jakub
> > >> >
> > >> > On Mon, Jun 5, 2023 at 3:39 PM Mickael Maison 
> > >> wrote:
> > >> >
> > >> > > Hello Kafka users, developers and client-developers,
> > >> > >
> > >> > > This is the second candidate for release of Apache Kafka 3.5.0.
> Some
> > >> > > of the major features include:
> > >> > > - KIP-710: Full support for distributed mode in dedicated
> MirrorMaker
> > >> > > 2.0 clusters
> > >> > > - KIP-881: Rack-aware Partition Assignment for Kafka Consumers
> > >> > > - KIP-887: Add ConfigProvider to make use of environment variables
> > >> > > - KIP-889: Versioned State Stores
> > >> > > - KIP-894: Use incrementalAlterConfig for syncing topic
> configurations
> > >> > > - KIP-900: KRaft kafka-storage.sh API additions to support SCRAM
> for
> > >> > > Kafka Brokers
> > >> > >
> > >> > > Release notes for the 3.5.0 release:
> > >> > >
> https://home.apache.org/~mimaison/kafka-3.5.0-rc1/RELEASE_NOTES.html
> > >> > >
> > >> > > *** Please download, test and vote by Friday June 9, 5pm PT
> > >> > >
> > >> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > >> > > https://kafka.apache.org/KEYS
> > >> > >
> > >> > > * Release artifacts to be voted upon (source and binary):
> > >> > > https://home.apache.org/~mimaison/kafka-3.5.0-rc1/
> > >> > >
> > >> > > * Maven artifacts to be voted upon:
> > >> > >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >> > >
> > >> > > * Javadoc:
> > >> > > https://home.apache.org/~mimaison/kafka-3.5.0-rc1/javadoc/
> > >> > >
> > >> > > * Tag to be voted upon (off 3.5 branch) is the 3.5.0 tag:
> > >> > > https://github.com/apache/kafka/releases/tag/3.5.0-rc1
> > >> > >
> > >> > > * Documentation:
> > >> > > https://kafka.apache.org/35/documentation.html
> > >> > >
> > >> > > * Protocol:
> > >> > > https://kafka.apache.org/35/protocol.html
> > >> > >
> > >> > > * Successful Jenkins builds for the 3.5 branch:
> > >> > > Unit/integration tests: I'm struggling to get all tests to pass
> in the
> > >> > > same build. I'll run a few more builds to ensure each test pass at
> > >> > > least once in the CI. All tests passed locally.
> > >> > > System tests: The build is still running, I'll send an update
> once I
> > >> > > have the results.
> > >> > >
> > >> > > Thanks,
> > >> > > Mickael
> > >> > >
> > >>
> > >
> > >
> > > --
> > > [image: Aiven] 
> > >
> > > *Josep Prat*
> > > Open Source Engineering Director, *Aiven*
> > > josep.p...@aiven.io   |   +491715557497
> > > aiven.io    |   <
> 

Re: [VOTE] KIP-917: Additional custom metadata for remote log segment

[Divij Vaidya]

I have reviewed the proposal and feel that it would be beneficial to
implement.

Vote +1 (non-binding)


--
Divij Vaidya



On Mon, Jun 12, 2023 at 4:39 PM Ivan Yurchenko 
wrote:

> Hello,
>
> Some interest in KIP-917 was expressed in the discussion thread [1]. After
> addressing all the comments there, I'm putting it up to a vote.
> Thanks!
>
> Best,
> Ivan
>
> [1] https://lists.apache.org/thread/qpccqd3jy5rzvbt5ngtzo3dg9pzp722y
>

[jira] [Resolved] (KAFKA-15076) KRaft should prefer snapshots when listeners are at the start of the log

[Jira]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

José Armando García Sancio resolved KAFKA-15076.

Resolution: Fixed

> KRaft should prefer snapshots when listeners are at the start of the log
> 
>
> Key: KAFKA-15076
> URL: https://issues.apache.org/jira/browse/KAFKA-15076
> Project: Kafka
>  Issue Type: Improvement
>  Components: kraft
>Reporter: José Armando García Sancio
>Assignee: José Armando García Sancio
>Priority: Major
> Fix For: 3.6.0
>
>
> If the KRaft log has a segment/record batch  at offset 0 it will send that 
> batch in Listener.handleCommit even if there is a snapshot that contains that 
> record batch. It is more efficient to have the Listener start with a 
> handleSnapshot if a snapshot exist.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[VOTE] KIP-917: Additional custom metadata for remote log segment

[Ivan Yurchenko]

Hello,

Some interest in KIP-917 was expressed in the discussion thread [1]. After
addressing all the comments there, I'm putting it up to a vote.
Thanks!

Best,
Ivan

[1] https://lists.apache.org/thread/qpccqd3jy5rzvbt5ngtzo3dg9pzp722y

Re: [DISCUSS] KIP-935: Extend AlterConfigPolicy with existing configurations

[Jorge Esteban Quilcate Otoya]

Thanks Colin! You're right. I have added some notes about this to the KIP,
and clarify how this KIP is related to legacy and incremental alter config
APIs.

Let me know if there's any gaps on the current proposal.

Many thanks,
Jorge.

On Mon, 12 Jun 2023 at 11:04, Colin McCabe  wrote:

> See KAFKA-14195. Some deletions are not handled correctly. And this cannot
> be fixed without a kip because of backwards compatibility.
>
> Colin
>
> On Wed, Jun 7, 2023, at 17:07, Jorge Esteban Quilcate Otoya wrote:
> > Thank Colin.
> >
> > I've took a closer look on how configs are passed to the policy when
> > delete
> > configs are requested, and either null (KRaft) or empty values
> > (ZkAdminManager) are passed:
> > - ZkAdminManager passes empty values:
> >   - Config Entries definition:
> >
> https://github.com/apache/kafka/blob/513e1c641d63c5e15144f9fcdafa1b56c5e5ba09/core/src/main/scala/kafka/server/ZkAdminManager.scala#L503
> >   - and passed to policy without changes:
> >
> https://github.com/apache/kafka/blob/513e1c641d63c5e15144f9fcdafa1b56c5e5ba09/core/src/main/scala/kafka/server/ZkAdminManager.scala#L495
> > - Similar with ConfigurationControlManager (KRaft) passes null values:
> >   - Config entries added regardless of value==null:
> >
> https://github.com/apache/kafka/blob/513e1c641d63c5e15144f9fcdafa1b56c5e5ba09/metadata/src/main/java/org/apache/kafka/controller/ConfigurationControlManager.java#L281
> >   - And passed to policy:
> >
> https://github.com/apache/kafka/blob/513e1c641d63c5e15144f9fcdafa1b56c5e5ba09/metadata/src/main/java/org/apache/kafka/controller/ConfigurationControlManager.java#L295
> >
> > So, instead of passing (requested config + requested config to delete +
> > existing configs), the new metadata type is including (requested
> > configs--which already include deleted configs-- + _resulting_ configs)
> so
> > users could evaluate the whole (resulting) config map similar to
> > CreateTopicPolicy; and check only requested configs if needed (as with
> > current metadata).
> >
> > I've also added a rejected alternative to consider the scenario of
> > piggybacking on the existing map and just including the resulting config
> > instead, though this would break compatibility with existing
> > implementations.
> >
> > Thanks,
> > Jorge.
> >
> >
> > On Wed, 7 Jun 2023 at 08:38, Colin McCabe  wrote:
> >
> >> On Tue, Jun 6, 2023, at 06:57, Jorge Esteban Quilcate Otoya wrote:
> >> > Thanks Colin.
> >> >
> >> >> I would suggest renaming the "configs" parameter to
> "proposedConfigs,"
> >> in
> >> > both the new and old RequestMetadata constructors, to make things
> >> clearer.
> >> > This would be a binary and API-compatible change in Java.
> >> >
> >> > Sure, fully agree. KIP is updated with this suggestion.
> >> >
> >> >> We should also clarify that if configurations are being proposed for
> >> > deletion, they won't appear in proposedConfigs.
> >> >
> >> > Great catch. Wasn't aware of this, but I think it's valuable for the
> API
> >> to
> >> > also include the list of configurations to be deleted.
> >> > For this, I have extended the `RequestMetadata` type with a list of
> >> > proposed configs to delete:
> >> >
> >>
> >> Hi Jorge,
> >>
> >> Thanks for the reply.
> >>
> >> Rather than having a separate list of proposedConfigsToDelete, it seems
> >> like we could have an accessor function that calculates this when
> needed.
> >> After all, it's completely determined by existingConfigs and
> >> proposedConfigs. And some plugins will want the list and some won't (or
> >> will want to do a slightly different analysis)
> >>
> >> regards,
> >> Colin
> >>
> >>
> >> > ```
> >> > class RequestMetadata {
> >> >
> >> > private final ConfigResource resource;
> >> > private final Map proposedConfigs;
> >> > private final List proposedConfigsToDelete;
> >> > private final Map existingConfigs;
> >> > ```
> >> >
> >> > Looking forward to your feedback.
> >> >
> >> > Cheers,
> >> > Jorge.
> >> >
> >> > On Fri, 2 Jun 2023 at 22:42, Colin McCabe  wrote:
> >> >
> >> >> Hi Jorge,
> >> >>
> >> >> This is a good KIP which addresses a real gap we have today.
> >> >>
> >> >> I would suggest renaming the "configs" parameter to
> "proposedConfigs,"
> >> in
> >> >> both the new and old RequestMetadata constructors, to make things
> >> clearer.
> >> >> This would be a binary and API-compatible change in Java. We should
> also
> >> >> clarify that if configurations are being proposed for deletion, they
> >> won't
> >> >> appear in proposedConfigs.
> >> >>
> >> >> best,
> >> >> Colin
> >> >>
> >> >>
> >> >> On Tue, May 23, 2023, at 03:03, Christo Lolov wrote:
> >> >> > Hello!
> >> >> >
> >> >> > This proposal will address problems with configuration dependencies
> >> >> which I
> >> >> > run into very frequently, so I am fully supporting the development
> of
> >> >> this
> >> >> > feature!
> >> >> >
> >> >> > Best,
> >> >> > Christo
> >> >> >
> >> >> > On Mon, 22 May 2023 at 17:18, Jorge 

Re: Permissions to contribute to Apache Kafka.

[Josep Prat]

Hi Vaibhav,

You are all set. Thanks for your interest in Apache Kafka!

Best,

On Mon, Jun 12, 2023 at 3:21 PM Vaibhav Nazare
 wrote:

> HI Team,
>
> Following are the details for
>
> Wiki id
> Email: vaibhav.naz...@ibm.com
> Username: vnazare
>
> Jira id
> Email: vaibhav.naz...@ibm.com
> Username: vnazare
>
>
> Thanks,
> Vaibhav Nazare
>


-- 
[image: Aiven] 

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io   |   +491715557497
aiven.io    |   
     
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

Permissions to contribute to Apache Kafka.

[Vaibhav Nazare]

HI Team,

Following are the details for

Wiki id
Email: vaibhav.naz...@ibm.com
Username: vnazare

Jira id
Email: vaibhav.naz...@ibm.com
Username: vnazare


Thanks,
Vaibhav Nazare

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

Hi,
I have created a release plan for Apache Kafka version 3.6.0 on the
wiki. You can access the release plan and all related information by
following this link:
https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.6.0

The release plan outlines the key milestones and important dates for
version 3.6.0. Currently, the following dates have been set for the
release:

KIP Freeze: 26th July 23
Feature Freeze : 16th Aug 23
Code Freeze : 30th Aug 23

Please review the release plan and provide any additional information
or updates regarding KIPs targeting version 3.6.0. If you have
authored any KIPs that are missing a status or if there are incorrect
status details, please make the necessary updates and inform me so
that I can keep the plan accurate and up to date.

Thanks,
Satish.

On Mon, 17 Apr 2023 at 21:17, Luke Chen  wrote:
>
> Thanks for volunteering!
>
> +1
>
> Luke
>
> On Mon, Apr 17, 2023 at 2:03 AM Ismael Juma  wrote:
>
> > Thanks for volunteering Satish. +1.
> >
> > Ismael
> >
> > On Sun, Apr 16, 2023 at 10:08 AM Satish Duggana 
> > wrote:
> >
> > > Hi,
> > > I would like to volunteer as release manager for the next release,
> > > which will be Apache Kafka 3.6.0.
> > >
> > > If there are no objections, I will start a release plan a week after
> > > 3.5.0 release(around early May).
> > >
> > > Thanks,
> > > Satish.
> > >
> >

[VOTE] KIP-858: Handle JBOD broker disk failure in KRaft

[Igor Soarez]

Hi everyone,

We're getting closer to dropping ZooKeeper support, and JBOD
in KRaft mode is one of the outstanding big missing features.

It's been a while since there was new feedback on KIP-858 [1]
which aims to address this gap, so I'm calling for a vote.

A huge thank you to everyone who has reviewed this KIP, and
participated in the discussion thread! [2]

I'd also like to thank you in advance for taking the time to vote.

Best,

--
Igor

[1] 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft
[2] https://lists.apache.org/thread/8dqvfhzcyy87zyy12837pxx9lgsdhvft

Re: [VOTE] 3.5.0 RC1

[Satish Duggana]

Hi Mickael,
Thanks for running the release.

+1 (non-binding)

- Ran testAll/releaseTarGzAll successfully with no failures.
- Ran through quickstart of core/streams on builds generated from 3.5.0-rc1 tag.
- Ran a few internal apps targeting to topics on a 3 node cluster.

On Mon, 12 Jun 2023 at 12:31, Josep Prat  wrote:
>
> Hi MIckael,
> Just for the record, I +1'd this release. However, my vote falls onto the
> non-binding ones as well.
>
> Best,
>
> On Sun, Jun 11, 2023 at 11:38 AM Mickael Maison 
> wrote:
>
> > Hi,
> >
> > We've passed the deadline but we are still missing votes.
> > So far we have have 2 binding votes (Luke and John) and 2 non-binding
> > votes (Federico and Jakub).
> >
> > Please take time to test this release candidate and vote.
> >
> > Thanks,
> > Mickael
> >
> > On Wed, Jun 7, 2023 at 6:26 PM John Roesler  wrote:
> > >
> > > Thanks for running this release, Mickael!
> > >
> > > I've verified:
> > > * the signature
> > > * that I can compile the project
> > > * that I can run the tests. I saw one flaky test failure, but I don't
> > think it should block us. Reported as
> > https://issues.apache.org/jira/browse/KAFKA-13531?focusedCommentId=17730190
> > > * the Kafka, Consumer, and Streams quickstarts with ZK and KRaft
> > >
> > > I'm +1 (binding)
> > >
> > > Thanks,
> > > -John
> > >
> > > On Wed, Jun 7, 2023, at 06:16, Josep Prat wrote:
> > > > Hi MIckael,
> > > >
> > > > Apparently you did it in this PR already :) :
> > > > https://github.com/apache/kafka/pull/13749 (this PR among other things
> > > > removes classgraph.
> > > >
> > > > Without being a lawyer, I think I agree with you as stating we depend
> > on
> > > > something we don't would be less problematic than the other way around.
> > > >
> > > > Best,
> > > >
> > > > On Wed, Jun 7, 2023 at 12:14 PM Mickael Maison <
> > mickael.mai...@gmail.com>
> > > > wrote:
> > > >
> > > >> Hi Josep,
> > > >>
> > > >> Thanks for spotting this. If not already done, can you open a
> > > >> ticket/PR to fix this on trunk? It looks like the last couple of
> > > >> releases already had that issue. Since we're including a license for a
> > > >> dependency we don't ship, I think we can consider this non blocking.
> > > >> The other way around (shipping a dependency without its license) would
> > > >> be blocking.
> > > >>
> > > >> Thanks,
> > > >> Mickael
> > > >>
> > > >> On Tue, Jun 6, 2023 at 10:10 PM Jakub Scholz  wrote:
> > > >> >
> > > >> > +1 (non-binding) ... I used the staged binaries with Scala 2.13 and
> > > >> staged
> > > >> > artifacts to run my tests. All seems to work fine.
> > > >> >
> > > >> > Thanks for running the release Mickael!
> > > >> >
> > > >> > Jakub
> > > >> >
> > > >> > On Mon, Jun 5, 2023 at 3:39 PM Mickael Maison 
> > > >> wrote:
> > > >> >
> > > >> > > Hello Kafka users, developers and client-developers,
> > > >> > >
> > > >> > > This is the second candidate for release of Apache Kafka 3.5.0.
> > Some
> > > >> > > of the major features include:
> > > >> > > - KIP-710: Full support for distributed mode in dedicated
> > MirrorMaker
> > > >> > > 2.0 clusters
> > > >> > > - KIP-881: Rack-aware Partition Assignment for Kafka Consumers
> > > >> > > - KIP-887: Add ConfigProvider to make use of environment variables
> > > >> > > - KIP-889: Versioned State Stores
> > > >> > > - KIP-894: Use incrementalAlterConfig for syncing topic
> > configurations
> > > >> > > - KIP-900: KRaft kafka-storage.sh API additions to support SCRAM
> > for
> > > >> > > Kafka Brokers
> > > >> > >
> > > >> > > Release notes for the 3.5.0 release:
> > > >> > >
> > https://home.apache.org/~mimaison/kafka-3.5.0-rc1/RELEASE_NOTES.html
> > > >> > >
> > > >> > > *** Please download, test and vote by Friday June 9, 5pm PT
> > > >> > >
> > > >> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > >> > > https://kafka.apache.org/KEYS
> > > >> > >
> > > >> > > * Release artifacts to be voted upon (source and binary):
> > > >> > > https://home.apache.org/~mimaison/kafka-3.5.0-rc1/
> > > >> > >
> > > >> > > * Maven artifacts to be voted upon:
> > > >> > >
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > >> > >
> > > >> > > * Javadoc:
> > > >> > > https://home.apache.org/~mimaison/kafka-3.5.0-rc1/javadoc/
> > > >> > >
> > > >> > > * Tag to be voted upon (off 3.5 branch) is the 3.5.0 tag:
> > > >> > > https://github.com/apache/kafka/releases/tag/3.5.0-rc1
> > > >> > >
> > > >> > > * Documentation:
> > > >> > > https://kafka.apache.org/35/documentation.html
> > > >> > >
> > > >> > > * Protocol:
> > > >> > > https://kafka.apache.org/35/protocol.html
> > > >> > >
> > > >> > > * Successful Jenkins builds for the 3.5 branch:
> > > >> > > Unit/integration tests: I'm struggling to get all tests to pass
> > in the
> > > >> > > same build. I'll run a few more builds to ensure each test pass at
> > > >> > > least once in the CI. All tests passed locally.
> > > >> > > System tests: The build is still running, 

Re: [DISCUSS] KIP-916: MM2 distributed mode flow log context

[Dániel Urbán]

Updated the KIP with a few example log lines before/after the change.
Daniel

Dániel Urbán  ezt írta (időpont: 2023. jún. 7., Sze,
13:59):

> Hi Chris,
>
> Thank you for your comments! I updated the KIP. I still need to add the
> example before/after log lines, will do that soon, but I addressed all the
> other points.
> 1. Added more details on thread renaming under Public Interfaces, removed
> pseudo code.
> 2. Removed the stale header - originally, client.id related changes were
> part of the KIP, and I failed removing all leftovers of that version.
> 3. Threads listed under Public Interfaces with current/proposed names.
> 4. Added a comment in the connect-log4j.properties, similar to the one
> introduced in KIP-449. We don't have a dedicated MM2 log4j config, not sure
> if we should introduce it here.
> 5. Clarified testing section - I think thread names should not be tested
> (they never were), but testing will focus on the new MDC context value.
>
> Thanks,
> Daniel
>
> Chris Egerton  ezt írta (időpont: 2023. jún. 5.,
> H, 16:46):
>
>> Hi Daniel,
>>
>> Thanks for the updates! A few more thoughts:
>>
>> 1. The "Proposed Changes" section seems a bit like a pseudocode
>> implementation of the KIP. We don't really need this level of detail; most
>> of the time, we're just looking for implementation details that don't
>> directly affect the user-facing changes proposed in the "Public
>> Interfaces"
>> section but are worth mentioning because they (1) demonstrate how the
>> user-facing changes are made possible, (2) indirectly affect user-facing
>> behavior, or (3) go into more detail (like providing examples) about the
>> user-facing behavior. For this KIP, I think examples of user-facing
>> behavior (like before/after of thread names and log messages) and possibly
>> an official description of the scope of the changes (which threads are
>> going to be renamed and/or include the new MDC key, and which aren't?) are
>> all that we'd really need in this section; everything else is fairly clear
>> IMO. FWIW, the reason we want to discourage going into too much detail
>> with
>> KIPs is that it can quickly devolve into code review over mailing list,
>> which can hold KIPs up for longer than necessary when the core design
>> changes they contain are already basically accepted by everyone.
>>
>> 2. The "MM2 distributed mode client.id and log change" header seems like
>> it
>> may no longer be accurate; the contents do not mention any changes to
>> client IDs. I might be missing something though; please correct me if I
>> am.
>>
>> 3. Can you provide some before/after examples of what thread names and log
>> messages will look like? I'm wondering about the thread that the
>> distributed herder runs on, threads for connectors and tasks, and threads
>> for polling internal topics (which we currently manage with the
>> KafkaBasedLog class). It's fine if some of these are unchanged, I just
>> want
>> to better understand the scope of the proposed changes and get an idea of
>> how they may appear to users.
>>
>> 4. There's no mention of changes to the default Log4j config files that we
>> ship. Is this intentional? I feel like we need some way for users to
>> easily
>> discover this feature; if we're not going to touch our default Log4j
>> config
>> files, is there another way that we can expect users to find out about the
>> new MDC key?
>>
>> 5. RE the "Test Plan" section: can you provide a little more detail of
>> which cases we'll be covering with the proposed unit tests? Will we be
>> verifying that the MDC context is set in various places? If so, which
>> places? And the same with thread names? (There doesn't have to be a ton of
>> detail, but a little more than "unit tests" would be nice )
>>
>> Cheers,
>>
>> Chris
>>
>> On Mon, Jun 5, 2023 at 9:45 AM Dániel Urbán 
>> wrote:
>>
>> > I updated the KIP accordingly. Tried to come up with more generic terms
>> in
>> > the Connect code instead of referring to "flow" anywhere.
>> > Daniel
>> >
>> > Dániel Urbán  ezt írta (időpont: 2023. jún. 5.,
>> H,
>> > 14:49):
>> >
>> > > Hi Chris,
>> > >
>> > > Thank you for your comments!
>> > >
>> > > I agree that the toString based logging is not ideal, and I believe
>> all
>> > > occurrences are within a proper logging context, so they can be
>> ignored.
>> > > If thread names can be changed unconditionally, I agree, using a new
>> MDC
>> > > key is the ideal solution.
>> > >
>> > > Will update the KIP accordingly.
>> > >
>> > > Thanks,
>> > > Daniel
>> > >
>> > > Chris Egerton  ezt írta (időpont: 2023. jún.
>> > 2.,
>> > > P, 22:23):
>> > >
>> > >> Hi Daniel,
>> > >>
>> > >> Are there any cases of Object::toString being used by internal
>> classes
>> > for
>> > >> logging context that can't be covered by MDC contexts? For example,
>> > >> anything logged by WorkerSinkTask (or any WorkerTask subclass)
>> already
>> > has
>> > >> the MDC set for the task [1]. Since the Object::toString-prefixed
>> style
>> > of
>> > >> 

Re: [DISCUSS] KIP-935: Extend AlterConfigPolicy with existing configurations

[Colin McCabe]

See KAFKA-14195. Some deletions are not handled correctly. And this cannot be 
fixed without a kip because of backwards compatibility.

Colin

On Wed, Jun 7, 2023, at 17:07, Jorge Esteban Quilcate Otoya wrote:
> Thank Colin.
>
> I've took a closer look on how configs are passed to the policy when 
> delete
> configs are requested, and either null (KRaft) or empty values
> (ZkAdminManager) are passed:
> - ZkAdminManager passes empty values:
>   - Config Entries definition:
> https://github.com/apache/kafka/blob/513e1c641d63c5e15144f9fcdafa1b56c5e5ba09/core/src/main/scala/kafka/server/ZkAdminManager.scala#L503
>   - and passed to policy without changes:
> https://github.com/apache/kafka/blob/513e1c641d63c5e15144f9fcdafa1b56c5e5ba09/core/src/main/scala/kafka/server/ZkAdminManager.scala#L495
> - Similar with ConfigurationControlManager (KRaft) passes null values:
>   - Config entries added regardless of value==null:
> https://github.com/apache/kafka/blob/513e1c641d63c5e15144f9fcdafa1b56c5e5ba09/metadata/src/main/java/org/apache/kafka/controller/ConfigurationControlManager.java#L281
>   - And passed to policy:
> https://github.com/apache/kafka/blob/513e1c641d63c5e15144f9fcdafa1b56c5e5ba09/metadata/src/main/java/org/apache/kafka/controller/ConfigurationControlManager.java#L295
>
> So, instead of passing (requested config + requested config to delete +
> existing configs), the new metadata type is including (requested
> configs--which already include deleted configs-- + _resulting_ configs) so
> users could evaluate the whole (resulting) config map similar to
> CreateTopicPolicy; and check only requested configs if needed (as with
> current metadata).
>
> I've also added a rejected alternative to consider the scenario of
> piggybacking on the existing map and just including the resulting config
> instead, though this would break compatibility with existing
> implementations.
>
> Thanks,
> Jorge.
>
>
> On Wed, 7 Jun 2023 at 08:38, Colin McCabe  wrote:
>
>> On Tue, Jun 6, 2023, at 06:57, Jorge Esteban Quilcate Otoya wrote:
>> > Thanks Colin.
>> >
>> >> I would suggest renaming the "configs" parameter to "proposedConfigs,"
>> in
>> > both the new and old RequestMetadata constructors, to make things
>> clearer.
>> > This would be a binary and API-compatible change in Java.
>> >
>> > Sure, fully agree. KIP is updated with this suggestion.
>> >
>> >> We should also clarify that if configurations are being proposed for
>> > deletion, they won't appear in proposedConfigs.
>> >
>> > Great catch. Wasn't aware of this, but I think it's valuable for the API
>> to
>> > also include the list of configurations to be deleted.
>> > For this, I have extended the `RequestMetadata` type with a list of
>> > proposed configs to delete:
>> >
>>
>> Hi Jorge,
>>
>> Thanks for the reply.
>>
>> Rather than having a separate list of proposedConfigsToDelete, it seems
>> like we could have an accessor function that calculates this when needed.
>> After all, it's completely determined by existingConfigs and
>> proposedConfigs. And some plugins will want the list and some won't (or
>> will want to do a slightly different analysis)
>>
>> regards,
>> Colin
>>
>>
>> > ```
>> > class RequestMetadata {
>> >
>> > private final ConfigResource resource;
>> > private final Map proposedConfigs;
>> > private final List proposedConfigsToDelete;
>> > private final Map existingConfigs;
>> > ```
>> >
>> > Looking forward to your feedback.
>> >
>> > Cheers,
>> > Jorge.
>> >
>> > On Fri, 2 Jun 2023 at 22:42, Colin McCabe  wrote:
>> >
>> >> Hi Jorge,
>> >>
>> >> This is a good KIP which addresses a real gap we have today.
>> >>
>> >> I would suggest renaming the "configs" parameter to "proposedConfigs,"
>> in
>> >> both the new and old RequestMetadata constructors, to make things
>> clearer.
>> >> This would be a binary and API-compatible change in Java. We should also
>> >> clarify that if configurations are being proposed for deletion, they
>> won't
>> >> appear in proposedConfigs.
>> >>
>> >> best,
>> >> Colin
>> >>
>> >>
>> >> On Tue, May 23, 2023, at 03:03, Christo Lolov wrote:
>> >> > Hello!
>> >> >
>> >> > This proposal will address problems with configuration dependencies
>> >> which I
>> >> > run into very frequently, so I am fully supporting the development of
>> >> this
>> >> > feature!
>> >> >
>> >> > Best,
>> >> > Christo
>> >> >
>> >> > On Mon, 22 May 2023 at 17:18, Jorge Esteban Quilcate Otoya <
>> >> > quilcate.jo...@gmail.com> wrote:
>> >> >
>> >> >> Hi everyone,
>> >> >>
>> >> >> I'd like to start a discussion for KIP-935 <
>> >> >>
>> >> >>
>> >>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-935%3A+Extend+AlterConfigPolicy+with+existing+configurations
>> >> >> >
>> >> >> which proposes extend AlterConfigPolicy with existing configuration
>> to
>> >> >> enable more complex policies.
>> >> >>
>> >> >> There have been related KIPs in the past that haven't been accepted
>> and
>> >> >> seem 

[jira] [Created] (KAFKA-15080) Fetcher's lag never set when partition is idle

[David Jacot (Jira)]

David Jacot created KAFKA-15080:
---

 Summary: Fetcher's lag never set when partition is idle
 Key: KAFKA-15080
 URL: https://issues.apache.org/jira/browse/KAFKA-15080
 Project: Kafka
  Issue Type: Bug
Reporter: David Jacot
Assignee: David Jacot


The PartitionFetchState's lag field is set to None when the state is created 
and it is updated when bytes are received for a partition. For idle partitions 
(newly created or not), the lag is never updated because `validBytes > 0` is 
never true. As a side effect, the partition is considered out-of-sync and could 
be incorrectly throttled.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] 3.5.0 RC1

[Josep Prat]

Hi MIckael,
Just for the record, I +1'd this release. However, my vote falls onto the
non-binding ones as well.

Best,

On Sun, Jun 11, 2023 at 11:38 AM Mickael Maison 
wrote:

> Hi,
>
> We've passed the deadline but we are still missing votes.
> So far we have have 2 binding votes (Luke and John) and 2 non-binding
> votes (Federico and Jakub).
>
> Please take time to test this release candidate and vote.
>
> Thanks,
> Mickael
>
> On Wed, Jun 7, 2023 at 6:26 PM John Roesler  wrote:
> >
> > Thanks for running this release, Mickael!
> >
> > I've verified:
> > * the signature
> > * that I can compile the project
> > * that I can run the tests. I saw one flaky test failure, but I don't
> think it should block us. Reported as
> https://issues.apache.org/jira/browse/KAFKA-13531?focusedCommentId=17730190
> > * the Kafka, Consumer, and Streams quickstarts with ZK and KRaft
> >
> > I'm +1 (binding)
> >
> > Thanks,
> > -John
> >
> > On Wed, Jun 7, 2023, at 06:16, Josep Prat wrote:
> > > Hi MIckael,
> > >
> > > Apparently you did it in this PR already :) :
> > > https://github.com/apache/kafka/pull/13749 (this PR among other things
> > > removes classgraph.
> > >
> > > Without being a lawyer, I think I agree with you as stating we depend
> on
> > > something we don't would be less problematic than the other way around.
> > >
> > > Best,
> > >
> > > On Wed, Jun 7, 2023 at 12:14 PM Mickael Maison <
> mickael.mai...@gmail.com>
> > > wrote:
> > >
> > >> Hi Josep,
> > >>
> > >> Thanks for spotting this. If not already done, can you open a
> > >> ticket/PR to fix this on trunk? It looks like the last couple of
> > >> releases already had that issue. Since we're including a license for a
> > >> dependency we don't ship, I think we can consider this non blocking.
> > >> The other way around (shipping a dependency without its license) would
> > >> be blocking.
> > >>
> > >> Thanks,
> > >> Mickael
> > >>
> > >> On Tue, Jun 6, 2023 at 10:10 PM Jakub Scholz  wrote:
> > >> >
> > >> > +1 (non-binding) ... I used the staged binaries with Scala 2.13 and
> > >> staged
> > >> > artifacts to run my tests. All seems to work fine.
> > >> >
> > >> > Thanks for running the release Mickael!
> > >> >
> > >> > Jakub
> > >> >
> > >> > On Mon, Jun 5, 2023 at 3:39 PM Mickael Maison 
> > >> wrote:
> > >> >
> > >> > > Hello Kafka users, developers and client-developers,
> > >> > >
> > >> > > This is the second candidate for release of Apache Kafka 3.5.0.
> Some
> > >> > > of the major features include:
> > >> > > - KIP-710: Full support for distributed mode in dedicated
> MirrorMaker
> > >> > > 2.0 clusters
> > >> > > - KIP-881: Rack-aware Partition Assignment for Kafka Consumers
> > >> > > - KIP-887: Add ConfigProvider to make use of environment variables
> > >> > > - KIP-889: Versioned State Stores
> > >> > > - KIP-894: Use incrementalAlterConfig for syncing topic
> configurations
> > >> > > - KIP-900: KRaft kafka-storage.sh API additions to support SCRAM
> for
> > >> > > Kafka Brokers
> > >> > >
> > >> > > Release notes for the 3.5.0 release:
> > >> > >
> https://home.apache.org/~mimaison/kafka-3.5.0-rc1/RELEASE_NOTES.html
> > >> > >
> > >> > > *** Please download, test and vote by Friday June 9, 5pm PT
> > >> > >
> > >> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > >> > > https://kafka.apache.org/KEYS
> > >> > >
> > >> > > * Release artifacts to be voted upon (source and binary):
> > >> > > https://home.apache.org/~mimaison/kafka-3.5.0-rc1/
> > >> > >
> > >> > > * Maven artifacts to be voted upon:
> > >> > >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > >> > >
> > >> > > * Javadoc:
> > >> > > https://home.apache.org/~mimaison/kafka-3.5.0-rc1/javadoc/
> > >> > >
> > >> > > * Tag to be voted upon (off 3.5 branch) is the 3.5.0 tag:
> > >> > > https://github.com/apache/kafka/releases/tag/3.5.0-rc1
> > >> > >
> > >> > > * Documentation:
> > >> > > https://kafka.apache.org/35/documentation.html
> > >> > >
> > >> > > * Protocol:
> > >> > > https://kafka.apache.org/35/protocol.html
> > >> > >
> > >> > > * Successful Jenkins builds for the 3.5 branch:
> > >> > > Unit/integration tests: I'm struggling to get all tests to pass
> in the
> > >> > > same build. I'll run a few more builds to ensure each test pass at
> > >> > > least once in the CI. All tests passed locally.
> > >> > > System tests: The build is still running, I'll send an update
> once I
> > >> > > have the results.
> > >> > >
> > >> > > Thanks,
> > >> > > Mickael
> > >> > >
> > >>
> > >
> > >
> > > --
> > > [image: Aiven] 
> > >
> > > *Josep Prat*
> > > Open Source Engineering Director, *Aiven*
> > > josep.p...@aiven.io   |   +491715557497
> > > aiven.io    |   <
> https://www.facebook.com/aivencloud>
> > >      <
> https://twitter.com/aiven_io>
> > > *Aiven Deutschland GmbH*
> > > Alexanderufer 3-7, 10117 Berlin
> > > Geschäftsführer: Oskari