[jira] [Resolved] (KAFKA-15669) Implement telemetry naming strategy

[Matthias J. Sax (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-15669.
-
Fix Version/s: 3.7.0
   Resolution: Fixed

> Implement telemetry naming strategy
> ---
>
> Key: KAFKA-15669
> URL: https://issues.apache.org/jira/browse/KAFKA-15669
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Apoorv Mittal
>Assignee: Apoorv Mittal
>Priority: Major
> Fix For: 3.7.0
>
>
> Define classes and implement telemetry metrics naming strategy for the 
> KIP-714 as defined here: 
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability#KIP714:Clientmetricsandobservability-Metricsnamingandformat]
>  
> The naming strategy must also support delta temporality metrics with a suffix 
> in original metric name.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15668) Add Opentelemetry Proto library with shadowed classes

[Matthias J. Sax (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax resolved KAFKA-15668.
-
Fix Version/s: 3.7.0
   Resolution: Fixed

> Add Opentelemetry Proto library with shadowed classes
> -
>
> Key: KAFKA-15668
> URL: https://issues.apache.org/jira/browse/KAFKA-15668
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Apoorv Mittal
>Assignee: Apoorv Mittal
>Priority: Major
> Fix For: 3.7.0
>
>
> The KIP-714 requires addition of [Java client 
> dependency|https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability#KIP714:Clientmetricsandobservability-Javaclientdependencies]
>  of {{{}opentelemetry-proto{}}}, also brings transitive dependency of 
> {{protobuf-java.}} The dependencies should be shadowed to avoid JVM 
> versioning conflicts.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: ACCESS to Apache Pony Mail

[Arpit Goyal]

Thanks Justin and Joseph. This was helpful.

On Wed, Nov 1, 2023, 22:39 Justine Olshan 
wrote:

> If you would like to read any historical conversation you can do so from
> the archive here: https://lists.apache.org/list.html?dev@kafka.apache.org
>
> As Josep said, in order to reply, you can use your own client without
> logging in.
> Hope this helps!
>
> Justine
>
>
>
> On Wed, Nov 1, 2023 at 10:01 AM Josep Prat 
> wrote:
>
> > Hi Arpit,
> >
> > Pony Mail can be seen as the archive of the mailing list. We usually
> share
> > these links because they are always accessible.
> >
> > That being said, if you want to reply to an email that you can only find
> on
> > Pony Mail (maybe because the mail was sent before you subscribed or
> because
> > you deleted the email), there is a button with a pen icon that lets you
> > reply with your preferred mail client.
> >
> > Best,
> >
> > ———
> > Josep Prat
> >
> > Aiven Deutschland GmbH
> >
> > Alexanderufer 3-7, 10117 Berlin
> >
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >
> > m: +491715557497
> >
> > w: aiven.io
> >
> > e: josep.p...@aiven.io
> >
> > On Wed, Nov 1, 2023, 16:42 Arpit Goyal  wrote:
> >
> > > Thanks Joseph for providing detailed information.
> > > I recently started contributing in the Kafka project and i observe
> > > developers shares pony mail link for discussion around the design.How
> > could
> > > i be part of the thread and able to  share the opinion around  the
> > design.
> > >
> > > On Wed, Nov 1, 2023, 17:22 Josep Prat 
> > wrote:
> > >
> > > > Hi Arpit,
> > > >
> > > > By committer it is meant a person with write access to an ASF project
> > > (for
> > > > example Apache Kafka). Towards the end of this page you can see what
> > > needs
> > > > to be done to become a committer:
> > > > https://kafka.apache.org/contributing.html
> > > > .
> > > > Committership happens on invite basis and it's done by the merits
> > > described
> > > > in the link above.
> > > >
> > > > Best,
> > > >
> > > > ———
> > > > Josep Prat
> > > >
> > > > Aiven Deutschland GmbH
> > > >
> > > > Alexanderufer 3-7, 10117 Berlin
> > > >
> > > > Amtsgericht Charlottenburg, HRB 209739 B
> > > >
> > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > >
> > > > m: +491715557497
> > > >
> > > > w: aiven.io
> > > >
> > > > e: josep.p...@aiven.io
> > > >
> > > > On Wed, Nov 1, 2023, 04:02 Arpit Goyal 
> > wrote:
> > > >
> > > > > I am already a committer of Apache Kafka.
> > > > > On Wed, Nov 1, 2023, 05:18 Matthias J. Sax 
> wrote:
> > > > >
> > > > > > Only committers can login using their ASF account.
> > > > > >
> > > > > > -Matthias
> > > > > >
> > > > > > On 10/30/23 10:19 PM, Arpit Goyal wrote:
> > > > > > > Hi
> > > > > > > Can anyone help me provide access to Apache Pony Mail. I tried
> > > login
> > > > > > using
> > > > > > > the jira credential but it didn't work.
> > > > > > > Thanks and Regards
> > > > > > > Arpit Goyal
> > > > > > > 8861094754
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[jira] [Resolved] (KAFKA-15731) KRaft support in ListOffsetsIntegrationTest

[Sameer Tejani (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sameer Tejani resolved KAFKA-15731.
---
Fix Version/s: 3.7.0
   Resolution: Fixed

> KRaft support in ListOffsetsIntegrationTest
> ---
>
> Key: KAFKA-15731
> URL: https://issues.apache.org/jira/browse/KAFKA-15731
> Project: Kafka
>  Issue Type: Task
>  Components: core
>Reporter: Sameer Tejani
>Priority: Minor
>  Labels: kraft, kraft-test, newbie
> Fix For: 3.7.0
>
>
> The following tests in ListOffsetsIntegrationTest in 
> core/src/test/scala/integration/kafka/admin/ListOffsetsIntegrationTest.scala 
> need to be updated to support KRaft
> 55 : def testEarliestOffset(): Unit = {
> 61 : def testLatestOffset(): Unit = {
> 67 : def testMaxTimestampOffset(): Unit = {
> Scanned 96 lines. Found 0 KRaft tests out of 3 tests



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15730) KRaft support in ProducerFailureHandlingTest

[Sameer Tejani (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sameer Tejani resolved KAFKA-15730.
---
Fix Version/s: 3.7.0
   Resolution: Fixed

> KRaft support in ProducerFailureHandlingTest
> 
>
> Key: KAFKA-15730
> URL: https://issues.apache.org/jira/browse/KAFKA-15730
> Project: Kafka
>  Issue Type: Task
>  Components: core
>Reporter: Sameer Tejani
>Priority: Minor
>  Labels: kraft, kraft-test, newbie
> Fix For: 3.7.0
>
>
> The following tests in ProducerFailureHandlingTest in 
> core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
> need to be updated to support KRaft
> 87 : def testTooLargeRecordWithAckZero(): Unit = {
> 104 : def testTooLargeRecordWithAckOne(): Unit = {
> 133 : def testPartitionTooLargeForReplicationWithAckAll(): Unit = {
> 139 : def testResponseTooLargeForReplicationWithAckAll(): Unit = {
> 147 : def testNonExistentTopic(): Unit = {
> 164 : def testWrongBrokerList(): Unit = {
> 181 : def testInvalidPartition(): Unit = {
> 195 : def testSendAfterClosed(): Unit = {
> 215 : def testCannotSendToInternalTopic(): Unit = {
> 223 : def testNotEnoughReplicas(): Unit = {
> 236 : def testNotEnoughReplicasAfterBrokerShutdown(): Unit = {
> Scanned 260 lines. Found 0 KRaft tests out of 11 tests



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15732) KRaft support in LogAppendTimeTest

[Sameer Tejani (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sameer Tejani resolved KAFKA-15732.
---
Fix Version/s: 3.7.0
   Resolution: Fixed

> KRaft support in LogAppendTimeTest
> --
>
> Key: KAFKA-15732
> URL: https://issues.apache.org/jira/browse/KAFKA-15732
> Project: Kafka
>  Issue Type: Task
>  Components: core
>Reporter: Sameer Tejani
>Priority: Minor
>  Labels: kraft, kraft-test, newbie
> Fix For: 3.7.0
>
>
> The following tests in LogAppendTimeTest in 
> core/src/test/scala/integration/kafka/api/LogAppendTimeTest.scala need to be 
> updated to support KRaft
> 50 : def testProduceConsume(): Unit = {
> Scanned 78 lines. Found 0 KRaft tests out of 1 tests



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15733) KRaft support in FetchRequestMaxBytesTest

[Sameer Tejani (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sameer Tejani resolved KAFKA-15733.
---
Fix Version/s: 3.7.0
   Resolution: Fixed

> KRaft support in FetchRequestMaxBytesTest
> -
>
> Key: KAFKA-15733
> URL: https://issues.apache.org/jira/browse/KAFKA-15733
> Project: Kafka
>  Issue Type: Task
>  Components: core
>Reporter: Sameer Tejani
>Priority: Minor
>  Labels: kraft, kraft-test, newbie
> Fix For: 3.7.0
>
>
> The following tests in FetchRequestMaxBytesTest in 
> core/src/test/scala/unit/kafka/server/FetchRequestMaxBytesTest.scala need to 
> be updated to support KRaft
> 105 : def testConsumeMultipleRecords(): Unit = {
> Scanned 134 lines. Found 0 KRaft tests out of 1 tests



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2352

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2351

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-968: Support single-key_multi-timestamp interactive queries (IQv2) for versioned state stores

[Alieh Saeedi]

Hi all,
@Matthias: I think Victoria was right. I must add the method `get(key,
fromTime, toTime)` to the interface `VersionedKeyValueStore`. Right now,
having the method only in `RocksDBVersionedStore`, made me to have an
instance of `RocksDBVersionedStore` (instead of `VersionedKeyValueStore`)
in `StoreQueryUtils.runMultiVersionedKeyQuery()` method. In future, we are
going to use the same method for In-memory/SPDB/blaBla versioned stores.
Then either this method won't work any more, or we have to add code (if
clauses) for each type of versioned stores. What do you think about that?

Bests,
Alieh

On Tue, Oct 24, 2023 at 10:01 PM Alieh Saeedi  wrote:

> Thank you, Matthias, Bruno, and Guozhang for keeping the discussion going.
>
> Here is the list of changes I made:
>
>1. I enriched the "Example" section as Bruno suggested. Do you please
>have a look at that section? I think I devised an interesting one ;-)
>2. As Matthias and Guozhang suggested, I renamed variables and methods
>as follows:
>   - "fromTimestamp" -> "fromTime"
>   - "asOfTimestamp" -> "toTime"
>   - "from(Instant fromTime)" -> "fromTime(Instant fromTime)"
>   - "asOf(Instant toTime)" -> "toTime(Instant toTime)"
>3.
>
>About throwing an NPE when time bounds are `null`: Ok, Matthias, I am
>convinced by your idea. I consider a null timestamp as "no bound".  But I
>had to change KIP-960 and the corresponding PR to be consistent as well.
>
> Answering questions and some more discussion points:
>
>1.
>
>For now, I keep the class names as they are.
>2.
>
>About the new field "validTo" in VersionedRecord. Yes Matthias I keep
>the old constructor, which does not have `validTo` as an input parameter.
>But in the body of the old constructor, I consider the default value MAX
>for the validTo field. I think MAX must be the default value for `validTo`
>since before inserting a tombstone or a new value for the same key, the
>value must be valid till iternity.
>3.
>
>Regarding changing the ValueIterator interface from `public interface
>ValueIterator extends Iterator` to `public interface
>ValueIterator extends Iterator>`: Matthias, I do not
>know how it improves type safety because the MultiVersionedKeyQuery class
>returns a ValueIterator of VersionedRecord any way. But if we want to be
>consistent with KeyValueIterator, we must apply the changes you suggested.
>4.
>
>Regarding adding the new `get(key, fromTime, toTime)` method to the
>public interface `VersionedKeyValueStore` or only adding it to the
>class `RocksDBVersionedStore`: In the KIP, I changed the interface as
>Victoria suggested. But still, I am not convinced why we do that. 
> @Victoria:
>Do you please clarify why we have to define it in the interface? More
>specifically, why do we need to use generic VersionedKeyValueStore
>instead of simply using the implementing classes?
>
> Cheers,
> Alieh
>
> On Sat, Oct 14, 2023 at 3:35 AM Guozhang Wang 
> wrote:
>
>> Thanks Alieh for the KIP, as well as a nice summary of all the
>> discussions! Just my 2c regarding your open questions:
>>
>> 1. I just checked KIP-889
>> (
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-889%3A+Versioned+State+Stores
>> )
>> and we used "VersionedRecord get(K key, long asOfTimestamp);", so I
>> feel to be consistent with this API is better compared with being
>> consistent with "WindowKeyQuery"?
>>
>> 3. I agree with Matthias that naming is always tricky, and I also tend
>> to be consistent with what we already have (even if in retro it may
>> not be the best idea :P and if that was really becoming a complaint,
>> we would change it across the board in one shot as well later).
>>
>> Guozhang
>>
>> On Wed, Oct 11, 2023 at 9:12 PM Matthias J. Sax  wrote:
>> >
>> > Thanks for the update!
>> >
>> >
>> >
>> > Some thoughts about changes you made and open questions you raised:
>> >
>> >
>> > 10) About asOf vs until: I was just looking into `WindowKeyQuery`,
>> > `WindowRangeQuery` and also `ReadOnlyWindowStore` interfaces. For those,
>> > we use "timeFrom" and "timeTo", so it seems best to actually use
>> > `to(Instant toTime)` to keep the naming consistent across the board?
>> >
>> > If yes, we should also do `from (Instant fromTime)` and use getters
>> > `fromTime()` and `toTime()` -- given that it's range bounds it seems
>> > acceptable to me, to diverge a little bit from KIP-960 `asOfTimestamp()`
>> > -- but we could also rename it to `asOfTime()`? -- Given that we
>> > strongly type with `Instant` I am not worried about semantic ambiguity.
>> >
>> >
>> >
>> > 20) About throwing a NPE when time bounds are `null` -- why? (For the
>> > key it makes sense as is mandatory to have a key.) Could we not
>> > interpret `null` as "no bound". We did KIP-941 to add `null` for
>> > open-ended `RangeQueries`, so I am wondering if we should just stick to
>> > the same semantics?
>> >

Re: ACCESS to Apache Pony Mail

[Justine Olshan]

If you would like to read any historical conversation you can do so from
the archive here: https://lists.apache.org/list.html?dev@kafka.apache.org

As Josep said, in order to reply, you can use your own client without
logging in.
Hope this helps!

Justine



On Wed, Nov 1, 2023 at 10:01 AM Josep Prat 
wrote:

> Hi Arpit,
>
> Pony Mail can be seen as the archive of the mailing list. We usually share
> these links because they are always accessible.
>
> That being said, if you want to reply to an email that you can only find on
> Pony Mail (maybe because the mail was sent before you subscribed or because
> you deleted the email), there is a button with a pen icon that lets you
> reply with your preferred mail client.
>
> Best,
>
> ———
> Josep Prat
>
> Aiven Deutschland GmbH
>
> Alexanderufer 3-7, 10117 Berlin
>
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>
> m: +491715557497
>
> w: aiven.io
>
> e: josep.p...@aiven.io
>
> On Wed, Nov 1, 2023, 16:42 Arpit Goyal  wrote:
>
> > Thanks Joseph for providing detailed information.
> > I recently started contributing in the Kafka project and i observe
> > developers shares pony mail link for discussion around the design.How
> could
> > i be part of the thread and able to  share the opinion around  the
> design.
> >
> > On Wed, Nov 1, 2023, 17:22 Josep Prat 
> wrote:
> >
> > > Hi Arpit,
> > >
> > > By committer it is meant a person with write access to an ASF project
> > (for
> > > example Apache Kafka). Towards the end of this page you can see what
> > needs
> > > to be done to become a committer:
> > > https://kafka.apache.org/contributing.html
> > > .
> > > Committership happens on invite basis and it's done by the merits
> > described
> > > in the link above.
> > >
> > > Best,
> > >
> > > ———
> > > Josep Prat
> > >
> > > Aiven Deutschland GmbH
> > >
> > > Alexanderufer 3-7, 10117 Berlin
> > >
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > >
> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > >
> > > m: +491715557497
> > >
> > > w: aiven.io
> > >
> > > e: josep.p...@aiven.io
> > >
> > > On Wed, Nov 1, 2023, 04:02 Arpit Goyal 
> wrote:
> > >
> > > > I am already a committer of Apache Kafka.
> > > > On Wed, Nov 1, 2023, 05:18 Matthias J. Sax  wrote:
> > > >
> > > > > Only committers can login using their ASF account.
> > > > >
> > > > > -Matthias
> > > > >
> > > > > On 10/30/23 10:19 PM, Arpit Goyal wrote:
> > > > > > Hi
> > > > > > Can anyone help me provide access to Apache Pony Mail. I tried
> > login
> > > > > using
> > > > > > the jira credential but it didn't work.
> > > > > > Thanks and Regards
> > > > > > Arpit Goyal
> > > > > > 8861094754
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] KIP-992 Proposal to introduce IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery

[Hanyu (Peter) Zheng]

Hello everyone,

I would like to start a vote for KIP-992: Proposal to introduce IQv2 Query
Types: TimestampedKeyQuery and TimestampedRangeQuery.

Sincerely,
Hanyu

On Wed, Nov 1, 2023 at 10:00 AM Hanyu (Peter) Zheng 
wrote:

>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-992%3A+Proposal+to+introduce+IQv2+Query+Types%3A+TimestampedKeyQuery+and+TimestampedRangeQuery
>
> --
>
> [image: Confluent] 
> Hanyu (Peter) Zheng he/him/his
> Software Engineer Intern
> +1 (213) 431-7193 <+1+(213)+431-7193>
> Follow us: [image: Blog]
> [image:
> Twitter] [image: LinkedIn]
> [image: Slack]
> [image: YouTube]
> 
>
> [image: Try Confluent Cloud for Free]
> 
>


-- 

[image: Confluent] 
Hanyu (Peter) Zheng he/him/his
Software Engineer Intern
+1 (213) 431-7193 <+1+(213)+431-7193>
Follow us: [image: Blog]
[image:
Twitter] [image: LinkedIn]
[image: Slack]
[image: YouTube]


[image: Try Confluent Cloud for Free]

[VOTE] KIP-992 Proposal to introduce IQv2 Query Types: TimestampedKeyQuery and TimestampedRangeQuery

[Hanyu (Peter) Zheng]

https://cwiki.apache.org/confluence/display/KAFKA/KIP-992%3A+Proposal+to+introduce+IQv2+Query+Types%3A+TimestampedKeyQuery+and+TimestampedRangeQuery

-- 

[image: Confluent] 
Hanyu (Peter) Zheng he/him/his
Software Engineer Intern
+1 (213) 431-7193 <+1+(213)+431-7193>
Follow us: [image: Blog]
[image:
Twitter] [image: LinkedIn]
[image: Slack]
[image: YouTube]


[image: Try Confluent Cloud for Free]

Re: ACCESS to Apache Pony Mail

[Josep Prat]

Hi Arpit,

Pony Mail can be seen as the archive of the mailing list. We usually share
these links because they are always accessible.

That being said, if you want to reply to an email that you can only find on
Pony Mail (maybe because the mail was sent before you subscribed or because
you deleted the email), there is a button with a pen icon that lets you
reply with your preferred mail client.

Best,

———
Josep Prat

Aiven Deutschland GmbH

Alexanderufer 3-7, 10117 Berlin

Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen

m: +491715557497

w: aiven.io

e: josep.p...@aiven.io

On Wed, Nov 1, 2023, 16:42 Arpit Goyal  wrote:

> Thanks Joseph for providing detailed information.
> I recently started contributing in the Kafka project and i observe
> developers shares pony mail link for discussion around the design.How could
> i be part of the thread and able to  share the opinion around  the design.
>
> On Wed, Nov 1, 2023, 17:22 Josep Prat  wrote:
>
> > Hi Arpit,
> >
> > By committer it is meant a person with write access to an ASF project
> (for
> > example Apache Kafka). Towards the end of this page you can see what
> needs
> > to be done to become a committer:
> > https://kafka.apache.org/contributing.html
> > .
> > Committership happens on invite basis and it's done by the merits
> described
> > in the link above.
> >
> > Best,
> >
> > ———
> > Josep Prat
> >
> > Aiven Deutschland GmbH
> >
> > Alexanderufer 3-7, 10117 Berlin
> >
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >
> > m: +491715557497
> >
> > w: aiven.io
> >
> > e: josep.p...@aiven.io
> >
> > On Wed, Nov 1, 2023, 04:02 Arpit Goyal  wrote:
> >
> > > I am already a committer of Apache Kafka.
> > > On Wed, Nov 1, 2023, 05:18 Matthias J. Sax  wrote:
> > >
> > > > Only committers can login using their ASF account.
> > > >
> > > > -Matthias
> > > >
> > > > On 10/30/23 10:19 PM, Arpit Goyal wrote:
> > > > > Hi
> > > > > Can anyone help me provide access to Apache Pony Mail. I tried
> login
> > > > using
> > > > > the jira credential but it didn't work.
> > > > > Thanks and Regards
> > > > > Arpit Goyal
> > > > > 8861094754
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

[Qichao Chu]

Hi Divij,

Thank you for the very quick response and the nice suggestions. I have
updated the KIP with the following thoughts.

1. I checked the Java documentation and it seems the regex engine in utils

is
not 100% compatible with PCRE, though it is very close. I stated
the Java implementation as the requirement since we are most likely to
target a JVM language.
2. Agreed with the filter limitation. For now, let's keep it topic only.
With that in mind, I feel we do have cases where a user wants to list many
topics. Although regex is also possible, an array will make things faster.
This makes me add two options for the topic filter.
3. It seems not many configs are using JSON, this was the intention for me
to use a compound string. However since JSON is used widely in the project,
and given the benefits you mentioned earlier, I tried to make the config a
JSON array. The change is to make it compatible with multi-level settings.

Let me know what you think. Many thanks!

Best,
Qichao Chu
Software Engineer | Data - Kafka
[image: Uber] 


On Wed, Nov 1, 2023 at 9:43 PM Divij Vaidya  wrote:

> Thank you for making the changes Qichao.
>
> We are now entering in the territory of defining a declarative schema for
> filters. In the new input format, the type is string but we are imposing a
> schema for the string and we should clearly call out the schema. You can
> perhaps choose to adopt a schema such as below:
>
> metricLevel = High | Low (default: Low)
> metricNameRegEx = regEx (default: .*)
> nameOfDimension = string
> dimensionRegEx = regEx
> dimensionFilter = [=] (default: [])
>
> Final Value schema = "level"=$metricLevel, "name"=$metricNameRegEx,
> $dimensionFilter
>
> Further we need to answer questions such as :
> 1. which regEx format do we support (it should probably be Perl-compatible
> regular expressions (PCRE) because Java's regEx is compatible with it)
> 2. should we restrict the dimensionFilter to at max length 1 and value
> "topic" only for now. Later when we want to expand, we can expand filters
> for other dimensions as well such as partitions.
> 3. if we are coming up with our stringified-schema, why not use json? It
> would save us from building a parsing utility for the schema. (I like it in
> its current format but there is a case to be made for json as well)
> 4. what happens when there are contradictory regEx rules, e.g. a topic
> defined in high as well as low. It is generally solved by defining
> precedence. In our case, we can choose that high has more precedence than
> low.
>
> What do you think?
>
> --
> Divij Vaidya
>
>
>
> On Wed, Nov 1, 2023 at 2:07 PM Qichao Chu  wrote:
>
> > Hi Divij,
> >
> > Thank you for the review and the great suggestions, again. I have updated
> > the corresponding content, can you take another look?
> > Regarding the KIP-544 style regex, I have added it to the new property
> too.
> > It's expanded to include multiple sections for better future extension.
> >
> > Best,
> > Qichao Chu
> > Software Engineer | Data - Kafka
> > [image: Uber] 
> >
> >
> > On Mon, Oct 30, 2023 at 6:26 PM Divij Vaidya 
> > wrote:
> >
> > > Hey *Qichao*
> > >
> > > Thank you for the update on the KIP. I like the idea of incremental
> > > delivery and adding which metrics support this verbosity as a later
> KIP.
> > > But I also want to ensure that we wouldn't have to change the current
> > > config when adding that in future. Hence, we need some discussion on it
> > in
> > > the scope of the KIP.
> > >
> > > About the dynamic configuration:
> > > Do we need to add the "default" mode? I am asking because it may
> inhibit
> > us
> > > from adding the allowList option in future. Instead if we could
> rephrase
> > > the config as: "metric.verbosity.high" which takes values as a regEx
> > > (default will be empty), then we wouldn't have to worry about
> > > future-proofness of this KIP. Notably this is an existing pattern used
> by
> > > KIP-544.
> > > Alternatively, if you choose to stick to the current configuration
> > pattern,
> > > please provide information on how this config will look like when we
> add
> > > allow listing in future.
> > >
> > > About the perf test:
> > > Motivation - The motivation of perf test is to provide users with a
> hint
> > on
> > > what perf penalty they can expect and whether default has degraded perf
> > > (due to additional "empty" labels).
> > > Dimensions of the test could be - scrape interval, utilization of
> broker
> > > (no traffic vs. heavy traffic), number of partitions (small/200 to
> > > large/2k).
> > > Things to collect during perf test - number of mbeans registered with
> > JMX,
> > > CPU, heap utilization
> > > Expected results - As long as we can prove that there is no additional
> > > usage (significant) of CPU or heap after this change for the "default
> > > mode", we should be good. For the "high" mode, 

Re: ACCESS to Apache Pony Mail

[Arpit Goyal]

Thanks Joseph for providing detailed information.
I recently started contributing in the Kafka project and i observe
developers shares pony mail link for discussion around the design.How could
i be part of the thread and able to  share the opinion around  the design.

On Wed, Nov 1, 2023, 17:22 Josep Prat  wrote:

> Hi Arpit,
>
> By committer it is meant a person with write access to an ASF project (for
> example Apache Kafka). Towards the end of this page you can see what needs
> to be done to become a committer:
> https://kafka.apache.org/contributing.html
> .
> Committership happens on invite basis and it's done by the merits described
> in the link above.
>
> Best,
>
> ———
> Josep Prat
>
> Aiven Deutschland GmbH
>
> Alexanderufer 3-7, 10117 Berlin
>
> Amtsgericht Charlottenburg, HRB 209739 B
>
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>
> m: +491715557497
>
> w: aiven.io
>
> e: josep.p...@aiven.io
>
> On Wed, Nov 1, 2023, 04:02 Arpit Goyal  wrote:
>
> > I am already a committer of Apache Kafka.
> > On Wed, Nov 1, 2023, 05:18 Matthias J. Sax  wrote:
> >
> > > Only committers can login using their ASF account.
> > >
> > > -Matthias
> > >
> > > On 10/30/23 10:19 PM, Arpit Goyal wrote:
> > > > Hi
> > > > Can anyone help me provide access to Apache Pony Mail. I tried login
> > > using
> > > > the jira credential but it didn't work.
> > > > Thanks and Regards
> > > > Arpit Goyal
> > > > 8861094754
> > > >
> > >
> >
>

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2350

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

[Divij Vaidya]

Thank you for making the changes Qichao.

We are now entering in the territory of defining a declarative schema for
filters. In the new input format, the type is string but we are imposing a
schema for the string and we should clearly call out the schema. You can
perhaps choose to adopt a schema such as below:

metricLevel = High | Low (default: Low)
metricNameRegEx = regEx (default: .*)
nameOfDimension = string
dimensionRegEx = regEx
dimensionFilter = [=] (default: [])

Final Value schema = "level"=$metricLevel, "name"=$metricNameRegEx,
$dimensionFilter

Further we need to answer questions such as :
1. which regEx format do we support (it should probably be Perl-compatible
regular expressions (PCRE) because Java's regEx is compatible with it)
2. should we restrict the dimensionFilter to at max length 1 and value
"topic" only for now. Later when we want to expand, we can expand filters
for other dimensions as well such as partitions.
3. if we are coming up with our stringified-schema, why not use json? It
would save us from building a parsing utility for the schema. (I like it in
its current format but there is a case to be made for json as well)
4. what happens when there are contradictory regEx rules, e.g. a topic
defined in high as well as low. It is generally solved by defining
precedence. In our case, we can choose that high has more precedence than
low.

What do you think?

--
Divij Vaidya



On Wed, Nov 1, 2023 at 2:07 PM Qichao Chu  wrote:

> Hi Divij,
>
> Thank you for the review and the great suggestions, again. I have updated
> the corresponding content, can you take another look?
> Regarding the KIP-544 style regex, I have added it to the new property too.
> It's expanded to include multiple sections for better future extension.
>
> Best,
> Qichao Chu
> Software Engineer | Data - Kafka
> [image: Uber] 
>
>
> On Mon, Oct 30, 2023 at 6:26 PM Divij Vaidya 
> wrote:
>
> > Hey *Qichao*
> >
> > Thank you for the update on the KIP. I like the idea of incremental
> > delivery and adding which metrics support this verbosity as a later KIP.
> > But I also want to ensure that we wouldn't have to change the current
> > config when adding that in future. Hence, we need some discussion on it
> in
> > the scope of the KIP.
> >
> > About the dynamic configuration:
> > Do we need to add the "default" mode? I am asking because it may inhibit
> us
> > from adding the allowList option in future. Instead if we could rephrase
> > the config as: "metric.verbosity.high" which takes values as a regEx
> > (default will be empty), then we wouldn't have to worry about
> > future-proofness of this KIP. Notably this is an existing pattern used by
> > KIP-544.
> > Alternatively, if you choose to stick to the current configuration
> pattern,
> > please provide information on how this config will look like when we add
> > allow listing in future.
> >
> > About the perf test:
> > Motivation - The motivation of perf test is to provide users with a hint
> on
> > what perf penalty they can expect and whether default has degraded perf
> > (due to additional "empty" labels).
> > Dimensions of the test could be - scrape interval, utilization of broker
> > (no traffic vs. heavy traffic), number of partitions (small/200 to
> > large/2k).
> > Things to collect during perf test - number of mbeans registered with
> JMX,
> > CPU, heap utilization
> > Expected results - As long as we can prove that there is no additional
> > usage (significant) of CPU or heap after this change for the "default
> > mode", we should be good. For the "high" mode, we should document the
> > expected increase for users but it is not a blocker to implement this
> KIP.
> >
> >
> > *Kirk*, I have tried to clarify the expectation on performance, does that
> > address your question earlier? Also, I am happy with having a Kafka level
> > dynamic config that we can use to filter our metric/dimensionality since
> we
> > have a precedence at KIP-544. Hence, my suggestion to push this filtering
> > to metric library can be ignored.
> >
> > --
> > Divij Vaidya
> >
> >
> >
> > On Sat, Oct 28, 2023 at 11:37 AM Qichao Chu 
> > wrote:
> >
> > > Hello Everyone,
> > >
> > > Can I ask for some feedback regarding KIP-977
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-977%3A+Partition-Level+Throughput+Metrics
> > > >
> > > ?
> > >
> > > Best,
> > > Qichao Chu
> > > Software Engineer | Data - Kafka
> > > [image: Uber] 
> > >
> > >
> > > On Mon, Oct 16, 2023 at 7:34 PM Qichao Chu  wrote:
> > >
> > > > Hi Divij and Kirk,
> > > >
> > > > Thank you both for providing the valuable feedback and sorry for the
> > > > delay. I have just updated the KIP to address the comments.
> > > >
> > > >1. Instead of using a topic-level control, global verbosity
> control
> > > >makes more sense if we want to extend it in the future. It would
> be
> > > very
> > > >difficult if we want to apply the topic allowlist everywhere

Re: [DISCUSS] KIP-963: Upload and delete lag metrics in Tiered Storage

[Jorge Esteban Quilcate Otoya]

Thanks, Christo!

1. Agree. Having a further look into how many latency metrics are included
on the broker side there are only a few of them (e.g. request lifecycle) —
but seems mostly delegated to clients, or plugin in this case, to measure
this.

3.2. Personally, I find the record-based lag less useful as records can't
be relied as a stable unit of measure. So, if we can keep bytes- and
segment-based lag, LGTM.
3.4.  Agree, these metrics should be on the broker side. Though if plugin
decides to take deletion as a background process, then it should have it's
own metrics. That's why I was thinking the calculation should be fairly
similar to the CopyLag: "these segments are available for deletion but
haven't been deleted yet"
3.5. For lag metrics: could we add an explanation on how each lag will be
calculated, e.g. using which values, from which components, under which
circumstances do we expect these values to increase/decrease, etc. This
will clarify 3.4. and make it easier to agree and eventually test.

4. Sorry I wasn't clear. I meant similar to `RemoteCopyBytesPerSec` and
`RemoteFetchBytesPerSec`, we could consider to include
`RemoteDeleteBytesPerSec`.

5. and 6. Thanks for the explanation! It surely benefits to have these as
part of the set of metrics.

Cheers,
Jorge.

On Mon, 30 Oct 2023 at 16:07, Christo Lolov  wrote:

> Heya Jorge,
>
> Thank you for the insightful comments!
>
> 1. I see a value in such latency metrics but in my opinion the correct
> location for such metrics is in the plugins providing the underlying
> functionality. What are your thoughts on the matter?
>
> 2. Okay, I will look for and adjust the formatting today/tomorrow!
>
> 3.1 Done.
> 3.2 Sure, I will add this to the KIP later today, the suggestion makes
> sense to me. However, my question is, would you still find value in
> emitting metrics for all three i.e. RemoteCopyLagRecords,
> RemoteCopyLagBytes and RemoteCopyLagSegments or would you only keep
> RemoteCopyLagBytes and RemoteCopyLagSegments?
> 3.3. Yes, RemoteDeleteLagRecords was supposed to be an equivalent of
> RemoteCopyLagRecords. Once I have your opinion on 3.2 I will make the
> respective changes.
> 3.4. I envision these metrics to be added to Kafka rather than the plugins.
> Today Kafka sends deletes to remote storage but does not know whether those
> segments have been deleted immediately when the request has been sent or
> have been given to a background process to carry out the actual reclamation
> of space. The purpose of this metric is to give an estimate in time which
> says "hey, we have called this many segments or bytes to be deleted".
>
> 4. I believe this goes down the same line of thinking as what you mentioned
> in 3.3 - have I misunderstood something?
>
> 5. I have on a number of occasions found I do not have a metric to quickly
> point me to what part of tiered storage functionality is experiencing an
> issue, in some scenarios a follower failing to build an auxiliary state. An
> increase in the number of BuildRemoteLogAuxState requests per second can
> uncover problems for specific topics warranting a further investigation,
> which I tend to find difficult to judge purely based on parsing log
> statements. An increase in the number of errors can quickly zone in on
> followers failing as part of tiered storage and point me to look in the
> logs specifically for that component.
>
> 6. I find it useful start my investigations with respect to tiering
> problems by checking the rough size distribution of topics in remote. From
> then on I try to correlate whether a historically high-volume topic started
> experiencing a decrease in volume due to a decrease in produce traffic to
> that topic or due to an increase in lag on local storage due to the broker
> slowing down for whatever reason. Besides correlation I would use such a
> metric to also confirm whether my rate calculations are correct i.e. if
> topic A receives X MB/s and rolls a segment every Y seconds with an upload
> rate of Z MB/s do I see that much data actually being written in remote
> storage. Do these two scenarios demonstrate the usefulness I would have
> from such a metric and do the benefits make sense to you?
>
> 7. I agree. I have changed TotalRemoteLogSizeComputationTime,
> TotalRemoteLogSizeBytes, and TotalRemoteLogMetadataCount to
> RemoteLogSizeComputationTime, RemoteLogSizeBytes and RemoteLogMetadataCount
> respectively.
>
> On Fri, 27 Oct 2023 at 15:24, Jorge Esteban Quilcate Otoya <
> quilcate.jo...@gmail.com> wrote:
>
> > Hi Christo,
> >
> > Thanks for proposing KIP, this metrics will certainly be useful to
> operate
> > Kafka Tiered Storage as it becomes production-ready.
> >
> > 1. Given that the scope of the KIPs has grown to cover more metrics, what
> > do you think about introducing latency metrics for RSM operations?
> > Copy and delete time metrics are quite obvious/simple on what they
> > represent; but fetch latency metrics would be helpful as remote fetching
> > 

Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

[Qichao Chu]

Hi Divij,

Thank you for the review and the great suggestions, again. I have updated
the corresponding content, can you take another look?
Regarding the KIP-544 style regex, I have added it to the new property too.
It's expanded to include multiple sections for better future extension.

Best,
Qichao Chu
Software Engineer | Data - Kafka
[image: Uber] 


On Mon, Oct 30, 2023 at 6:26 PM Divij Vaidya 
wrote:

> Hey *Qichao*
>
> Thank you for the update on the KIP. I like the idea of incremental
> delivery and adding which metrics support this verbosity as a later KIP.
> But I also want to ensure that we wouldn't have to change the current
> config when adding that in future. Hence, we need some discussion on it in
> the scope of the KIP.
>
> About the dynamic configuration:
> Do we need to add the "default" mode? I am asking because it may inhibit us
> from adding the allowList option in future. Instead if we could rephrase
> the config as: "metric.verbosity.high" which takes values as a regEx
> (default will be empty), then we wouldn't have to worry about
> future-proofness of this KIP. Notably this is an existing pattern used by
> KIP-544.
> Alternatively, if you choose to stick to the current configuration pattern,
> please provide information on how this config will look like when we add
> allow listing in future.
>
> About the perf test:
> Motivation - The motivation of perf test is to provide users with a hint on
> what perf penalty they can expect and whether default has degraded perf
> (due to additional "empty" labels).
> Dimensions of the test could be - scrape interval, utilization of broker
> (no traffic vs. heavy traffic), number of partitions (small/200 to
> large/2k).
> Things to collect during perf test - number of mbeans registered with JMX,
> CPU, heap utilization
> Expected results - As long as we can prove that there is no additional
> usage (significant) of CPU or heap after this change for the "default
> mode", we should be good. For the "high" mode, we should document the
> expected increase for users but it is not a blocker to implement this KIP.
>
>
> *Kirk*, I have tried to clarify the expectation on performance, does that
> address your question earlier? Also, I am happy with having a Kafka level
> dynamic config that we can use to filter our metric/dimensionality since we
> have a precedence at KIP-544. Hence, my suggestion to push this filtering
> to metric library can be ignored.
>
> --
> Divij Vaidya
>
>
>
> On Sat, Oct 28, 2023 at 11:37 AM Qichao Chu 
> wrote:
>
> > Hello Everyone,
> >
> > Can I ask for some feedback regarding KIP-977
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-977%3A+Partition-Level+Throughput+Metrics
> > >
> > ?
> >
> > Best,
> > Qichao Chu
> > Software Engineer | Data - Kafka
> > [image: Uber] 
> >
> >
> > On Mon, Oct 16, 2023 at 7:34 PM Qichao Chu  wrote:
> >
> > > Hi Divij and Kirk,
> > >
> > > Thank you both for providing the valuable feedback and sorry for the
> > > delay. I have just updated the KIP to address the comments.
> > >
> > >1. Instead of using a topic-level control, global verbosity control
> > >makes more sense if we want to extend it in the future. It would be
> > very
> > >difficult if we want to apply the topic allowlist everywhere
> > >2. Also, the topic allowlist was not dynamic which makes everything
> > >quite complex, especially for the topic lifecycle management. By
> > using the
> > >dynamic global config, debugging could be easier, and management of
> > the
> > >config is also made easier.
> > >3. More details are included in the test section.
> > >
> > > One thing that still misses is the performance numbers. I will get it
> > > ready with our internal clusters and share out soon.
> > >
> > > Many thanks for the review!
> > > Qichao
> > >
> > > On Tue, Sep 12, 2023 at 8:31 AM Kirk True  wrote:
> > >
> > >> Oh, and does metrics.partition.level.reporting.topics allow for regex?
> > >>
> > >> > On Sep 12, 2023, at 8:26 AM, Kirk True  wrote:
> > >> >
> > >> > Hi Qichao,
> > >> >
> > >> > Thanks for the KIP!
> > >> >
> > >> > Divij—questions/comments inline...
> > >> >
> > >> >> On Sep 11, 2023, at 4:32 AM, Divij Vaidya  >
> > >> wrote:
> > >> >>
> > >> >> Thank you for the proposal Qichao.
> > >> >>
> > >> >> I agree with the motivation here and understand the tradeoff here
> > >> >> between observability vs. increased metric dimensions (metric
> fan-out
> > >> >> as you say in the KIP).
> > >> >>
> > >> >> High level comments:
> > >> >>
> > >> >> 1. I would urge you to consider the extensibility of the proposal
> for
> > >> >> other types of metrics. Tomorrow, if we want to selectively add
> > >> >> "partition" dimension to another metric, would we have to modify
> the
> > >> >> code where each metric is emitted? Alternatively, could we abstract
> > >> >> out this config in a "Kafka Metrics" library. The code provides all
> > >> >> 

Re: ACCESS to Apache Pony Mail

[Josep Prat]

Hi Arpit,

By committer it is meant a person with write access to an ASF project (for
example Apache Kafka). Towards the end of this page you can see what needs
to be done to become a committer: https://kafka.apache.org/contributing.html
.
Committership happens on invite basis and it's done by the merits described
in the link above.

Best,

———
Josep Prat

Aiven Deutschland GmbH

Alexanderufer 3-7, 10117 Berlin

Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen

m: +491715557497

w: aiven.io

e: josep.p...@aiven.io

On Wed, Nov 1, 2023, 04:02 Arpit Goyal  wrote:

> I am already a committer of Apache Kafka.
> On Wed, Nov 1, 2023, 05:18 Matthias J. Sax  wrote:
>
> > Only committers can login using their ASF account.
> >
> > -Matthias
> >
> > On 10/30/23 10:19 PM, Arpit Goyal wrote:
> > > Hi
> > > Can anyone help me provide access to Apache Pony Mail. I tried login
> > using
> > > the jira credential but it didn't work.
> > > Thanks and Regards
> > > Arpit Goyal
> > > 8861094754
> > >
> >
>

Re: [DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker

[Federico Valeri]

Hi Krishna, thanks for the updates, LGTM.

On Mon, Oct 30, 2023 at 3:36 PM Krishna Agarwal
 wrote:
>
> Hi Federico,
> Thanks for the feedback.
>
>1. Yes, we will add the building, testing and scanning automation for
>this Docker Image along with the flow mentioned in KIP-975. (Updated in the
>KIP)
>2. Added the other alternatives to the "rejected alternatives" section,
>instead of the main sections. (Updated in the KIP)
>3. Regarding the release process- In the KIP-975, it was concluded that
>there shouldn't be any docker specific release process. If there is a high
>severity CVE, we should release a new version of Kafka for the affected
>branch. It would include the latest Kafka code from the branch. In my
>opinion we should keep the same release process here for consistency.
>(Updated in the KIP)
>KIP-975 Release Process:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka#KIP975:DockerImageforApacheKafka-ReleaseProcess
>Discussion thread for the same:
>https://lists.apache.org/thread/05t8ccvhp3fotfftgm7dzn8wobkl59l4
>
>
> Regards,
> Krishna
>
> On Wed, Oct 25, 2023 at 9:50 PM Federico Valeri 
> wrote:
>
> > Hi Krishna, thanks for updating the KIP and all the work you are
> > putting into that.
> >
> > The release process LGTM. In the other KIP I see that there will be
> > some automation for building, testing and scanning for CVEs. Is this
> > also true for native images?
> >
> > I see you are proposing to use Alpine as the base image. I would add
> > Distroless to the rejected alternatives with the motivation. Maybe we
> > can do the same for the GraalVM distribution of choice.
> >
> > On Fri, Oct 20, 2023 at 12:02 PM Manikumar 
> > wrote:
> > >
> > > Hi,
> > >
> > > > For the native AK docker image, we are considering '*kafka-local*' as
> > it
> > > clearly signifies that this image is intended exclusively for local
> > >
> > > I am not sure, if there is any naming pattern for graalvm based images.
> > Can
> > > we include "graalvm" to the image name like "kafka-graalvm-native".
> > > This will clearly indicate this is graalvm based image.
> > >
> > >
> > > Thanks. Regards
> > >
> > >
> > >
> > >
> > > On Wed, Oct 18, 2023 at 9:26 PM Krishna Agarwal <
> > > krishna0608agar...@gmail.com> wrote:
> > >
> > > > Hi Federico,
> > > > Thanks for the feedback and apologies for the delay.
> > > >
> > > > I've included a section in the KIP on the release process. I would
> > greatly
> > > > appreciate your insights after reviewing it.
> > > >
> > > > Regards,
> > > > Krishna
> > > >
> > > > On Fri, Sep 8, 2023 at 3:08 PM Federico Valeri 
> > > > wrote:
> > > >
> > > > > Hi Krishna, thanks for opening this discussion.
> > > > >
> > > > > I see you created two separate KIPs (974 and 975), but there are some
> > > > > common points (build system and test plan).
> > > > >
> > > > > Currently, the Docker image used for system tests is only supported
> > in
> > > > > that limited scope, so the maintenance burden is minimal. Providing
> > > > > official Kafka images would be much more complicated. Have you
> > > > > considered how the image rebuild process would work in case a high
> > > > > severity CVE comes out for a non Kafka image dependency? In that
> > case,
> > > > > there will be no Kafka release.
> > > > >
> > > > > Br
> > > > > Fede
> > > > >
> > > > > On Fri, Sep 8, 2023 at 9:17 AM Krishna Agarwal
> > > > >  wrote:
> > > > > >
> > > > > > Hi,
> > > > > > I want to submit a KIP to deliver an experimental Apache Kafka
> > docker
> > > > > image.
> > > > > > The proposed docker image can launch brokers with sub-second
> > startup
> > > > time
> > > > > > and minimal memory footprint by leveraging a GraalVM based native
> > Kafka
> > > > > > binary.
> > > > > >
> > > > > > KIP-974: Docker Image for GraalVM based Native Kafka Broker
> > > > > > <
> > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker
> > > > > >
> > > > > >
> > > > > > Regards,
> > > > > > Krishna
> > > > >
> > > >
> >

[jira] [Created] (KAFKA-15771) ProduceRequest#partitionSizes() is not an atomic operation

[Xiaobing Fang (Jira)]

Xiaobing Fang created KAFKA-15771:
-

 Summary:  ProduceRequest#partitionSizes() is not an atomic 
operation
 Key: KAFKA-15771
 URL: https://issues.apache.org/jira/browse/KAFKA-15771
 Project: Kafka
  Issue Type: Bug
  Components: producer 
Reporter: Xiaobing Fang


Encountered a concurrency issue in method ProduceRequest#partitionSizes() while 
developing with Kafka. When both Thread 1 and Thread 2 concurrently call method 
ProduceRequest#partitionSizes(), Thread 2 may receive an incomplete or empty 
result if Thread 1 is still in the process of initializing partitionSizes. This 
is an incorrect state. the code to ensure that Thread 2 obtains the final state 
rather than an intermediate one.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2349

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 437410 lines...]

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldProcessTasks() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldProcessTasks() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldPunctuateStreamTime() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldPunctuateStreamTime() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldShutdownTaskExecutor() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldShutdownTaskExecutor() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldAwaitProcessableTasksIfNoneAssignable() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldAwaitProcessableTasksIfNoneAssignable() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > 
shouldRespectPunctuationDisabledByTaskExecutionMetadata() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > 
shouldRespectPunctuationDisabledByTaskExecutionMetadata() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldSetTaskTimeoutOnTimeoutException() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldSetTaskTimeoutOnTimeoutException() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldPunctuateSystemTime() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldPunctuateSystemTime() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldUnassignTaskWhenNotProgressing() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldUnassignTaskWhenNotProgressing() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldNotFlushOnException() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > shouldNotFlushOnException() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > 
shouldRespectProcessingDisabledByTaskExecutionMetadata() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskExecutorTest > 
shouldRespectProcessingDisabledByTaskExecutionMetadata() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > shouldLockAnEmptySetOfTasks() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > shouldLockAnEmptySetOfTasks() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > shouldAssignTasksThatCanBeSystemTimePunctuated() 
STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > shouldAssignTasksThatCanBeSystemTimePunctuated() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > shouldNotUnassignNotOwnedTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > shouldNotUnassignNotOwnedTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > shouldNotSetUncaughtExceptionsTwice() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > shouldNotSetUncaughtExceptionsTwice() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > shouldReturnFromAwaitOnAdding() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > shouldReturnFromAwaitOnAdding() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > shouldShutdownTaskExecutors() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > shouldShutdownTaskExecutors() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > 
shouldNotAssignTasksForPunctuationIfPunctuationDisabled() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > 
shouldNotAssignTasksForPunctuationIfPunctuationDisabled() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > shouldReturnFromAwaitOnSignalProcessableTasks() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest > shouldReturnFromAwaitOnSignalProcessableTasks() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 88 > 
DefaultTaskManagerTest