Re: [DISCUSS] KIP-974 Docker Image for GraalVM based Native Kafka Broker

[Krishna Agarwal]

Hi Ismael,

In my pursuit of a lightweight base image, I initially considered Alpine
and Distroless

   1. The next best option I explored is the Ubuntu Docker image(
   https://hub.docker.com/_/ubuntu/tags) which is a more complete image.
   It has a size of 70MB compared to the 15MB of the Alpine image
   (post-installation of glibc and bash), resulting in a difference of 55MB.
   2. To assess performance, I executed produce/consume performance scripts
   on the Kafka native Docker image using both Alpine and Ubuntu, and the
   results indicated comparable performance between the two.

I wanted to check if there's any other image you'd like me to assess for
consideration. Your input would be greatly appreciated.

Regards,
Krishna

On Thu, Nov 23, 2023 at 2:31 AM Ismael Juma  wrote:

> Hi Krishna,
>
> I am still finding it difficult to evaluate this choice. A couple of things
> would help:
>
> 1. How much smaller is the alpine image compared to the best alternative?
> 2. Is there any performance impact of going with Alpine?
>
> Ismael
>
>
> On Wed, Nov 22, 2023, 8:42 AM Krishna Agarwal <
> krishna0608agar...@gmail.com>
> wrote:
>
> > Hi Ismael,
> > Thanks for the feedback.
> >
> > The alpine image does present a few drawbacks, such as the use of musl
> libc
> > instead of glibc, the absence of bash, and reliance on the less popular
> > package manager "apk". Considering the advantage of a smaller image size
> > and installing the missing packages(glibc and bash), I have proposed the
> > alpine image as the base image. Let me know if you have any suggestions.
> > I have added a detailed section for the same in the KIP.
> >
> > Regards,
> > Krishna
> >
> > On Wed, Nov 22, 2023 at 8:08 PM Ismael Juma  wrote:
> >
> > > Hi,
> > >
> > > One question I have is regarding the choice to use alpine - it would be
> > > good to clarify if there are downsides (the upside was explained -
> images
> > > are smaller).
> > >
> > > Ismael
> > >
> > > On Fri, Sep 8, 2023, 12:17 AM Krishna Agarwal <
> > > krishna0608agar...@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > > I want to submit a KIP to deliver an experimental Apache Kafka docker
> > > > image.
> > > > The proposed docker image can launch brokers with sub-second startup
> > time
> > > > and minimal memory footprint by leveraging a GraalVM based native
> Kafka
> > > > binary.
> > > >
> > > > KIP-974: Docker Image for GraalVM based Native Kafka Broker
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-974%3A+Docker+Image+for+GraalVM+based+Native+Kafka+Broker
> > > > >
> > > >
> > > > Regards,
> > > > Krishna
> > > >
> > >
> >
>

Re: [VOTE] KIP-963: Additional metrics in Tiered Storage

[Jorge Esteban Quilcate Otoya]

A bit late, but happy this KIP is being adopted.

Thanks, Christo!

On Thu 23. Nov 2023 at 15.15, Christo Lolov  wrote:

> Hello all,
>
> With 3 +1 binding and 1 +1 non-binding votes KIP-963 is adopted 拾!
> I will get down to implementing it.
>
> Best,
> Christo
>
> On Tue, 21 Nov 2023 at 07:22, Luke Chen  wrote:
>
> > +1 (binding) from me.
> > Thanks for the KIP.
> >
> > Luke
> >
> > On Tue, Nov 21, 2023 at 11:53 AM Satish Duggana <
> satish.dugg...@gmail.com>
> > wrote:
> >
> > > +1 (binding)
> > > Thanks for the KIP and the discussion.
> > >
> > > Discussion mail thread for the KIP:
> > > https://lists.apache.org/thread/40vsyc240hyody37mf2f0pn90shkzb45
> > >
> > >
> > >
> > > On Tue, 21 Nov 2023 at 05:21, Kamal Chandraprakash
> > >  wrote:
> > > >
> > > > +1 (non-binding). Thanks for the KIP!
> > > >
> > > > On Tue, Nov 21, 2023, 03:04 Divij Vaidya 
> > > wrote:
> > > >
> > > > > + 1 (binding)
> > > > >
> > > > > This Kip will greatly improve Tiered Storage troubleshooting. Thank
> > you
> > > > > Christo.
> > > > >
> > > > > On Mon 20. Nov 2023 at 17:21, Christo Lolov <
> christolo...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hello all!
> > > > > >
> > > > > > Now that the discussion for KIP-963 has winded down, I would like
> > to
> > > open
> > > > > > it for a vote targeting 3.7.0 as the release. You can find the
> > > current
> > > > > > version of the KIP at
> > > > > >
> > > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-963%3A+Additional+metrics+in+Tiered+Storage
> > > > > >
> > > > > > Best,
> > > > > > Christo
> > > > > >
> > > > >
> > >
> >
>

Re: Apache Kafka 3.6.1 release

[Ismael Juma]

I agree that we should not block 3.6.1 if it's not a regression.

Ismael

On Thu, Nov 23, 2023 at 8:13 AM Mickael Maison 
wrote:

> Hi Kirk,
>
> It's pretty late, I was hoping to build RC0 tomorrow morning (EU
> time). This does not seem like it's a regression or a major bug so I
> don't consider it a blocker.
> If you can get it merged and backported to 3.6 overnight, fine, but I
> don't think we need to wait for it.
>
> Thanks,
> Mickael
>
>
> On Wed, Nov 22, 2023 at 11:33 PM Kirk True  wrote:
> >
> > Hi Mickael,
> >
> > Is there still time to put in another fix? I'd like to propose we
> include https://issues.apache.org/jira/browse/KAFKA-15817.
> >
> > Thanks,
> > Kirk
> >
> > On Fri, Nov 17, 2023, at 6:15 AM, Mickael Maison wrote:
> > > Hi,
> > >
> > > Quick update on 3.6.1:
> > > We have 1 blocker issue left: KAFKA-15802. Once it's fixed I'll start
> > > building the first release candidate.
> > >
> > > Thanks,
> > > Mickael
> > >
> > > On Mon, Nov 13, 2023 at 6:01 PM Mickael Maison <
> mickael.mai...@gmail.com> wrote:
> > > >
> > > > Hi,
> > > >
> > > > Ok, I've put together a release plan:
> > > > https://cwiki.apache.org/confluence/display/KAFKA/Release+plan+3.6.1
> > > >
> > > > I'll start chasing the owners of the few open issues. If there's any
> > > > other issues you'd like to have in 3.6.1, please let me know.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > > On Mon, Nov 13, 2023 at 4:26 PM Divij Vaidya <
> divijvaidy...@gmail.com> wrote:
> > > > >
> > > > > Thanks for volunteering Mickael. Please feel free to take over
> this thread.
> > > > >
> > > > > From a Tiered Storage perspective, there is a long list of known
> bugs in
> > > > > 3.6.0 [1] but we shouldn't wait on fixing them all for 3.6.1. This
> should
> > > > > be ok since this feature is in early access. We will do a
> best-effort to
> > > > > merge some of the critical ones by next week. I will nudge the
> contributors
> > > > > where things are pending for a while.
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/KAFKA-15420
> > > > >
> > > > > --
> > > > > Divij Vaidya
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Nov 13, 2023 at 4:10 PM Mickael Maison <
> mickael.mai...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Divij,
> > > > > >
> > > > > > You beat me to it, I was about to propose doing a 3.6.1 release
> later this
> > > > > > week.
> > > > > > While there's only a dozen or so issues fixed since 3.6.0, as
> > > > > > mentioned there's a few important dependency upgrades that would
> be
> > > > > > good to release.
> > > > > >
> > > > > > I'm happy to volunteer to run the release if we agree to
> releasing
> > > > > > sooner than initially proposed.
> > > > > > There seems to only be a few unresolved Jiras targeting 3.6.1
> [0] (all
> > > > > > have PRs with some of them even already merged!).
> > > > > >
> > > > > > 0:
> > > > > >
> https://issues.apache.org/jira/browse/KAFKA-15552?jql=project%20%3D%20KAFKA%20AND%20resolution%20%3D%20Unresolved%20AND%20fixVersion%20%3D%203.6.1%20ORDER%20BY%20priority%20DESC%2C%20status%20DESC%2C%20updated%20DESC
> > > > > >
> > > > > > Thanks,
> > > > > > Mickael
> > > > > >
> > > > > > On Mon, Nov 13, 2023 at 3:57 PM Divij Vaidya <
> divijvaidy...@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi Ismael, I am all-in favour for frequent releases. Sooner is
> always
> > > > > > > better. Unfortunately, I won't have bandwidth to volunteer for
> a release
> > > > > > in
> > > > > > > December. If someone else volunteers to be RM prior to this
> timeline, I
> > > > > > > would be happy to ceed the RM role to them but in the worst
> case
> > > > > > scenario,
> > > > > > > my offer to volunteer for Jan release could be considered as a
> backup.
> > > > > > >
> > > > > > > --
> > > > > > > Divij Vaidya
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Nov 13, 2023 at 3:40 PM Ismael Juma 
> wrote:
> > > > > > >
> > > > > > > > Hi Divij,
> > > > > > > >
> > > > > > > > I think we should be releasing 3.6.1 this year rather than
> next. There
> > > > > > are
> > > > > > > > some critical bugs in 3.6.0 and I don't think we should be
> waiting that
> > > > > > > > long to fix them. What do you think?
> > > > > > > >
> > > > > > > > Ismael
> > > > > > > >
> > > > > > > > On Mon, Nov 13, 2023 at 6:32 AM Divij Vaidya <
> divijvaidy...@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hey folks,
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I'd like to volunteer to be the release manager for a bug
> fix
> > > > > > release of
> > > > > > > > > the 3.6 line. This will be the first bug fix release of
> this line and
> > > > > > > > will
> > > > > > > > > be version 3.6.1. It would contain critical bug fixes for
> features
> > > > > > such
> > > > > > > > as
> > > > > > > > > Transaction verification [1], will stabilize Tiered
> Storage early
> > > > > > access
> > > > > > > > > release [2] [3] and upgrade dependencies to fix 

Re: Apache Kafka 3.6.1 release

[Mickael Maison]

Hi Kirk,

It's pretty late, I was hoping to build RC0 tomorrow morning (EU
time). This does not seem like it's a regression or a major bug so I
don't consider it a blocker.
If you can get it merged and backported to 3.6 overnight, fine, but I
don't think we need to wait for it.

Thanks,
Mickael


On Wed, Nov 22, 2023 at 11:33 PM Kirk True  wrote:
>
> Hi Mickael,
>
> Is there still time to put in another fix? I'd like to propose we include 
> https://issues.apache.org/jira/browse/KAFKA-15817.
>
> Thanks,
> Kirk
>
> On Fri, Nov 17, 2023, at 6:15 AM, Mickael Maison wrote:
> > Hi,
> >
> > Quick update on 3.6.1:
> > We have 1 blocker issue left: KAFKA-15802. Once it's fixed I'll start
> > building the first release candidate.
> >
> > Thanks,
> > Mickael
> >
> > On Mon, Nov 13, 2023 at 6:01 PM Mickael Maison  
> > wrote:
> > >
> > > Hi,
> > >
> > > Ok, I've put together a release plan:
> > > https://cwiki.apache.org/confluence/display/KAFKA/Release+plan+3.6.1
> > >
> > > I'll start chasing the owners of the few open issues. If there's any
> > > other issues you'd like to have in 3.6.1, please let me know.
> > >
> > > Thanks,
> > > Mickael
> > >
> > > On Mon, Nov 13, 2023 at 4:26 PM Divij Vaidya  
> > > wrote:
> > > >
> > > > Thanks for volunteering Mickael. Please feel free to take over this 
> > > > thread.
> > > >
> > > > From a Tiered Storage perspective, there is a long list of known bugs in
> > > > 3.6.0 [1] but we shouldn't wait on fixing them all for 3.6.1. This 
> > > > should
> > > > be ok since this feature is in early access. We will do a best-effort to
> > > > merge some of the critical ones by next week. I will nudge the 
> > > > contributors
> > > > where things are pending for a while.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/KAFKA-15420
> > > >
> > > > --
> > > > Divij Vaidya
> > > >
> > > >
> > > >
> > > > On Mon, Nov 13, 2023 at 4:10 PM Mickael Maison 
> > > > 
> > > > wrote:
> > > >
> > > > > Hi Divij,
> > > > >
> > > > > You beat me to it, I was about to propose doing a 3.6.1 release later 
> > > > > this
> > > > > week.
> > > > > While there's only a dozen or so issues fixed since 3.6.0, as
> > > > > mentioned there's a few important dependency upgrades that would be
> > > > > good to release.
> > > > >
> > > > > I'm happy to volunteer to run the release if we agree to releasing
> > > > > sooner than initially proposed.
> > > > > There seems to only be a few unresolved Jiras targeting 3.6.1 [0] (all
> > > > > have PRs with some of them even already merged!).
> > > > >
> > > > > 0:
> > > > > https://issues.apache.org/jira/browse/KAFKA-15552?jql=project%20%3D%20KAFKA%20AND%20resolution%20%3D%20Unresolved%20AND%20fixVersion%20%3D%203.6.1%20ORDER%20BY%20priority%20DESC%2C%20status%20DESC%2C%20updated%20DESC
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > > > > On Mon, Nov 13, 2023 at 3:57 PM Divij Vaidya 
> > > > > wrote:
> > > > > >
> > > > > > Hi Ismael, I am all-in favour for frequent releases. Sooner is 
> > > > > > always
> > > > > > better. Unfortunately, I won't have bandwidth to volunteer for a 
> > > > > > release
> > > > > in
> > > > > > December. If someone else volunteers to be RM prior to this 
> > > > > > timeline, I
> > > > > > would be happy to ceed the RM role to them but in the worst case
> > > > > scenario,
> > > > > > my offer to volunteer for Jan release could be considered as a 
> > > > > > backup.
> > > > > >
> > > > > > --
> > > > > > Divij Vaidya
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Nov 13, 2023 at 3:40 PM Ismael Juma  
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Divij,
> > > > > > >
> > > > > > > I think we should be releasing 3.6.1 this year rather than next. 
> > > > > > > There
> > > > > are
> > > > > > > some critical bugs in 3.6.0 and I don't think we should be 
> > > > > > > waiting that
> > > > > > > long to fix them. What do you think?
> > > > > > >
> > > > > > > Ismael
> > > > > > >
> > > > > > > On Mon, Nov 13, 2023 at 6:32 AM Divij Vaidya 
> > > > > > > 
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hey folks,
> > > > > > > >
> > > > > > > >
> > > > > > > > I'd like to volunteer to be the release manager for a bug fix
> > > > > release of
> > > > > > > > the 3.6 line. This will be the first bug fix release of this 
> > > > > > > > line and
> > > > > > > will
> > > > > > > > be version 3.6.1. It would contain critical bug fixes for  
> > > > > > > > features
> > > > > such
> > > > > > > as
> > > > > > > > Transaction verification [1], will stabilize Tiered Storage 
> > > > > > > > early
> > > > > access
> > > > > > > > release [2] [3] and upgrade dependencies to fix CVEs such as 
> > > > > > > > Netty
> > > > > [4]
> > > > > > > and
> > > > > > > > Zookeeper [5].
> > > > > > > >
> > > > > > > > If no one has any objections, I will send out a release plan 
> > > > > > > > latest
> > > > > by
> > > > > > > 23rd
> > > > > > > > Dec 2023 with a tentative release in mid-Jan 2024. The release 
> > > > > > > > 

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2411

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 438041 lines...]

Gradle Test Run :streams:test > Gradle Test Executor 91 > 
DefaultTaskManagerTest > shouldReturnFromAwaitOnUnlocking() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 91 > 
DefaultTaskManagerTest > shouldReturnFromAwaitOnUnlocking() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 91 > 
DefaultTaskManagerTest > 
shouldNotAssignTasksForProcessingIfProcessingDisabled() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 91 > 
DefaultTaskManagerTest > 
shouldNotAssignTasksForProcessingIfProcessingDisabled() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 91 > 
DefaultTaskManagerTest > shouldNotSetUncaughtExceptionsForUnassignedTasks() 
STARTED

Gradle Test Run :streams:test > Gradle Test Executor 91 > 
DefaultTaskManagerTest > shouldNotSetUncaughtExceptionsForUnassignedTasks() 
PASSED

Gradle Test Run :streams:test > Gradle Test Executor 91 > 
DefaultTaskManagerTest > shouldNotAssignLockedTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 91 > 
DefaultTaskManagerTest > shouldNotAssignLockedTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 91 > 
DefaultTaskManagerTest > shouldUnassignLockingTask() STARTED

Gradle Test Run :streams:test > Gradle Test Executor 91 > 
DefaultTaskManagerTest > shouldUnassignLockingTask() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 91 > 
DefaultTaskManagerTest > shouldAssignTasksThatCanBeStreamTimePunctuated() 
STARTED

Gradle Test Run :streams:test > Gradle Test Executor 91 > 
DefaultTaskManagerTest > shouldAssignTasksThatCanBeStreamTimePunctuated() PASSED

Gradle Test Run :streams:test > Gradle Test Executor 91 > 
RocksDBBlockCacheMetricsTest > shouldRecordCorrectBlockCacheUsage(RocksDBStore, 
StateStoreContext) > [1] 
org.apache.kafka.streams.state.internals.RocksDBStore@7c828eea, 
org.apache.kafka.test.MockInternalProcessorContext@3f820ad7 STARTED

Gradle Test Run :streams:test > Gradle Test Executor 91 > 
RocksDBBlockCacheMetricsTest > shouldRecordCorrectBlockCacheUsage(RocksDBStore, 
StateStoreContext) > [1] 
org.apache.kafka.streams.state.internals.RocksDBStore@7c828eea, 
org.apache.kafka.test.MockInternalProcessorContext@3f820ad7 PASSED

Gradle Test Run :streams:test > Gradle Test Executor 91 > 
RocksDBBlockCacheMetricsTest > shouldRecordCorrectBlockCacheUsage(RocksDBStore, 
StateStoreContext) > [2] 
org.apache.kafka.streams.state.internals.RocksDBTimestampedStore@28767c35, 
org.apache.kafka.test.MockInternalProcessorContext@3514849e STARTED

Gradle Test Run :streams:test > Gradle Test Executor 91 > 
RocksDBBlockCacheMetricsTest > shouldRecordCorrectBlockCacheUsage(RocksDBStore, 
StateStoreContext) > [2] 
org.apache.kafka.streams.state.internals.RocksDBTimestampedStore@28767c35, 
org.apache.kafka.test.MockInternalProcessorContext@3514849e PASSED

Gradle Test Run :streams:test > Gradle Test Executor 91 > 
RocksDBBlockCacheMetricsTest > 
shouldRecordCorrectBlockCachePinnedUsage(RocksDBStore, StateStoreContext) > [1] 
org.apache.kafka.streams.state.internals.RocksDBStore@41ac4724, 
org.apache.kafka.test.MockInternalProcessorContext@3e2502fe STARTED

Gradle Test Run :streams:test > Gradle Test Executor 91 > 
RocksDBBlockCacheMetricsTest > 
shouldRecordCorrectBlockCachePinnedUsage(RocksDBStore, StateStoreContext) > [1] 
org.apache.kafka.streams.state.internals.RocksDBStore@41ac4724, 
org.apache.kafka.test.MockInternalProcessorContext@3e2502fe PASSED

Gradle Test Run :streams:test > Gradle Test Executor 91 > 
RocksDBBlockCacheMetricsTest > 
shouldRecordCorrectBlockCachePinnedUsage(RocksDBStore, StateStoreContext) > [2] 
org.apache.kafka.streams.state.internals.RocksDBTimestampedStore@6a23615e, 
org.apache.kafka.test.MockInternalProcessorContext@70ee9438 STARTED

Gradle Test Run :streams:test > Gradle Test Executor 91 > 
RocksDBBlockCacheMetricsTest > 
shouldRecordCorrectBlockCachePinnedUsage(RocksDBStore, StateStoreContext) > [2] 
org.apache.kafka.streams.state.internals.RocksDBTimestampedStore@6a23615e, 
org.apache.kafka.test.MockInternalProcessorContext@70ee9438 PASSED

Gradle Test Run :streams:test > Gradle Test Executor 91 > 
RocksDBBlockCacheMetricsTest > 
shouldRecordCorrectBlockCacheCapacity(RocksDBStore, StateStoreContext) > [1] 
org.apache.kafka.streams.state.internals.RocksDBStore@64d2ac95, 
org.apache.kafka.test.MockInternalProcessorContext@792ca388 STARTED

Gradle Test Run :streams:test > Gradle Test Executor 91 > 
RocksDBBlockCacheMetricsTest > 
shouldRecordCorrectBlockCacheCapacity(RocksDBStore, StateStoreContext) > [1] 
org.apache.kafka.streams.state.internals.RocksDBStore@64d2ac95, 
org.apache.kafka.test.MockInternalProcessorContext@792ca388 PASSED

Gradle Test Run :streams:test > Gradle Test Executor 91 > 
RocksDBBlockCacheMetricsTest > 
shouldRecordCorrectBlockCacheCapacity(RocksDBStore, 

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.6 #119

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 415268 lines...]
> Task :connect:api:compileTestJava UP-TO-DATE
> Task :connect:api:testClasses UP-TO-DATE
> Task :connect:json:generateMetadataFileForMavenJavaPublication
> Task :connect:json:testSrcJar
> Task :connect:api:testJar
> Task :connect:api:testSrcJar
> Task :connect:api:publishMavenJavaPublicationToMavenLocal
> Task :connect:api:publishToMavenLocal
> Task :connect:json:publishMavenJavaPublicationToMavenLocal
> Task :connect:json:publishToMavenLocal
> Task :clients:generateMetadataFileForMavenJavaPublication
> Task :storage:api:compileTestJava
> Task :storage:api:testClasses
> Task :server-common:compileTestJava
> Task :server-common:testClasses
> Task :raft:compileTestJava
> Task :raft:testClasses
> Task :core:compileScala
> Task :group-coordinator:compileTestJava
> Task :group-coordinator:testClasses

> Task :clients:javadoc
/home/jenkins/workspace/Kafka_kafka_3.6/clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java:81:
 warning - Tag @link:illegal character: "60" in "#define(String, Type, 
Importance, String, String, int, Width, String, List)"
/home/jenkins/workspace/Kafka_kafka_3.6/clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java:81:
 warning - Tag @link:illegal character: "62" in "#define(String, Type, 
Importance, String, String, int, Width, String, List)"
/home/jenkins/workspace/Kafka_kafka_3.6/clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java:81:
 warning - Tag @link: can't find define(String, Type, Importance, String, 
String, int, Width, String, List) in 
org.apache.kafka.common.config.ConfigDef
/home/jenkins/workspace/Kafka_kafka_3.6/clients/src/main/java/org/apache/kafka/clients/admin/ScramMechanism.java:32:
 warning - Tag @see: missing final '>': "https://cwiki.apache.org/confluence/display/KAFKA/KIP-554%3A+Add+Broker-side+SCRAM+Config+API;>KIP-554:
 Add Broker-side SCRAM Config API

 This code is duplicated in 
org.apache.kafka.common.security.scram.internals.ScramMechanism.
 The type field in both files must match and must not change. The type field
 is used both for passing ScramCredentialUpsertion and for the internal
 UserScramCredentialRecord. Do not change the type field."
/home/jenkins/workspace/Kafka_kafka_3.6/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/package-info.java:21:
 warning - Tag @link: reference not found: 
org.apache.kafka.common.security.oauthbearer

> Task :metadata:compileTestJava
> Task :metadata:testClasses

> Task :clients:javadoc
5 warnings

> Task :clients:javadocJar
> Task :clients:srcJar
> Task :clients:testJar
> Task :clients:testSrcJar
> Task :clients:publishMavenJavaPublicationToMavenLocal
> Task :clients:publishToMavenLocal
> Task :core:classes
> Task :core:compileTestJava NO-SOURCE
> Task :core:compileTestScala
> Task :core:testClasses
> Task :streams:compileTestJava
> Task :streams:testClasses
> Task :streams:testJar
> Task :streams:testSrcJar
> Task :streams:publishMavenJavaPublicationToMavenLocal
> Task :streams:publishToMavenLocal

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings 
and determine if they come from your own scripts or plugins.

For more on this, please refer to 
https://docs.gradle.org/8.2.1/userguide/command_line_interface.html#sec:command_line_warnings
 in the Gradle documentation.

BUILD SUCCESSFUL in 7m 27s
94 actionable tasks: 41 executed, 53 up-to-date

Publishing build scan...
https://ge.apache.org/s/dqpvwvxejrwma

[Pipeline] sh
+ + grep ^version= gradle.propertiescut
 -d= -f 2
[Pipeline] dir
Running in /home/jenkins/workspace/Kafka_kafka_3.6/streams/quickstart
[Pipeline] {
[Pipeline] sh
+ mvn clean install -Dgpg.skip
[INFO] Scanning for projects...
[INFO] 
[INFO] Reactor Build Order:
[INFO] 
[INFO] Kafka Streams :: Quickstart[pom]
[INFO] streams-quickstart-java[maven-archetype]
[INFO] 
[INFO] < org.apache.kafka:streams-quickstart >-
[INFO] Building Kafka Streams :: Quickstart 3.6.1-SNAPSHOT[1/2]
[INFO]   from pom.xml
[INFO] [ pom ]-
[INFO] 
[INFO] --- clean:3.0.0:clean (default-clean) @ streams-quickstart ---
[INFO] 
[INFO] --- remote-resources:1.5:process (process-resource-bundles) @ 
streams-quickstart ---
[INFO] 
[INFO] --- site:3.5.1:attach-descriptor (attach-descriptor) @ 
streams-quickstart ---
[INFO] 
[INFO] --- gpg:1.6:sign (sign-artifacts) @ streams-quickstart ---
[INFO] 
[INFO] --- install:2.5.2:install (default-install) @ streams-quickstart ---
[INFO] Installing 

Re: Apache Kafka 3.7.0 Release

[Nick Telford]

Hi Stan,

I'd like to propose including KIP-892 in the 3.7 release. The KIP has been
accepted and I'm just working on rebasing the implementation against trunk
before I open a PR.

Regards,
Nick

On Tue, 21 Nov 2023 at 11:27, Mayank Shekhar Narula <
mayanks.nar...@gmail.com> wrote:

> Hi Stan
>
> Can you include KIP-951 to the 3.7 release plan? All PRs are merged in the
> trunk.
>
> On Wed, Nov 15, 2023 at 4:05 PM Stanislav Kozlovski
>  wrote:
>
> > Friendly reminder to everybody that the KIP Freeze is *exactly 7 days
> away*
> > - November 22.
> >
> > A KIP must be accepted by this date in order to be considered for this
> > release. Note, any KIP that may not be implemented in time, or otherwise
> > risks heavily destabilizing the release, should be deferred.
> >
> > Best,
> > Stan
> >
> > On Fri, Nov 3, 2023 at 6:03 AM Sophie Blee-Goldman <
> sop...@responsive.dev>
> > wrote:
> >
> > > Looks great, thank you! +1
> > >
> > > On Thu, Nov 2, 2023 at 10:21 AM David Jacot
>  > >
> > > wrote:
> > >
> > > > +1 from me as well. Thanks, Stan!
> > > >
> > > > David
> > > >
> > > > On Thu, Nov 2, 2023 at 6:04 PM Ismael Juma 
> wrote:
> > > >
> > > > > Thanks Stanislav, +1
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Thu, Nov 2, 2023 at 7:01 AM Stanislav Kozlovski
> > > > >  wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > Given the discussion here and the lack of any pushback, I have
> > > changed
> > > > > the
> > > > > > dates of the release:
> > > > > > - KIP Freeze - *November 22 *(moved 4 days later)
> > > > > > - Feature Freeze - *December 6 *(moved 2 days earlier)
> > > > > > - Code Freeze - *December 20*
> > > > > >
> > > > > > If anyone has any thoughts against this proposal - please let me
> > > know!
> > > > It
> > > > > > would be good to settle on this early. These will be the dates
> > we're
> > > > > going
> > > > > > with
> > > > > >
> > > > > > Best,
> > > > > > Stanislav
> > > > > >
> > > > > > On Thu, Oct 26, 2023 at 12:15 AM Sophie Blee-Goldman <
> > > > > > sop...@responsive.dev>
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks for the response and explanations -- I think the main
> > > question
> > > > > for
> > > > > > > me
> > > > > > > was whether we intended to permanently increase the KF -- FF
> gap
> > > from
> > > > > the
> > > > > > > historical 1 week to 3 weeks? Maybe this was a conscious
> decision
> > > > and I
> > > > > > > just
> > > > > > >  missed the memo, hopefully someone else can chime in here. I'm
> > all
> > > > for
> > > > > > > additional though. And looking around at some of the recent
> > > releases,
> > > > > it
> > > > > > > seems like we haven't been consistently following the "usual"
> > > > schedule
> > > > > > > since
> > > > > > > the 2.x releases.
> > > > > > >
> > > > > > > Anyways, my main concern was making sure to leave a full 2
> weeks
> > > > > between
> > > > > > > feature freeze and code freeze, so I'm generally happy with the
> > new
> > > > > > > proposal.
> > > > > > > Although I would still prefer to have the KIP freeze fall on a
> > > > > Wednesday
> > > > > > --
> > > > > > > Ismael actually brought up the same thing during the 3.5.0
> > release
> > > > > > > planning,
> > > > > > > so I'll just refer to his explanation for this:
> > > > > > >
> > > > > > > We typically choose a Wednesday for the various freeze dates -
> > > there
> > > > > are
> > > > > > > > often 1-2 day slips and it's better if that doesn't require
> > > people
> > > > > > > > working through the weekend.
> > > > > > > >
> > > > > > >
> > > > > > > (From this mailing list thread
> > > > > > > <
> > https://lists.apache.org/thread/dv1rym2jkf0141sfsbkws8ckkzw7st5h
> > > >)
> > > > > > >
> > > > > > > Thanks for driving the release!
> > > > > > > Sophie
> > > > > > >
> > > > > > > On Wed, Oct 25, 2023 at 8:13 AM Stanislav Kozlovski
> > > > > > >  wrote:
> > > > > > >
> > > > > > > > Thanks for the thorough response, Sophie.
> > > > > > > >
> > > > > > > > - Added to the "Future Release Plan"
> > > > > > > >
> > > > > > > > > 1. Why is the KIP freeze deadline on a Saturday?
> > > > > > > >
> > > > > > > > It was simply added as a starting point - around 30 days from
> > the
> > > > > > > > announcement. We can move it earlier to the 15th of November,
> > but
> > > > my
> > > > > > > > thinking is later is better with these things - it's already
> > > > > aggressive
> > > > > > > > enough. e.g given the choice of Nov 15 vs Nov 18, I don't
> > > > necessarily
> > > > > > > see a
> > > > > > > > strong reason to choose 15.
> > > > > > > >
> > > > > > > > If people feel strongly about this, to make up for this, we
> can
> > > eat
> > > > > > into
> > > > > > > > the KF-FF time as I'll touch upon later, and move FF a few
> days
> > > > > earlier
> > > > > > > to
> > > > > > > > land on a Wednesday.
> > > > > > > >
> > > > > > > > This reduces the time one has to get their feature complete
> > after
> > > > KF,
> > > > > > but
> > > > > > > > allows for longer 

Re: [VOTE] KIP-963: Additional metrics in Tiered Storage

[Christo Lolov]

Hello all,

With 3 +1 binding and 1 +1 non-binding votes KIP-963 is adopted 拾!
I will get down to implementing it.

Best,
Christo

On Tue, 21 Nov 2023 at 07:22, Luke Chen  wrote:

> +1 (binding) from me.
> Thanks for the KIP.
>
> Luke
>
> On Tue, Nov 21, 2023 at 11:53 AM Satish Duggana 
> wrote:
>
> > +1 (binding)
> > Thanks for the KIP and the discussion.
> >
> > Discussion mail thread for the KIP:
> > https://lists.apache.org/thread/40vsyc240hyody37mf2f0pn90shkzb45
> >
> >
> >
> > On Tue, 21 Nov 2023 at 05:21, Kamal Chandraprakash
> >  wrote:
> > >
> > > +1 (non-binding). Thanks for the KIP!
> > >
> > > On Tue, Nov 21, 2023, 03:04 Divij Vaidya 
> > wrote:
> > >
> > > > + 1 (binding)
> > > >
> > > > This Kip will greatly improve Tiered Storage troubleshooting. Thank
> you
> > > > Christo.
> > > >
> > > > On Mon 20. Nov 2023 at 17:21, Christo Lolov 
> > > > wrote:
> > > >
> > > > > Hello all!
> > > > >
> > > > > Now that the discussion for KIP-963 has winded down, I would like
> to
> > open
> > > > > it for a vote targeting 3.7.0 as the release. You can find the
> > current
> > > > > version of the KIP at
> > > > >
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-963%3A+Additional+metrics+in+Tiered+Storage
> > > > >
> > > > > Best,
> > > > > Christo
> > > > >
> > > >
> >
>

[jira] [Resolved] (KAFKA-15840) Correct initialization of ConsumerGroupHeartbeat by client

[Andrew Schofield (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Schofield resolved KAFKA-15840.
--
Resolution: Duplicate

> Correct initialization of ConsumerGroupHeartbeat by client
> --
>
> Key: KAFKA-15840
> URL: https://issues.apache.org/jira/browse/KAFKA-15840
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, consumer
>Reporter: Andrew Schofield
>Assignee: Andrew Schofield
>Priority: Major
>  Labels: kip-848-client-support, kip-848-e2e, kip-848-preview
>
> The new consumer using the KIP-848 protocol currently leaves the 
> TopicPartitions set to null for the ConsumerGroupHeartbeat request, even when 
> the MemberEpoch is zero. This violates the KIP which expects the list to be 
> empty (but not null).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15890) kafka-console-consumer.sh problems with new consumer

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-15890:


 Summary: kafka-console-consumer.sh problems with new consumer
 Key: KAFKA-15890
 URL: https://issues.apache.org/jira/browse/KAFKA-15890
 Project: Kafka
  Issue Type: Bug
  Components: clients, consumer
Reporter: Andrew Schofield
Assignee: Andrew Schofield


Various problems found testing `kafka-console-consumer.sh` with the new 
consumer, including NPEs, never-ending reconcilation states and failure to 
fetch records.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15856) Add integration tests for JoinGroup API and SyncGroup API

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-15856.
-
Fix Version/s: 3.7.0
   Resolution: Fixed

> Add integration tests for JoinGroup API and SyncGroup API
> -
>
> Key: KAFKA-15856
> URL: https://issues.apache.org/jira/browse/KAFKA-15856
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Dongnuo Lyu
>Assignee: Dongnuo Lyu
>Priority: Major
> Fix For: 3.7.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15555) Ensure wakeups are handled correctly in PrototypeAsyncConsumer.poll()

[Bruno Cadonna (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bruno Cadonna resolved KAFKA-1.
---
Resolution: Fixed

> Ensure wakeups are handled correctly in PrototypeAsyncConsumer.poll()
> -
>
> Key: KAFKA-1
> URL: https://issues.apache.org/jira/browse/KAFKA-1
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, consumer
>Reporter: Kirk True
>Assignee: Bruno Cadonna
>Priority: Minor
>  Labels: consumer-threading-refactor, kip-848-preview
>
> The implementation of the {{poll()}} method in {{PrototypeAsyncConsumer}} 
> does not disable wakeups in the same manner that {{KafkaConsumer}} does. 
> Investigate how to make the new implementation consistent with the 
> functionality of the existing implementation.
> There was a comment in the code that I plan to remove, but I will leave it 
> here for reference:
> {quote}// TODO: Once we implement poll(), clear wakeupTrigger in a finally 
> block: wakeupTrigger.clearActiveTask();{quote}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15484) Implement general uniform broker side assignor

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-15484.
-
Fix Version/s: 3.7.0
   Resolution: Fixed

> Implement general uniform broker side assignor
> --
>
> Key: KAFKA-15484
> URL: https://issues.apache.org/jira/browse/KAFKA-15484
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Ritika Reddy
>Assignee: Ritika Reddy
>Priority: Major
>  Labels: kip-848, kip-848-preview
> Fix For: 3.7.0
>
>
> Part 2 of the Uniform broker side assignor for consumer groups that have 
> different members subscribed to different topics



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15889) Make General Assignor's rebalance incremental

[David Jacot (Jira)]

David Jacot created KAFKA-15889:
---

 Summary: Make General Assignor's rebalance incremental
 Key: KAFKA-15889
 URL: https://issues.apache.org/jira/browse/KAFKA-15889
 Project: Kafka
  Issue Type: Sub-task
Reporter: David Jacot


While reviewing [https://github.com/apache/kafka/pull/14481,] we found out that 
it may be possible to make the 
`GeneralUniformAssignmentBuilder.performReassignments` incremental. As it is 
big change, we agreed on looking into this afterwards.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2410

[Apache Jenkins Server]

See