Re: [ANNOUNCE] New Kafka PMC Member: Satish Duggana

[Guozhang Wang]

Congratulations Satish!

On Sat, Oct 28, 2023 at 12:59 AM Luke Chen  wrote:
>
> Congrats Satish!
>
> Luke
>
> On Sat, Oct 28, 2023 at 11:16 AM ziming deng 
> wrote:
>
> > Congratulations Satish!
> >
> > > On Oct 27, 2023, at 23:03, Jun Rao  wrote:
> > >
> > > Hi, Everyone,
> > >
> > > Satish Duggana has been a Kafka committer since 2022. He has been very
> > > instrumental to the community since becoming a committer. It's my
> > pleasure
> > > to announce that Satish is now a member of Kafka PMC.
> > >
> > > Congratulations Satish!
> > >
> > > Jun
> > > on behalf of Apache Kafka PMC
> >
> >

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2338

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

[Qichao Chu]

Hello Everyone,

Can I ask for some feedback regarding KIP-977

?

Best,
Qichao Chu
Software Engineer | Data - Kafka
[image: Uber] 


On Mon, Oct 16, 2023 at 7:34 PM Qichao Chu  wrote:

> Hi Divij and Kirk,
>
> Thank you both for providing the valuable feedback and sorry for the
> delay. I have just updated the KIP to address the comments.
>
>1. Instead of using a topic-level control, global verbosity control
>makes more sense if we want to extend it in the future. It would be very
>difficult if we want to apply the topic allowlist everywhere
>2. Also, the topic allowlist was not dynamic which makes everything
>quite complex, especially for the topic lifecycle management. By using the
>dynamic global config, debugging could be easier, and management of the
>config is also made easier.
>3. More details are included in the test section.
>
> One thing that still misses is the performance numbers. I will get it
> ready with our internal clusters and share out soon.
>
> Many thanks for the review!
> Qichao
>
> On Tue, Sep 12, 2023 at 8:31 AM Kirk True  wrote:
>
>> Oh, and does metrics.partition.level.reporting.topics allow for regex?
>>
>> > On Sep 12, 2023, at 8:26 AM, Kirk True  wrote:
>> >
>> > Hi Qichao,
>> >
>> > Thanks for the KIP!
>> >
>> > Divij—questions/comments inline...
>> >
>> >> On Sep 11, 2023, at 4:32 AM, Divij Vaidya 
>> wrote:
>> >>
>> >> Thank you for the proposal Qichao.
>> >>
>> >> I agree with the motivation here and understand the tradeoff here
>> >> between observability vs. increased metric dimensions (metric fan-out
>> >> as you say in the KIP).
>> >>
>> >> High level comments:
>> >>
>> >> 1. I would urge you to consider the extensibility of the proposal for
>> >> other types of metrics. Tomorrow, if we want to selectively add
>> >> "partition" dimension to another metric, would we have to modify the
>> >> code where each metric is emitted? Alternatively, could we abstract
>> >> out this config in a "Kafka Metrics" library. The code provides all
>> >> information about this library and this library can choose which
>> >> dimensions it wants to add to the final metrics that are emitted based
>> >> on declarative configuration.
>> >
>> > I’d agree with this if it doesn’t place a burden on the callers. Are
>> there any potential call sites that don’t have the partition information
>> readily available?
>> >
>> >> 2. Can we offload the handling of this dimension filtering to the
>> >> metric framework? Have you explored whether prometheus or other
>> >> libraries provide the ability to dynamically change dimensions
>> >> associated with metrics?
>> >
>> > I’m not familiar with the downstream metrics providers’ capabilities.
>> This is a greatest common denominator scenario, right? We’d have to be
>> reasonable sure that the heavily used providers *all* support such dynamic
>> filtering.
>> >
>> > Also—and correct me as needed as I’m not familiar with the area—if we
>> relegate partition filtering to a lower layer, we’d still need to store the
>> metric data in memory until it’s flushed, yes? If so, is that overhead of
>> any concern?
>> >
>> >> Implementation level comments:
>> >>
>> >> 1. In the test plan section, please mention what kind of integ and/or
>> >> unit tests will be added and what they will assert. As an example, you
>> >> can add a section, "functionality tests", which would assert that new
>> >> metric config is being respected and another section, "performance
>> >> tests", which could be a system test and assert that no regression
>> >> caused wrt resources occupied by metrics from one version to another.
>> >> 2. Please mention why or why not are we considering dynamically
>> >> setting the configuration (i.e. without a broker restart)? I would
>> >> imagine that the ability to dynamically configure for a specific topic
>> >> will be very useful especially to debug production situations that you
>> >> mention in the motivation.
>> >
>> > +1
>> >
>> >> 3. You mention that we want to start with metrics closely related to
>> >> producer & consumers first, which is fair. Could you please add a
>> >> statement on the work required to extend this to other metrics in
>> >> future?
>> >
>> > +1
>> >
>> >> 4. In the compatibility section, you mention that this change is
>> >> backward compatible. I don't fully understand that. During a version
>> >> upgrade, we will start with an empty list of topics to maintain
>> >> backward compatibility. I assume after the upgrade, we will update the
>> >> new config with topic names that we desire to monitor. But updating
>> >> the config will require a broker restart (a rolling restart since
>> >> config is read-only). We will be in a situation where some brokers are
>> >> sending metrics with a new "partition" dimension and some brokers are
>> >> sending metrics with no partition dimension. Is 

Re: [ANNOUNCE] New Kafka PMC Member: Satish Duggana

[Luke Chen]

Congrats Satish!

Luke

On Sat, Oct 28, 2023 at 11:16 AM ziming deng 
wrote:

> Congratulations Satish!
>
> > On Oct 27, 2023, at 23:03, Jun Rao  wrote:
> >
> > Hi, Everyone,
> >
> > Satish Duggana has been a Kafka committer since 2022. He has been very
> > instrumental to the community since becoming a committer. It's my
> pleasure
> > to announce that Satish is now a member of Kafka PMC.
> >
> > Congratulations Satish!
> >
> > Jun
> > on behalf of Apache Kafka PMC
>
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2337

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 337289 lines...]
> Task :connect:json:testClasses UP-TO-DATE
> Task :connect:api:jar UP-TO-DATE
> Task :connect:json:testJar
> Task :connect:api:generateMetadataFileForMavenJavaPublication
> Task :connect:json:copyDependantLibs UP-TO-DATE
> Task :connect:api:compileTestJava UP-TO-DATE
> Task :connect:api:testClasses UP-TO-DATE
> Task :connect:json:jar UP-TO-DATE
> Task :connect:json:generateMetadataFileForMavenJavaPublication
> Task :connect:api:testJar
> Task :connect:json:testSrcJar
> Task :connect:api:testSrcJar
> Task :connect:api:publishMavenJavaPublicationToMavenLocal
> Task :connect:api:publishToMavenLocal
> Task :connect:json:publishMavenJavaPublicationToMavenLocal
> Task :connect:json:publishToMavenLocal
> Task :clients:generateMetadataFileForMavenJavaPublication
> Task :core:compileScala
> Task :storage:storage-api:compileTestJava
> Task :storage:storage-api:testClasses
> Task :server-common:compileTestJava
> Task :server-common:testClasses
> Task :raft:compileTestJava
> Task :raft:testClasses
> Task :group-coordinator:compileTestJava
> Task :group-coordinator:testClasses

> Task :clients:javadoc
/home/jenkins/workspace/Kafka_kafka_trunk/clients/src/main/java/org/apache/kafka/clients/admin/ScramMechanism.java:32:
 warning - Tag @see: missing final '>': "https://cwiki.apache.org/confluence/display/KAFKA/KIP-554%3A+Add+Broker-side+SCRAM+Config+API;>KIP-554:
 Add Broker-side SCRAM Config API

 This code is duplicated in 
org.apache.kafka.common.security.scram.internals.ScramMechanism.
 The type field in both files must match and must not change. The type field
 is used both for passing ScramCredentialUpsertion and for the internal
 UserScramCredentialRecord. Do not change the type field."
/home/jenkins/workspace/Kafka_kafka_trunk/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/package-info.java:21:
 warning - Tag @link: reference not found: 
org.apache.kafka.common.security.oauthbearer
2 warnings

> Task :metadata:compileTestJava
> Task :metadata:testClasses
> Task :clients:javadocJar
> Task :clients:srcJar
> Task :clients:testJar
> Task :clients:testSrcJar
> Task :clients:publishMavenJavaPublicationToMavenLocal
> Task :clients:publishToMavenLocal
> Task :core:classes
> Task :core:compileTestJava NO-SOURCE
> Task :core:compileTestScala
> Task :core:testClasses
> Task :streams:compileTestJava
> Task :streams:testClasses
> Task :streams:testJar
> Task :streams:testSrcJar
> Task :streams:publishMavenJavaPublicationToMavenLocal
> Task :streams:publishToMavenLocal

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings 
and determine if they come from your own scripts or plugins.

For more on this, please refer to 
https://docs.gradle.org/8.3/userguide/command_line_interface.html#sec:command_line_warnings
 in the Gradle documentation.

BUILD SUCCESSFUL in 7m 53s
94 actionable tasks: 41 executed, 53 up-to-date

Publishing build scan...
https://ge.apache.org/s/qbxgped4ggo4w

[Pipeline] sh
+ grep ^version= gradle.properties
+ cut -d= -f 2
[Pipeline] dir
Running in /home/jenkins/workspace/Kafka_kafka_trunk/streams/quickstart
[Pipeline] {
[Pipeline] sh
+ mvn clean install -Dgpg.skip
[INFO] Scanning for projects...
[INFO] 
[INFO] Reactor Build Order:
[INFO] 
[INFO] Kafka Streams :: Quickstart[pom]
[INFO] streams-quickstart-java[maven-archetype]
[INFO] 
[INFO] < org.apache.kafka:streams-quickstart >-
[INFO] Building Kafka Streams :: Quickstart 3.7.0-SNAPSHOT[1/2]
[INFO]   from pom.xml
[INFO] [ pom ]-
[INFO] 
[INFO] --- clean:3.0.0:clean (default-clean) @ streams-quickstart ---
[INFO] 
[INFO] --- remote-resources:1.5:process (process-resource-bundles) @ 
streams-quickstart ---
[INFO] 
[INFO] --- site:3.5.1:attach-descriptor (attach-descriptor) @ 
streams-quickstart ---
[INFO] 
[INFO] --- gpg:1.6:sign (sign-artifacts) @ streams-quickstart ---
[INFO] 
[INFO] --- install:2.5.2:install (default-install) @ streams-quickstart ---
[INFO] Installing 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/quickstart/pom.xml to 
/home/jenkins/.m2/repository/org/apache/kafka/streams-quickstart/3.7.0-SNAPSHOT/streams-quickstart-3.7.0-SNAPSHOT.pom
[INFO] 
[INFO] --< org.apache.kafka:streams-quickstart-java >--
[INFO] Building streams-quickstart-java 3.7.0-SNAPSHOT[2/2]
[INFO]   from java/pom.xml
[INFO] --[ maven-archetype ]---
[INFO] 
[INFO] --- clean:3.0.0:clean (default-clean) @