Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #2661

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 2574 lines...]
[2024-02-21T05:34:55.249Z] > Task :tools:spotbugsMain
[2024-02-21T05:34:55.793Z] > Task :server-common:checkstyleMain
[2024-02-21T05:34:56.369Z] > Task :streams:examples:spotbugsMain
[2024-02-21T05:34:57.703Z] > Task :connect:mirror:spotbugsMain
[2024-02-21T05:34:59.775Z] > Task :streams:streams-scala:compileScala
[2024-02-21T05:35:00.896Z] > Task :streams:test-utils:spotbugsMain
[2024-02-21T05:35:02.106Z] > Task :streams:compileJava
[2024-02-21T05:35:02.106Z] > Task :clients:checkstyleMain
[2024-02-21T05:35:02.106Z] > Task :clients:compileTestJava
[2024-02-21T05:35:04.656Z] > Task :log4j-appender:spotbugsMain
[2024-02-21T05:35:04.656Z] > Task :examples:spotbugsMain
[2024-02-21T05:35:04.656Z] > Task :tools:tools-api:spotbugsMain
[2024-02-21T05:35:04.656Z] > Task :connect:mirror-client:spotbugsMain
[2024-02-21T05:35:06.115Z] > Task :connect:api:spotbugsMain
[2024-02-21T05:35:06.115Z] > Task :tools:tools-api:check
[2024-02-21T05:35:07.239Z] > Task :connect:file:spotbugsMain
[2024-02-21T05:35:07.239Z] > Task :connect:basic-auth-extension:spotbugsMain
[2024-02-21T05:35:07.239Z] > Task :connect:json:spotbugsMain
[2024-02-21T05:35:08.370Z] > Task :connect:runtime:compileJava
[2024-02-21T05:35:08.370Z] > Task :connect:transforms:spotbugsMain
[2024-02-21T05:35:08.370Z] > Task :raft:checkstyleMain
[2024-02-21T05:35:08.370Z] > Task :server-common:spotbugsMain
[2024-02-21T05:35:08.370Z] > Task :trogdor:checkstyleMain
[2024-02-21T05:35:09.681Z] > Task :connect:runtime:classes
[2024-02-21T05:35:09.681Z] > Task :storage:storage-api:spotbugsMain
[2024-02-21T05:35:10.906Z] > Task :connect:test-plugins:spotbugsMain
[2024-02-21T05:35:13.158Z] > Task :trogdor:spotbugsMain
[2024-02-21T05:35:14.699Z] > Task :clients:spotbugsMain
[2024-02-21T05:35:15.817Z] > Task :streams:streams-scala:classes
[2024-02-21T05:35:15.817Z] > Task :streams:streams-scala:checkstyleMain 
NO-SOURCE
[2024-02-21T05:35:16.033Z] > Task :metadata:compileJava
[2024-02-21T05:35:16.033Z] > Task :raft:spotbugsMain
[2024-02-21T05:35:17.433Z] > Task :metadata:classes
[2024-02-21T05:35:24.091Z] > Task :streams:streams-scala:spotbugsMain
[2024-02-21T05:35:24.959Z] > Task :storage:checkstyleMain
[2024-02-21T05:35:24.959Z] > Task :storage:spotbugsMain
[2024-02-21T05:35:24.959Z] > Task :connect:mirror:compileJava
[2024-02-21T05:35:24.959Z] > Task :connect:mirror:classes
[2024-02-21T05:35:39.233Z] > Task :tools:compileJava
[2024-02-21T05:35:39.233Z] > Task :connect:runtime:checkstyleMain
[2024-02-21T05:35:40.631Z] > Task :connect:runtime:spotbugsMain
[2024-02-21T05:35:40.631Z] > Task :tools:classes
[2024-02-21T05:35:42.205Z] > Task :connect:mirror:checkstyleMain
[2024-02-21T05:35:42.205Z] > Task :examples:check
[2024-02-21T05:35:45.002Z] > Task :group-coordinator:compileJava
[2024-02-21T05:35:45.002Z] > Task :group-coordinator:classes
[2024-02-21T05:35:46.576Z] > Task :metadata:checkstyleMain
[2024-02-21T05:35:46.576Z] > Task :metadata:spotbugsMain
[2024-02-21T05:35:48.150Z] > Task :streams:classes
[2024-02-21T05:35:48.150Z] > Task :streams:streams-scala:compileJava NO-SOURCE
[2024-02-21T05:35:49.733Z] 
[2024-02-21T05:35:49.733Z] > Task :core:compileScala
[2024-02-21T05:35:49.733Z] Unexpected javac output: warning: [options] 
bootstrap class path not set in conjunction with -source 8
[2024-02-21T05:35:49.733Z] warning: [options] source value 8 is obsolete and 
will be removed in a future release
[2024-02-21T05:35:49.733Z] warning: [options] target value 8 is obsolete and 
will be removed in a future release
[2024-02-21T05:35:49.733Z] warning: [options] To suppress warnings about 
obsolete options, use -Xlint:-options.
[2024-02-21T05:35:49.733Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/main/java/kafka/log/remote/RemoteLogManager.java:234:
 warning: [removal] AccessController in java.security has been deprecated and 
marked for removal
[2024-02-21T05:35:49.733Z] return 
java.security.AccessController.doPrivileged(new 
PrivilegedAction() {
[2024-02-21T05:35:49.733Z] ^
[2024-02-21T05:35:49.733Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/main/java/kafka/log/remote/RemoteLogManager.java:256:
 warning: [removal] AccessController in java.security has been deprecated and 
marked for removal
[2024-02-21T05:35:49.733Z] return 
java.security.AccessController.doPrivileged(new 
PrivilegedAction() {
[2024-02-21T05:35:49.733Z] ^
[2024-02-21T05:35:49.733Z] Note: Some input files use or override a deprecated 
API.
[2024-02-21T05:35:49.733Z] Note: Recompile with -Xlint:deprecation for details.
[2024-02-21T05:35:49.733Z] Note: 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/core/src/main/java/kafka/log/remote/RemoteLogManager.java
 uses unchecked or unsafe operations.

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2660

[Apache Jenkins Server]

See 

Re: [VOTE] 3.7.0 RC4

[Luke Chen]

Hi all,

I found there is a bug (KAFKA-16283
) in the built-in
`RoundRobinPartitioner`, and it will cause only half of the partitions
receiving the records. (I'm surprised our tests didn't catch it!?)
After further testing, I found this issue already existed in v3.0.0. (I
didn't test 2.x versions)
I think this should not be a blocker to v3.7.0 since it's been there for
years.
But I think we should, at least, document it to notify users about this
known issue.
I've created 2 PRs to document it:
https://github.com/apache/kafka/pull/15400
https://github.com/apache/kafka-site/pull/585

Let me know what you think.

Thanks.
Luke

On Wed, Feb 21, 2024 at 10:52 AM Proven Provenzano
 wrote:

> HI,
>
> I've downloaded, built from source and then validated JBOD with KRaft works
> along with migrating a cluster with JBOD from ZK to KRaft works.
>
> +1 (nonbinding) from me.
>
> --Proven
>
> On Tue, Feb 20, 2024 at 2:13 PM Justine Olshan
> 
> wrote:
>
> > Hey folks,
> >
> > I've done the following to validate the release:
> >
> > -- validated the keys for all the artifacts
> > -- built from source and started a ZK cluster -- ran a few workloads on
> it.
> > -- ran 2.12 Kraft cluster and ran a few workloads on it
> >
> > I see there is a lot of ongoing discussion about the upgrade notes. +1
> > (binding) from me given Mickael is voting +1 as well.
> >
> > Justine
> >
> > On Tue, Feb 20, 2024 at 6:18 AM Divij Vaidya 
> > wrote:
> >
> > > > I am a bit unclear on the precise process regarding what parts of
> this
> > > get merged at what time, and whether the release first needs to be done
> > or
> > > not.
> > >
> > > The order is as follows:
> > >
> > > 1. Release approved as part of this vote. After this we follow the
> > > steps from here:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Process#ReleaseProcess-Afterthevotepasses
> > >
> > > 2. Upload artifacts to maven etc. These artifacts do not have RC suffix
> > in
> > > them. You need a PMC member to mark these artifacts as "production" in
> > > apache svn.
> > > 3. Update website changes (docs, blog etc.). This is where your PRs
> > > on kafka-site repo get merged.
> > > 4. Send a release announcement by email.
> > >
> > > --
> > > Divij Vaidya
> > >
> > >
> > >
> > > On Tue, Feb 20, 2024 at 3:02 PM Stanislav Kozlovski
> > >  wrote:
> > >
> > > > Thanks for testing the release! And thanks for the review on the
> > > > documentation. Good catch on the license too.
> > > >
> > > > I have addressed the comments in the blog PR, and opened a few other
> > PRs
> > > to
> > > > the website in relation to the release.
> > > >
> > > > - 37: Add download section for the latest 3.7 release
> > > > 
> > > > - 37: Update default docs to point to the 3.7.0 release docs
> > > > 
> > > > - 3.7: Add blog post for Kafka 3.7
> > > > 
> > > > - MINOR: Update stale upgrade_3_6_0 header links in documentation
> > > > 
> > > > - 37: Add upgrade notes for the 3.7.0 release
> > > > 
> > > >
> > > > I am a bit unclear on the precise process regarding what parts of
> this
> > > get
> > > > merged at what time, and whether the release first needs to be done
> or
> > > not.
> > > >
> > > > Best,
> > > > Stanislav
> > > >
> > > > On Mon, Feb 19, 2024 at 8:34 PM Divij Vaidya <
> divijvaidy...@gmail.com>
> > > > wrote:
> > > >
> > > > > Great. In that case we can fix the license issue retrospectively. I
> > > have
> > > > > created a JIRA for it
> > > https://issues.apache.org/jira/browse/KAFKA-16278
> > > > > and
> > > > > also updated the release process (which redirects to
> > > > > https://issues.apache.org/jira/browse/KAFKA-12622) to check for
> the
> > > > > correct
> > > > > license in both the kafka binaries.
> > > > >
> > > > > I am +1 (binding) assuming Mickael's concerns about update notes to
> > 3.7
> > > > are
> > > > > addressed before release.
> > > > >
> > > > > --
> > > > > Divij Vaidya
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Feb 19, 2024 at 6:08 PM Mickael Maison <
> > > mickael.mai...@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I agree with Josep, I don't think it's worth making a new RC just
> > for
> > > > > this.
> > > > > >
> > > > > > Thanks Stanislav for sharing the test results. The last thing
> > holding
> > > > > > me from casting my vote is the missing upgrade notes for 3.7.0.
> > > > > >
> > > > > > Thanks,
> > > > > > Mickael
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Feb 19, 2024 at 4:28 PM Josep Prat
> > >  > > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > I think I remember finding a similar problem (NOTICE_binary)
> and
> > it
> > > > > > didn't
> > > > > > > qualify for an extra 

[PR] KAFKA-16283: notify users about RoundRobinPartitioner bug [kafka-site]

[via GitHub]

showuon opened a new pull request, #585:
URL: https://github.com/apache/kafka-site/pull/585

   Add notes in "config doc" to notify users about the bug: KAFKA-16283 and not 
to use `RoundRobinPartitioner`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [VOTE] 3.7.0 RC4

[Proven Provenzano]

HI,

I've downloaded, built from source and then validated JBOD with KRaft works
along with migrating a cluster with JBOD from ZK to KRaft works.

+1 (nonbinding) from me.

--Proven

On Tue, Feb 20, 2024 at 2:13 PM Justine Olshan 
wrote:

> Hey folks,
>
> I've done the following to validate the release:
>
> -- validated the keys for all the artifacts
> -- built from source and started a ZK cluster -- ran a few workloads on it.
> -- ran 2.12 Kraft cluster and ran a few workloads on it
>
> I see there is a lot of ongoing discussion about the upgrade notes. +1
> (binding) from me given Mickael is voting +1 as well.
>
> Justine
>
> On Tue, Feb 20, 2024 at 6:18 AM Divij Vaidya 
> wrote:
>
> > > I am a bit unclear on the precise process regarding what parts of this
> > get merged at what time, and whether the release first needs to be done
> or
> > not.
> >
> > The order is as follows:
> >
> > 1. Release approved as part of this vote. After this we follow the
> > steps from here:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Process#ReleaseProcess-Afterthevotepasses
> >
> > 2. Upload artifacts to maven etc. These artifacts do not have RC suffix
> in
> > them. You need a PMC member to mark these artifacts as "production" in
> > apache svn.
> > 3. Update website changes (docs, blog etc.). This is where your PRs
> > on kafka-site repo get merged.
> > 4. Send a release announcement by email.
> >
> > --
> > Divij Vaidya
> >
> >
> >
> > On Tue, Feb 20, 2024 at 3:02 PM Stanislav Kozlovski
> >  wrote:
> >
> > > Thanks for testing the release! And thanks for the review on the
> > > documentation. Good catch on the license too.
> > >
> > > I have addressed the comments in the blog PR, and opened a few other
> PRs
> > to
> > > the website in relation to the release.
> > >
> > > - 37: Add download section for the latest 3.7 release
> > > 
> > > - 37: Update default docs to point to the 3.7.0 release docs
> > > 
> > > - 3.7: Add blog post for Kafka 3.7
> > > 
> > > - MINOR: Update stale upgrade_3_6_0 header links in documentation
> > > 
> > > - 37: Add upgrade notes for the 3.7.0 release
> > > 
> > >
> > > I am a bit unclear on the precise process regarding what parts of this
> > get
> > > merged at what time, and whether the release first needs to be done or
> > not.
> > >
> > > Best,
> > > Stanislav
> > >
> > > On Mon, Feb 19, 2024 at 8:34 PM Divij Vaidya 
> > > wrote:
> > >
> > > > Great. In that case we can fix the license issue retrospectively. I
> > have
> > > > created a JIRA for it
> > https://issues.apache.org/jira/browse/KAFKA-16278
> > > > and
> > > > also updated the release process (which redirects to
> > > > https://issues.apache.org/jira/browse/KAFKA-12622) to check for the
> > > > correct
> > > > license in both the kafka binaries.
> > > >
> > > > I am +1 (binding) assuming Mickael's concerns about update notes to
> 3.7
> > > are
> > > > addressed before release.
> > > >
> > > > --
> > > > Divij Vaidya
> > > >
> > > >
> > > >
> > > > On Mon, Feb 19, 2024 at 6:08 PM Mickael Maison <
> > mickael.mai...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I agree with Josep, I don't think it's worth making a new RC just
> for
> > > > this.
> > > > >
> > > > > Thanks Stanislav for sharing the test results. The last thing
> holding
> > > > > me from casting my vote is the missing upgrade notes for 3.7.0.
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Feb 19, 2024 at 4:28 PM Josep Prat
> >  > > >
> > > > > wrote:
> > > > > >
> > > > > > I think I remember finding a similar problem (NOTICE_binary) and
> it
> > > > > didn't
> > > > > > qualify for an extra RC
> > > > > >
> > > > > > Best,
> > > > > >
> > > > > > On Mon, Feb 19, 2024 at 3:44 PM Divij Vaidya <
> > > divijvaidy...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I have performed the following checks. The only thing I would
> > like
> > > to
> > > > > call
> > > > > > > out is the missing licenses before providing a vote. How do we
> > want
> > > > > > > to proceed on this? What have we done in the past? (Creating a
> > new
> > > RC
> > > > > is
> > > > > > > overkill IMO for this license issue).
> > > > > > >
> > > > > > > ## License check
> > > > > > >
> > > > > > > Test: Validate license of dependencies for both 2.12 & 2.13
> > binary.
> > > > > > > Result: Missing license for some scala* libraries specifically
> > for
> > > > > 2.12.
> > > > > > > Seems like we have been missing these licenses for quite some
> > > version
> > > > > now.
> > > > > > >
> > > > > > > ```
> > > > > > > for f in $(ls libs | grep -v "^kafka\|connect\|trogdor"); do
> if !
> > > > grep
> > > > > -q
> > > > > > > ${f%.*} LICENSE; then echo 

[jira] [Resolved] (KAFKA-6675) Connect workers should log plugin path and available plugins more clearly

[Greg Harris (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Greg Harris resolved KAFKA-6675.

Fix Version/s: 3.6.0
 Assignee: Greg Harris  (was: Valeria Vasylieva)
   Resolution: Fixed

This was incorporated into the bin/connect-plugin-path.sh list command, as 
specified in KIP-898: 
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-898%3A+Modernize+Connect+plugin+discovery]
 . This can be used offline without starting the connect worker or loading any 
live configurations.

> Connect workers should log plugin path and available plugins more clearly
> -
>
> Key: KAFKA-6675
> URL: https://issues.apache.org/jira/browse/KAFKA-6675
> Project: Kafka
>  Issue Type: Improvement
>  Components: connect
>Affects Versions: 0.11.0.1
>Reporter: Randall Hauch
>Assignee: Greg Harris
>Priority: Minor
> Fix For: 3.6.0
>
>
> Users struggle with setting the plugin path and properly installing plugins. 
> If users get any of this wrong, they get strange errors only after they run 
> the worker and attempt to deploy connectors or use transformations. 
> The Connect worker should more obviously output the plugin path directories 
> and the available plugins. For example, if the {{plugin.path}} were:
> {code}
> plugin.path=/usr/local/share/java,/usr/local/plugins
> {code}
> then the worker might output something like the following information in the 
> log:
> {noformat}
> Looking for plugins on classpath and inside plugin.path directories:
>   /usr/local/share/java
>   /usr/local/plugins
>  
> Source Connector(s):
>   FileStreamSource  (org.apache.kafka.connect.file.FileStreamSourceConnector) 
>   @ classpath
>   FileStreamSink(org.apache.kafka.connect.file.FileStreamSinkConnector)   
>   @ classpath
>   JdbcSource(io.confluent.connect.jdbc.JdbcSourceConnector)   
>   @ /usr/local/share/java/kafka-connect-jdbc
>   MySql (io.debezium.connector.mysql.MySqlConnector)  
>   @ /usr/local/plugins/debezium-connector-mysql
> Converter(s):
>   JsonConverter (org.apache.kafka.connect.json.JsonConverter) 
>   @ classpath
>   ByteArrayConverter
> (org.apache.kafka.connect.converters.ByteArrayConverter)@ classpath
>   SimpleHeaderConverter 
> (org.apache.kafka.connect.converters.SimpleHeaderConverter) @ classpath
>   AvroConverter (io.confluent.connect.avro.AvroConverter) 
>   @ /usr/local/share/java/kafka-serde-tools
> Transformation(s):
>   InsertField   (org.apache.kafka.connect.transforms.InsertField) 
>   @ classpath
>   ReplaceField  (org.apache.kafka.connect.transforms.ReplaceField)
>   @ classpath
>   MaskField (org.apache.kafka.connect.transforms.MaskField)   
>   @ classpath
>   ValueToKey(org.apache.kafka.connect.transforms.ValueToKey)  
>   @ classpath
>   HoistField(org.apache.kafka.connect.transforms.HoistField)  
>   @ classpath
>   ExtractField  (org.apache.kafka.connect.transforms.ExtractField)
>   @ classpath
>   SetSchemaMetadata (org.apache.kafka.connect.transforms.SetSchemaMetadata)   
>   @ classpath
>   RegexRouter   (org.apache.kafka.connect.transforms.RegexRouter) 
>   @ classpath
>   TimestampRouter   (org.apache.kafka.connect.transforms.TimestampRouter) 
>   @ classpath
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16289) Values.parseString on heterogeneous lists and maps sometimes corrupts data by inferring incorrect schema

[Greg Harris (Jira)]

Greg Harris created KAFKA-16289:
---

 Summary: Values.parseString on heterogeneous lists and maps 
sometimes corrupts data by inferring incorrect schema
 Key: KAFKA-16289
 URL: https://issues.apache.org/jira/browse/KAFKA-16289
 Project: Kafka
  Issue Type: Bug
  Components: connect
Reporter: Greg Harris
Assignee: Greg Harris


The Values.parseString function makes a best-effort conversion of strings to 
Connect-schema'd data. It supports reading arrays and maps as delimited by 
`[,]` and `\{:,}` characters, and attempts to infer the common type of these 
structures from the types of the elements. The algorithm it follows is:

1. Parse the elements of the list in one-pass. Infer the smallest/strictest 
type which can contain each value individually.
2. Iterate over the schemas inferred for each element, and repeatedly merge two 
schemas together to the smallest type which covers both element schemas.
3. Convert the parsed elements to the common element schema.

The implementation of step 2 here: 
[https://github.com/apache/kafka/blob/ead2431c37ace9255df88ffe819bb905311af088/connect/api/src/main/java/org/apache/kafka/connect/data/Values.java#L805-L823]
 has a flaw in it however. The `elementSchema` variable has `null` as a 
sentinel both of the situations "no elements seen so far" and "no common schema 
possible" among the seen elements.

When processing the first element of the list, `null` is used to adopt the 
schema of the first element as the initial common schema. Later when an 
incompatible element is found, the common schema is set to null to indicate 
that there is no common element schema. However, a following iteration can 
misinterpret the `null` as being at the start of the list again, and inject a 
schema which works for some of the elements and not others.

When the values are converted in step 3, each element has one of the following 
happen:
1. The value is left-as is (e.g. no common schema inferred)
2. The value is converted correctly to the destination type (e.g. int -> long)
3. An exception is thrown because the type could not be converted (e.g. string 
-> struct)
4. The value is silently corrupted (e.g. long -> int, decimal -> long)

In normal circumstances either case (1) happens to all of the elements, or 
case(2) does, depending on if a common schema was found. But when this bug is 
triggered by having heterogeneous types, case (2) or case (3) can happen to 
some of the elements in the array.

The effects depend on the order of elements in the array, as the sentinel value 
bug is dependent on the iteration order of the elements. For example:

* `[1,2,"3"]` returns Byte, Byte, String
* `["1",2,3]` returns Byte, Byte, Byte (safely converts the data, case 2)
* `[1,2,{}]` returns Byte, Byte, Map
* `[{},2,3]` experiences an exception and returns String (exception, case 3)
* `[1, 2, 10]` returns Byte, Byte, BigDecimal
* `[10, 1, 2]` returns Byte, Byte, Byte (corruption, case 4)

Fixing this bug would entail changing all of these to return heterogeneous 
lists without a common schema, and not convert the values at all. However, this 
is a backwards-incompatible change because these are all situations in which we 
return data without an exception, so downstream users could be relying on the 
result.

However, this behavior is very opaque and unpredictable, and I think anyone 
that observes this in the wild would need to work-around it or avoid it, rather 
than rely on it happening. I think that fixing it to address the silent 
corruption case is a bigger benefit to users than the harm done by changing the 
other cases.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [Discuss] KIP-1019: Expose method to determine Metric Measurability

[Jun Rao]

Hi, Apoorv,

Thanks for the KIP.

Could we document how we plan to use the new isMeasurable() method? For
example, gauge is another type of metric. Do we plan to add an isGauge()
method too? If so, is it better to add a separate method for each metric
type or is it better to have a single method like metricType() that returns
an enum?

Jun


On Wed, Feb 14, 2024 at 6:56 AM Apoorv Mittal 
wrote:

> Hi,
> I would like to start discussion of a small KIP which fills a gap in
> determining Kafka Metric measurability.
>
> KIP-1019: Expose method to determine Metric Measurability
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1019%3A+Expose+method+to+determine+Metric+Measurability
> >
>
> Regards,
> Apoorv Mittal
>

[jira] [Resolved] (KAFKA-15475) Request might retry forever even if the user API timeout expires

[Kirk True (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kirk True resolved KAFKA-15475.
---
Resolution: Fixed

> Request might retry forever even if the user API timeout expires
> 
>
> Key: KAFKA-15475
> URL: https://issues.apache.org/jira/browse/KAFKA-15475
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, consumer
>Reporter: Philip Nee
>Assignee: Kirk True
>Priority: Critical
>  Labels: consumer-threading-refactor, timeout
> Fix For: 3.8.0
>
>
> If the request timeout in the background thread, it will be completed with 
> TimeoutException, which is Retriable.  In the TopicMetadataRequestManager and 
> possibly other managers, the request might continue to be retried forever.
>  
> There are two ways to fix this
>  # Pass a timer to the manager to remove the inflight requests when it is 
> expired.
>  # Pass the future to the application layer and continue to retry.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (KAFKA-16200) Enforce that RequestManager implementations respect user-provided timeout

[Kirk True (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kirk True reopened KAFKA-16200:
---

> Enforce that RequestManager implementations respect user-provided timeout
> -
>
> Key: KAFKA-16200
> URL: https://issues.apache.org/jira/browse/KAFKA-16200
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, consumer
>Reporter: Kirk True
>Assignee: Kirk True
>Priority: Blocker
>  Labels: consumer-threading-refactor, timeout
> Fix For: 3.8.0
>
>
> The intention of the {{CompletableApplicationEvent}} is for a {{Consumer}} to 
> block waiting for the event to complete. The application thread will block 
> for the timeout, but there is not yet a consistent manner in which events are 
> timed out.
> Enforce at the request manager layer that timeouts are respected per the 
> design in KAFKA-15848.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-1019: Expose method to determine Metric Measurability

[Matthias J. Sax]

+1 (binding)

On 2/20/24 2:55 AM, Manikumar wrote:

+1 (binding).

Thanks for the KIP.

Manikumar

On Tue, Feb 20, 2024 at 2:31 PM Andrew Schofield <
andrew_schofield_j...@outlook.com> wrote:


Hi Apoorv,
Thanks for the KIP.

+1 (non-binding)

Thanks,
Andrew


On 19 Feb 2024, at 22:31, Apoorv Mittal 

wrote:


Hi,
I’d like to start the voting for KIP-1019: Expose method to determine
Metric Measurability.



https://cwiki.apache.org/confluence/display/KAFKA/KIP-1019%3A+Expose+method+to+determine+Metric+Measurability


Regards,
Apoorv Mittal

[jira] [Resolved] (KAFKA-16199) Prune the event queue if event timeout expired before starting

[Kirk True (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kirk True resolved KAFKA-16199.
---
Resolution: Duplicate

> Prune the event queue if event timeout expired before starting
> --
>
> Key: KAFKA-16199
> URL: https://issues.apache.org/jira/browse/KAFKA-16199
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, consumer
>Reporter: Kirk True
>Assignee: Kirk True
>Priority: Major
>  Labels: consumer-threading-refactor, timeout
> Fix For: 3.8.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16200) Enforce that RequestManager implementations respect user-provided timeout

[Kirk True (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kirk True resolved KAFKA-16200.
---
Resolution: Duplicate

> Enforce that RequestManager implementations respect user-provided timeout
> -
>
> Key: KAFKA-16200
> URL: https://issues.apache.org/jira/browse/KAFKA-16200
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, consumer
>Reporter: Kirk True
>Assignee: Kirk True
>Priority: Blocker
>  Labels: consumer-threading-refactor, timeout
> Fix For: 3.8.0
>
>
> The intention of the {{CompletableApplicationEvent}} is for a {{Consumer}} to 
> block waiting for the event to complete. The application thread will block 
> for the timeout, but there is not yet a consistent manner in which events are 
> timed out.
> Enforce at the request manager layer that timeouts are respected per the 
> design in KAFKA-15848.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16019) Some of the tests in PlaintextConsumer can't seem to deterministically invoke and verify the consumer callback

[Kirk True (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kirk True resolved KAFKA-16019.
---
Resolution: Fixed

> Some of the tests in PlaintextConsumer can't seem to deterministically invoke 
> and verify the consumer callback
> --
>
> Key: KAFKA-16019
> URL: https://issues.apache.org/jira/browse/KAFKA-16019
> Project: Kafka
>  Issue Type: Test
>  Components: clients, consumer
>Reporter: Philip Nee
>Assignee: Kirk True
>Priority: Critical
>  Labels: consumer-threading-refactor, integration-tests, timeout
> Fix For: 3.8.0
>
>
> I was running the PlaintextConsumer to test the async consumer; however, a 
> few tests were failing with not being able to verify the listener is invoked 
> correctly
> For example `testPerPartitionLeadMetricsCleanUpWithSubscribe`
> Around 50% of the time, the listener's callsToAssigned was never incremented 
> correctly.  Event changing it to awaitUntilTrue it was still the same case
> {code:java}
> consumer.subscribe(List(topic, topic2).asJava, listener)
> val records = awaitNonEmptyRecords(consumer, tp)
> assertEquals(1, listener.callsToAssigned, "should be assigned once") {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16023) PlaintextConsumerTest needs to wait for reconciliation to complete before proceeding

[Kirk True (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kirk True resolved KAFKA-16023.
---
Resolution: Fixed

{{testPerPartitionLagMetricsCleanUpWithSubscribe}} is now passing consistently, 
so marking this as fixed.

> PlaintextConsumerTest needs to wait for reconciliation to complete before 
> proceeding
> 
>
> Key: KAFKA-16023
> URL: https://issues.apache.org/jira/browse/KAFKA-16023
> Project: Kafka
>  Issue Type: Test
>  Components: clients, consumer
>Reporter: Philip Nee
>Assignee: Kirk True
>Priority: Critical
>  Labels: consumer-threading-refactor, integration-tests, timeout
> Fix For: 3.8.0
>
>
> Several tests in PlaintextConsumerTest.scala (such as 
> testPerPartitionLagMetricsCleanUpWithSubscribe) uses:
> assertEquals(1, listener.callsToAssigned, "should be assigned once")
> However, as the timing for reconciliation completion is not deterministic due 
> to asynchronous processing. We actually need to wait until the condition to 
> happen.
> However, another issue is the timeout - some of these tasks might not 
> complete within the 600ms timeout, so the tests are deemed to be flaky.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15993) Enable max poll integration tests that depend on callback invocation

[Kirk True (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kirk True resolved KAFKA-15993.
---
Resolution: Duplicate

> Enable max poll integration tests that depend on callback invocation
> 
>
> Key: KAFKA-15993
> URL: https://issues.apache.org/jira/browse/KAFKA-15993
> Project: Kafka
>  Issue Type: Test
>  Components: clients, consumer
>Reporter: Philip Nee
>Assignee: Kirk True
>Priority: Blocker
>  Labels: consumer-threading-refactor, integration-tests, timeout
> Fix For: 3.8.0
>
>
> We will enable integration tests using the async consumer in KAFKA-15971.  
> However, we should also enable tests that rely on rebalance listeners after 
> KAFKA-15628 is closed.  One example would be testMaxPollIntervalMs, that I 
> relies on the listener to verify the correctness.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16208) Design new Consumer timeout policy

[Kirk True (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kirk True resolved KAFKA-16208.
---
Resolution: Duplicate

> Design new Consumer timeout policy
> --
>
> Key: KAFKA-16208
> URL: https://issues.apache.org/jira/browse/KAFKA-16208
> Project: Kafka
>  Issue Type: Task
>  Components: clients, consumer, documentation
>Affects Versions: 3.7.0
>Reporter: Kirk True
>Assignee: Kirk True
>Priority: Blocker
>  Labels: consumer-threading-refactor, timeout
> Fix For: 3.8.0
>
>
> This task is to design and document the timeout policy for the new Consumer 
> implementation.
> The documentation lives here: 
> https://cwiki.apache.org/confluence/display/KAFKA/Java+client+Consumer+timeouts



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16288) Values.convertToDecimal throws ClassCastExceptions on String inputs

[Greg Harris (Jira)]

Greg Harris created KAFKA-16288:
---

 Summary: Values.convertToDecimal throws ClassCastExceptions on 
String inputs
 Key: KAFKA-16288
 URL: https://issues.apache.org/jira/browse/KAFKA-16288
 Project: Kafka
  Issue Type: Bug
  Components: connect
Affects Versions: 1.1.0
Reporter: Greg Harris
Assignee: Greg Harris


The convertToDecimal function does a best-effort conversion of an arbitrary 
Object to a BigDecimal. Generally when a conversion cannot take place (such as 
when an unknown subclass is passed-in) the function throws a DataException. 
However, specifically for String inputs with valid number within, a 
ClassCastException is thrown.

This is because there is an extra "doubleValue" call in the implementation: 
[https://github.com/apache/kafka/blob/ead2431c37ace9255df88ffe819bb905311af088/connect/api/src/main/java/org/apache/kafka/connect/data/Values.java#L427]
 which immediately causes a ClassCastException in the caller: 
[https://github.com/apache/kafka/blob/ead2431c37ace9255df88ffe819bb905311af088/connect/api/src/main/java/org/apache/kafka/connect/data/Values.java#L305]
 

This appears accidental, because the case for String is explicitly handled, it 
just behaves poorly. Instead of the ClassCastException, the number should be 
parsed correctly.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16287) Implement example test for common rebalance callback scenarios

[Kirk True (Jira)]

Kirk True created KAFKA-16287:
-

 Summary: Implement example test for common rebalance callback 
scenarios
 Key: KAFKA-16287
 URL: https://issues.apache.org/jira/browse/KAFKA-16287
 Project: Kafka
  Issue Type: Test
  Components: clients, consumer
Reporter: Kirk True
Assignee: Lucas Brutschy
 Fix For: 3.8.0


There is justified concern that the new threading model may not play well with 
"tricky" {{ConsumerRebalanceListener}} callbacks. We need to provide some 
assurance that it will support complicated patterns.
 # Design and implement test scenarios
 # Update and document any design changes with the callback sub-system where 
needed
 # Provide fix(es) to the {{AsyncKafkaConsumer}} implementation to abide by 
said design



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-939: Support Participation in 2PC

[Jun Rao]

Hi, Artem,

Thanks for the reply.

20. "Say if an application
currently uses initTransactions() to achieve the current semantics, it
would need to be rewritten to use initTransactions() + abort to achieve the
same semantics if the config is changed. "

This only takes care of the abort case. The application still needs to be
changed to handle the commit case properly
if transaction.two.phase.commit.enable is set to true.

"Even when KIP-939 is implemented,
there would be situations when 2PC is disabled by the admin (e.g. Kafka
service providers may be reluctant to enable 2PC for Flink services that
users host themselves), so we either have to perpetuate the
reflection-based implementation in Flink or enable keepPreparedTxn=true
without 2PC."

Hmm, if the admin disables 2PC, there is likely a reason behind that. I am
not sure that we should provide an API to encourage the application to
circumvent that.

32. Ok. That's the kafka metric. In that case, the metric name has a group
and a name. There is no type and no package name.

Jun


On Thu, Feb 15, 2024 at 8:23 PM Artem Livshits
 wrote:

> Hi Jun,
>
> Thank you for your questions.
>
> > 20. So to abort a prepared transaction after the producer start, we could
> use ...
>
> I agree, initTransaction(true) + abort would accomplish the behavior of
> initTransactions(false), so we could technically have fewer ways to achieve
> the same thing, which is generally valuable.  I wonder, though, if that
> would be intuitive from the application perspective.  Say if an application
> currently uses initTransactions() to achieve the current semantics, it
> would need to be rewritten to use initTransactions() + abort to achieve the
> same semantics if the config is changed.  I think this could create
> subtle confusion, as the config change is generally decoupled from changing
> application implementation.
>
> >  The use case mentioned for keepPreparedTxn=true without 2PC doesn't seem
> very important
>
> I agree, it's not a strict requirement.  It is, however, a missing option
> in the public API, so currently Flink has to use reflection to emulate this
> functionality without 2PC support.   Even when KIP-939 is implemented,
> there would be situations when 2PC is disabled by the admin (e.g. Kafka
> service providers may be reluctant to enable 2PC for Flink services that
> users host themselves), so we either have to perpetuate the
> reflection-based implementation in Flink or enable keepPreparedTxn=true
> without 2PC.
>
> > 32.
>
> kafka.server:type=transaction-coordinator-metrics,name=active-transaction-open-time-max
>
> I just followed the existing metric implementation example
>
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/coordinator/transaction/TransactionStateManager.scala#L95
> ,
> which maps to
>
> kafka.server:type=transaction-coordinator-metrics,name=partition-load-time-max.
>
> > 33. "If the value is 'true' then the corresponding field is set
>
> That's correct.  Updated the KIP.
>
> -Artem
>
> On Wed, Feb 7, 2024 at 10:06 AM Jun Rao  wrote:
>
> > Hi, Artem,
> >
> > Thanks for the reply.
> >
> > 20. So to abort a prepared transaction after producer start, we could use
> > either
> >   producer.initTransactions(false)
> > or
> >   producer.initTransactions(true)
> >   producer.abortTransaction
> > Could we just always use the latter API? If we do this, we could
> > potentially eliminate the keepPreparedTxn flag in initTransactions().
> After
> > the initTransactions() call, the outstanding txn is always preserved if
> 2pc
> > is enabled and aborted if 2pc is disabled. The use case mentioned for
> > keepPreparedTxn=true without 2PC doesn't seem very important. If we could
> > do that, it seems that we have (1) less redundant and simpler APIs; (2)
> > more symmetric syntax for aborting/committing a prepared txn after
> producer
> > restart.
> >
> > 32.
> >
> >
> kafka.server:type=transaction-coordinator-metrics,name=active-transaction-open-time-max
> > Is this a Yammer or kafka metric? The former uses the camel case for name
> > and type. The latter uses the hyphen notation, but doesn't have the type
> > attribute.
> >
> > 33. "If the value is 'true' then the corresponding field is set in the
> > InitProducerIdRequest and the KafkaProducer object is set into a state
> > which only allows calling .commitTransaction or .abortTransaction."
> > We should also allow .completeTransaction, right?
> >
> > Jun
> >
> >
> > On Tue, Feb 6, 2024 at 3:29 PM Artem Livshits
> >  wrote:
> >
> > > Hi Jun,
> > >
> > > > 20. For Flink usage, it seems that the APIs used to abort and commit
> a
> > > prepared txn are not symmetric.
> > >
> > > For Flink it is expected that Flink would call .commitTransaction or
> > > .abortTransaction directly, it wouldn't need to deal with
> > PreparedTxnState,
> > > the outcome is actually determined by the Flink's job manager, not by
> > > comparison of PreparedTxnState.  So for Flink, if the Kafka sync
> crashes
> > > 

Re: [VOTE] 3.7.0 RC4

[Justine Olshan]

Hey folks,

I've done the following to validate the release:

-- validated the keys for all the artifacts
-- built from source and started a ZK cluster -- ran a few workloads on it.
-- ran 2.12 Kraft cluster and ran a few workloads on it

I see there is a lot of ongoing discussion about the upgrade notes. +1
(binding) from me given Mickael is voting +1 as well.

Justine

On Tue, Feb 20, 2024 at 6:18 AM Divij Vaidya 
wrote:

> > I am a bit unclear on the precise process regarding what parts of this
> get merged at what time, and whether the release first needs to be done or
> not.
>
> The order is as follows:
>
> 1. Release approved as part of this vote. After this we follow the
> steps from here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/Release+Process#ReleaseProcess-Afterthevotepasses
>
> 2. Upload artifacts to maven etc. These artifacts do not have RC suffix in
> them. You need a PMC member to mark these artifacts as "production" in
> apache svn.
> 3. Update website changes (docs, blog etc.). This is where your PRs
> on kafka-site repo get merged.
> 4. Send a release announcement by email.
>
> --
> Divij Vaidya
>
>
>
> On Tue, Feb 20, 2024 at 3:02 PM Stanislav Kozlovski
>  wrote:
>
> > Thanks for testing the release! And thanks for the review on the
> > documentation. Good catch on the license too.
> >
> > I have addressed the comments in the blog PR, and opened a few other PRs
> to
> > the website in relation to the release.
> >
> > - 37: Add download section for the latest 3.7 release
> > 
> > - 37: Update default docs to point to the 3.7.0 release docs
> > 
> > - 3.7: Add blog post for Kafka 3.7
> > 
> > - MINOR: Update stale upgrade_3_6_0 header links in documentation
> > 
> > - 37: Add upgrade notes for the 3.7.0 release
> > 
> >
> > I am a bit unclear on the precise process regarding what parts of this
> get
> > merged at what time, and whether the release first needs to be done or
> not.
> >
> > Best,
> > Stanislav
> >
> > On Mon, Feb 19, 2024 at 8:34 PM Divij Vaidya 
> > wrote:
> >
> > > Great. In that case we can fix the license issue retrospectively. I
> have
> > > created a JIRA for it
> https://issues.apache.org/jira/browse/KAFKA-16278
> > > and
> > > also updated the release process (which redirects to
> > > https://issues.apache.org/jira/browse/KAFKA-12622) to check for the
> > > correct
> > > license in both the kafka binaries.
> > >
> > > I am +1 (binding) assuming Mickael's concerns about update notes to 3.7
> > are
> > > addressed before release.
> > >
> > > --
> > > Divij Vaidya
> > >
> > >
> > >
> > > On Mon, Feb 19, 2024 at 6:08 PM Mickael Maison <
> mickael.mai...@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I agree with Josep, I don't think it's worth making a new RC just for
> > > this.
> > > >
> > > > Thanks Stanislav for sharing the test results. The last thing holding
> > > > me from casting my vote is the missing upgrade notes for 3.7.0.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > >
> > > >
> > > > On Mon, Feb 19, 2024 at 4:28 PM Josep Prat
>  > >
> > > > wrote:
> > > > >
> > > > > I think I remember finding a similar problem (NOTICE_binary) and it
> > > > didn't
> > > > > qualify for an extra RC
> > > > >
> > > > > Best,
> > > > >
> > > > > On Mon, Feb 19, 2024 at 3:44 PM Divij Vaidya <
> > divijvaidy...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > I have performed the following checks. The only thing I would
> like
> > to
> > > > call
> > > > > > out is the missing licenses before providing a vote. How do we
> want
> > > > > > to proceed on this? What have we done in the past? (Creating a
> new
> > RC
> > > > is
> > > > > > overkill IMO for this license issue).
> > > > > >
> > > > > > ## License check
> > > > > >
> > > > > > Test: Validate license of dependencies for both 2.12 & 2.13
> binary.
> > > > > > Result: Missing license for some scala* libraries specifically
> for
> > > > 2.12.
> > > > > > Seems like we have been missing these licenses for quite some
> > version
> > > > now.
> > > > > >
> > > > > > ```
> > > > > > for f in $(ls libs | grep -v "^kafka\|connect\|trogdor"); do if !
> > > grep
> > > > -q
> > > > > > ${f%.*} LICENSE; then echo "${f%.*} is missing in license file";
> > fi;
> > > > done
> > > > > > scala-collection-compat_2.12-2.10.0 is missing in license file
> > > > > > scala-java8-compat_2.12-1.0.2 is missing in license file
> > > > > > scala-library-2.12.18 is missing in license file
> > > > > > scala-logging_2.12-3.9.4 is missing in license file
> > > > > > scala-reflect-2.12.18 is missing in license file
> > > > > > ```
> > > > > >
> > > > > > ## Long running tests for memory leak (on ARM machine with zstd)
> > > > > >
> > > > > > Test: Run produce/consume for a few 

[jira] [Created] (KAFKA-16286) KRaft doesn't always notify listener of latest leader

[Jira]

José Armando García Sancio created KAFKA-16286:
--

 Summary: KRaft doesn't always notify listener of latest leader
 Key: KAFKA-16286
 URL: https://issues.apache.org/jira/browse/KAFKA-16286
 Project: Kafka
  Issue Type: Bug
  Components: kraft
Reporter: José Armando García Sancio
Assignee: José Armando García Sancio


If a listener register with RaftClient after the KRaft replica has transition 
to follower it will not get notified of the current leader until it has 
transitioned to another state.

In a stable cluster the listeners that are not the active leader (inactive 
controllers and brokers) will only get notified when then leader changes.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [PR] 37: remove jbod as missing feature in kraft [kafka-site]

[via GitHub]

divijvaidya commented on PR #584:
URL: https://github.com/apache/kafka-site/pull/584#issuecomment-1954810785

   Hey @gaurav-narula 
   Thanks for making this change but I think we already got it covered in 
https://github.com/apache/kafka-site/pull/578/files#diff-592961a45575efec39dbcc010a6e0c7f40c81a0afa61ad0c7d4a8fe13739f44a
 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #2659

[Apache Jenkins Server]

See 

[PR] 37: remove jbod as missing feature in kraft [kafka-site]

[via GitHub]

gaurav-narula opened a new pull request, #584:
URL: https://github.com/apache/kafka-site/pull/584

   Thanks @divijvaidya for pointing this out in 
[JIRA](https://issues.apache.org/jira/browse/KAFKA-14127?focusedCommentId=17816038=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17816038)!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] 3.7: Add blog post for Kafka 3.7 [kafka-site]

[via GitHub]

divijvaidya commented on code in PR #578:
URL: https://github.com/apache/kafka-site/pull/578#discussion_r1496098814


##
37/upgrade.html:
##
@@ -18,8 +18,85 @@
 
 
 

Re: [DISCUSS] KIP-932: Queues for Kafka

[Andrew Schofield]

Hi Manikumar,
Thanks for your comments.

1. I believe that in general, there are not situations in which a dynamic config
change is prevented because of the existence of a resource. So, if we prevented
setting config `group.type=consumer` on resource G of GROUP type
if there was a share group G in existence, it would be a bit weird.

I wonder whether changing the config name to `new.group.type` would help. It’s
ensuring the type of a new group created.

2. The behaviour for a DEAD share group is intended to be the same as a DEAD
consumer group. The group cannot be “reused” again as such, but the group ID
can be used by a new group.

3. Yes. AlterShareGroupOffsets will cause a new SHARE_CHECKPOINT.

4. In common with Admin.deleteConsumerGroups, the underlying Kafka RPC
for Admin.deleteShareGroups is DeleteGroups. This is handled by the group
coordinator and it does this by writing control records (a tombstone in this 
case).
The KIP doesn’t say anything about this because it’s the same as consumer 
groups.
Perhaps it would be sensible to add a GroupType to DeleteGroupsRequest so we can
make sure we are deleting the correct type of group. The fact that there is not 
a specific
RPC for DeleteShareGroups seems correct to me.

5. I prefer using “o.a.k.clients.consumer” because it’s already a public 
package and
many of the classes and interfaces such as ConsumerRecord are in that package.

I definitely need to add more information about how the Admin operations work.
I will add a section to the KIP in the next version, later today. This will 
fill in details for
your questions (3) and (4).

Thanks,
Andrew

> On 14 Feb 2024, at 18:04, Manikumar  wrote:
> 
> Hi Andrew,
> 
> Thanks for the KIP. A few comments below.
> 
> 1. kafka-configs.sh (incrementalAlterConfigs) allows you to dynamically
> change the configs. Maybe in this case, we should not allow the user to
> change `group.type` if it's already set.
> 2. What's the behaviour after a group transitions into DEAD state. Do we
> add new control records to reset the state? Can we reuse the group again?
> 3. Are we going to write new control records after the
> AlterShareGroupOffsets API to reset the state?
> 4. Is there any API for DeleteShareGroups? I assume, group co-ordinator is
> going to handle the API. If so, Does this mean the group co-ordinator also
> needs to write control records?
> 5. How about using "org.apache.kafka.clients.consumer.share" package for
> new interfaces/classes?
> 
> 
> Thanks,
> Manikumar

Re: [PR] 3.7: Add blog post for Kafka 3.7 [kafka-site]

[via GitHub]

soarez commented on code in PR #578:
URL: https://github.com/apache/kafka-site/pull/578#discussion_r1495950924


##
37/upgrade.html:
##
@@ -18,8 +18,85 @@
 
 
 

Re: [VOTE] 3.7.0 RC4

[Divij Vaidya]

> I am a bit unclear on the precise process regarding what parts of this
get merged at what time, and whether the release first needs to be done or
not.

The order is as follows:

1. Release approved as part of this vote. After this we follow the
steps from here:
https://cwiki.apache.org/confluence/display/KAFKA/Release+Process#ReleaseProcess-Afterthevotepasses

2. Upload artifacts to maven etc. These artifacts do not have RC suffix in
them. You need a PMC member to mark these artifacts as "production" in
apache svn.
3. Update website changes (docs, blog etc.). This is where your PRs
on kafka-site repo get merged.
4. Send a release announcement by email.

--
Divij Vaidya



On Tue, Feb 20, 2024 at 3:02 PM Stanislav Kozlovski
 wrote:

> Thanks for testing the release! And thanks for the review on the
> documentation. Good catch on the license too.
>
> I have addressed the comments in the blog PR, and opened a few other PRs to
> the website in relation to the release.
>
> - 37: Add download section for the latest 3.7 release
> 
> - 37: Update default docs to point to the 3.7.0 release docs
> 
> - 3.7: Add blog post for Kafka 3.7
> 
> - MINOR: Update stale upgrade_3_6_0 header links in documentation
> 
> - 37: Add upgrade notes for the 3.7.0 release
> 
>
> I am a bit unclear on the precise process regarding what parts of this get
> merged at what time, and whether the release first needs to be done or not.
>
> Best,
> Stanislav
>
> On Mon, Feb 19, 2024 at 8:34 PM Divij Vaidya 
> wrote:
>
> > Great. In that case we can fix the license issue retrospectively. I have
> > created a JIRA for it https://issues.apache.org/jira/browse/KAFKA-16278
> > and
> > also updated the release process (which redirects to
> > https://issues.apache.org/jira/browse/KAFKA-12622) to check for the
> > correct
> > license in both the kafka binaries.
> >
> > I am +1 (binding) assuming Mickael's concerns about update notes to 3.7
> are
> > addressed before release.
> >
> > --
> > Divij Vaidya
> >
> >
> >
> > On Mon, Feb 19, 2024 at 6:08 PM Mickael Maison  >
> > wrote:
> >
> > > Hi,
> > >
> > > I agree with Josep, I don't think it's worth making a new RC just for
> > this.
> > >
> > > Thanks Stanislav for sharing the test results. The last thing holding
> > > me from casting my vote is the missing upgrade notes for 3.7.0.
> > >
> > > Thanks,
> > > Mickael
> > >
> > >
> > >
> > > On Mon, Feb 19, 2024 at 4:28 PM Josep Prat  >
> > > wrote:
> > > >
> > > > I think I remember finding a similar problem (NOTICE_binary) and it
> > > didn't
> > > > qualify for an extra RC
> > > >
> > > > Best,
> > > >
> > > > On Mon, Feb 19, 2024 at 3:44 PM Divij Vaidya <
> divijvaidy...@gmail.com>
> > > > wrote:
> > > >
> > > > > I have performed the following checks. The only thing I would like
> to
> > > call
> > > > > out is the missing licenses before providing a vote. How do we want
> > > > > to proceed on this? What have we done in the past? (Creating a new
> RC
> > > is
> > > > > overkill IMO for this license issue).
> > > > >
> > > > > ## License check
> > > > >
> > > > > Test: Validate license of dependencies for both 2.12 & 2.13 binary.
> > > > > Result: Missing license for some scala* libraries specifically for
> > > 2.12.
> > > > > Seems like we have been missing these licenses for quite some
> version
> > > now.
> > > > >
> > > > > ```
> > > > > for f in $(ls libs | grep -v "^kafka\|connect\|trogdor"); do if !
> > grep
> > > -q
> > > > > ${f%.*} LICENSE; then echo "${f%.*} is missing in license file";
> fi;
> > > done
> > > > > scala-collection-compat_2.12-2.10.0 is missing in license file
> > > > > scala-java8-compat_2.12-1.0.2 is missing in license file
> > > > > scala-library-2.12.18 is missing in license file
> > > > > scala-logging_2.12-3.9.4 is missing in license file
> > > > > scala-reflect-2.12.18 is missing in license file
> > > > > ```
> > > > >
> > > > > ## Long running tests for memory leak (on ARM machine with zstd)
> > > > >
> > > > > Test: Run produce/consume for a few hours and verify no gradual
> > > increase in
> > > > > heap.
> > > > > Result: No heap increase observed. The overall CPU utilization is
> > lower
> > > > > compared to 3.5.1.
> > > > >
> > > > > ## Verify system test results
> > > > >
> > > > > Test: Spot check the results of system tests.
> > > > > Result: I have verified that the system tests are passing across
> > > different
> > > > > runs.
> > > > >
> > > > > --
> > > > > Divij Vaidya
> > > > >
> > > > >
> > > > >
> > > > > On Sun, Feb 18, 2024 at 2:50 PM Stanislav Kozlovski
> > > > >  wrote:
> > > > >
> > > > > > The latest system test build completed successfully -
> > > > > >
> > > > > >
> > > > >
> > >
> >
> 

Re: [PR] 37: Add upgrade notes for the 3.7.0 release [kafka-site]

[via GitHub]

divijvaidya commented on code in PR #581:
URL: https://github.com/apache/kafka-site/pull/581#discussion_r1495890168


##
37/upgrade.html:
##
@@ -18,8 +18,85 @@
 
 
 

Re: [VOTE] 3.7.0 RC4

[Stanislav Kozlovski]

Thanks for testing the release! And thanks for the review on the
documentation. Good catch on the license too.

I have addressed the comments in the blog PR, and opened a few other PRs to
the website in relation to the release.

- 37: Add download section for the latest 3.7 release

- 37: Update default docs to point to the 3.7.0 release docs

- 3.7: Add blog post for Kafka 3.7

- MINOR: Update stale upgrade_3_6_0 header links in documentation

- 37: Add upgrade notes for the 3.7.0 release


I am a bit unclear on the precise process regarding what parts of this get
merged at what time, and whether the release first needs to be done or not.

Best,
Stanislav

On Mon, Feb 19, 2024 at 8:34 PM Divij Vaidya 
wrote:

> Great. In that case we can fix the license issue retrospectively. I have
> created a JIRA for it https://issues.apache.org/jira/browse/KAFKA-16278
> and
> also updated the release process (which redirects to
> https://issues.apache.org/jira/browse/KAFKA-12622) to check for the
> correct
> license in both the kafka binaries.
>
> I am +1 (binding) assuming Mickael's concerns about update notes to 3.7 are
> addressed before release.
>
> --
> Divij Vaidya
>
>
>
> On Mon, Feb 19, 2024 at 6:08 PM Mickael Maison 
> wrote:
>
> > Hi,
> >
> > I agree with Josep, I don't think it's worth making a new RC just for
> this.
> >
> > Thanks Stanislav for sharing the test results. The last thing holding
> > me from casting my vote is the missing upgrade notes for 3.7.0.
> >
> > Thanks,
> > Mickael
> >
> >
> >
> > On Mon, Feb 19, 2024 at 4:28 PM Josep Prat 
> > wrote:
> > >
> > > I think I remember finding a similar problem (NOTICE_binary) and it
> > didn't
> > > qualify for an extra RC
> > >
> > > Best,
> > >
> > > On Mon, Feb 19, 2024 at 3:44 PM Divij Vaidya 
> > > wrote:
> > >
> > > > I have performed the following checks. The only thing I would like to
> > call
> > > > out is the missing licenses before providing a vote. How do we want
> > > > to proceed on this? What have we done in the past? (Creating a new RC
> > is
> > > > overkill IMO for this license issue).
> > > >
> > > > ## License check
> > > >
> > > > Test: Validate license of dependencies for both 2.12 & 2.13 binary.
> > > > Result: Missing license for some scala* libraries specifically for
> > 2.12.
> > > > Seems like we have been missing these licenses for quite some version
> > now.
> > > >
> > > > ```
> > > > for f in $(ls libs | grep -v "^kafka\|connect\|trogdor"); do if !
> grep
> > -q
> > > > ${f%.*} LICENSE; then echo "${f%.*} is missing in license file"; fi;
> > done
> > > > scala-collection-compat_2.12-2.10.0 is missing in license file
> > > > scala-java8-compat_2.12-1.0.2 is missing in license file
> > > > scala-library-2.12.18 is missing in license file
> > > > scala-logging_2.12-3.9.4 is missing in license file
> > > > scala-reflect-2.12.18 is missing in license file
> > > > ```
> > > >
> > > > ## Long running tests for memory leak (on ARM machine with zstd)
> > > >
> > > > Test: Run produce/consume for a few hours and verify no gradual
> > increase in
> > > > heap.
> > > > Result: No heap increase observed. The overall CPU utilization is
> lower
> > > > compared to 3.5.1.
> > > >
> > > > ## Verify system test results
> > > >
> > > > Test: Spot check the results of system tests.
> > > > Result: I have verified that the system tests are passing across
> > different
> > > > runs.
> > > >
> > > > --
> > > > Divij Vaidya
> > > >
> > > >
> > > >
> > > > On Sun, Feb 18, 2024 at 2:50 PM Stanislav Kozlovski
> > > >  wrote:
> > > >
> > > > > The latest system test build completed successfully -
> > > > >
> > > > >
> > > >
> >
> https://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/system-test-kafka-branch-builder--1708250728--apache--3.7--02197edaaa/2024-02-18--001./2024-02-18--001./report.html
> > > > >
> > > > > *System tests are therefore all good*. We just have some flakes
> > > > >
> > > > > On Sun, Feb 18, 2024 at 10:45 AM Stanislav Kozlovski <
> > > > > stanis...@confluent.io>
> > > > > wrote:
> > > > >
> > > > > > The upgrade test passed ->
> > > > > >
> > > > >
> > > >
> >
> https://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/system-test-kafka-branch-builder--1708103771--apache--3.7--bb6990114b/2024-02-16--001./2024-02-16--001./report.html
> > > > > >
> > > > > > The replica verification test succeeded in ZK mode, but failed in
> > > > > > ISOLATED_KRAFT. It just seems to be very flaky.
> > > > > >
> > > > >
> > > >
> >
> https://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/system-test-kafka-branch-builder--1708100119--apache--3.7--bb6990114b/2024-02-16--001./2024-02-16--001./report.html
> > > > > >
> > 

Re: [PR] 37: Add download section for the latest 3.7 release [kafka-site]

[via GitHub]

divijvaidya commented on PR #583:
URL: https://github.com/apache/kafka-site/pull/583#issuecomment-1954277178

   > But I assume also we have time between the actual release and the 
announcement.
   
   Yes, we can publish this PR *after* uploading artifacts to maven etc.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] 37: Add download section for the latest 3.7 release [kafka-site]

[via GitHub]

divijvaidya commented on code in PR #583:
URL: https://github.com/apache/kafka-site/pull/583#discussion_r1495864194


##
downloads.html:
##
@@ -6,12 +6,45 @@

 Download
 
-3.6.1 is the latest release. The current stable version is 3.6.1
+3.7.0 is the latest release. The current stable version is 3.7.0
 
 
 You can verify your download by following these https://www.apache.org/info/verification.html;>procedures and using 
these https://downloads.apache.org/kafka/KEYS;>KEYS.
 
 
+
+
+3.7.0
+
+
+Released Feb 21, 2024
+
+
+https://downloads.apache.org/kafka/3.7.0/RELEASE_NOTES.html;>Release 
Notes
+
+
+Docker image: https://hub.docker.com/layers/apache/kafka/3.7.0-rc4/images/sha256-3e324d2bd331570676436b24f625e5dcf1facdfbd62efcffabc6b69b1abc13cc?context=explore;>apache/kafka:3.7.0.
+
+
+Source download: https://downloads.apache.org/kafka/3.7.0/kafka-3.7.0-src.tgz;>kafka-3.7.0-src.tgz
 (https://downloads.apache.org/kafka/3.7.0/kafka-3.7.0-src.tgz.asc;>asc,
 https://downloads.apache.org/kafka/3.7.0/kafka-3.7.0-src.tgz.sha512;>sha512)
+
+
+Binary downloads:
+
+Scala 2.12 - https://downloads.apache.org/kafka/3.7.0/kafka_2.12-3.7.0.tgz;>kafka_2.12-3.7.0.tgz
 (https://downloads.apache.org/kafka/3.7.0/kafka_2.12-3.7.0.tgz.asc;>asc,
 https://downloads.apache.org/kafka/3.7.0/kafka_2.12-3.7.0.tgz.sha512;>sha512)
+Scala 2.13 - https://downloads.apache.org/kafka/3.7.0/kafka_2.13-3.7.0.tgz;>kafka_2.13-3.7.0.tgz
 (https://downloads.apache.org/kafka/3.7.0/kafka_2.13-3.7.0.tgz.asc;>asc,
 https://downloads.apache.org/kafka/3.7.0/kafka_2.13-3.7.0.tgz.sha512;>sha512)
+
+We build for multiple versions of Scala. This only matters if you 
are using Scala and you want a version
+built for the same Scala version you use. Otherwise any version 
should work (2.13 is recommended).
+
+
+
+
+Kafka 3.7.0 includes a significant number of new features and fixes.
+For more information, please read our https://kafka.apache.org/blog#apache_kafka_360_release_announcement; 
target="_blank">blog post

Review Comment:
   typo: this should point to 370 release blog



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] 3.7: Add blog post for Kafka 3.7 [kafka-site]

[via GitHub]

divijvaidya commented on code in PR #578:
URL: https://github.com/apache/kafka-site/pull/578#discussion_r1495858647


##
blog.html:
##
@@ -22,6 +22,125 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.7.0 Release Announcement
+
+February 2024 - Stanislav Kozlovski (https://twitter.com/BdKozlovski;>@BdKozlovski)
+We are proud to announce the release of Apache Kafka 3.7.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.7.0/RELEASE_NOTES.html;>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_7_0;>Upgrading 
to 3.7.0 from any version 0.8.x through 3.6.x section in the documentation 
for the list of notable changes and detailed upgrade steps.
+
+In the last release, 3.6,
+https://kafka.apache.org/documentation/#kraft_zk_migration;>the ability 
to migrate Kafka clusters from a ZooKeeper metadata system
+to a KRaft metadata system was ready for usage in 
production environments with one caveat -- JBOD was not yet available for KRaft 
clusters.
+In this release, we are shipping an early access release 
of JBOD in KRaft. (See https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft;>KIP-858
 for details)
+
+
+Additionally, client APIs released prior to Apache Kafka 
2.1 are now marked deprecated in 3.7 and will be removed in Apache Kafka 4.0. 
See https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0;>KIP-896
 for details and RPC versions that are now deprecated.
+
+
+Java 11 support for the Kafka broker is also marked 
deprecated in 3.7, and is planned to be removed in Kafka 4.0. See https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510;>KIP-1013
 for more details
+
+
+Note: ZooKeeper is marked as deprecated since the 3.5.0 
release. ZooKeeper is planned to be removed in Apache Kafka 4.0. For more 
information, please see the documentation for https://kafka.apache.org/documentation/#zk_depr;>ZooKeeper 
Deprecation.
+
+
+Kafka Broker, Controller, Producer, Consumer and Admin 
Client
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft;>(Early
 Access) KIP-858 Handle JBOD broker disk failure in KRaft:
+This update closes the gap on one of the last 
major missing features in KRaft by adding JBOD support in KRaft-based clusters.
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability;>KIP-714
 Client metrics and observability:
+With KIP-714, operators get better visibility 
into the clients connecting to their cluster with broker-side support of 
client-level metrics via a standardized telemetry interface.
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-1000%3A+List+Client+Metrics+Configuration+Resources;>KIP-1000
 List Client Metrics Configuration Resources:
+KIP-1000 supports KIP-714 by introducing a way 
to create, read, update, and delete the client metrics configuration resources 
using the existing RPCs and the kafka-configs.sh tool.
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol;>(Early
 Access) KIP-848 The Next Generation of the Consumer Rebalance Protocol:
+The new simplified Consumer Rebalance Protocol 
moves complexity away from the consumer and into the Group Coordinator within 
the broker and completely revamps the protocol to be incremental in nature. It 
provides the same guarantee as the current protocol––but better and more 
efficient, including no longer relying on a global synchronization barrier. https://cwiki.apache.org/confluence/display/KAFKA/The+Next+Generation+of+the+Consumer+Rebalance+Protocol+%28KIP-848%29+-+Early+Access+Release+Notes;>See
 the early access release notes for more information.
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-951%3A+Leader+discovery+optimisations+for+the+client;>KIP-951
 Leader discovery optimisations for the client:
+KIP-951 optimizes the time it takes for a 
client to discover the new leader of a partition, 

Re: [PR] 37: Add download section for the latest 3.7 release [kafka-site]

[via GitHub]

stanislavkozlovski commented on code in PR #583:
URL: https://github.com/apache/kafka-site/pull/583#discussion_r1495863880


##
downloads.html:
##
@@ -6,12 +6,45 @@

 Download
 
-3.6.1 is the latest release. The current stable version is 3.6.1
+3.7.0 is the latest release. The current stable version is 3.7.0
 
 
 You can verify your download by following these https://www.apache.org/info/verification.html;>procedures and using 
these https://downloads.apache.org/kafka/KEYS;>KEYS.
 
 
+
+
+3.7.0
+
+
+Released Feb 21, 2024
+
+
+https://downloads.apache.org/kafka/3.7.0/RELEASE_NOTES.html;>Release 
Notes
+
+
+Docker image: https://hub.docker.com/layers/apache/kafka/3.7.0-rc4/images/sha256-3e324d2bd331570676436b24f625e5dcf1facdfbd62efcffabc6b69b1abc13cc?context=explore;>apache/kafka:3.7.0.

Review Comment:
   this needs to get updated 100%, since I don't have the image hash yet



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] 37: Add download section for the latest 3.7 release [kafka-site]

[via GitHub]

stanislavkozlovski commented on PR #583:
URL: https://github.com/apache/kafka-site/pull/583#issuecomment-1954264735

   so far these links don't exist and i haven't verified any of them, since the 
release is not done.
   
   Not sure what the actual process is - whether we:
   1) pre-approve, and I change the links & merge once finished releasing
   2) wait for the release, update this, and then get an approval and merge
   
   I would assume 1) is more flexible and preferable, since it allows me to 
update the website right after the release (avoiding any inconsistency). But I 
assume also we have time between the actual release and the announcement.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] 37: Add download section for the latest 3.7 release [kafka-site]

[via GitHub]

stanislavkozlovski commented on code in PR #583:
URL: https://github.com/apache/kafka-site/pull/583#discussion_r1495863565


##
downloads.html:
##
@@ -6,12 +6,45 @@

 Download
 
-3.6.1 is the latest release. The current stable version is 3.6.1
+3.7.0 is the latest release. The current stable version is 3.7.0
 
 
 You can verify your download by following these https://www.apache.org/info/verification.html;>procedures and using 
these https://downloads.apache.org/kafka/KEYS;>KEYS.
 
 
+
+
+3.7.0
+
+
+Released Feb 21, 2024

Review Comment:
   This will change to the date of the **release** announcement



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] 3.7: Add blog post for Kafka 3.7 [kafka-site]

[via GitHub]

divijvaidya commented on code in PR #578:
URL: https://github.com/apache/kafka-site/pull/578#discussion_r1495858647


##
blog.html:
##
@@ -22,6 +22,125 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.7.0 Release Announcement
+
+February 2024 - Stanislav Kozlovski (https://twitter.com/BdKozlovski;>@BdKozlovski)
+We are proud to announce the release of Apache Kafka 3.7.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.7.0/RELEASE_NOTES.html;>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_7_0;>Upgrading 
to 3.7.0 from any version 0.8.x through 3.6.x section in the documentation 
for the list of notable changes and detailed upgrade steps.
+
+In the last release, 3.6,
+https://kafka.apache.org/documentation/#kraft_zk_migration;>the ability 
to migrate Kafka clusters from a ZooKeeper metadata system
+to a KRaft metadata system was ready for usage in 
production environments with one caveat -- JBOD was not yet available for KRaft 
clusters.
+In this release, we are shipping an early access release 
of JBOD in KRaft. (See https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft;>KIP-858
 for details)
+
+
+Additionally, client APIs released prior to Apache Kafka 
2.1 are now marked deprecated in 3.7 and will be removed in Apache Kafka 4.0. 
See https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0;>KIP-896
 for details and RPC versions that are now deprecated.
+
+
+Java 11 support for the Kafka broker is also marked 
deprecated in 3.7, and is planned to be removed in Kafka 4.0. See https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510;>KIP-1013
 for more details
+
+
+Note: ZooKeeper is marked as deprecated since the 3.5.0 
release. ZooKeeper is planned to be removed in Apache Kafka 4.0. For more 
information, please see the documentation for https://kafka.apache.org/documentation/#zk_depr;>ZooKeeper 
Deprecation.
+
+
+Kafka Broker, Controller, Producer, Consumer and Admin 
Client
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft;>(Early
 Access) KIP-858 Handle JBOD broker disk failure in KRaft:
+This update closes the gap on one of the last 
major missing features in KRaft by adding JBOD support in KRaft-based clusters.
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability;>KIP-714
 Client metrics and observability:
+With KIP-714, operators get better visibility 
into the clients connecting to their cluster with broker-side support of 
client-level metrics via a standardized telemetry interface.
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-1000%3A+List+Client+Metrics+Configuration+Resources;>KIP-1000
 List Client Metrics Configuration Resources:
+KIP-1000 supports KIP-714 by introducing a way 
to create, read, update, and delete the client metrics configuration resources 
using the existing RPCs and the kafka-configs.sh tool.
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol;>(Early
 Access) KIP-848 The Next Generation of the Consumer Rebalance Protocol:
+The new simplified Consumer Rebalance Protocol 
moves complexity away from the consumer and into the Group Coordinator within 
the broker and completely revamps the protocol to be incremental in nature. It 
provides the same guarantee as the current protocol––but better and more 
efficient, including no longer relying on a global synchronization barrier. https://cwiki.apache.org/confluence/display/KAFKA/The+Next+Generation+of+the+Consumer+Rebalance+Protocol+%28KIP-848%29+-+Early+Access+Release+Notes;>See
 the early access release notes for more information.
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-951%3A+Leader+discovery+optimisations+for+the+client;>KIP-951
 Leader discovery optimisations for the client:
+KIP-951 optimizes the time it takes for a 
client to discover the new leader of a partition, 

[PR] 37: Update default docs to point to the 3.7.0 release docs [kafka-site]

[via GitHub]

stanislavkozlovski opened a new pull request, #582:
URL: https://github.com/apache/kafka-site/pull/582

   This patch updates the default documentation links to link to the 3.7 
documentation


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] 3.7: Add blog post for Kafka 3.7 [kafka-site]

[via GitHub]

stanislavkozlovski commented on code in PR #578:
URL: https://github.com/apache/kafka-site/pull/578#discussion_r1495828306


##
blog.html:
##
@@ -22,6 +22,125 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.7.0 Release Announcement
+
+February 2024 - Stanislav Kozlovski (https://twitter.com/BdKozlovski;>@BdKozlovski)
+We are proud to announce the release of Apache Kafka 3.7.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.7.0/RELEASE_NOTES.html;>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_7_0;>Upgrading 
to 3.7.0 from any version 0.8.x through 3.6.x section in the documentation 
for the list of notable changes and detailed upgrade steps.
+
+In the last release, 3.6,
+https://kafka.apache.org/documentation/#kraft_zk_migration;>the ability 
to migrate Kafka clusters from a ZooKeeper metadata system
+to a KRaft metadata system was ready for usage in 
production environments with one caveat -- JBOD was not yet available for KRaft 
clusters.
+In this release, we are shipping an early access release 
of JBOD in KRaft. (See https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft;>KIP-858
 for details)
+
+
+Additionally, client APIs released prior to Apache Kafka 
2.1 are now marked deprecated in 3.7 and will be removed in Apache Kafka 4.0. 
See https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0;>KIP-896
 for details and RPC versions that are now deprecated.
+
+
+Java 11 support for the Kafka broker is also marked 
deprecated in 3.7, and is planned to be removed in Kafka 4.0. See https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510;>KIP-1013
 for more details
+
+
+Note: ZooKeeper is marked as deprecated since the 3.5.0 
release. ZooKeeper is planned to be removed in Apache Kafka 4.0. For more 
information, please see the documentation for https://kafka.apache.org/documentation/#zk_depr;>ZooKeeper 
Deprecation.
+
+
+Kafka Broker, Controller, Producer, Consumer and Admin 
Client
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft;>(Early
 Access) KIP-858 Handle JBOD broker disk failure in KRaft:
+This update closes the gap on one of the last 
major missing features in KRaft by adding JBOD support in KRaft-based clusters.
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability;>KIP-714
 Client metrics and observability:
+With KIP-714, operators get better visibility 
into the clients connecting to their cluster with broker-side support of 
client-level metrics via a standardized telemetry interface.
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-1000%3A+List+Client+Metrics+Configuration+Resources;>KIP-1000
 List Client Metrics Configuration Resources:
+KIP-1000 supports KIP-714 by introducing a way 
to create, read, update, and delete the client metrics configuration resources 
using the existing RPCs and the kafka-configs.sh tool.
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol;>(Early
 Access) KIP-848 The Next Generation of the Consumer Rebalance Protocol:
+The new simplified Consumer Rebalance Protocol 
moves complexity away from the consumer and into the Group Coordinator within 
the broker and completely revamps the protocol to be incremental in nature. It 
provides the same guarantee as the current protocol––but better and more 
efficient, including no longer relying on a global synchronization barrier. https://cwiki.apache.org/confluence/display/KAFKA/The+Next+Generation+of+the+Consumer+Rebalance+Protocol+%28KIP-848%29+-+Early+Access+Release+Notes;>See
 the early access release notes for more information.
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-951%3A+Leader+discovery+optimisations+for+the+client;>KIP-951
 Leader discovery optimisations for the client:
+KIP-951 optimizes the time it takes for a 
client to discover the new leader of a 

Re: [PR] 3.7: Add blog post for Kafka 3.7 [kafka-site]

[via GitHub]

stanislavkozlovski commented on code in PR #578:
URL: https://github.com/apache/kafka-site/pull/578#discussion_r1495827960


##
blog.html:
##
@@ -22,6 +22,125 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.7.0 Release Announcement
+
+February 2024 - Stanislav Kozlovski (https://twitter.com/BdKozlovski;>@BdKozlovski)
+We are proud to announce the release of Apache Kafka 3.7.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.7.0/RELEASE_NOTES.html;>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_7_0;>Upgrading 
to 3.7.0 from any version 0.8.x through 3.6.x section in the documentation 
for the list of notable changes and detailed upgrade steps.
+
+In the last release, 3.6,
+https://kafka.apache.org/documentation/#kraft_zk_migration;>the ability 
to migrate Kafka clusters from a ZooKeeper metadata system
+to a KRaft metadata system was ready for usage in 
production environments with one caveat -- JBOD was not yet available for KRaft 
clusters.
+In this release, we are shipping an early access release 
of JBOD in KRaft. (See https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft;>KIP-858
 for details)
+
+
+Additionally, client APIs released prior to Apache Kafka 
2.1 are now marked deprecated in 3.7 and will be removed in Apache Kafka 4.0. 
See https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0;>KIP-896
 for details and RPC versions that are now deprecated.
+
+
+Java 11 support for the Kafka broker is also marked 
deprecated in 3.7, and is planned to be removed in Kafka 4.0. See https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510;>KIP-1013
 for more details
+
+
+Note: ZooKeeper is marked as deprecated since the 3.5.0 
release. ZooKeeper is planned to be removed in Apache Kafka 4.0. For more 
information, please see the documentation for https://kafka.apache.org/documentation/#zk_depr;>ZooKeeper 
Deprecation.
+
+
+Kafka Broker, Controller, Producer, Consumer and Admin 
Client
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft;>(Early
 Access) KIP-858 Handle JBOD broker disk failure in KRaft:
+This update closes the gap on one of the last 
major missing features in KRaft by adding JBOD support in KRaft-based clusters.

Review Comment:
   great point! modified it to `Supporting JBOD configurations with 
multiple storage directories in production environments. Note that an Early 
Access release is supported in 3.7 as per https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft;>KIP-858`
 and will update it once we have some clarity regarding 
https://github.com/apache/kafka-site/pull/578#discussion_r1495481232 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] 3.7: Add blog post for Kafka 3.7 [kafka-site]

[via GitHub]

stanislavkozlovski commented on code in PR #578:
URL: https://github.com/apache/kafka-site/pull/578#discussion_r1495824410


##
blog.html:
##
@@ -22,6 +22,125 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.7.0 Release Announcement
+
+February 2024 - Stanislav Kozlovski (https://twitter.com/BdKozlovski;>@BdKozlovski)
+We are proud to announce the release of Apache Kafka 3.7.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.7.0/RELEASE_NOTES.html;>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_7_0;>Upgrading 
to 3.7.0 from any version 0.8.x through 3.6.x section in the documentation 
for the list of notable changes and detailed upgrade steps.
+
+In the last release, 3.6,
+https://kafka.apache.org/documentation/#kraft_zk_migration;>the ability 
to migrate Kafka clusters from a ZooKeeper metadata system
+to a KRaft metadata system was ready for usage in 
production environments with one caveat -- JBOD was not yet available for KRaft 
clusters.
+In this release, we are shipping an early access release 
of JBOD in KRaft. (See https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft;>KIP-858
 for details)
+
+
+Additionally, client APIs released prior to Apache Kafka 
2.1 are now marked deprecated in 3.7 and will be removed in Apache Kafka 4.0. 
See https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0;>KIP-896
 for details and RPC versions that are now deprecated.
+
+
+Java 11 support for the Kafka broker is also marked 
deprecated in 3.7, and is planned to be removed in Kafka 4.0. See https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510;>KIP-1013
 for more details
+
+
+Note: ZooKeeper is marked as deprecated since the 3.5.0 
release. ZooKeeper is planned to be removed in Apache Kafka 4.0. For more 
information, please see the documentation for https://kafka.apache.org/documentation/#zk_depr;>ZooKeeper 
Deprecation.
+
+
+Kafka Broker, Controller, Producer, Consumer and Admin 
Client
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft;>(Early
 Access) KIP-858 Handle JBOD broker disk failure in KRaft:
+This update closes the gap on one of the last 
major missing features in KRaft by adding JBOD support in KRaft-based clusters.
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability;>KIP-714
 Client metrics and observability:
+With KIP-714, operators get better visibility 
into the clients connecting to their cluster with broker-side support of 
client-level metrics via a standardized telemetry interface.
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-1000%3A+List+Client+Metrics+Configuration+Resources;>KIP-1000
 List Client Metrics Configuration Resources:
+KIP-1000 supports KIP-714 by introducing a way 
to create, read, update, and delete the client metrics configuration resources 
using the existing RPCs and the kafka-configs.sh tool.
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol;>(Early
 Access) KIP-848 The Next Generation of the Consumer Rebalance Protocol:
+The new simplified Consumer Rebalance Protocol 
moves complexity away from the consumer and into the Group Coordinator within 
the broker and completely revamps the protocol to be incremental in nature. It 
provides the same guarantee as the current protocol––but better and more 
efficient, including no longer relying on a global synchronization barrier. https://cwiki.apache.org/confluence/display/KAFKA/The+Next+Generation+of+the+Consumer+Rebalance+Protocol+%28KIP-848%29+-+Early+Access+Release+Notes;>See
 the early access release notes for more information.
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-951%3A+Leader+discovery+optimisations+for+the+client;>KIP-951
 Leader discovery optimisations for the client:
+KIP-951 optimizes the time it takes for a 
client to discover the new leader of a 

Re: [PR] MINOR: Update stale `upgrade_3_6_0` header links in documentation [kafka-site]

[via GitHub]

mimaison commented on code in PR #580:
URL: https://github.com/apache/kafka-site/pull/580#discussion_r1495809050


##
blog.html:
##
@@ -55,7 +55,7 @@ 
 
 10 Oct 2023 - Satish Duggana (https://twitter.com/0xeed;>@SatishDuggana)
 We are proud to announce the release of Apache Kafka 3.6.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.6.0/RELEASE_NOTES.html;>release 
notes.
-See the https://kafka.apache.org/36/documentation.html#upgrade_3_6_0;>Upgrading 
to 3.6.0 from any version 0.8.x through 3.5.x section in the documentation 
for the list of notable changes and detailed upgrade steps.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_6_1;>Upgrading 
to 3.6.0 from any version 0.8.x through 3.5.x section in the documentation 
for the list of notable changes and detailed upgrade steps.

Review Comment:
   Let's update the title as well `Upgrading to 3.6.0` -> `Upgrading to 3.6.1`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] MINOR: Update stale `upgrade_3_6_0` header links in documentation [kafka-site]

[via GitHub]

mimaison commented on code in PR #580:
URL: https://github.com/apache/kafka-site/pull/580#discussion_r1495807353


##
37/upgrade.html:
##
@@ -19,9 +19,9 @@
 
 

Re: [PR] 37: Add upgrade notes for the 3.7.0 releaase [kafka-site]

[via GitHub]

stanislavkozlovski commented on code in PR #581:
URL: https://github.com/apache/kafka-site/pull/581#discussion_r1495781312


##
37/upgrade.html:
##
@@ -18,8 +18,85 @@
 
 
 

[jira] [Created] (KAFKA-16285) Make group metadata available when a new assignment is set in async Kafka consumer

[Bruno Cadonna (Jira)]

Bruno Cadonna created KAFKA-16285:
-

 Summary: Make group metadata available when a new assignment is 
set in async Kafka consumer
 Key: KAFKA-16285
 URL: https://issues.apache.org/jira/browse/KAFKA-16285
 Project: Kafka
  Issue Type: Improvement
  Components: clients, consumer
Reporter: Bruno Cadonna


Currently, the new async Kafka consumer sends an event from the background 
thread to the application thread when the group metadata is updated. Group 
metadata is updated when the background thread receives a new assignment. More 
specifically, the member epoch is updated each time a new assignment is 
received and and the member ID is updated with the first assignment. 
In contrast to the group metadata update, the assignment is directly set in the 
subscription without sending an update event from the background thread to the 
application thread. That means that there is a delay between the application 
thread being aware of the update to the assignment and the application thread 
being aware of the update to the group metadata. This behavior differs with 
respect to the legacy consumer were the assignment and the group metadata is 
updated at the same time.
We should make the update to the group metadata available to the application 
thread when the update to the assignment is made available to the application 
thread so that assignment an group metadata are in sync. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-16284) Performance regression in RocksDB

[Lucas Brutschy (Jira)]

Lucas Brutschy created KAFKA-16284:
--

 Summary: Performance regression in RocksDB
 Key: KAFKA-16284
 URL: https://issues.apache.org/jira/browse/KAFKA-16284
 Project: Kafka
  Issue Type: Task
  Components: streams
Reporter: Lucas Brutschy


In benchmarks, we are noticing a performance regression in the performance of 
`RocksDBStore`.

The regression happens between those two commits:
 
{code:java}
trunk - 70c8b8d0af - regressed - 2024-01-06T14:00:20Z
trunk - d5aa341a18 - not regressed - 2023-12-31T11:47:14Z
{code}

The regression can be reproduced by the following test:
 
{code:java}
package org.apache.kafka.streams.state.internals;

import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.common.utils.Bytes;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.processor.StateStoreContext;
import org.apache.kafka.test.InternalMockProcessorContext;
import org.apache.kafka.test.MockRocksDbConfigSetter;
import org.apache.kafka.test.StreamsTestUtils;
import org.apache.kafka.test.TestUtils;
import org.junit.Before;
import org.junit.Test;

import java.io.File;
import java.nio.ByteBuffer;
import java.util.Properties;

public class RocksDBStorePerfTest {

InternalMockProcessorContext context;
RocksDBStore rocksDBStore;
final static String DB_NAME = "db-name";
final static String METRICS_SCOPE = "metrics-scope";

RocksDBStore getRocksDBStore() {
return new RocksDBStore(DB_NAME, METRICS_SCOPE);
}
@Before
public void setUp() {
final Properties props = StreamsTestUtils.getStreamsConfig();
props.put(StreamsConfig.ROCKSDB_CONFIG_SETTER_CLASS_CONFIG, 
MockRocksDbConfigSetter.class);
File dir = TestUtils.tempDirectory();
context = new InternalMockProcessorContext<>(
dir,
Serdes.String(),
Serdes.String(),
new StreamsConfig(props)
);
}

@Test
public void testPerf() {
long start = System.currentTimeMillis();
for (int i = 0; i < 10; i++) {
System.out.println("Iteration: "+i+" Time: " + 
(System.currentTimeMillis() - start));
RocksDBStore rocksDBStore = getRocksDBStore();
rocksDBStore.init((StateStoreContext) context, rocksDBStore);
for (int j = 0; j < 100; j++) {
rocksDBStore.put(new 
Bytes(ByteBuffer.allocate(4).putInt(j).array()), "perf".getBytes());
}
rocksDBStore.close();
}
long end = System.currentTimeMillis();
System.out.println("Time: " + (end - start));
}

}
 {code}
 
I have isolated the regression to commit 
[5bc3aa4|https://github.com/apache/kafka/commit/5bc3aa428067dff1f2b9075ff5d1351fb05d4b10].
 On my machine, the test takes ~8 seconds before 
[5bc3aa4|https://github.com/apache/kafka/commit/5bc3aa428067dff1f2b9075ff5d1351fb05d4b10]
 and ~30 seconds after 
[5bc3aa4|https://github.com/apache/kafka/commit/5bc3aa428067dff1f2b9075ff5d1351fb05d4b10].




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [PR] 3.7: Add blog post for Kafka 3.7 [kafka-site]

[via GitHub]

stanislavkozlovski commented on code in PR #578:
URL: https://github.com/apache/kafka-site/pull/578#discussion_r1495737109


##
blog.html:
##
@@ -22,6 +22,125 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.7.0 Release Announcement
+
+February 2024 - Stanislav Kozlovski (https://twitter.com/BdKozlovski;>@BdKozlovski)
+We are proud to announce the release of Apache Kafka 3.7.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.7.0/RELEASE_NOTES.html;>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_7_0;>Upgrading 
to 3.7.0 from any version 0.8.x through 3.6.x section in the documentation 
for the list of notable changes and detailed upgrade steps.
+
+In the last release, 3.6,
+https://kafka.apache.org/documentation/#kraft_zk_migration;>the ability 
to migrate Kafka clusters from a ZooKeeper metadata system
+to a KRaft metadata system was ready for usage in 
production environments with one caveat -- JBOD was not yet available for KRaft 
clusters.
+In this release, we are shipping an early access release 
of JBOD in KRaft. (See https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft;>KIP-858
 for details)
+
+
+Additionally, client APIs released prior to Apache Kafka 
2.1 are now marked deprecated in 3.7 and will be removed in Apache Kafka 4.0. 
See https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0;>KIP-896
 for details and RPC versions that are now deprecated.
+
+
+Java 11 support for the Kafka broker is also marked 
deprecated in 3.7, and is planned to be removed in Kafka 4.0. See https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510;>KIP-1013
 for more details

Review Comment:
   good point, thanks



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] 3.7: Add blog post for Kafka 3.7 [kafka-site]

[via GitHub]

stanislavkozlovski commented on code in PR #578:
URL: https://github.com/apache/kafka-site/pull/578#discussion_r1495733083


##
blog.html:
##
@@ -22,6 +22,125 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.7.0 Release Announcement
+
+February 2024 - Stanislav Kozlovski (https://twitter.com/BdKozlovski;>@BdKozlovski)
+We are proud to announce the release of Apache Kafka 3.7.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.7.0/RELEASE_NOTES.html;>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_7_0;>Upgrading 
to 3.7.0 from any version 0.8.x through 3.6.x section in the documentation 
for the list of notable changes and detailed upgrade steps.

Review Comment:
   
   - https://kafka.apache.org/documentation.html#upgrade_3_6_0 nor 
https://kafka.apache.org/36/documentation.html#upgrade_3_6_0 link to anything 
(this is what the last PR linked to - 
https://github.com/apache/kafka-site/pull/547/files#diff-f0599f46f79e7231037e2b37dbc5c73cd0cab4b18d6121f1fee6fe4e2a78ef7cR32).
 I think this was done when you changed the section to 3.6.1 - 
https://github.com/apache/kafka-site/commit/c1a96e48b0106763f0a358c59bb4b8c23dda9850#diff-fc250e11862028fbf00a12f45b3a71c9f63da43921bb1399745a06270a97ef09L22
 (pic in case github takes too long to load) 
   
![image](https://github.com/apache/kafka-site/assets/13639618/08ae0b8f-daf3-4f10-a04a-bb6bde251eae)
   
   Here is a PR to fix that: https://github.com/apache/kafka-site/pull/580/files
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] MINOR: Update stale `upgrade_3_6_0` header links in documentation [kafka-site]

[via GitHub]

stanislavkozlovski commented on code in PR #580:
URL: https://github.com/apache/kafka-site/pull/580#discussion_r1495732697


##
37/upgrade.html:
##
@@ -19,9 +19,9 @@
 
 

[PR] MINOR: Update stale `upgrade_3_6_0` header links in documentation [kafka-site]

[via GitHub]

stanislavkozlovski opened a new pull request, #580:
URL: https://github.com/apache/kafka-site/pull/580

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [PR] 3.7: Add blog post for Kafka 3.7 [kafka-site]

[via GitHub]

stanislavkozlovski commented on code in PR #578:
URL: https://github.com/apache/kafka-site/pull/578#discussion_r1495710631


##
blog.html:
##
@@ -22,6 +22,125 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.7.0 Release Announcement
+
+February 2024 - Stanislav Kozlovski (https://twitter.com/BdKozlovski;>@BdKozlovski)
+We are proud to announce the release of Apache Kafka 3.7.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.7.0/RELEASE_NOTES.html;>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_7_0;>Upgrading 
to 3.7.0 from any version 0.8.x through 3.6.x section in the documentation 
for the list of notable changes and detailed upgrade steps.

Review Comment:
   good point!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] KIP-932: Queues for Kafka

[Andrew Schofield]

Hi Jun,
Thanks for your comments. A couple of your questions I will leave unanswered for
now because there is some research needed.

10. This area needs more work. Using a share group surely gets the broker to do
more manipulation of the data that it fetches than a regular consumer. I want 
to minimise
this and need to research before providing a comprehensive answer. I suspect 
zero-copy
is lost and that we do not cache records in heap. I will confirm later on.

12. I propose INCONSISTENT_GROUP_PROTOCOL. The alternative which comes to
mind is GROUP_ID_NOT_FOUND, but I prefer the former personally.

17. Let’s imagine supporting `group.share.auto.offset.reset=none` so that the
SPSO doesn’t auto-reset if the consumption is slow and unconsumed data is
deleted because of retention. The consumers in a share group are kind of 
“sharing”
the position of the SPSO, so there’s no KafkaShareConsumer.commit(EARLIEST).
Instead, you need to use KafkaAdminClient.alterShareGroupOffsets or the
kafka-share-groups.sh tool. The effect is for all consumers, not just one. So, 
I don’t
favour having the “none” option here because it would essentially “break” the 
share
group until an administrative action was taken.

I could imagine trying to acknowledge a record that has evaporated could throw a
specific exception. The KIP currently says that the broker just accepts the 
acknowledgement.
Would this be a helpful improvement in your mind?

18.2. Yes, you are correct that to server a ShareAcknowledgeRequest, the leader
needs to write the records and wait for the records to be fully acknowledged 
before
sending a response. I have updated the KIP accordingly.

18.3. I view the choice between SHARE_CHECKPOINT and SHARE_DELTA as
a choice the broker will make dynamically.

19. The KIP does say that the delivery behaviour is at-least-once.

21. This is related to 10. I need to do more research. You are most likely 
correct
but I want to confirm.

24. Yes, the index layout needs to be documented. I’ve added a to-do to the KIP.
I plan to put some concerted effort into the share-partition state in the next 
week or so.

25. This is a good point. I have updated the callback interface so the offset 
information
is available.

30.1. You are correct. `states` should be `types`.
30.2. There is no overall `GroupStates` at the moment because the different 
group types
have different state enums.

31. The answer to 30.2 is also why there’s no state in the GroupListing.

I’ll push an updated KIP later today.

Thanks,
Andrew

> On 13 Feb 2024, at 23:15, Jun Rao  wrote:
> 
> Hi, Andrew,
> 
> Thanks for the reply.
> 
> 10. The impact from doing server side filtering is that we lose zero-copy
> transfer, which provides two potential benefits: (1) more direct/efficient
> transfer from disk to network; (2) less memory usage in heap since we don't
> need to copy the records to heap to send a fetch response. (1) may already
> be lost if SSL is used. However, it would be useful to outline the impact
> of (2). For example, do we need to use memory_records in fetch responses?
> How much additional heap memory will the server use? Do we need to cache
> records in heap? If so, is the cache bounded?
> 
> 12. If the group is configured with share and a client tries to join as a
> consumer, what error do we return in the RPC and in the client API? Ditto
> for the reverse case where a share client tries to join a group configured
> with consumer.
> 
> 17. "What is the client going to do with the exception?" Well, the reason
> that we added this option was for the case where the same data could be
> obtained from some other place. Suppose that we use CDC to get
> database changes into a Kafka topic. If the consumer is slow
> and unconsumed data is deleted in Kafka because of retention, by receiving
> an exception, the consumer will know that there is potential missing data
> in Kafka and could bootstrap all data from the database first before
> resuming the consumption from Kafka.
> 
> 18.2 So, to serve a ShareAcknowledgeRequest, the leader needs to write
> share records for the acknowledgement and wait for the records to be fully
> replicated before sending a response? It would be useful to document that
> in the section of handling ShareAcknowledgeRequest.
> 18.3 Could we document when the leader writes SHARE_CHECKPOINT vs
> SHARE_DELTA?
> 
> 19. Could we document the difference in delivery guarantee between
> share and regular consumer?
> 
> 21. Do we lose zero-copy transfer for all consumers because of that (since
> we don't know which topics contain control records?
> 
> 24. Could we document the index layout?
> 
> 25. "The callback tells you whether the acknowledgements for the entire
> topic-partition succeeded or failed, rather than each record individually."
> The issue is that with implicit acknowledgement, it's not clear which
> records are in the batch. For example, if the client just keeps calling
> poll() with no explicit commits, when a callback is 

[jira] [Created] (KAFKA-16283) RoundRobinPartitioner will only send to half of the partitions in a topic

[Luke Chen (Jira)]

Luke Chen created KAFKA-16283:
-

 Summary: RoundRobinPartitioner will only send to half of the 
partitions in a topic
 Key: KAFKA-16283
 URL: https://issues.apache.org/jira/browse/KAFKA-16283
 Project: Kafka
  Issue Type: Bug
Reporter: Luke Chen


When using `org.apache.kafka.clients.producer.RoundRobinPartitioner`, we expect 
data are send to all partitions in round-robin manner. But we found there are 
only half of the partitions got the data. This causes half of the 
resources(storage, consumer...) are wasted.


{code:java}
bin/kafka-topics.sh --create --topic quickstart-events4 --bootstrap-server 
localhost:9092 --partitions 2 

Created topic quickstart-events4.

bin/kafka-producer-perf-test.sh --topic quickstart-events4 --num-records 10 
--record-size 100 --throughput -1 --producer-props 
bootstrap.servers=localhost:9092 
partitioner.class=org.apache.kafka.clients.producer.RoundRobinPartitioner

10 records sent, 72.463768 records/sec (0.01 MB/sec), 35.10 ms avg latency, 
132.00 ms max latency, 24 ms 50th, 132 ms 95th, 132 ms 99th, 132 ms 99.9th.

lukchen@lukchen-mac kafka % ls -al /tmp/kafka-logs/quickstart-events4-0
total 24
drwxr-xr-x   7 lukchen  wheel   224  2 20 19:53 .
drwxr-xr-x  70 lukchen  wheel  2240  2 20 19:53 ..
-rw-r--r--   1 lukchen  wheel  10485760  2 20 19:53 .index
-rw-r--r--   1 lukchen  wheel  1151  2 20 19:53 .log
-rw-r--r--   1 lukchen  wheel  10485756  2 20 19:53 
.timeindex
-rw-r--r--   1 lukchen  wheel 8  2 20 19:53 leader-epoch-checkpoint
-rw-r--r--   1 lukchen  wheel43  2 20 19:53 partition.metadata
lukchen@lukchen-mac kafka % ls -al /tmp/kafka-logs/quickstart-events4-1
total 8
drwxr-xr-x   7 lukchen  wheel   224  2 20 19:53 .
drwxr-xr-x  70 lukchen  wheel  2240  2 20 19:53 ..
-rw-r--r--   1 lukchen  wheel  10485760  2 20 19:53 .index
-rw-r--r--   1 lukchen  wheel 0  2 20 19:53 .log
-rw-r--r--   1 lukchen  wheel  10485756  2 20 19:53 
.timeindex
-rw-r--r--   1 lukchen  wheel 0  2 20 19:53 leader-epoch-checkpoint
-rw-r--r--   1 lukchen  wheel43  2 20 19:53 partition.metadata
{code}




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-1019: Expose method to determine Metric Measurability

[Manikumar]

+1 (binding).

Thanks for the KIP.

Manikumar

On Tue, Feb 20, 2024 at 2:31 PM Andrew Schofield <
andrew_schofield_j...@outlook.com> wrote:

> Hi Apoorv,
> Thanks for the KIP.
>
> +1 (non-binding)
>
> Thanks,
> Andrew
>
> > On 19 Feb 2024, at 22:31, Apoorv Mittal 
> wrote:
> >
> > Hi,
> > I’d like to start the voting for KIP-1019: Expose method to determine
> > Metric Measurability.
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1019%3A+Expose+method+to+determine+Metric+Measurability
> >
> > Regards,
> > Apoorv Mittal
>
>

Re: [PR] 3.7: Add blog post for Kafka 3.7 [kafka-site]

[via GitHub]

stanislavkozlovski commented on code in PR #578:
URL: https://github.com/apache/kafka-site/pull/578#discussion_r1495481232


##
blog.html:
##
@@ -22,6 +22,125 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.7.0 Release Announcement
+
+February 2024 - Stanislav Kozlovski (https://twitter.com/BdKozlovski;>@BdKozlovski)
+We are proud to announce the release of Apache Kafka 3.7.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.7.0/RELEASE_NOTES.html;>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_7_0;>Upgrading 
to 3.7.0 from any version 0.8.x through 3.6.x section in the documentation 
for the list of notable changes and detailed upgrade steps.
+
+In the last release, 3.6,
+https://kafka.apache.org/documentation/#kraft_zk_migration;>the ability 
to migrate Kafka clusters from a ZooKeeper metadata system
+to a KRaft metadata system was ready for usage in 
production environments with one caveat -- JBOD was not yet available for KRaft 
clusters.
+In this release, we are shipping an early access release 
of JBOD in KRaft. (See https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft;>KIP-858
 for details)
+
+
+Additionally, client APIs released prior to Apache Kafka 
2.1 are now marked deprecated in 3.7 and will be removed in Apache Kafka 4.0. 
See https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0;>KIP-896
 for details and RPC versions that are now deprecated.
+
+
+Java 11 support for the Kafka broker is also marked 
deprecated in 3.7, and is planned to be removed in Kafka 4.0. See https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=284789510;>KIP-1013
 for more details
+
+
+Note: ZooKeeper is marked as deprecated since the 3.5.0 
release. ZooKeeper is planned to be removed in Apache Kafka 4.0. For more 
information, please see the documentation for https://kafka.apache.org/documentation/#zk_depr;>ZooKeeper 
Deprecation.
+
+
+Kafka Broker, Controller, Producer, Consumer and Admin 
Client
+
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft;>(Early
 Access) KIP-858 Handle JBOD broker disk failure in KRaft:

Review Comment:
   I have been contacting @pprovenzano about this too. I think we should add 
something just like we did for KIP-848. cc @soarez 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Community Over Code Asia 2024 Travel Assistance Applications now open!

[Gavin McDonald]

Hello to all users, contributors and Committers!

The Travel Assistance Committee (TAC) are pleased to announce that
travel assistance applications for Community over Code Asia 2024 are now
open!

We will be supporting Community over Code Asia, Hangzhou, China
July 26th - 28th, 2024.

TAC exists to help those that would like to attend Community over Code
events, but are unable to do so for financial reasons. For more info
on this year's applications and qualifying criteria, please visit the
TAC website at < https://tac.apache.org/ >. Applications are already
open on https://tac-apply.apache.org/, so don't delay!

The Apache Travel Assistance Committee will only be accepting
applications from those people that are able to attend the full event.

Important: Applications close on Friday, May 10th, 2024.

Applicants have until the the closing date above to submit their
applications (which should contain as much supporting material as
required to efficiently and accurately process their request), this
will enable TAC to announce successful applications shortly
afterwards.

As usual, TAC expects to deal with a range of applications from a
diverse range of backgrounds; therefore, we encourage (as always)
anyone thinking about sending in an application to do so ASAP.

For those that will need a Visa to enter the Country - we advise you to
apply
now so that you have enough time in case of interview delays. So do not
wait until you know if you have been accepted or not.

We look forward to greeting many of you in Hangzhou, China in July, 2024!

Kind Regards,

Gavin

(On behalf of the Travel Assistance Committee)

[jira] [Created] (KAFKA-16282) Allow to get last stable offset (LSO) in kafka-get-offsets.sh

[Luke Chen (Jira)]

Luke Chen created KAFKA-16282:
-

 Summary: Allow to get last stable offset (LSO) in 
kafka-get-offsets.sh
 Key: KAFKA-16282
 URL: https://issues.apache.org/jira/browse/KAFKA-16282
 Project: Kafka
  Issue Type: Improvement
Reporter: Luke Chen


Currently, when using `kafka-get-offsets.sh` to get the offset by time, we have 
these choices:


{code:java}
--time  /  timestamp of the offsets before 
that. 
  -1 or latest /   [Note: No offset is returned, if the
  -2 or earliest / timestamp greater than recently
  -3 or max-timestamp /committed record timestamp is
  -4 or earliest-local /   given.] (default: latest)
  -5 or latest-tiered  

{code}

For the latest choice, it'll always return the "high watermark" because we 
always send with the default option: *IsolationLevel.READ_UNCOMMITTED*. It 
would be good if the command can support to get the last stable offset (LSO) 
for transaction support.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-1019: Expose method to determine Metric Measurability

[Andrew Schofield]

Hi Apoorv,
Thanks for the KIP.

+1 (non-binding)

Thanks,
Andrew

> On 19 Feb 2024, at 22:31, Apoorv Mittal  wrote:
>
> Hi,
> I’d like to start the voting for KIP-1019: Expose method to determine
> Metric Measurability.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-1019%3A+Expose+method+to+determine+Metric+Measurability
>
> Regards,
> Apoorv Mittal

[REVIEW REQUEST] Tests of ConsoleGroupCommand moved to tools

[Николай Ижиков]

Hello. 

I’m working on moving `ConsoleGroupCommand` to `tools`.
There are two PR’s that moves all tests of ConsoleGroupCommand to tools.
Please, take a look.

[1] https://github.com/apache/kafka/pull/15365
[2] https://github.com/apache/kafka/pull/15363

BIG PR here - https://github.com/apache/kafka/pull/14471

[jira] [Created] (KAFKA-16281) Probable IllegalState possible with KIP-966

[Jack Vanlightly (Jira)]

Jack Vanlightly created KAFKA-16281:
---

 Summary: Probable IllegalState possible with KIP-966
 Key: KAFKA-16281
 URL: https://issues.apache.org/jira/browse/KAFKA-16281
 Project: Kafka
  Issue Type: Task
  Components: kraft
Reporter: Jack Vanlightly


I have a TLA+ model of KIP-966 and I have identified an IllegalState exception 
that would occur with the existing MaybeHandleCommonResponse behavior.

The issue stems from the fact that a leader, let's call it r1, can resign 
(either due to a restart or check quorum) and then later initiate a pre-vote 
where it ends up in the same epoch as before, but a cleared local leader id. 
When r1 transitions to Prospective it clears its local leader id. When r1 
receives a response from r2 who believes that r1 is still the leader, the logic 
in MaybeHandleCommonResponse tries to transition r1 to follower of itself, 
causing an IllegalState exception to be raised.

This is an example history:
 # r1 is the leader in epoch 1.
 # r1 quorum resigns, or restarts and resigns.
 # r1 experiences an election timeout and transitions to Prospective clearing 
its local leader id.
 # r1 sends a pre vote request to its peers.
 # r2 thinks r1 is still the leader, sends a vote response, not granting its 
vote and setting leaderId=r1 and epoch=1.
 # r1 receives the vote response and executes MaybeHandleCommonResponse which 
tries to transition r1 to Follower of itself and an illegal state occurs.

The relevant else if statement in MaybeHandleCommonResponse is here: 
https://github.com/apache/kafka/blob/a26a1d847f1884a519561e7a4fb4cd13e051c824/raft/src/main/java/org/apache/kafka/raft/KafkaRaftClient.java#L1538

In the TLA+ specification, I fixed this issue by adding a fourth condition to 
this statement, that the leaderId also does not equal this server's id. 
[https://github.com/Vanlightly/kafka-tlaplus/blob/9b2600d1cd5c65930d666b12792d47362b64c015/kraft/kip_996/kraft_kip_996_functions.tla#L336]

We should probably create a test to confirm the issue first and then look at 
using the fix I made in the TLA+, though there may be other options.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)