[jira] [Created] (KAFKA-15958) Add tests to validate telemetry requests with different version

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-15958:
-

 Summary: Add tests to validate telemetry requests with different 
version
 Key: KAFKA-15958
 URL: https://issues.apache.org/jira/browse/KAFKA-15958
 Project: Kafka
  Issue Type: Sub-task
Reporter: Apoorv Mittal
Assignee: Apoorv Mittal


Details: https://github.com/apache/kafka/pull/14724#discussion_r1412530561



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: fine-gained acls

[Haruki Okada]

Hi.

KafkaConsumer can subscribe to topics by pattern:
https://kafka.apache.org/36/javadoc/org/apache/kafka/clients/consumer/Consumer.html#subscribe(java.util.regex.Pattern)

2023年12月1日(金) 22:05 Chunlin Yang :

> Hi team,
>
> My use case is I have a central controller to manage tens of thousands of
> clusters. Each cluster can receive and send the message via Kafka. but each
> cluster can only consume its own message. The controller can consume all
> the messages from each cluster.
>
> I checked the Kafka document and know that there is no limitation for Kafka
> topics and the Kafka provides the ACLs per topic so my idea is to create
> one topic per cluster. but it seems Kafka cannot support subscript topics
> with wildcard. Is that true?
>
> I guess I do not use Kafka correctly. Could you share your best practices
> which can address my case? Thanks in advance.
>


-- 

Okada Haruki
ocadar...@gmail.com

[jira] [Created] (KAFKA-15957) ConsistencyVectorIntegrationTest.shouldHaveSamePositionBoundActiveAndStandBy broken

[Justine Olshan (Jira)]

Justine Olshan created KAFKA-15957:
--

 Summary: 
ConsistencyVectorIntegrationTest.shouldHaveSamePositionBoundActiveAndStandBy 
broken
 Key: KAFKA-15957
 URL: https://issues.apache.org/jira/browse/KAFKA-15957
 Project: Kafka
  Issue Type: Bug
Reporter: Justine Olshan






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15956) MetadataShell must take the directory lock when reading

[Colin McCabe (Jira)]

Colin McCabe created KAFKA-15956:


 Summary: MetadataShell must take the directory lock when reading
 Key: KAFKA-15956
 URL: https://issues.apache.org/jira/browse/KAFKA-15956
 Project: Kafka
  Issue Type: Improvement
Reporter: Colin McCabe


MetadataShell must take the directory lock when reading files, to avoid 
unpleasant surprises from concurrent reads and writes.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-994: Minor Enhancements to ListTransactions and DescribeTransactions APIs

[Jun Rao]

Hi, Raman,

Thanks for the KIP. A couple of comments.

10. From the discussion of KIP-951, it seems that our convention is to
always bump up the request version even for just adding tagged fields in
the response.

11. "In case a new AdminClient is sending durationFilter (greater than 0)
to an older broker, ListTransactionsRequest will fail to build at the
client side. This will require some check to be made at
ListTransactionsRequest.Builder.build(short version) method. A new
AdminClient can still generate older version of ListTransactionsRequest
when it sets durationFilter to 0." This seems weird. If a user specifies a
durationFilter but the server doesn't support it, it seems that we should
throw an exception to the user instead of silently changing durationFilter.

Jun

On Wed, Nov 29, 2023 at 5:05 PM Justine Olshan 
wrote:

> Hey Raman,
> Thanks for the KIP! I think this will be super useful.
>
> Given https://issues.apache.org/jira/browse/KAFKA-15546 -- do you think it
> would be useful to specify the duration of the completed transaction rather
> than the time since the start in the describe output?
> We would probably want to specify the transaction is completed as well to
> differentiate from the response that does not have the tagged field. This
> would more clearly resolve the ticket.
> Maybe this was the plan but it wasn't clear from the KIP.
>
> Thanks,
> Justine
>
> On Tue, Nov 28, 2023 at 12:38 PM Jason Gustafson
> 
> wrote:
>
> > Hey Raman,
> >
> > Thanks for the KIP! I think it makes sense. I agree that this becomes
> > especially useful in the context of KIP-939 because transactions can last
> > an indefinite amount of time, but it is useful even today. A large
> cluster
> > may have a very large number of ongoing transactions at any time, so
> > providing a better way to filter by time will make the tools much more
> > efficient for this common use case.
> >
> > I have just a couple small comments.
> >
> > 1. In `ListTransactionsOptions`, we use a long in the setter for the
> > duration filter. Can we use `Duration` instead?
> > 2. I think we need to expose `TransactionLastUpdateTimeMs` on
> > `TransactionDescription` as well, right?
> >
> > Thanks,
> > Jason
> >
> > On Tue, Nov 28, 2023 at 8:34 AM Kirk True  wrote:
> >
> > > Hi Raman,
> > >
> > > Thanks for the KIP!
> > >
> > > Questions/comments:
> > >
> > >  1. Is there a Jira created for this? The Jira link still points to
> > > KAFKA-1.
> > >  2. There's a minor typo in the ListTransactionsRequest documentation
> for
> > > the DurationFilter: trsanactions.
> > >  3. Is the TransactionStartTimeMs return value in the
> > > DescribeTransactionsResponse nullable?
> > >  4. The API uses the general terminology of a "duration filter" but the
> > > CLI uses the specific phrase "running longer than ms." These are both
> > > referring to the same value, right? Can the naming of these two be more
> > > consistent?
> > >  5. What happens when a user runs the updated kafka-transactions.sh
> > script
> > > (using the new argument) against an older broker that doesn't support
> the
> > > new filter? Does the user get an error, a warning, or a silent ignoring
> > of
> > > the filter?
> > >
> > > Thanks,
> > > Kirk
> > >
> > > On Wed, Nov 15, 2023, at 3:03 PM, Raman Verma wrote:
> > > > Thanks Artem,
> > > >
> > > > I have made changes to the `Public Interfaces` and `Compatibility...`
> > > > sections to incorporate your comment.
> > > >
> > > > On Mon, Nov 6, 2023 at 3:44 PM Raman Verma 
> > > wrote:
> > > >
> > > > > I would like to start a discussion on KIP-994
> > > > >
> > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-994%3A+Minor+Enhancements+to+ListTransactions+and+DescribeTransactions+APIs
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] 3.6.1 RC0

[Bill Bejeck]

Hi Mickael,

I did the following:

   1. Validated all checksums, signatures, and hashes
   2. Built from source
   3. Ran all the unit tests
   4. Spot checked the documentation and Javadoc
   5. Ran the ZK, Kraft, and Kafka Streams quickstart guides

I did notice that the `fillDotVersion` in `js/templateData.js` needs
updating to `3.6.1`, but this is minor and should not block the release.

It's a +1(binding) for me, pending the successful system test run

Thanks,
Bill

On Fri, Dec 1, 2023 at 1:49 PM Justine Olshan 
wrote:

> I've started a system test run on my end.
>
> Justine
>
> On Wed, Nov 29, 2023 at 1:55 PM Justine Olshan 
> wrote:
>
> > I built from source and ran a simple transactional produce bench. I ran a
> > handful of unit tests as well.
> > I scanned the docs and everything looked reasonable.
> >
> > I was wondering if we got the system test results mentioned > System
> > tests: Still running I'll post an update once they complete.
> >
> > Justine
> >
> > On Wed, Nov 29, 2023 at 6:33 AM Mickael Maison  >
> > wrote:
> >
> >> Hi Josep,
> >>
> >> Good catch!
> >> If it's the only issue we find, I don't think we should block the
> >> release just to fix that.
> >>
> >> If we find another issue, I'll backport it before running another RC,
> >> otherwise I'll backport it once 3.6.1 is released.
> >>
> >> Thanks,
> >> Mickael
> >>
> >> On Wed, Nov 29, 2023 at 11:55 AM Josep Prat  >
> >> wrote:
> >> >
> >> > Hi Mickael,
> >> > This PR[1] made me realize NOTICE-binary is missing the notice for
> >> > commons-io. I don't know if it's a blocker or not. I can cherry pick
> the
> >> > commit to the 3.6 branch if you want.
> >> >
> >> > Best,
> >> >
> >> >
> >> > [1]: https://github.com/apache/kafka/pull/14865
> >> >
> >> > On Tue, Nov 28, 2023 at 10:25 AM Josep Prat 
> >> wrote:
> >> >
> >> > > Hi Mickael,
> >> > > Thanks for running the release. It's a +1 for me (non-binding).
> >> > > I did the following:
> >> > > - Verified artifact's signatures and hashes
> >> > > - Checked JavaDoc (with navigation to Oracle JavaDoc)
> >> > > - Compiled source code
> >> > > - Run unit tests and integration tests
> >> > > - Run getting started with ZK and KRaft
> >> > >
> >> > > Best,
> >> > >
> >> > > On Tue, Nov 28, 2023 at 8:51 AM Kamal Chandraprakash <
> >> > > kamal.chandraprak...@gmail.com> wrote:
> >> > >
> >> > >> +1 (non-binding)
> >> > >>
> >> > >> 1. Built the source from 3.6.1-rc0 tag in scala 2.12 and 2.13
> >> > >> 2. Ran all the unit and integration tests.
> >> > >> 3. Ran quickstart and verified the produce-consume on a 3 node
> >> cluster.
> >> > >> 4. Verified the tiered storage functionality with local-tiered
> >> storage.
> >> > >>
> >> > >> On Tue, Nov 28, 2023 at 12:55 AM Federico Valeri <
> >> fedeval...@gmail.com>
> >> > >> wrote:
> >> > >>
> >> > >> > Hi Mickael,
> >> > >> >
> >> > >> > - Build from source (Java 17, Scala 2.13)
> >> > >> > - Run unit and integration tests
> >> > >> > - Run custom client apps using staging artifacts
> >> > >> >
> >> > >> > +1 (non binding)
> >> > >> >
> >> > >> > Thanks
> >> > >> > Fede
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >> > On Sun, Nov 26, 2023 at 11:34 AM Jakub Scholz 
> >> wrote:
> >> > >> > >
> >> > >> > > +1 non-binding. I used the staged Scala 2.13 artifacts and the
> >> staged
> >> > >> > Maven
> >> > >> > > repo for my tests. All seems to work fine.
> >> > >> > >
> >> > >> > > Thanks
> >> > >> > > Jakub
> >> > >> > >
> >> > >> > > On Fri, Nov 24, 2023 at 4:37 PM Mickael Maison <
> >> mimai...@apache.org>
> >> > >> > wrote:
> >> > >> > >
> >> > >> > > > Hello Kafka users, developers and client-developers,
> >> > >> > > >
> >> > >> > > > This is the first candidate for release of Apache Kafka
> 3.6.1.
> >> > >> > > >
> >> > >> > > > This is a bugfix release with several fixes, including
> >> dependency
> >> > >> > > > version bumps for CVEs.
> >> > >> > > >
> >> > >> > > > Release notes for the 3.6.1 release:
> >> > >> > > >
> >> > >>
> https://home.apache.org/~mimaison/kafka-3.6.1-rc0/RELEASE_NOTES.html
> >> > >> > > >
> >> > >> > > > *** Please download, test and vote by Friday, December 1
> >> > >> > > >
> >> > >> > > > Kafka's KEYS file containing PGP keys we use to sign the
> >> release:
> >> > >> > > > https://kafka.apache.org/KEYS
> >> > >> > > >
> >> > >> > > > * Release artifacts to be voted upon (source and binary):
> >> > >> > > > https://home.apache.org/~mimaison/kafka-3.6.1-rc0/
> >> > >> > > >
> >> > >> > > > * Maven artifacts to be voted upon:
> >> > >> > > >
> >> > >>
> >> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> >> > >> > > >
> >> > >> > > > * Javadoc:
> >> > >> > > > https://home.apache.org/~mimaison/kafka-3.6.1-rc0/javadoc/
> >> > >> > > >
> >> > >> > > > * Tag to be voted upon (off 3.6 branch) is the 3.6.1 tag:
> >> > >> > > > https://github.com/apache/kafka/releases/tag/3.6.1-rc0
> >> > >> > > >
> >> > >> > > > PR for updating docs:
> >> > >> > > > 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2439

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.6 #123

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 310028 lines...]
> Task :connect:json:testJar
> Task :connect:json:copyDependantLibs UP-TO-DATE
> Task :connect:api:compileTestJava UP-TO-DATE
> Task :connect:api:testClasses UP-TO-DATE
> Task :connect:json:jar UP-TO-DATE
> Task :connect:json:generateMetadataFileForMavenJavaPublication
> Task :connect:json:testSrcJar
> Task :connect:api:testJar
> Task :connect:api:testSrcJar
> Task :clients:generateMetadataFileForMavenJavaPublication
> Task :connect:api:publishMavenJavaPublicationToMavenLocal
> Task :connect:json:publishMavenJavaPublicationToMavenLocal
> Task :connect:api:publishToMavenLocal
> Task :connect:json:publishToMavenLocal
> Task :storage:api:compileTestJava
> Task :storage:api:testClasses
> Task :server-common:compileTestJava
> Task :server-common:testClasses
> Task :raft:compileTestJava
> Task :raft:testClasses
> Task :core:compileScala
> Task :group-coordinator:compileTestJava
> Task :group-coordinator:testClasses

> Task :clients:javadoc
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.6/clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java:81:
 warning - Tag @link:illegal character: "60" in "#define(String, Type, 
Importance, String, String, int, Width, String, List)"
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.6/clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java:81:
 warning - Tag @link:illegal character: "62" in "#define(String, Type, 
Importance, String, String, int, Width, String, List)"
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.6/clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java:81:
 warning - Tag @link: can't find define(String, Type, Importance, String, 
String, int, Width, String, List) in 
org.apache.kafka.common.config.ConfigDef
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.6/clients/src/main/java/org/apache/kafka/clients/admin/ScramMechanism.java:32:
 warning - Tag @see: missing final '>': "https://cwiki.apache.org/confluence/display/KAFKA/KIP-554%3A+Add+Broker-side+SCRAM+Config+API;>KIP-554:
 Add Broker-side SCRAM Config API

 This code is duplicated in 
org.apache.kafka.common.security.scram.internals.ScramMechanism.
 The type field in both files must match and must not change. The type field
 is used both for passing ScramCredentialUpsertion and for the internal
 UserScramCredentialRecord. Do not change the type field."
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.6/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/package-info.java:21:
 warning - Tag @link: reference not found: 
org.apache.kafka.common.security.oauthbearer
5 warnings

> Task :metadata:compileTestJava
> Task :metadata:testClasses
> Task :clients:javadocJar
> Task :clients:srcJar
> Task :clients:testJar
> Task :clients:testSrcJar
> Task :clients:publishMavenJavaPublicationToMavenLocal
> Task :clients:publishToMavenLocal
> Task :core:classes
> Task :core:compileTestJava NO-SOURCE
> Task :core:compileTestScala
> Task :core:testClasses
> Task :streams:compileTestJava
> Task :streams:testClasses
> Task :streams:testJar
> Task :streams:testSrcJar
> Task :streams:publishMavenJavaPublicationToMavenLocal
> Task :streams:publishToMavenLocal

Deprecated Gradle features were used in this build, making it incompatible with 
Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings 
and determine if they come from your own scripts or plugins.

For more on this, please refer to 
https://docs.gradle.org/8.2.1/userguide/command_line_interface.html#sec:command_line_warnings
 in the Gradle documentation.

BUILD SUCCESSFUL in 5m 27s
94 actionable tasks: 41 executed, 53 up-to-date

Publishing build scan...
https://ge.apache.org/s/pyw2ka6rvj6js

[Pipeline] sh
+ grep ^version= gradle.properties
+ cut -d= -f 2
[Pipeline] dir
Running in 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_3.6/streams/quickstart
[Pipeline] {
[Pipeline] sh
+ mvn clean install -Dgpg.skip
[INFO] Scanning for projects...
[INFO] 
[INFO] Reactor Build Order:
[INFO] 
[INFO] Kafka Streams :: Quickstart[pom]
[INFO] streams-quickstart-java[maven-archetype]
[INFO] 
[INFO] < org.apache.kafka:streams-quickstart >-
[INFO] Building Kafka Streams :: Quickstart 3.6.1-SNAPSHOT[1/2]
[INFO]   from pom.xml
[INFO] [ pom ]-
[INFO] 
[INFO] --- clean:3.0.0:clean (default-clean) @ streams-quickstart ---
[INFO] 
[INFO] --- remote-resources:1.5:process (process-resource-bundles) @ 
streams-quickstart ---
[INFO] 
[INFO] --- site:3.5.1:attach-descriptor (attach-descriptor) @ 
streams-quickstart ---
[INFO] 
[INFO] --- gpg:1.6:sign 

Re: [DISCUSS] KIP-969: Support range interactive queries for versioned state stores

[Alieh Saeedi]

Hi all,
I updated the KIP based on the changes made in the former KIP (KIP-968). So
with the `ResultOrder` enum, the class `MultiVersionedRangeQuery` had some
changes both in the defined fields and defined methods.

Based on the PoC PR, what we currently promise in the KIP about ordering
seems like a dream. I intended to enable the user to have a global ordering
based on the key or timestamp (using `orderByKey()` or
`orderByTimestamp()`) and then even have a partial ordering based on the
other parameter.  For example, if the user specifies
`orderByKey().withDescendingKey().withAscendingTimestamps()`, then the
global ordering is based on keys in a descending order, and then all the
records with the same key are ordered ascendingly based on ts. The result
will be something like (k3,v1,t1), (k3,v2,t2), (k2,v2,t3), (k1.v1.t1)
(assuming that k1 wrote:

> Thank you, Bruno and Matthias.
> I updated the KIP as follows:
> 1. The one remaining `asOf` word in the KIP is removed.
> 2. Example 2 is updated. Thanks, Bruno for the correction.
>
> Discussions and open questions
> 1. Yes, Bruno. We need `orderByKey()` and `orderByTimestamp()` as well.
> Because the results must have a global ordering. Either based on key or
> based on ts. For example, we can have
> `orderByKey().withDescendingKey().withAscendingTimestamps()`. Then the
> global ordering is based on keys in a descending order, and then all the
> records with the same key are ordered ascendingly based on ts. The result
> will be something like (k3,v1,t1), (k3,v2,t2), (k3,v1,t1), (k2,v2,t2),
> (k1.v1.t1) (assuming that k1 2. About having the `latest()` method: it seems like we are undecided yet.
> Adding a new class or ignoring `latest()` for VersionedRangeQuery and
> instead using the `TimestampedRangeQuery` as Matthias suggested.
>
> Cheers,
> Alieh
>
> On Sat, Nov 4, 2023 at 1:38 AM Matthias J. Sax  wrote:
>
>> Great discussion. Seems we are making good progress.
>>
>> I see advantages and disadvantages in splitting out a "single-ts
>> key-range" query type. I guess, the main question might be, what
>> use-cases do we anticipate and how common we believe they are?
>>
>> We should also take KIP-992 (adding `TimestampedRangeQuery`) into account.
>>
>> (1) The most common use case seems to be, a key-range over latest. For
>> this one, we could use `TimestampedRangeQuery` -- it would return a
>> `ValueAndTimestamp` instead of a `VersionedRecord` but the won't
>> be any information loss, because "toTimestamp" would be `null` anyway.
>>
>>
>> (2) The open question is, how common is a key-range in a point in the
>> past? For this case, using
>> `MultiVersionedRangeQuery.withKeyRange().from(myTs).to(myTs)` seems
>> actually not to be a bad UX, and also does not really need to be
>> explained how to do this (compared to "latest" that required to pass in
>> MAX_INT).
>>
>>
>> If we add a new query type, we avoid both issues (are they actually
>> issues?) and add some nice syntactic sugar to the API. The question is,
>> if it's worth the effort and expanded API surface area?
>>
>> To summarize:
>>
>> Add new query type:
>>
>> > // queries latest; returns VersionedRecords
>> > VersionedRangeQuery.withKeyRange(...);
>> >
>> > VersionedRangeQuery.withKeyRange(...).asOf(ts);
>>
>> vs
>>
>> No new query type:
>>
>> > // queries latest; returns ValueAndTimestamps
>> > TimestampedRangeQuery.withRange(...);
>> >
>> > MultiVersionedRangeQuery.withKeyRange(...).from(myTs).to(myTs)
>>
>>
>>
>> I guess, bottom line, I would be ok with either one and I am actually
>> not even sure which one I prefer personally. Just wanted to lay out the
>> tradeoffs I see. Not sure if three are other considerations that would
>> tip the scale into either direction?
>>
>>
>>
>> -Matthias
>>
>> On 11/3/23 3:43 AM, Bruno Cadonna wrote:
>> > Hi Alieh,
>> >
>> > I like the examples!
>> >
>> >
>> > 1.
>> > Some terms like `asOf` in the descriptions still need to be replaced in
>> > the KIP.
>> >
>> >
>> > 2.
>> > In your last e-mail you state:
>> >
>> > "How can a user retrieve the latest value? We have the same issue with
>> > kip-968 as well."
>> >
>> > Why do we have the same issue in KIP-968?
>> > If users need to retrieve the latest value for a specific key, they
>> > should use KIP-960.
>> >
>> >
>> > 3.
>> > Regarding querying the latest version (or an asOf version) of records
>> in
>> > a given key range, that is exactly why I proposed to split the query
>> > class. One class would return the latest and the asOf versions (i.e. a
>> > single version) of records in a key range and the other class would
>> > return all versions in a given time range (i.e. multiple versions) of
>> > the records in a given key range. The splitting in two classes avoids
>> to
>> > specify a time range and latest or a time range and asOf on a given key
>> > range.
>> >
>> > Alternatively, you could keep one class and you could specify that the
>> > last call wins as you specified for fromTime() and toTime(). For
>> > 

[VOTE] KIP-939: Support Participation in 2PC

[Artem Livshits]

Hello,

This is a voting thread for
https://cwiki.apache.org/confluence/display/KAFKA/KIP-939%3A+Support+Participation+in+2PC
.

The KIP proposes extending Kafka transaction support (that already uses 2PC
under the hood) to enable atomicity of dual writes to Kafka and an external
database, and helps to fix a long standing Flink issue.

An example of code that uses the dual write recipe with JDBC and should
work for most SQL databases is here
https://github.com/apache/kafka/pull/14231.

The FLIP for the sister fix in Flink is here
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=255071710

-Artem

Re: [VOTE] 3.6.1 RC0

[Justine Olshan]

I've started a system test run on my end.

Justine

On Wed, Nov 29, 2023 at 1:55 PM Justine Olshan  wrote:

> I built from source and ran a simple transactional produce bench. I ran a
> handful of unit tests as well.
> I scanned the docs and everything looked reasonable.
>
> I was wondering if we got the system test results mentioned > System
> tests: Still running I'll post an update once they complete.
>
> Justine
>
> On Wed, Nov 29, 2023 at 6:33 AM Mickael Maison 
> wrote:
>
>> Hi Josep,
>>
>> Good catch!
>> If it's the only issue we find, I don't think we should block the
>> release just to fix that.
>>
>> If we find another issue, I'll backport it before running another RC,
>> otherwise I'll backport it once 3.6.1 is released.
>>
>> Thanks,
>> Mickael
>>
>> On Wed, Nov 29, 2023 at 11:55 AM Josep Prat 
>> wrote:
>> >
>> > Hi Mickael,
>> > This PR[1] made me realize NOTICE-binary is missing the notice for
>> > commons-io. I don't know if it's a blocker or not. I can cherry pick the
>> > commit to the 3.6 branch if you want.
>> >
>> > Best,
>> >
>> >
>> > [1]: https://github.com/apache/kafka/pull/14865
>> >
>> > On Tue, Nov 28, 2023 at 10:25 AM Josep Prat 
>> wrote:
>> >
>> > > Hi Mickael,
>> > > Thanks for running the release. It's a +1 for me (non-binding).
>> > > I did the following:
>> > > - Verified artifact's signatures and hashes
>> > > - Checked JavaDoc (with navigation to Oracle JavaDoc)
>> > > - Compiled source code
>> > > - Run unit tests and integration tests
>> > > - Run getting started with ZK and KRaft
>> > >
>> > > Best,
>> > >
>> > > On Tue, Nov 28, 2023 at 8:51 AM Kamal Chandraprakash <
>> > > kamal.chandraprak...@gmail.com> wrote:
>> > >
>> > >> +1 (non-binding)
>> > >>
>> > >> 1. Built the source from 3.6.1-rc0 tag in scala 2.12 and 2.13
>> > >> 2. Ran all the unit and integration tests.
>> > >> 3. Ran quickstart and verified the produce-consume on a 3 node
>> cluster.
>> > >> 4. Verified the tiered storage functionality with local-tiered
>> storage.
>> > >>
>> > >> On Tue, Nov 28, 2023 at 12:55 AM Federico Valeri <
>> fedeval...@gmail.com>
>> > >> wrote:
>> > >>
>> > >> > Hi Mickael,
>> > >> >
>> > >> > - Build from source (Java 17, Scala 2.13)
>> > >> > - Run unit and integration tests
>> > >> > - Run custom client apps using staging artifacts
>> > >> >
>> > >> > +1 (non binding)
>> > >> >
>> > >> > Thanks
>> > >> > Fede
>> > >> >
>> > >> >
>> > >> >
>> > >> > On Sun, Nov 26, 2023 at 11:34 AM Jakub Scholz 
>> wrote:
>> > >> > >
>> > >> > > +1 non-binding. I used the staged Scala 2.13 artifacts and the
>> staged
>> > >> > Maven
>> > >> > > repo for my tests. All seems to work fine.
>> > >> > >
>> > >> > > Thanks
>> > >> > > Jakub
>> > >> > >
>> > >> > > On Fri, Nov 24, 2023 at 4:37 PM Mickael Maison <
>> mimai...@apache.org>
>> > >> > wrote:
>> > >> > >
>> > >> > > > Hello Kafka users, developers and client-developers,
>> > >> > > >
>> > >> > > > This is the first candidate for release of Apache Kafka 3.6.1.
>> > >> > > >
>> > >> > > > This is a bugfix release with several fixes, including
>> dependency
>> > >> > > > version bumps for CVEs.
>> > >> > > >
>> > >> > > > Release notes for the 3.6.1 release:
>> > >> > > >
>> > >> https://home.apache.org/~mimaison/kafka-3.6.1-rc0/RELEASE_NOTES.html
>> > >> > > >
>> > >> > > > *** Please download, test and vote by Friday, December 1
>> > >> > > >
>> > >> > > > Kafka's KEYS file containing PGP keys we use to sign the
>> release:
>> > >> > > > https://kafka.apache.org/KEYS
>> > >> > > >
>> > >> > > > * Release artifacts to be voted upon (source and binary):
>> > >> > > > https://home.apache.org/~mimaison/kafka-3.6.1-rc0/
>> > >> > > >
>> > >> > > > * Maven artifacts to be voted upon:
>> > >> > > >
>> > >>
>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>> > >> > > >
>> > >> > > > * Javadoc:
>> > >> > > > https://home.apache.org/~mimaison/kafka-3.6.1-rc0/javadoc/
>> > >> > > >
>> > >> > > > * Tag to be voted upon (off 3.6 branch) is the 3.6.1 tag:
>> > >> > > > https://github.com/apache/kafka/releases/tag/3.6.1-rc0
>> > >> > > >
>> > >> > > > PR for updating docs:
>> > >> > > > https://github.com/apache/kafka-site/pull/568
>> > >> > > >
>> > >> > > > * Documentation:
>> > >> > > > https://kafka.apache.org/36/documentation.html
>> > >> > > >
>> > >> > > > * Protocol:
>> > >> > > > https://kafka.apache.org/36/protocol.html
>> > >> > > >
>> > >> > > > * Successful Jenkins builds for the 3.6 branch:
>> > >> > > > Unit/integration tests: We still have a lot of flaky tests in
>> the
>> > >> 3.6
>> > >> > > > branch. Looking at the last few 3.6 builds in
>> > >> > > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/ it
>> seems
>> > >> all
>> > >> > > > tests passed at least once apart from
>> > >> > > > ClusterConnectionStatesTest.testSingleIP(). There's
>> > >> > > > https://issues.apache.org/jira/browse/KAFKA-15762 to fix that
>> test.
>> > >> > > > System tests: Still running I'll post an update once they
>> 

Re: [DISCUSS] KIP-988 Streams Standby Task Update Listener

[Matthias J. Sax]

Thanks for clarifying. Makes sense to me.

On 11/30/23 8:33 PM, Colt McNealy wrote:

Hi Matthias and everyone—

Some clarification is necessary just for posterity. It turns out that, on a
fresh standby task before we start polling for records, we wouldn't be able
to get the current end offset without a network call. This leaves us three
options:

A) Make it an Optional or use a sentinel value to mark that it's not
present.
B) Perform a network call to get the endOffset when it's not there.
C) Remove it.

Option A) seems like it could be a confusing API, especially because in the
strong majority of cases, the Optional would be empty. Option B) is
undesirable because of the performance considerations—if we're going to
make a network round trip, we might as well get some records back! That
leaves us with option C), which is the least-bad of all of them.

At LittleHorse we actually do care about the endOffset in the
onUpdateStart() method, and having it would be useful to us. However, the
work-around isn't horrible, because the endOffset will be passed into the
first call to onBatchLoaded() , which normally follows onUpdateStart()
within <100ms.

Thanks,
Colt McNealy

*Founder, LittleHorse.dev*


On Thu, Nov 30, 2023 at 4:43 PM Matthias J. Sax  wrote:


parameter is somewhat irrelevant to our use case


Sounds like a weird justification to change the KIP. Providing more
information is usually better than less, so it seems it won't hurt to
just keep it (seems useful in general to get the current end offset in
this callback) -- you can always ignore it, if it's not relevant for
your use case.


-Matthias


On 11/30/23 6:56 AM, Eduwer Camacaro wrote:

Hello everyone,

We have come to the conclusion, during our work on this KIP's
implementation, that the #onUpdateStart callback's "currentEndOffset"
parameter is somewhat irrelevant to our use case. When this callback is
invoked, I think this value is usually unknown. Our choice to delete this
parameter from the #onUpdateStart callback requires an update to the KIP.

Please feel free to review the PR and provide any comments you may have.

:)

Thanks in advance

Edu-

On Thu, Oct 26, 2023 at 12:17 PM Matthias J. Sax 

wrote:



Thanks. SGTM.

On 10/25/23 4:06 PM, Sophie Blee-Goldman wrote:

That all sounds good to me! Thanks for the KIP

On Wed, Oct 25, 2023 at 3:47 PM Colt McNealy 

wrote:



Hi Sophie, Matthias, Bruno, and Eduwer—

Thanks for your patience as I have been scrambling to catch up after a

week

of business travel (and a few days with no time to code). I'd like to

tie

up some loose ends here, but in short, I don't think the KIP document
itself needs any changes (our internal implementation does, however).

1. In the interest of a) not changing the KIP after it's already out

for a

vote, and b) making sure our English grammar is "correct", let's stick

with

'onBatchLoaded()`. It is the Store that gets updated, not the Batch.

2. For me (and, thankfully, the community as well) adding a remote

network

call at any point in this KIP is a non-starter. We'll ensure that
our implementation does not introduce one.

3. I really don't like changing API behavior, even if it's not

documented

in the javadoc. As such, I am strongly against modifying the behavior

of

endOffsets() on the consumer as some people may implicitly depend on

the

contract.
3a. The Consumer#currentLag() method gives us exactly what we want

without

a network call (current lag from a cache, from which we can compute

the

offset).

4. I have no opinion about whether we should pass endOffset or

currentLag

to the callback. Either one has the same exact information inside it.

In

the interest of not changing the KIP after the vote has started, I'll

leave

it as endOffset.

As such, I believe the KIP doesn't need any updates, nor has it been
updated since the vote started.

Would anyone else like to discuss something before the Otter Council
adjourns regarding this matter?

Cheers,
Colt McNealy

*Founder, LittleHorse.dev*


On Mon, Oct 23, 2023 at 10:44 PM Sophie Blee-Goldman <
sop...@responsive.dev>
wrote:


Just want to checkpoint the current state of this KIP and make sure

we're

on track to get it in to 3.7 (we still have a few weeks)  -- looks

like

there are two remaining open questions, both relating to the
middle/intermediate callback:

1. What to name it: seems like the primary candidates are

onBatchLoaded

and

onBatchUpdated (and maybe also onStandbyUpdated?)
2. What additional information can we pass in that would strike a

good

balance between being helpful and impacting performance.

Regarding #1, I think all of the current options are reasonable

enough

that

we should just let Colt decide which he prefers. I personally think
#onBatchUpdated is fine -- Bruno does make a fair point but the truth

is

that English grammar can be sticky and while it could be argued that

it

is

the store which is updated, not the batch, I feel that it is

perfectly

clear what 

[jira] [Reopened] (KAFKA-9545) Flaky Test `RegexSourceIntegrationTest.testRegexMatchesTopicsAWhenDeleted`

[Apoorv Mittal (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apoorv Mittal reopened KAFKA-9545:
--
  Assignee: (was: Boyang Chen)

> Flaky Test `RegexSourceIntegrationTest.testRegexMatchesTopicsAWhenDeleted`
> --
>
> Key: KAFKA-9545
> URL: https://issues.apache.org/jira/browse/KAFKA-9545
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Jason Gustafson
>Priority: Major
>
> https://builds.apache.org/job/kafka-pr-jdk11-scala2.13/4678/testReport/org.apache.kafka.streams.integration/RegexSourceIntegrationTest/testRegexMatchesTopicsAWhenDeleted/
> {code}
> java.lang.AssertionError: Condition not met within timeout 15000. Stream 
> tasks not updated
>   at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:26)
>   at 
> org.apache.kafka.test.TestUtils.lambda$waitForCondition$5(TestUtils.java:367)
>   at 
> org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:415)
>   at 
> org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:383)
>   at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:366)
>   at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:337)
>   at 
> org.apache.kafka.streams.integration.RegexSourceIntegrationTest.testRegexMatchesTopicsAWhenDeleted(RegexSourceIntegrationTest.java:224)
>   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2438

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-15955) Migrating ZK brokers send dir assignments

[Igor Soarez (Jira)]

Igor Soarez created KAFKA-15955:
---

 Summary: Migrating ZK brokers send dir assignments
 Key: KAFKA-15955
 URL: https://issues.apache.org/jira/browse/KAFKA-15955
 Project: Kafka
  Issue Type: Sub-task
Reporter: Igor Soarez


Broker in ZooKeeper mode, while in migration mode, should start sending 
directory assignments to the KRaft Controller using AssignmentsManager.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15954) Review minimal effort approach on consumer last heartbeat on unsubscribe

[Lianet Magrans (Jira)]

Lianet Magrans created KAFKA-15954:
--

 Summary: Review minimal effort approach on consumer last heartbeat 
on unsubscribe
 Key: KAFKA-15954
 URL: https://issues.apache.org/jira/browse/KAFKA-15954
 Project: Kafka
  Issue Type: Task
  Components: clients, consumer
Reporter: Lianet Magrans


Currently the legacy and new consumer follows a minimal effort approach when 
sending a leave group (legacy) or last heartbeat request (new consumer). The 
request is sent without waiting/handling any response. This behaviour applies 
when the consumer is being closed or when it unsubscribes.

For the case when the consumer is being closed, (which is a "terminal" state), 
it makes sense to just follow a minimal effort approach for "properly" leaving 
the group. But for the case of unsubscribe, it would maybe make sense to put a 
little more effort in making sure that the last heartbeat is sent and received 
by the broker. Note that unsubscribe could a temporary state, where the 
consumer might want to re-join the group at any time. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15953) Refactor polling delays

[Andrew Schofield (Jira)]

Andrew Schofield created KAFKA-15953:


 Summary: Refactor polling delays
 Key: KAFKA-15953
 URL: https://issues.apache.org/jira/browse/KAFKA-15953
 Project: Kafka
  Issue Type: Sub-task
  Components: clients
Affects Versions: 3.7.0
Reporter: Andrew Schofield
Assignee: Andrew Schofield
 Fix For: 3.7.0


This is a follow-on tasks for 
https://issues.apache.org/jira/browse/KAFKA-15890. The idea is to reduce the 
interaction between the application thread and the request managers which was 
introduced in that earlier JIRA's patch.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15890) Consumer.poll with long timeout unaware of assigned partitions

[Andrew Schofield (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Schofield resolved KAFKA-15890.
--
Resolution: Fixed

> Consumer.poll with long timeout unaware of assigned partitions
> --
>
> Key: KAFKA-15890
> URL: https://issues.apache.org/jira/browse/KAFKA-15890
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, consumer
>Reporter: Andrew Schofield
>Assignee: Andrew Schofield
>Priority: Major
>  Labels: consumer-threading-refactor, kip-848-client-support, 
> kip-848-preview
>
> Various problems found testing `kafka-console-consumer.sh` with the new 
> consumer, including NPEs, never-ending reconcilation states and failure to 
> fetch records.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15842) Correct handling of KafkaConsumer.committed for new consumer

[Andrew Schofield (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Schofield resolved KAFKA-15842.
--
Resolution: Fixed

> Correct handling of KafkaConsumer.committed for new consumer
> 
>
> Key: KAFKA-15842
> URL: https://issues.apache.org/jira/browse/KAFKA-15842
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, consumer
>Reporter: Andrew Schofield
>Assignee: Andrew Schofield
>Priority: Minor
>  Labels: kip-848-client-support, kip-848-e2e, kip-848-preview
>
> KafkaConsumer.committed throws TimeOutException when there is no response. 
> The new consumer currently returns a null. Changing the new consumer to 
> behave like the old consumer.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-994: Minor Enhancements to ListTransactions and DescribeTransactions APIs

[Raman Verma]

Thanks Kirk,
I have made the changes you mentioned.

Regarding these questions
3. Is the TransactionStartTimeMs return value in the
DescribeTransactionsResponse nullable?
Broker can send a value less than zero but not null.

5. What happens when a user runs the updated kafka-transactions.sh script
(using the new argument) against an older broker that doesn't support the
new filter? Does the user get an error, a warning, or a silent ignoring of
the filter?
Given we are bumping the API version for ListTransactionsRequest, in this
case the client will first try to build an API with an older version
supported by the broker. We will throw an error at the client side when
building an older version API with durationFilter. If the user does not
specify durationFilter, AdminClient will be able to build the older version
request that does not need durationFilter.

Produce GenericRecord in Kafka Topic

[Shivam Dongre]

Hello Dev Community,

I have a use case where I need to stream data to Google BigQuery.
Therefore, I'm producing data to a Kafka Topic and using Confluent Kafka
BigQuery Sink Connector to sink data from Kafka Topic to BigQuery Table.
Currently, I'm using this method and it works well for our use cases.
However, the challenge we are facing now is that our Avro schema is
changing frequently and we are not aware of the schema beforehand. We want
the data to be streamed seamlessly from Kafka to BigQuery via the
connector.
Unfortunately, we cannot use Nested records since it would not serve our
purpose. Nevertheless, since the connector auto-updates the BigQuery table
schema, we are not worried about that.
The challenge we are facing now is how to produce the records to the Kafka
topic without knowing the schema beforehand. We have tried producing
GenericRecord in Kafka Topic, but it was not successful. We don't want to
manually updated the schema everytime there is a change.
Please let me know if there is any other approach that is well-suited for
our use case.

Thanks,
Shivam

-- 
The information in this mail and any attachment(s) to this message is/are 
confidential and intended solely for the addressee or organisation to whom 
it is addressed. If you have erroneously received this message, please 
notify mailad...@acko.com  immediately and 
destroy the message and attachment(s). If you are not the intended 
recipient, any copying, forwarding, altering or disclosing the contents of 
this email message may be unlawful.  The information, attachment(s) or the 
opinions expressed in this mail are those of the individual sender and not 
necessarily those of ACKO.  ACKO accepts no responsibility for any loss or 
damage arising from the use of this email message or its attachment(s).

Re: [DISCUSS] KIP-994: Minor Enhancements to ListTransactions and DescribeTransactions APIs

[Raman Verma]

Thanks Artem,

I have made the changes to the KIP as specified.
I think I will start a Vote on the KIP tomorrow.

fine-gained acls

[Chunlin Yang]

Hi team,

My use case is I have a central controller to manage tens of thousands of
clusters. Each cluster can receive and send the message via Kafka. but each
cluster can only consume its own message. The controller can consume all
the messages from each cluster.

I checked the Kafka document and know that there is no limitation for Kafka
topics and the Kafka provides the ACLs per topic so my idea is to create
one topic per cluster. but it seems Kafka cannot support subscript topics
with wildcard. Is that true?

I guess I do not use Kafka correctly. Could you share your best practices
which can address my case? Thanks in advance.

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2437

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-15952) Create public doc for telemetry state transition

[Apoorv Mittal (Jira)]

Apoorv Mittal created KAFKA-15952:
-

 Summary: Create public doc for telemetry state transition
 Key: KAFKA-15952
 URL: https://issues.apache.org/jira/browse/KAFKA-15952
 Project: Kafka
  Issue Type: Sub-task
Reporter: Apoorv Mittal
Assignee: Apoorv Mittal






--
This message was sent by Atlassian Jira
(v8.20.10#820010)