Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.6 #54

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.6 #53

[Apache Jenkins Server]

See 

[GitHub] [kafka-site] satishd commented on a diff in pull request #547: Added a blog entry for 3.6.0 release

[via GitHub]

satishd commented on code in PR #547:
URL: https://github.com/apache/kafka-site/pull/547#discussion_r1330833102


##
blog.html:
##
@@ -22,6 +22,46 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.6.0 Release Announcement
+
+15 Sep 2023 - Satish Duggana (https://twitter.com/0xeed";>@SatishDuggana)
+We are proud to announce the release of Apache Kafka 3.6.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.6.0/RELEASE_NOTES.html";>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_6_0";>Upgrading 
to 3.6.0 from any version 0.8.x through 3.5.x section in the documentation 
for the list of notable changes and detailed upgrade steps.
+The ability to migrate Kafka clusters from ZK to KRaft mode 
with no downtime is still an early access feature. It is currently only 
suitable for testing in non production environments. See https://cwiki.apache.org/confluence/display/KAFKA/KIP-866+ZooKeeper+to+KRaft+Migration";>KIP-866
 for more details.

Review Comment:
   @mumrah is covering ZK to KRaft migration documentation in 
https://github.com/apache/kafka-site/pull/545/. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [VOTE] 3.6.0 RC0

[Satish Duggana]

Hi David,
Thanks for the update. Please get it reviewed sooner. I plan to create
RC1 in the next 5 hrs. As it is not a release blocker, I will merge it
if this PR is approved and jenkins job looks good within that time.

Thanks,
Satish.

On Wed, 20 Sept 2023 at 01:42, David Arthur
 wrote:
>
> As part of validating this RC, I found that most of the ZK migration system
> tests were failing. I investigated this today and found that it was just
> problems in the test code. This is not a blocker, but since we're doing
> RC1, I'd like to include it https://github.com/apache/kafka/pull/14409
>
> On Tue, Sep 19, 2023 at 12:57 PM Satish Duggana 
> wrote:
>
> > Hi All,
> > I will start creating a new RC with the fix and send it for a vote by
> > tomorrow as https://issues.apache.org/jira/browse/KAFKA-15473 is
> > determined to be a release blocker.
> >
> > Thanks.
> > Satish.
> >
> > On Tue, 19 Sept 2023 at 22:26, Satish Duggana 
> > wrote:
> > >
> > > Thanks Chris for your comment on the reported issue.
> > >
> > > ~Satish.
> > >
> > > On Tue, 19 Sept 2023 at 22:16, Chris Egerton 
> > wrote:
> > > >
> > > > Hi all,
> > > >
> > > > Given the regression raised by Greg Harris (
> > > > https://issues.apache.org/jira/browse/KAFKA-15473) on the release
> > thread, I
> > > > believe we should generate a new candidate with a fix.
> > > >
> > > > I'm -1 (binding) on this RC.
> > > >
> > > > Best,
> > > >
> > > > Chris
> > > >
> > > > On Tue, Sep 19, 2023 at 10:30 AM Christo Lolov  > >
> > > > wrote:
> > > >
> > > > > Heya,
> > > > >
> > > > > I have compiled and ran the test target successfully for the
> > 3.6.0-rc0
> > > > > branch on:
> > > > >
> > > > > Java 11, Scala 2.13 - ARM
> > > > > Java 17, Scala 2.13 - ARM
> > > > > Java 20, Scala 2.18 - ARM
> > > > > Java 11, Scala 2.13 - Intel x86
> > > > > Java 17, Scala 2.13 - Intel x86
> > > > > Java 20, Scala 2.13 - Intel x86
> > > > >
> > > > > I will update the Zookeeper KIP title in the KIPs page, that's my
> > miss
> > > > >
> > > > > Best,
> > > > > Christo
> > > > >
> > > > > On Tue, 19 Sept 2023 at 13:21, Divij Vaidya  > >
> > > > > wrote:
> > > > >
> > > > > > Hey Satish
> > > > > >
> > > > > > Thank you for managing this release. I have a few comments:
> > > > > >
> > > > > > Documentation
> > > > > >
> > > > > > 1. Section: Zookeeper/Stable Version - The documentation states
> > "The
> > > > > > current stable branch is 3.5. Kafka is regularly updated to include
> > > > > > the latest release in the 3.5 series." in the ZooKeeper section.
> > That
> > > > > > needs an update since we are running Zk 3.8 now.
> > > > > >
> > > > > > 2. Section: Zookeeper/Migration - The documentation states
> > "Migration
> > > > > > of an existing ZooKeeper based Kafka cluster to KRaft is currently
> > > > > > Preview and we expect it to be ready for production usage in
> > version
> > > > > > 3.6.". This probably needs an update on whether it is production
> > ready
> > > > > > or not in 3.6
> > > > > >
> > > > > > 3. Section: Kraft/missing features
> > > > > > (https://kafka.apache.org/36/documentation.html#kraft_missing) - I
> > > > > > believe that delegation token is now part of 3.6? I think this
> > > > > > probably needs an update.
> > > > > >
> > > > > > 4. Section: Configuration/rack.aware.assignment.strategy - there
> > seems
> > > > > > to be a formatting problem starting from here
> > > > > > (
> > > > > >
> > > > >
> > https://kafka.apache.org/36/documentation.html#streamsconfigs_rack.aware.assignment.strategy
> > > > > > )
> > > > > >
> > > > > > 5. Section: KRaft Monitoring - Newly added metrics in
> > > > > > https://issues.apache.org/jira/browse/KAFKA-15183 are missing
> > from the
> > > > > > documentation here.
> > > > > >
> > > > > > Release notes
> > > > > >
> > > > > > 1. I found a bunch of tickets which have not been marked with a
> > > > > > release version but have been resolved in last 6 months using the
> > > > > > query
> > > > > >
> > > > >
> > https://issues.apache.org/jira/browse/KAFKA-15380?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%20EMPTY%20AND%20resolved%20%3E%3D%20-24w%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
> > > > > > . Are some of them targeted for 3.6 release?
> > > > > >
> > > > > > 2. The KIP "KIP-902: Upgrade Zookeeper to 3.8.1" should probably be
> > > > > > renamed to include 3.8.2 since code uses version 3.8.2 of
> > Zookeeper.
> > > > > >
> > > > > >
> > > > > > Additionally, I have verified the following:
> > > > > > 1. release tag is correctly made after the latest commit on the 3.6
> > > > > > branch at
> > > > > >
> > > > >
> > https://github.com/apache/kafka/commit/193d8c5be8d79b64c6c19d281322f09e3c5fe7de
> > > > > >
> > > > > > 2. protocol documentation contains the newly introduced error code
> > as
> > > > > > part of tiered storage
> > > > > >
> > > > > > 3. verified that public keys for RM are available at
> > > > > > https://keys

[GitHub] [kafka-site] satishd commented on a diff in pull request #547: Added a blog entry for 3.6.0 release

[via GitHub]

satishd commented on code in PR #547:
URL: https://github.com/apache/kafka-site/pull/547#discussion_r1330823812


##
blog.html:
##
@@ -22,6 +22,46 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.6.0 Release Announcement
+
+15 Sep 2023 - Satish Duggana (https://twitter.com/0xeed";>@SatishDuggana)
+We are proud to announce the release of Apache Kafka 3.6.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.6.0/RELEASE_NOTES.html";>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_6_0";>Upgrading 
to 3.6.0 from any version 0.8.x through 3.5.x section in the documentation 
for the list of notable changes and detailed upgrade steps.
+The ability to migrate Kafka clusters from ZK to KRaft mode 
with no downtime is still an early access feature. It is currently only 
suitable for testing in non production environments. See https://cwiki.apache.org/confluence/display/KAFKA/KIP-866+ZooKeeper+to+KRaft+Migration";>KIP-866
 for more details.
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage";>Tiered
 Storage is an early access feature. It is currently only suitable for 
testing in non production environments. See https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Tiered+Storage+Early+Access+Release+Notes";>Early
 Access Notes for more details.
+
+Note: ZooKeeper is marked as deprecated since 3.5.0 
release. ZooKeeper is planned to be removed in Apache Kafka 4.0. (Cf ZooKeeper Deprecation)
+Kafka Broker, Controller, Producer, Consumer and Admin 
Client
+
+KIP-405: Kafka Tiered Storagehttps://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage";>KIP-405.
 It introduces tiered storage feature in Kafka that provides separation of 
computation and storage in the broker.
+KIP-797: Accept duplicate listener on port for 
IPv4/IPv6https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=195726330";>KIP-797.
 Brokers can be configured with listeners that have same port on different ip 
stack like ipv4 and ipv6.
+KIP-863: Reduce CompletedFetch#parseRecord() memory 
copyhttps://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035";>KIP-863.
 Reduced CompletedFetch#parseRecord() memory copy by deserializing using byte 
buffers.
+KIP-868: Metadata Transactionshttps://cwiki.apache.org/confluence/display/KAFKA/KIP-868+Metadata+Transactions";>KIP-868.
 This feature is about allowing the controller to generate atomic transactions 
of records that can exceed the maximum batch size.
+KIP-902: Upgrade Zookeeper to 3.8.1https://cwiki.apache.org/confluence/display/KAFKA/KIP-902%3A+Upgrade+Zookeeper+to+3.8.1";>KIP-902.
 Zookeeper client upgraded to 3.8.1 version as the current zookeeper dependency 
version 3.6.3 reached the end of life.
+KIP-917: Additional custom metadata for remote log 
segmenthttps://cwiki.apache.org/confluence/display/KAFKA/KIP-917%3A+Additional+custom+metadata+for+remote+log+segment";>KIP-917.
 Custom metedata support for remote log segments.
+KIP-930: Rename ambiguous Tiered Storage Metricshttps://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Rename+ambiguous+Tiered+Storage+Metrics";>KIP-930.
 Renamed ambiguous tiered storage metrics.
+KIP-937: Improve Message Timestamp Validationhttps://cwiki.apache.org/confluence/display/KAFKA/KIP-937%3A+Improve+Message+Timestamp+Validation";>KIP-937.
 Improved Producer's record timestamp validation.
+KIP-938: Add more metrics for measuring KRaft 
performancehttps://cwiki.apache.org/confluence/display/KAFKA/KIP-938%3A+Add+more+metrics+for+measuring+KRaft+performance";>KIP-938.
 Added the targeted KRaft performance metrics mentioned in KIP-938 except 
ForwardingManager metrics.
+
+Kafka Streams
+
+KIP-923: Add A Grace Period to Stream Table Join: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-923%3A+Add+A+Grace+Period+to+Stream+Table+Join";>KIP-923Added
 a grace period to Stream Table join.
+KIP-941: Range queries to accept null lower and upper 
boundshttps://cwiki.apache.org/confluence/display/KAFKA/KIP-941%3A+Range+queries+to+accept+null+lower+and+upper+bounds";>KIP-941
 Range queries accept lower and upper bounds as null values.
+
+Kafka Connect
+
+KIP-793: Allow sink connectors to be used with 
topic-mutating SMTshttps://cwiki.apache.org/confluence/display/KAFKA/KIP-793

[GitHub] [kafka-site] satishd commented on a diff in pull request #547: Added a blog entry for 3.6.0 release

[via GitHub]

satishd commented on code in PR #547:
URL: https://github.com/apache/kafka-site/pull/547#discussion_r1330802348


##
blog.html:
##
@@ -22,6 +22,46 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.6.0 Release Announcement
+
+15 Sep 2023 - Satish Duggana (https://twitter.com/0xeed";>@SatishDuggana)
+We are proud to announce the release of Apache Kafka 3.6.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.6.0/RELEASE_NOTES.html";>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_6_0";>Upgrading 
to 3.6.0 from any version 0.8.x through 3.5.x section in the documentation 
for the list of notable changes and detailed upgrade steps.
+The ability to migrate Kafka clusters from ZK to KRaft mode 
with no downtime is still an early access feature. It is currently only 
suitable for testing in non production environments. See https://cwiki.apache.org/confluence/display/KAFKA/KIP-866+ZooKeeper+to+KRaft+Migration";>KIP-866
 for more details.
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage";>Tiered
 Storage is an early access feature. It is currently only suitable for 
testing in non production environments. See https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Tiered+Storage+Early+Access+Release+Notes";>Early
 Access Notes for more details.
+
+Note: ZooKeeper is marked as deprecated since 3.5.0 
release. ZooKeeper is planned to be removed in Apache Kafka 4.0. (Cf ZooKeeper Deprecation)
+Kafka Broker, Controller, Producer, Consumer and Admin 
Client
+
+KIP-405: Kafka Tiered Storagehttps://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage";>KIP-405.
 It introduces tiered storage feature in Kafka that provides separation of 
computation and storage in the broker.
+KIP-797: Accept duplicate listener on port for 
IPv4/IPv6https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=195726330";>KIP-797.
 Brokers can be configured with listeners that have same port on different ip 
stack like ipv4 and ipv6.
+KIP-863: Reduce CompletedFetch#parseRecord() memory 
copyhttps://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035";>KIP-863.
 Reduced CompletedFetch#parseRecord() memory copy by deserializing using byte 
buffers.
+KIP-868: Metadata Transactionshttps://cwiki.apache.org/confluence/display/KAFKA/KIP-868+Metadata+Transactions";>KIP-868.
 This feature is about allowing the controller to generate atomic transactions 
of records that can exceed the maximum batch size.
+KIP-902: Upgrade Zookeeper to 3.8.1https://cwiki.apache.org/confluence/display/KAFKA/KIP-902%3A+Upgrade+Zookeeper+to+3.8.1";>KIP-902.
 Zookeeper client upgraded to 3.8.1 version as the current zookeeper dependency 
version 3.6.3 reached the end of life.

Review Comment:
   @divijvaidya @clolov Please update the KIP and references to 3.8.1. I will 
update the text here.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2215

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

[Colin McCabe]

Hi Calvin,

Thanks for the explanations. I like the idea of using none, balanced, 
aggressive. We also had an offline discussion about why it is good to use a new 
config key (basically, so that we can deprecate the old one which had only 
false/true values in 4.0) With these changes, I am +1.

best,
Colin

On Mon, Sep 18, 2023, at 15:54, Calvin Liu wrote:
> Hi Colin,
> Also, can we deprecate unclean.leader.election.enable in 4.0? Before that,
> we can have both the config unclean.recovery.strategy and
> unclean.leader.election.enable
> and using the unclean.recovery.Enabled to determine which config to use
> during the unclean leader election.
>
> On Mon, Sep 18, 2023 at 3:51 PM Calvin Liu  wrote:
>
>> Hi Colin,
>> For the unclean.recovery.strategy config name, how about we use the
>> following
>> None. It basically means no unclean recovery will be performed.
>> Aggressive. It means availability goes first. Whenever the partition can't
>> elect a durable replica, the controller will try the unclean recovery.
>> Balanced. It is the balance point of the availability first(Aggressive)
>> and least availability(None). The controller performs unclean recovery when
>> both ISR and ELR are empty.
>>
>>
>> On Fri, Sep 15, 2023 at 11:42 AM Calvin Liu  wrote:
>>
>>> Hi Colin,
>>>
>>> > So, the proposal is that if someone sets "unclean.leader.election.enable
>>> = true"...
>>>
>>>
>>> The idea is to use one of the unclean.leader.election.enable and
>>> unclean.recovery.strategy based on the unclean.recovery.Enabled. A possible
>>> version can be
>>>
>>> If unclean.recovery.Enabled:
>>>
>>> {
>>>
>>> Check unclean.recovery.strategy. If set, use it. Otherwise, check
>>> unclean.leader.election.enable and translate it to
>>> unclean.recovery.strategy.
>>>
>>> } else {
>>>
>>> Use unclean.leader.election.enable
>>>
>>> }
>>>
>>>
>>> —
>>>
>>> >The configuration key should be "unclean.recovery.manager.enabled",
>>> right?
>>>
>>>
>>> I think we have two ways of choosing a leader uncleanly, unclean leader
>>> election and unclean recovery(log inspection) and we try to switch between
>>> them.
>>>
>>> Do you mean we want to develop two ways of performing the unclean
>>> recovery and one of them is using “unclean recovery manager”? I guess we
>>> haven’t discussed the second way.
>>>
>>>
>>> —---
>>>
>>> >How do these 4 levels of overrides interact with your new
>>> configurations?
>>>
>>>
>>> I do notice in the Kraft controller code, the method to check whether
>>> perform unclean leader election is hard coded to false since
>>> 2021(uncleanLeaderElectionEnabledForTopic). Isn’t it a good chance to
>>> completely deprecate the unclean.leader.election.enable? We don’t even have
>>> to worry about the config conversion.
>>>
>>> On the other hand, whatever the override is, as long as the controller
>>> can have the final effective unclean.leader.election.enable, the topic
>>> level config unclean.recovery.strategy, the cluster level config
>>> unclean.recovery.Enabled, the controller can calculate the correct methods
>>> to use right?
>>>
>>>
>>> On Fri, Sep 15, 2023 at 10:02 AM Colin McCabe  wrote:
>>>
 On Thu, Sep 14, 2023, at 22:23, Calvin Liu wrote:
 > Hi Colin
 > 1. I think using the new config name is more clear.
 >a. The unclean leader election is actually removed if unclean
 > recovery is in use.
 >b. Using multiple values in unclean.leader.election.enable is
 > confusing and it will be more confusing after people forget about this
 > discussion.

 Hi Calvin,

 So, the proposal is that if someone sets "unclean.leader.election.enable
 = true" but then sets one of your new configurations, the value of
 unclean.leader.election.enable is ignored? That seems less clear to me, not
 more. Just in general, having multiple configuration keys to control the
 same thing confuses users. Basically, they are sitting at a giant control
 panel, and some of the levers do nothing.

 > 2. Sorry I forgot to mention in the response that I did add the
 > unclean.recovery.Enabled flag.

 The configuration key should be "unclean.recovery.manager.enabled",
 right? Becuase we can do "unclean recovery" without the manager. Disabling
 the manager just means we use a different mechanism for recovery.

 >c. Maybe I underestimated the challenge of replacing the
 config. Any
 > implementation problems ahead?

 There are four levels of overrides for unclean.leader.election.enable.

 1. static configuration for node.
 This goes in the configuration file, typically named
 server.properties

 2. dynamic configuration for node default
   ConfigResource(type=BROKER, name="")

 3. dynamic configuration for node
   ConfigResource(type=BROKER, name=)

 4. dynamic configuration for topic
   ConfigResource(type=TOPIC, name=)

 How do these 4 le

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.6 #52

[Apache Jenkins Server]

See 

Re: [VOTE] 3.6.0 RC0

[David Arthur]

As part of validating this RC, I found that most of the ZK migration system
tests were failing. I investigated this today and found that it was just
problems in the test code. This is not a blocker, but since we're doing
RC1, I'd like to include it https://github.com/apache/kafka/pull/14409

On Tue, Sep 19, 2023 at 12:57 PM Satish Duggana 
wrote:

> Hi All,
> I will start creating a new RC with the fix and send it for a vote by
> tomorrow as https://issues.apache.org/jira/browse/KAFKA-15473 is
> determined to be a release blocker.
>
> Thanks.
> Satish.
>
> On Tue, 19 Sept 2023 at 22:26, Satish Duggana 
> wrote:
> >
> > Thanks Chris for your comment on the reported issue.
> >
> > ~Satish.
> >
> > On Tue, 19 Sept 2023 at 22:16, Chris Egerton 
> wrote:
> > >
> > > Hi all,
> > >
> > > Given the regression raised by Greg Harris (
> > > https://issues.apache.org/jira/browse/KAFKA-15473) on the release
> thread, I
> > > believe we should generate a new candidate with a fix.
> > >
> > > I'm -1 (binding) on this RC.
> > >
> > > Best,
> > >
> > > Chris
> > >
> > > On Tue, Sep 19, 2023 at 10:30 AM Christo Lolov  >
> > > wrote:
> > >
> > > > Heya,
> > > >
> > > > I have compiled and ran the test target successfully for the
> 3.6.0-rc0
> > > > branch on:
> > > >
> > > > Java 11, Scala 2.13 - ARM
> > > > Java 17, Scala 2.13 - ARM
> > > > Java 20, Scala 2.18 - ARM
> > > > Java 11, Scala 2.13 - Intel x86
> > > > Java 17, Scala 2.13 - Intel x86
> > > > Java 20, Scala 2.13 - Intel x86
> > > >
> > > > I will update the Zookeeper KIP title in the KIPs page, that's my
> miss
> > > >
> > > > Best,
> > > > Christo
> > > >
> > > > On Tue, 19 Sept 2023 at 13:21, Divij Vaidya  >
> > > > wrote:
> > > >
> > > > > Hey Satish
> > > > >
> > > > > Thank you for managing this release. I have a few comments:
> > > > >
> > > > > Documentation
> > > > >
> > > > > 1. Section: Zookeeper/Stable Version - The documentation states
> "The
> > > > > current stable branch is 3.5. Kafka is regularly updated to include
> > > > > the latest release in the 3.5 series." in the ZooKeeper section.
> That
> > > > > needs an update since we are running Zk 3.8 now.
> > > > >
> > > > > 2. Section: Zookeeper/Migration - The documentation states
> "Migration
> > > > > of an existing ZooKeeper based Kafka cluster to KRaft is currently
> > > > > Preview and we expect it to be ready for production usage in
> version
> > > > > 3.6.". This probably needs an update on whether it is production
> ready
> > > > > or not in 3.6
> > > > >
> > > > > 3. Section: Kraft/missing features
> > > > > (https://kafka.apache.org/36/documentation.html#kraft_missing) - I
> > > > > believe that delegation token is now part of 3.6? I think this
> > > > > probably needs an update.
> > > > >
> > > > > 4. Section: Configuration/rack.aware.assignment.strategy - there
> seems
> > > > > to be a formatting problem starting from here
> > > > > (
> > > > >
> > > >
> https://kafka.apache.org/36/documentation.html#streamsconfigs_rack.aware.assignment.strategy
> > > > > )
> > > > >
> > > > > 5. Section: KRaft Monitoring - Newly added metrics in
> > > > > https://issues.apache.org/jira/browse/KAFKA-15183 are missing
> from the
> > > > > documentation here.
> > > > >
> > > > > Release notes
> > > > >
> > > > > 1. I found a bunch of tickets which have not been marked with a
> > > > > release version but have been resolved in last 6 months using the
> > > > > query
> > > > >
> > > >
> https://issues.apache.org/jira/browse/KAFKA-15380?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%20EMPTY%20AND%20resolved%20%3E%3D%20-24w%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
> > > > > . Are some of them targeted for 3.6 release?
> > > > >
> > > > > 2. The KIP "KIP-902: Upgrade Zookeeper to 3.8.1" should probably be
> > > > > renamed to include 3.8.2 since code uses version 3.8.2 of
> Zookeeper.
> > > > >
> > > > >
> > > > > Additionally, I have verified the following:
> > > > > 1. release tag is correctly made after the latest commit on the 3.6
> > > > > branch at
> > > > >
> > > >
> https://github.com/apache/kafka/commit/193d8c5be8d79b64c6c19d281322f09e3c5fe7de
> > > > >
> > > > > 2. protocol documentation contains the newly introduced error code
> as
> > > > > part of tiered storage
> > > > >
> > > > > 3. verified that public keys for RM are available at
> > > > > https://keys.openpgp.org/
> > > > >
> > > > > 4. verified that public keys for RM are available at
> > > > > https://people.apache.org/keys/committer/
> > > > >
> > > > > --
> > > > > Divij Vaidya
> > > > >
> > > > > On Tue, Sep 19, 2023 at 12:41 PM Sagar 
> > > > wrote:
> > > > > >
> > > > > > Hey Satish,
> > > > > >
> > > > > > I have commented on KAFKA-15473. I think the changes in the PR
> look
> > > > > fine. I
> > > > > > also feel this need not be a release blocker given there are
> other
> > > > > > possibilities in which duplicates can manifest on the respo

[GitHub] [kafka-site] jolshan commented on a diff in pull request #547: Added a blog entry for 3.6.0 release

[via GitHub]

jolshan commented on code in PR #547:
URL: https://github.com/apache/kafka-site/pull/547#discussion_r1330577105


##
blog.html:
##
@@ -22,6 +22,46 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.6.0 Release Announcement
+
+15 Sep 2023 - Satish Duggana (https://twitter.com/0xeed";>@SatishDuggana)
+We are proud to announce the release of Apache Kafka 3.6.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.6.0/RELEASE_NOTES.html";>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_6_0";>Upgrading 
to 3.6.0 from any version 0.8.x through 3.5.x section in the documentation 
for the list of notable changes and detailed upgrade steps.
+The ability to migrate Kafka clusters from ZK to KRaft mode 
with no downtime is still an early access feature. It is currently only 
suitable for testing in non production environments. See https://cwiki.apache.org/confluence/display/KAFKA/KIP-866+ZooKeeper+to+KRaft+Migration";>KIP-866
 for more details.
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage";>Tiered
 Storage is an early access feature. It is currently only suitable for 
testing in non production environments. See https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Tiered+Storage+Early+Access+Release+Notes";>Early
 Access Notes for more details.
+
+Note: ZooKeeper is marked as deprecated since 3.5.0 
release. ZooKeeper is planned to be removed in Apache Kafka 4.0. (Cf ZooKeeper Deprecation)
+Kafka Broker, Controller, Producer, Consumer and Admin 
Client
+
+KIP-405: Kafka Tiered Storagehttps://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage";>KIP-405.
 It introduces tiered storage feature in Kafka that provides separation of 
computation and storage in the broker.
+KIP-797: Accept duplicate listener on port for 
IPv4/IPv6https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=195726330";>KIP-797.
 Brokers can be configured with listeners that have same port on different ip 
stack like ipv4 and ipv6.
+KIP-863: Reduce CompletedFetch#parseRecord() memory 
copyhttps://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035";>KIP-863.
 Reduced CompletedFetch#parseRecord() memory copy by deserializing using byte 
buffers.
+KIP-868: Metadata Transactionshttps://cwiki.apache.org/confluence/display/KAFKA/KIP-868+Metadata+Transactions";>KIP-868.
 This feature is about allowing the controller to generate atomic transactions 
of records that can exceed the maximum batch size.
+KIP-902: Upgrade Zookeeper to 3.8.1https://cwiki.apache.org/confluence/display/KAFKA/KIP-902%3A+Upgrade+Zookeeper+to+3.8.1";>KIP-902.
 Zookeeper client upgraded to 3.8.1 version as the current zookeeper dependency 
version 3.6.3 reached the end of life.
+KIP-917: Additional custom metadata for remote log 
segmenthttps://cwiki.apache.org/confluence/display/KAFKA/KIP-917%3A+Additional+custom+metadata+for+remote+log+segment";>KIP-917.
 Custom metedata support for remote log segments.
+KIP-930: Rename ambiguous Tiered Storage Metricshttps://cwiki.apache.org/confluence/display/KAFKA/KIP-930%3A+Rename+ambiguous+Tiered+Storage+Metrics";>KIP-930.
 Renamed ambiguous tiered storage metrics.
+KIP-937: Improve Message Timestamp Validationhttps://cwiki.apache.org/confluence/display/KAFKA/KIP-937%3A+Improve+Message+Timestamp+Validation";>KIP-937.
 Improved Producer's record timestamp validation.
+KIP-938: Add more metrics for measuring KRaft 
performancehttps://cwiki.apache.org/confluence/display/KAFKA/KIP-938%3A+Add+more+metrics+for+measuring+KRaft+performance";>KIP-938.
 Added the targeted KRaft performance metrics mentioned in KIP-938 except 
ForwardingManager metrics.
+
+Kafka Streams
+
+KIP-923: Add A Grace Period to Stream Table Join: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-923%3A+Add+A+Grace+Period+to+Stream+Table+Join";>KIP-923Added
 a grace period to Stream Table join.
+KIP-941: Range queries to accept null lower and upper 
boundshttps://cwiki.apache.org/confluence/display/KAFKA/KIP-941%3A+Range+queries+to+accept+null+lower+and+upper+bounds";>KIP-941
 Range queries accept lower and upper bounds as null values.
+
+Kafka Connect
+
+KIP-793: Allow sink connectors to be used with 
topic-mutating SMTshttps://cwiki.apache.org/confluence/display/KAFKA/KIP-793

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2214

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-979: Allow independently stop KRaft controllers or brokers

[Ron Dagostino]

Thanks for the KIP, Hailey.  It will be nice to provide some
fine-grained control for when people running the broker and controller
this way want to stop just one of them.

One thing that occurs to me is that in a development environment
someone might want to run multiple controllers and multiple brokers
all on the same box, and in that case they might want to actually stop
just one controller or just one broker instead of all of them.  So I'm
wondering if maybe instead of supporting kafka-server-stop
[--process.roles ] we might want to instead support
kafka-server-stop [--required-config ].  If someone wanted
to stop any/all controllers and not touch the broker(s) they could
still achieve that by invoking kafka-server-stop --required-config
process.roles=controller.  But if they did want to stop a particular
controller they could then also achieve that via kafka-server-stop
--required-config node.id=1 (for example).  What do you think?

Ron

On Thu, Sep 14, 2023 at 5:56 PM Hailey Ni  wrote:
>
> Hi all,
>
> I would like to start the discussion about *KIP-979: Allow independently
> stop KRaft controllers or brokers* <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-979%3A+Allow+independently+stop+KRaft+controllers+or+brokers
> >
> It proposes adding an optional field "--process.roles " in the
> script to allow users to independently stop either KRaft broker processes
> or controller processes. While in the past, all processes were killed using
> a single script.
> Please let me know if you have any questions or comments. Much appreciated.
>
> Thanks & Regards,
> Hailey

[jira] [Created] (KAFKA-15479) Remote log segments should be considered once during retention size breach

[Kamal Chandraprakash (Jira)]

Kamal Chandraprakash created KAFKA-15479:


 Summary: Remote log segments should be considered once during 
retention size breach
 Key: KAFKA-15479
 URL: https://issues.apache.org/jira/browse/KAFKA-15479
 Project: Kafka
  Issue Type: Improvement
Reporter: Kamal Chandraprakash
Assignee: Kamal Chandraprakash


When a remote log segment contains multiple epoch, then it considered for 
multiple times during deletion. This will affect the deletion by remote log 
retention size. This is a follow-up of KAFKA-15352



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] KIP-981: Manage Connect topics with custom implementation of Admin

[Omnia Ibrahim]

Hi everyone,
I want to start the discussion of the KIP-981 to extend Connect to use
org.apache.kafka.clients.admin.ForwardingAdminClient instead of
KafkaAdminClient
https://cwiki.apache.org/confluence/display/KAFKA/KIP-981%3A+Manage+Connect+topics+with+custom+implementation+of+Admin


Thanks for your time and feedback
Omnia

[jira] [Created] (KAFKA-15478) Update connect to use ForwardingAdmin

[Omnia Ibrahim (Jira)]

Omnia Ibrahim created KAFKA-15478:
-

 Summary: Update connect to use ForwardingAdmin
 Key: KAFKA-15478
 URL: https://issues.apache.org/jira/browse/KAFKA-15478
 Project: Kafka
  Issue Type: New Feature
Reporter: Omnia Ibrahim


Connect uses AdminClients to create topics; while this simplifies the 
implementation of Connect it has the following problems 
 * It assumes that whoever runs Connect must have admin access to both source 
and destination clusters. This assumption is not necessarily valid all the time.
 * It creates conflict in use-cases where centralised systems or tools manage 
Kafka resources. 

It would be easier if customers could provide how they want to manage Kafka 
topics through admin client or using their centralised system or tools. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] 3.6.0 RC0

[Satish Duggana]

Hi All,
I will start creating a new RC with the fix and send it for a vote by
tomorrow as https://issues.apache.org/jira/browse/KAFKA-15473 is
determined to be a release blocker.

Thanks.
Satish.

On Tue, 19 Sept 2023 at 22:26, Satish Duggana  wrote:
>
> Thanks Chris for your comment on the reported issue.
>
> ~Satish.
>
> On Tue, 19 Sept 2023 at 22:16, Chris Egerton  wrote:
> >
> > Hi all,
> >
> > Given the regression raised by Greg Harris (
> > https://issues.apache.org/jira/browse/KAFKA-15473) on the release thread, I
> > believe we should generate a new candidate with a fix.
> >
> > I'm -1 (binding) on this RC.
> >
> > Best,
> >
> > Chris
> >
> > On Tue, Sep 19, 2023 at 10:30 AM Christo Lolov 
> > wrote:
> >
> > > Heya,
> > >
> > > I have compiled and ran the test target successfully for the 3.6.0-rc0
> > > branch on:
> > >
> > > Java 11, Scala 2.13 - ARM
> > > Java 17, Scala 2.13 - ARM
> > > Java 20, Scala 2.18 - ARM
> > > Java 11, Scala 2.13 - Intel x86
> > > Java 17, Scala 2.13 - Intel x86
> > > Java 20, Scala 2.13 - Intel x86
> > >
> > > I will update the Zookeeper KIP title in the KIPs page, that's my miss
> > >
> > > Best,
> > > Christo
> > >
> > > On Tue, 19 Sept 2023 at 13:21, Divij Vaidya 
> > > wrote:
> > >
> > > > Hey Satish
> > > >
> > > > Thank you for managing this release. I have a few comments:
> > > >
> > > > Documentation
> > > >
> > > > 1. Section: Zookeeper/Stable Version - The documentation states "The
> > > > current stable branch is 3.5. Kafka is regularly updated to include
> > > > the latest release in the 3.5 series." in the ZooKeeper section. That
> > > > needs an update since we are running Zk 3.8 now.
> > > >
> > > > 2. Section: Zookeeper/Migration - The documentation states "Migration
> > > > of an existing ZooKeeper based Kafka cluster to KRaft is currently
> > > > Preview and we expect it to be ready for production usage in version
> > > > 3.6.". This probably needs an update on whether it is production ready
> > > > or not in 3.6
> > > >
> > > > 3. Section: Kraft/missing features
> > > > (https://kafka.apache.org/36/documentation.html#kraft_missing) - I
> > > > believe that delegation token is now part of 3.6? I think this
> > > > probably needs an update.
> > > >
> > > > 4. Section: Configuration/rack.aware.assignment.strategy - there seems
> > > > to be a formatting problem starting from here
> > > > (
> > > >
> > > https://kafka.apache.org/36/documentation.html#streamsconfigs_rack.aware.assignment.strategy
> > > > )
> > > >
> > > > 5. Section: KRaft Monitoring - Newly added metrics in
> > > > https://issues.apache.org/jira/browse/KAFKA-15183 are missing from the
> > > > documentation here.
> > > >
> > > > Release notes
> > > >
> > > > 1. I found a bunch of tickets which have not been marked with a
> > > > release version but have been resolved in last 6 months using the
> > > > query
> > > >
> > > https://issues.apache.org/jira/browse/KAFKA-15380?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%20EMPTY%20AND%20resolved%20%3E%3D%20-24w%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
> > > > . Are some of them targeted for 3.6 release?
> > > >
> > > > 2. The KIP "KIP-902: Upgrade Zookeeper to 3.8.1" should probably be
> > > > renamed to include 3.8.2 since code uses version 3.8.2 of Zookeeper.
> > > >
> > > >
> > > > Additionally, I have verified the following:
> > > > 1. release tag is correctly made after the latest commit on the 3.6
> > > > branch at
> > > >
> > > https://github.com/apache/kafka/commit/193d8c5be8d79b64c6c19d281322f09e3c5fe7de
> > > >
> > > > 2. protocol documentation contains the newly introduced error code as
> > > > part of tiered storage
> > > >
> > > > 3. verified that public keys for RM are available at
> > > > https://keys.openpgp.org/
> > > >
> > > > 4. verified that public keys for RM are available at
> > > > https://people.apache.org/keys/committer/
> > > >
> > > > --
> > > > Divij Vaidya
> > > >
> > > > On Tue, Sep 19, 2023 at 12:41 PM Sagar 
> > > wrote:
> > > > >
> > > > > Hey Satish,
> > > > >
> > > > > I have commented on KAFKA-15473. I think the changes in the PR look
> > > > fine. I
> > > > > also feel this need not be a release blocker given there are other
> > > > > possibilities in which duplicates can manifest on the response of the
> > > end
> > > > > point in question (albeit we can potentially see more in number due to
> > > > > this).
> > > > >
> > > > > Would like to hear others' thoughts as well.
> > > > >
> > > > > Thanks!
> > > > > Sagar.
> > > > >
> > > > >
> > > > > On Tue, Sep 19, 2023 at 3:14 PM Satish Duggana <
> > > satish.dugg...@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Greg,
> > > > > > Thanks for reporting the KafkaConnect issue. I replied to this issue
> > > > > > on "Apache Kafka 3.6.0 release" email thread and on
> > > > > > https://issues.apache.org/jira/browse/KAFKA-15473.
> > > > > >
> > > > > > I would l

Re: [VOTE] 3.6.0 RC0

[Satish Duggana]

Thanks Chris for your comment on the reported issue.

~Satish.

On Tue, 19 Sept 2023 at 22:16, Chris Egerton  wrote:
>
> Hi all,
>
> Given the regression raised by Greg Harris (
> https://issues.apache.org/jira/browse/KAFKA-15473) on the release thread, I
> believe we should generate a new candidate with a fix.
>
> I'm -1 (binding) on this RC.
>
> Best,
>
> Chris
>
> On Tue, Sep 19, 2023 at 10:30 AM Christo Lolov 
> wrote:
>
> > Heya,
> >
> > I have compiled and ran the test target successfully for the 3.6.0-rc0
> > branch on:
> >
> > Java 11, Scala 2.13 - ARM
> > Java 17, Scala 2.13 - ARM
> > Java 20, Scala 2.18 - ARM
> > Java 11, Scala 2.13 - Intel x86
> > Java 17, Scala 2.13 - Intel x86
> > Java 20, Scala 2.13 - Intel x86
> >
> > I will update the Zookeeper KIP title in the KIPs page, that's my miss
> >
> > Best,
> > Christo
> >
> > On Tue, 19 Sept 2023 at 13:21, Divij Vaidya 
> > wrote:
> >
> > > Hey Satish
> > >
> > > Thank you for managing this release. I have a few comments:
> > >
> > > Documentation
> > >
> > > 1. Section: Zookeeper/Stable Version - The documentation states "The
> > > current stable branch is 3.5. Kafka is regularly updated to include
> > > the latest release in the 3.5 series." in the ZooKeeper section. That
> > > needs an update since we are running Zk 3.8 now.
> > >
> > > 2. Section: Zookeeper/Migration - The documentation states "Migration
> > > of an existing ZooKeeper based Kafka cluster to KRaft is currently
> > > Preview and we expect it to be ready for production usage in version
> > > 3.6.". This probably needs an update on whether it is production ready
> > > or not in 3.6
> > >
> > > 3. Section: Kraft/missing features
> > > (https://kafka.apache.org/36/documentation.html#kraft_missing) - I
> > > believe that delegation token is now part of 3.6? I think this
> > > probably needs an update.
> > >
> > > 4. Section: Configuration/rack.aware.assignment.strategy - there seems
> > > to be a formatting problem starting from here
> > > (
> > >
> > https://kafka.apache.org/36/documentation.html#streamsconfigs_rack.aware.assignment.strategy
> > > )
> > >
> > > 5. Section: KRaft Monitoring - Newly added metrics in
> > > https://issues.apache.org/jira/browse/KAFKA-15183 are missing from the
> > > documentation here.
> > >
> > > Release notes
> > >
> > > 1. I found a bunch of tickets which have not been marked with a
> > > release version but have been resolved in last 6 months using the
> > > query
> > >
> > https://issues.apache.org/jira/browse/KAFKA-15380?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%20EMPTY%20AND%20resolved%20%3E%3D%20-24w%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
> > > . Are some of them targeted for 3.6 release?
> > >
> > > 2. The KIP "KIP-902: Upgrade Zookeeper to 3.8.1" should probably be
> > > renamed to include 3.8.2 since code uses version 3.8.2 of Zookeeper.
> > >
> > >
> > > Additionally, I have verified the following:
> > > 1. release tag is correctly made after the latest commit on the 3.6
> > > branch at
> > >
> > https://github.com/apache/kafka/commit/193d8c5be8d79b64c6c19d281322f09e3c5fe7de
> > >
> > > 2. protocol documentation contains the newly introduced error code as
> > > part of tiered storage
> > >
> > > 3. verified that public keys for RM are available at
> > > https://keys.openpgp.org/
> > >
> > > 4. verified that public keys for RM are available at
> > > https://people.apache.org/keys/committer/
> > >
> > > --
> > > Divij Vaidya
> > >
> > > On Tue, Sep 19, 2023 at 12:41 PM Sagar 
> > wrote:
> > > >
> > > > Hey Satish,
> > > >
> > > > I have commented on KAFKA-15473. I think the changes in the PR look
> > > fine. I
> > > > also feel this need not be a release blocker given there are other
> > > > possibilities in which duplicates can manifest on the response of the
> > end
> > > > point in question (albeit we can potentially see more in number due to
> > > > this).
> > > >
> > > > Would like to hear others' thoughts as well.
> > > >
> > > > Thanks!
> > > > Sagar.
> > > >
> > > >
> > > > On Tue, Sep 19, 2023 at 3:14 PM Satish Duggana <
> > satish.dugg...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Hi Greg,
> > > > > Thanks for reporting the KafkaConnect issue. I replied to this issue
> > > > > on "Apache Kafka 3.6.0 release" email thread and on
> > > > > https://issues.apache.org/jira/browse/KAFKA-15473.
> > > > >
> > > > > I would like to hear other KafkaConnect experts' opinions on whether
> > > > > this issue is really a release blocker.
> > > > >
> > > > > Thanks,
> > > > > Satish.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Tue, 19 Sept 2023 at 00:27, Greg Harris
> > > 
> > > > > wrote:
> > > > > >
> > > > > > Hey all,
> > > > > >
> > > > > > I noticed this regression in RC0:
> > > > > > https://issues.apache.org/jira/browse/KAFKA-15473
> > > > > > I've mentioned it in the release thread, and I'm working on a fix.
> > > > > >
> > 

Re: [VOTE] 3.6.0 RC0

[Chris Egerton]

Hi all,

Given the regression raised by Greg Harris (
https://issues.apache.org/jira/browse/KAFKA-15473) on the release thread, I
believe we should generate a new candidate with a fix.

I'm -1 (binding) on this RC.

Best,

Chris

On Tue, Sep 19, 2023 at 10:30 AM Christo Lolov 
wrote:

> Heya,
>
> I have compiled and ran the test target successfully for the 3.6.0-rc0
> branch on:
>
> Java 11, Scala 2.13 - ARM
> Java 17, Scala 2.13 - ARM
> Java 20, Scala 2.18 - ARM
> Java 11, Scala 2.13 - Intel x86
> Java 17, Scala 2.13 - Intel x86
> Java 20, Scala 2.13 - Intel x86
>
> I will update the Zookeeper KIP title in the KIPs page, that's my miss
>
> Best,
> Christo
>
> On Tue, 19 Sept 2023 at 13:21, Divij Vaidya 
> wrote:
>
> > Hey Satish
> >
> > Thank you for managing this release. I have a few comments:
> >
> > Documentation
> >
> > 1. Section: Zookeeper/Stable Version - The documentation states "The
> > current stable branch is 3.5. Kafka is regularly updated to include
> > the latest release in the 3.5 series." in the ZooKeeper section. That
> > needs an update since we are running Zk 3.8 now.
> >
> > 2. Section: Zookeeper/Migration - The documentation states "Migration
> > of an existing ZooKeeper based Kafka cluster to KRaft is currently
> > Preview and we expect it to be ready for production usage in version
> > 3.6.". This probably needs an update on whether it is production ready
> > or not in 3.6
> >
> > 3. Section: Kraft/missing features
> > (https://kafka.apache.org/36/documentation.html#kraft_missing) - I
> > believe that delegation token is now part of 3.6? I think this
> > probably needs an update.
> >
> > 4. Section: Configuration/rack.aware.assignment.strategy - there seems
> > to be a formatting problem starting from here
> > (
> >
> https://kafka.apache.org/36/documentation.html#streamsconfigs_rack.aware.assignment.strategy
> > )
> >
> > 5. Section: KRaft Monitoring - Newly added metrics in
> > https://issues.apache.org/jira/browse/KAFKA-15183 are missing from the
> > documentation here.
> >
> > Release notes
> >
> > 1. I found a bunch of tickets which have not been marked with a
> > release version but have been resolved in last 6 months using the
> > query
> >
> https://issues.apache.org/jira/browse/KAFKA-15380?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%20EMPTY%20AND%20resolved%20%3E%3D%20-24w%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
> > . Are some of them targeted for 3.6 release?
> >
> > 2. The KIP "KIP-902: Upgrade Zookeeper to 3.8.1" should probably be
> > renamed to include 3.8.2 since code uses version 3.8.2 of Zookeeper.
> >
> >
> > Additionally, I have verified the following:
> > 1. release tag is correctly made after the latest commit on the 3.6
> > branch at
> >
> https://github.com/apache/kafka/commit/193d8c5be8d79b64c6c19d281322f09e3c5fe7de
> >
> > 2. protocol documentation contains the newly introduced error code as
> > part of tiered storage
> >
> > 3. verified that public keys for RM are available at
> > https://keys.openpgp.org/
> >
> > 4. verified that public keys for RM are available at
> > https://people.apache.org/keys/committer/
> >
> > --
> > Divij Vaidya
> >
> > On Tue, Sep 19, 2023 at 12:41 PM Sagar 
> wrote:
> > >
> > > Hey Satish,
> > >
> > > I have commented on KAFKA-15473. I think the changes in the PR look
> > fine. I
> > > also feel this need not be a release blocker given there are other
> > > possibilities in which duplicates can manifest on the response of the
> end
> > > point in question (albeit we can potentially see more in number due to
> > > this).
> > >
> > > Would like to hear others' thoughts as well.
> > >
> > > Thanks!
> > > Sagar.
> > >
> > >
> > > On Tue, Sep 19, 2023 at 3:14 PM Satish Duggana <
> satish.dugg...@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi Greg,
> > > > Thanks for reporting the KafkaConnect issue. I replied to this issue
> > > > on "Apache Kafka 3.6.0 release" email thread and on
> > > > https://issues.apache.org/jira/browse/KAFKA-15473.
> > > >
> > > > I would like to hear other KafkaConnect experts' opinions on whether
> > > > this issue is really a release blocker.
> > > >
> > > > Thanks,
> > > > Satish.
> > > >
> > > >
> > > >
> > > >
> > > > On Tue, 19 Sept 2023 at 00:27, Greg Harris
> > 
> > > > wrote:
> > > > >
> > > > > Hey all,
> > > > >
> > > > > I noticed this regression in RC0:
> > > > > https://issues.apache.org/jira/browse/KAFKA-15473
> > > > > I've mentioned it in the release thread, and I'm working on a fix.
> > > > >
> > > > > I'm -1 (non-binding) until we determine if this regression is a
> > blocker.
> > > > >
> > > > > Thanks!
> > > > >
> > > > > On Mon, Sep 18, 2023 at 10:56 AM Josep Prat
> > 
> > > > wrote:
> > > > > >
> > > > > > Hi Satish,
> > > > > > Thanks for running the release.
> > > > > >
> > > > > > I ran the following validation steps:
> > > > > > - Built from source with Java 11 and Scala 2.13
> > > > > >

[jira] [Resolved] (KAFKA-15473) Connect connector-plugins endpoint shows duplicate plugins

[Chris Egerton (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Egerton resolved KAFKA-15473.
---
Fix Version/s: 3.7.0
   (was: 3.6.0)
   Resolution: Fixed

> Connect connector-plugins endpoint shows duplicate plugins
> --
>
> Key: KAFKA-15473
> URL: https://issues.apache.org/jira/browse/KAFKA-15473
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 3.6.0
>Reporter: Greg Harris
>Assignee: Greg Harris
>Priority: Major
> Fix For: 3.7.0
>
>
> In <3.6.0-rc0, duplicates of a plugin would be shown if it subclassed 
> multiple interfaces. For example:
> {noformat}
>   {
> "class": "org.apache.kafka.connect.storage.StringConverter",
> "type": "converter"
>   },
>   { 
> "class": "org.apache.kafka.connect.storage.StringConverter",
> "type": "converter"
>   },{noformat}
> In 3.6.0-rc0, there are many more listings for the same plugin. For example:
> {noformat}
>   {
>     "class": "org.apache.kafka.connect.storage.StringConverter",
>     "type": "converter"
>   },
>   {
>     "class": "org.apache.kafka.connect.storage.StringConverter",
>     "type": "converter"
>   },
>   {
>     "class": "org.apache.kafka.connect.storage.StringConverter",
>     "type": "converter"
>   },
>   {
>     "class": "org.apache.kafka.connect.storage.StringConverter",
>     "type": "converter"
>   },
>   {
>     "class": "org.apache.kafka.connect.storage.StringConverter",
>     "type": "converter"
>   },
>   {
>     "class": "org.apache.kafka.connect.storage.StringConverter",
>     "type": "converter",
>     "version": "3.6.0"
>   },{noformat}
> These duplicates appear to happen when a plugin with the same class name 
> appears in multiple locations/classloaders.
> When interpreting a connector configuration, only one of these plugins will 
> be chosen, so only one is relevant to show to users. The REST API should only 
> display the plugins which are eligible to be loaded, and hide the duplicates.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-15463) StreamsException: Accessing from an unknown node

[Lucas Brutschy (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lucas Brutschy resolved KAFKA-15463.

Resolution: Not A Problem

>  StreamsException: Accessing from an unknown node
> -
>
> Key: KAFKA-15463
> URL: https://issues.apache.org/jira/browse/KAFKA-15463
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 3.2.1
>Reporter: Yevgeny
>Priority: Major
>
> After some time application was working fine, starting to get:
>  
> This is springboot application runs in kubernetes as stateful pod.
>  
>  
>  
> {code:java}
>   Exception in thread 
> "-ddf9819f-d6c7-46ce-930e-cd923e1b3c2c-StreamThread-1" 
> org.apache.kafka.streams.errors.StreamsException: Accessing from an unknown 
> node at 
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.getStateStore(ProcessorContextImpl.java:162)
>  at myclass1.java:28) at myclass2.java:48) at 
> java.base/java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90) at 
> java.base/java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1602)
>  at 
> java.base/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:129)
>  at 
> java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:527)
>  at 
> java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:513)
>  at 
> java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
>  at 
> java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230)
>  at 
> java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196)
>  at 
> java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
>  at 
> java.base/java.util.stream.ReferencePipeline.allMatch(ReferencePipeline.java:637)
>  at myclass3.java:48) at 
> org.apache.kafka.streams.kstream.internals.TransformerSupplierAdapter$1.transform(TransformerSupplierAdapter.java:49)
>  at 
> org.apache.kafka.streams.kstream.internals.TransformerSupplierAdapter$1.transform(TransformerSupplierAdapter.java:38)
>  at 
> org.apache.kafka.streams.kstream.internals.KStreamFlatTransform$KStreamFlatTransformProcessor.process(KStreamFlatTransform.java:66)
>  at 
> org.apache.kafka.streams.processor.internals.ProcessorNode.process(ProcessorNode.java:146)
>  at 
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forwardInternal(ProcessorContextImpl.java:275)
>  at 
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:254)
>  at 
> org.apache.kafka.streams.processor.internals.ProcessorContextImpl.forward(ProcessorContextImpl.java:213)
>  at 
> org.apache.kafka.streams.processor.internals.SourceNode.process(SourceNode.java:84)
>  at 
> org.apache.kafka.streams.processor.internals.StreamTask.lambda$doProcess$1(StreamTask.java:780)
>  at 
> org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl.maybeMeasureLatency(StreamsMetricsImpl.java:809)
>  at 
> org.apache.kafka.streams.processor.internals.StreamTask.doProcess(StreamTask.java:780)
>  at 
> org.apache.kafka.streams.processor.internals.StreamTask.process(StreamTask.java:711)
>  at 
> org.apache.kafka.streams.processor.internals.TaskExecutor.processTask(TaskExecutor.java:100)
>  at 
> org.apache.kafka.streams.processor.internals.TaskExecutor.process(TaskExecutor.java:81)
>  at 
> org.apache.kafka.streams.processor.internals.TaskManager.process(TaskManager.java:1177)
>  at 
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:769)
>  at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:589)
>  at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:551)
>    {code}
>  
> stream-thread 
> [-ddf9819f-d6c7-46ce-930e-cd923e1b3c2c-StreamThread-1] State 
> transition from PENDING_SHUTDOWN to DEAD
>  
>  
> Transformer is Prototype bean, the supplier supplys new instance of the 
> Transformer:
>  
>  
> {code:java}
> @Override public Transformer> get() 
> {     return ctx.getBean(MyTransformer.class); }{code}
>  
>  
> The only way to recover is to delete all topics used by kafkastreams, even if 
> application restarted same exception is thrown.
> *If messages in internal topics of 'store-changelog'  are deleted/offset 
> manipulated, can it cause the issue?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Apache Kafka 3.6.0 release

[Chris Egerton]

Hi Satish,

I think this qualifies as a blocker. This API has been around for years now
and, while we don't document it as not exposing duplicates*, it has come
with that implicit contract since its inception. More importantly, it has
also never exposed plugins that cannot be used on the worker. This change
in behavior not only introduces duplicates*, it causes unreachable plugins
to be displayed. With this in mind, it seems to qualify pretty clearly as a
regression and we should not put out a release that includes it.

* - Really, these aren't duplicates; rather, they're multiple copies of the
same plugin that come from different locations on the worker

Best,

Chris

On Tue, Sep 19, 2023 at 4:31 AM Satish Duggana 
wrote:

> Hi Greg,
> Is this API documented that it does not return duplicate entries?
>
> Can we also get an opinion from PMC/Committers who have KafkaConnect
> expertise on whether this issue is a release blocker?
>
> If we agree that it is not a release blocker then we can have a
> release note clarifying this behaviour and add a reference to the JIRA
> that follows up on the possible solutions.
>
> Thanks,
> Satish.
>
>
> On Tue, 19 Sept 2023 at 03:29, Greg Harris 
> wrote:
> >
> > Hey Satish,
> >
> > After investigating further, I believe that this is a regression, but
> > mostly a cosmetic one.
> > I don't think there is significant risk of breaking clients with this
> > change, but it would be confusing for users, so I'd still like to get
> > the fix into the next RC.
> > I've opened a PR here: https://github.com/apache/kafka/pull/14398 and
> > I'll work to get it merged promptly.
> >
> > Thanks!
> >
> > On Mon, Sep 18, 2023 at 11:54 AM Greg Harris 
> wrote:
> > >
> > > Hi Satish,
> > >
> > > While validating 3.6.0-rc0, I noticed this regression as compared to
> > > 3.5.1: https://issues.apache.org/jira/browse/KAFKA-15473
> > >
> > > Impact: The `connector-plugins` endpoint lists duplicates which may
> > > cause confusion for users, or poor behavior in clients.
> > > Using the other REST API endpoints appears unaffected.
> > > I'll open a PR for this later today.
> > >
> > > Thanks,
> > > Greg
> > >
> > > On Thu, Sep 14, 2023 at 11:56 AM Satish Duggana
> > >  wrote:
> > > >
> > > > Thanks Justine for the update. I saw in the morning that these
> changes
> > > > are pushed to trunk and 3.6.
> > > >
> > > > ~Satish.
> > > >
> > > > On Thu, 14 Sept 2023 at 21:54, Justine Olshan
> > > >  wrote:
> > > > >
> > > > > Hi Satish,
> > > > > We were able to merge
> > > > > https://issues.apache.org/jira/browse/KAFKA-15459 yesterday
> > > > > and pick to 3.6.
> > > > >
> > > > > Hopefully nothing more from me on this release.
> > > > >
> > > > > Thanks,
> > > > > Justine
> > > > >
> > > > > On Wed, Sep 13, 2023 at 9:51 PM Satish Duggana <
> satish.dugg...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Thanks Luke for the update.
> > > > > >
> > > > > > ~Satish.
> > > > > >
> > > > > > On Thu, 14 Sept 2023 at 07:29, Luke Chen 
> wrote:
> > > > > > >
> > > > > > > Hi Satish,
> > > > > > >
> > > > > > > Since this PR:
> > > > > > > https://github.com/apache/kafka/pull/14366 only changes the
> doc, I've
> > > > > > > backported to 3.6 branch. FYI.
> > > > > > >
> > > > > > > Thanks.
> > > > > > > Luke
> > > > > > >
> > > > > > > On Thu, Sep 14, 2023 at 12:15 AM Justine Olshan
> > > > > > >  wrote:
> > > > > > >
> > > > > > > > Hey Satish -- yes, you are correct. KAFKA-15459 only affects
> 3.6.
> > > > > > > > PR should be finalized soon.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Justine
> > > > > > > >
> > > > > > > > On Wed, Sep 13, 2023 at 1:41 AM Federico Valeri <
> fedeval...@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Satish, this is a small documentation fix about ZK to
> KRaft
> > > > > > > > > migration, that we would like to backport to 3.5 and 3.6
> branches.
> > > > > > Are
> > > > > > > > > you ok with that?
> > > > > > > > >
> > > > > > > > > https://github.com/apache/kafka/pull/14366
> > > > > > > > >
> > > > > > > > > On Wed, Sep 13, 2023 at 3:13 AM Satish Duggana <
> > > > > > satish.dugg...@gmail.com
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Thanks David for the quick resolution.
> > > > > > > > > >
> > > > > > > > > > ~Satish.
> > > > > > > > > >
> > > > > > > > > > On Tue, 12 Sept 2023 at 22:51, David Arthur
> > > > > > > > > >  wrote:
> > > > > > > > > > >
> > > > > > > > > > > Satish,
> > > > > > > > > > >
> > > > > > > > > > > KAFKA-15450 is merged to 3.6 (as well as trunk, 3.5,
> and 3.4)
> > > > > > > > > > >
> > > > > > > > > > > Thanks!
> > > > > > > > > > > David
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Sep 12, 2023 at 11:44 AM Ismael Juma <
> m...@ismaeljuma.com>
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Justine,
> > > > > > > > > > > >
> > > > > > > > > > > > Probably best to have the conversation in the JIRA
> ticket vs
> > > > > > the
> > > > > > > >

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2213

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-15477) Kafka won't shutdown when deleting remote segment

[Francois Visconte (Jira)]

Francois Visconte created KAFKA-15477:
-

 Summary: Kafka won't shutdown when deleting remote segment
 Key: KAFKA-15477
 URL: https://issues.apache.org/jira/browse/KAFKA-15477
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 3.6.0
Reporter: Francois Visconte


When brokers are busy deleting a bunch of segments (following a topic removal 
using tiered storage), brokers won't respond to sigterm signal and cleanly 
shutdown. 
Intead, they keep removing remote segment until it's fully completed (which can 
take time for topics with long retention).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] 3.6.0 RC0

[Christo Lolov]

Heya,

I have compiled and ran the test target successfully for the 3.6.0-rc0
branch on:

Java 11, Scala 2.13 - ARM
Java 17, Scala 2.13 - ARM
Java 20, Scala 2.18 - ARM
Java 11, Scala 2.13 - Intel x86
Java 17, Scala 2.13 - Intel x86
Java 20, Scala 2.13 - Intel x86

I will update the Zookeeper KIP title in the KIPs page, that's my miss

Best,
Christo

On Tue, 19 Sept 2023 at 13:21, Divij Vaidya  wrote:

> Hey Satish
>
> Thank you for managing this release. I have a few comments:
>
> Documentation
>
> 1. Section: Zookeeper/Stable Version - The documentation states "The
> current stable branch is 3.5. Kafka is regularly updated to include
> the latest release in the 3.5 series." in the ZooKeeper section. That
> needs an update since we are running Zk 3.8 now.
>
> 2. Section: Zookeeper/Migration - The documentation states "Migration
> of an existing ZooKeeper based Kafka cluster to KRaft is currently
> Preview and we expect it to be ready for production usage in version
> 3.6.". This probably needs an update on whether it is production ready
> or not in 3.6
>
> 3. Section: Kraft/missing features
> (https://kafka.apache.org/36/documentation.html#kraft_missing) - I
> believe that delegation token is now part of 3.6? I think this
> probably needs an update.
>
> 4. Section: Configuration/rack.aware.assignment.strategy - there seems
> to be a formatting problem starting from here
> (
> https://kafka.apache.org/36/documentation.html#streamsconfigs_rack.aware.assignment.strategy
> )
>
> 5. Section: KRaft Monitoring - Newly added metrics in
> https://issues.apache.org/jira/browse/KAFKA-15183 are missing from the
> documentation here.
>
> Release notes
>
> 1. I found a bunch of tickets which have not been marked with a
> release version but have been resolved in last 6 months using the
> query
> https://issues.apache.org/jira/browse/KAFKA-15380?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%20EMPTY%20AND%20resolved%20%3E%3D%20-24w%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
> . Are some of them targeted for 3.6 release?
>
> 2. The KIP "KIP-902: Upgrade Zookeeper to 3.8.1" should probably be
> renamed to include 3.8.2 since code uses version 3.8.2 of Zookeeper.
>
>
> Additionally, I have verified the following:
> 1. release tag is correctly made after the latest commit on the 3.6
> branch at
> https://github.com/apache/kafka/commit/193d8c5be8d79b64c6c19d281322f09e3c5fe7de
>
> 2. protocol documentation contains the newly introduced error code as
> part of tiered storage
>
> 3. verified that public keys for RM are available at
> https://keys.openpgp.org/
>
> 4. verified that public keys for RM are available at
> https://people.apache.org/keys/committer/
>
> --
> Divij Vaidya
>
> On Tue, Sep 19, 2023 at 12:41 PM Sagar  wrote:
> >
> > Hey Satish,
> >
> > I have commented on KAFKA-15473. I think the changes in the PR look
> fine. I
> > also feel this need not be a release blocker given there are other
> > possibilities in which duplicates can manifest on the response of the end
> > point in question (albeit we can potentially see more in number due to
> > this).
> >
> > Would like to hear others' thoughts as well.
> >
> > Thanks!
> > Sagar.
> >
> >
> > On Tue, Sep 19, 2023 at 3:14 PM Satish Duggana  >
> > wrote:
> >
> > > Hi Greg,
> > > Thanks for reporting the KafkaConnect issue. I replied to this issue
> > > on "Apache Kafka 3.6.0 release" email thread and on
> > > https://issues.apache.org/jira/browse/KAFKA-15473.
> > >
> > > I would like to hear other KafkaConnect experts' opinions on whether
> > > this issue is really a release blocker.
> > >
> > > Thanks,
> > > Satish.
> > >
> > >
> > >
> > >
> > > On Tue, 19 Sept 2023 at 00:27, Greg Harris
> 
> > > wrote:
> > > >
> > > > Hey all,
> > > >
> > > > I noticed this regression in RC0:
> > > > https://issues.apache.org/jira/browse/KAFKA-15473
> > > > I've mentioned it in the release thread, and I'm working on a fix.
> > > >
> > > > I'm -1 (non-binding) until we determine if this regression is a
> blocker.
> > > >
> > > > Thanks!
> > > >
> > > > On Mon, Sep 18, 2023 at 10:56 AM Josep Prat
> 
> > > wrote:
> > > > >
> > > > > Hi Satish,
> > > > > Thanks for running the release.
> > > > >
> > > > > I ran the following validation steps:
> > > > > - Built from source with Java 11 and Scala 2.13
> > > > > - Verified Signatures and hashes of the artifacts generated
> > > > > - Navigated through Javadoc including links to JDK classes
> > > > > - Run the unit tests
> > > > > - Run integration tests
> > > > > - Run the quickstart in KRaft and Zookeeper mode
> > > > > - Checked License-binary against libs and matched them
> > > > >
> > > > > I +1 this release (non-binding)
> > > > >
> > > > > Best,
> > > > >
> > > > > On Mon, Sep 18, 2023 at 6:02 PM David Arthur 
> wrote:
> > > > >
> > > > > > Hey Satish, thanks for getting the RC underway!
> > > > > >
> > > > > > I noticed that the PR for the 3.6 bl

Re: [VOTE] KIP-858: Handle JBOD broker disk failure in KRaft

[Ron Dagostino]

Ok, great, that makes sense, Igor.  Thanks.  +1 (binding) on the KIP from me.

Ron

> On Sep 13, 2023, at 11:58 AM, Igor Soarez  wrote:
> 
> Hi Ron,
> 
> Thanks for drilling down on this. I think the KIP isn't really clear here,
> and the metadata caching section you quoted needs clarification.
> 
> The "hosting broker's latest registration" refers to the previous,
> not the current registration. The registrations are only compared by
> the controller, when handling the broker registration request.
> 
> Suppose broker b1 hosts two partitions, t-1 and t-2, in two
> directories, d1 and d2. The broker is registered, and the
> metadata correlates the replicas to their respective directories.
> i.e. OnlineLogDirs=[d1,d2] and OfflineLogDirs=false
> 
> The broker is then reconfigured to remove t-2 from log.dirs, and at startup,
> the registration request shows OnlineLogDirs=[d1] and OfflineLogDirs=false.
> The previous registration will only be replaced after a new successful
> registration, regardless of how quickly or how often b1 restarts.
> The controller compares the previous registration, and notices
> that one of the directories has been removed.
> So for any replica hosted in the broker that is assigned to that
> missing log directory, a logical metadata update takes place
> that assigned them to Uuid.OfflineDir, so Assignment.Directory
> is updated for t-2. This value is indicates that the replica
> is offline — I have updated the section you quoted to address this.
> 
> Once the broker catches up with metadata, it will select the only
> configured log directory — d1 — for any partitions assigned to
> Uuid.OfflineDir, and update the assignment.
> 
> Best,
> 
> --
> Igor
> 
> 
> 

[GitHub] [kafka-site] divijvaidya commented on a diff in pull request #547: Added a blog entry for 3.6.0 release

[via GitHub]

divijvaidya commented on code in PR #547:
URL: https://github.com/apache/kafka-site/pull/547#discussion_r1330040052


##
blog.html:
##
@@ -22,6 +22,46 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.6.0 Release Announcement
+
+15 Sep 2023 - Satish Duggana (https://twitter.com/0xeed";>@SatishDuggana)
+We are proud to announce the release of Apache Kafka 3.6.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.6.0/RELEASE_NOTES.html";>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_6_0";>Upgrading 
to 3.6.0 from any version 0.8.x through 3.5.x section in the documentation 
for the list of notable changes and detailed upgrade steps.
+The ability to migrate Kafka clusters from ZK to KRaft mode 
with no downtime is still an early access feature. It is currently only 
suitable for testing in non production environments. See https://cwiki.apache.org/confluence/display/KAFKA/KIP-866+ZooKeeper+to+KRaft+Migration";>KIP-866
 for more details.
+https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage";>Tiered
 Storage is an early access feature. It is currently only suitable for 
testing in non production environments. See https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Tiered+Storage+Early+Access+Release+Notes";>Early
 Access Notes for more details.
+
+Note: ZooKeeper is marked as deprecated since 3.5.0 
release. ZooKeeper is planned to be removed in Apache Kafka 4.0. (Cf ZooKeeper Deprecation)
+Kafka Broker, Controller, Producer, Consumer and Admin 
Client
+
+KIP-405: Kafka Tiered Storagehttps://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage";>KIP-405.
 It introduces tiered storage feature in Kafka that provides separation of 
computation and storage in the broker.
+KIP-797: Accept duplicate listener on port for 
IPv4/IPv6https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=195726330";>KIP-797.
 Brokers can be configured with listeners that have same port on different ip 
stack like ipv4 and ipv6.
+KIP-863: Reduce CompletedFetch#parseRecord() memory 
copyhttps://cwiki.apache.org/confluence/pages/viewpage.action?pageId=225152035";>KIP-863.
 Reduced CompletedFetch#parseRecord() memory copy by deserializing using byte 
buffers.
+KIP-868: Metadata Transactionshttps://cwiki.apache.org/confluence/display/KAFKA/KIP-868+Metadata+Transactions";>KIP-868.
 This feature is about allowing the controller to generate atomic transactions 
of records that can exceed the maximum batch size.
+KIP-902: Upgrade Zookeeper to 3.8.1https://cwiki.apache.org/confluence/display/KAFKA/KIP-902%3A+Upgrade+Zookeeper+to+3.8.1";>KIP-902.
 Zookeeper client upgraded to 3.8.1 version as the current zookeeper dependency 
version 3.6.3 reached the end of life.

Review Comment:
   It's actually version 3.8.2. The KIP title is not updated.



##
blog.html:
##
@@ -22,6 +22,46 @@
 
 
 Blog
+
+
+
+Apache 
Kafka 3.6.0 Release Announcement
+
+15 Sep 2023 - Satish Duggana (https://twitter.com/0xeed";>@SatishDuggana)
+We are proud to announce the release of Apache Kafka 3.6.0. 
This release contains many new features and improvements. This blog post will 
highlight some of the more prominent features. For a full list of changes, be 
sure to check the https://downloads.apache.org/kafka/3.6.0/RELEASE_NOTES.html";>release 
notes.
+See the https://kafka.apache.org/36/documentation.html#upgrade_3_6_0";>Upgrading 
to 3.6.0 from any version 0.8.x through 3.5.x section in the documentation 
for the list of notable changes and detailed upgrade steps.
+The ability to migrate Kafka clusters from ZK to KRaft mode 
with no downtime is still an early access feature. It is currently only 
suitable for testing in non production environments. See https://cwiki.apache.org/confluence/display/KAFKA/KIP-866+ZooKeeper+to+KRaft+Migration";>KIP-866
 for more details.

Review Comment:
   Should we add a note about support of delegation token for kraft? [1] 
   Suggesting this because it takes us one step closer to completing the 
missing features for kraft 
https://kafka.apache.org/documentation.html#kraft_missing
   
   
   [1] https://issues.apache.org/jira/browse/KAFKA-15219 



##
blog.html:
##
@@ -22,6 +22,46 @@
 
 
 Blog
+
+
+   

Re: [VOTE] 3.6.0 RC0

[Divij Vaidya]

Hey Satish

Thank you for managing this release. I have a few comments:

Documentation

1. Section: Zookeeper/Stable Version - The documentation states "The
current stable branch is 3.5. Kafka is regularly updated to include
the latest release in the 3.5 series." in the ZooKeeper section. That
needs an update since we are running Zk 3.8 now.

2. Section: Zookeeper/Migration - The documentation states "Migration
of an existing ZooKeeper based Kafka cluster to KRaft is currently
Preview and we expect it to be ready for production usage in version
3.6.". This probably needs an update on whether it is production ready
or not in 3.6

3. Section: Kraft/missing features
(https://kafka.apache.org/36/documentation.html#kraft_missing) - I
believe that delegation token is now part of 3.6? I think this
probably needs an update.

4. Section: Configuration/rack.aware.assignment.strategy - there seems
to be a formatting problem starting from here
(https://kafka.apache.org/36/documentation.html#streamsconfigs_rack.aware.assignment.strategy)

5. Section: KRaft Monitoring - Newly added metrics in
https://issues.apache.org/jira/browse/KAFKA-15183 are missing from the
documentation here.

Release notes

1. I found a bunch of tickets which have not been marked with a
release version but have been resolved in last 6 months using the
query 
https://issues.apache.org/jira/browse/KAFKA-15380?jql=project%20%3D%20KAFKA%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%20EMPTY%20AND%20resolved%20%3E%3D%20-24w%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
. Are some of them targeted for 3.6 release?

2. The KIP "KIP-902: Upgrade Zookeeper to 3.8.1" should probably be
renamed to include 3.8.2 since code uses version 3.8.2 of Zookeeper.


Additionally, I have verified the following:
1. release tag is correctly made after the latest commit on the 3.6
branch at 
https://github.com/apache/kafka/commit/193d8c5be8d79b64c6c19d281322f09e3c5fe7de

2. protocol documentation contains the newly introduced error code as
part of tiered storage

3. verified that public keys for RM are available at https://keys.openpgp.org/

4. verified that public keys for RM are available at
https://people.apache.org/keys/committer/

--
Divij Vaidya

On Tue, Sep 19, 2023 at 12:41 PM Sagar  wrote:
>
> Hey Satish,
>
> I have commented on KAFKA-15473. I think the changes in the PR look fine. I
> also feel this need not be a release blocker given there are other
> possibilities in which duplicates can manifest on the response of the end
> point in question (albeit we can potentially see more in number due to
> this).
>
> Would like to hear others' thoughts as well.
>
> Thanks!
> Sagar.
>
>
> On Tue, Sep 19, 2023 at 3:14 PM Satish Duggana 
> wrote:
>
> > Hi Greg,
> > Thanks for reporting the KafkaConnect issue. I replied to this issue
> > on "Apache Kafka 3.6.0 release" email thread and on
> > https://issues.apache.org/jira/browse/KAFKA-15473.
> >
> > I would like to hear other KafkaConnect experts' opinions on whether
> > this issue is really a release blocker.
> >
> > Thanks,
> > Satish.
> >
> >
> >
> >
> > On Tue, 19 Sept 2023 at 00:27, Greg Harris 
> > wrote:
> > >
> > > Hey all,
> > >
> > > I noticed this regression in RC0:
> > > https://issues.apache.org/jira/browse/KAFKA-15473
> > > I've mentioned it in the release thread, and I'm working on a fix.
> > >
> > > I'm -1 (non-binding) until we determine if this regression is a blocker.
> > >
> > > Thanks!
> > >
> > > On Mon, Sep 18, 2023 at 10:56 AM Josep Prat 
> > wrote:
> > > >
> > > > Hi Satish,
> > > > Thanks for running the release.
> > > >
> > > > I ran the following validation steps:
> > > > - Built from source with Java 11 and Scala 2.13
> > > > - Verified Signatures and hashes of the artifacts generated
> > > > - Navigated through Javadoc including links to JDK classes
> > > > - Run the unit tests
> > > > - Run integration tests
> > > > - Run the quickstart in KRaft and Zookeeper mode
> > > > - Checked License-binary against libs and matched them
> > > >
> > > > I +1 this release (non-binding)
> > > >
> > > > Best,
> > > >
> > > > On Mon, Sep 18, 2023 at 6:02 PM David Arthur  wrote:
> > > >
> > > > > Hey Satish, thanks for getting the RC underway!
> > > > >
> > > > > I noticed that the PR for the 3.6 blog post is merged. This makes
> > the blog
> > > > > post live on the Kafka website https://kafka.apache.org/blog.html.
> > The
> > > > > blog
> > > > > post (along with other public announcements) is usually the last
> > thing we
> > > > > do as part of the release. I think we should probably take this down
> > until
> > > > > we're done with the release, otherwise users stumbling on this post
> > could
> > > > > get confused. It also contains some broken links.
> > > > >
> > > > > Thanks!
> > > > > David
> > > > >
> > > > > On Sun, Sep 17, 2023 at 1:31 PM Satish Duggana <
> > satish.dugg...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hello Kafka 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #2212

[Apache Jenkins Server]

See 

Re: [VOTE] 3.6.0 RC0

[Sagar]

Hey Satish,

I have commented on KAFKA-15473. I think the changes in the PR look fine. I
also feel this need not be a release blocker given there are other
possibilities in which duplicates can manifest on the response of the end
point in question (albeit we can potentially see more in number due to
this).

Would like to hear others' thoughts as well.

Thanks!
Sagar.


On Tue, Sep 19, 2023 at 3:14 PM Satish Duggana 
wrote:

> Hi Greg,
> Thanks for reporting the KafkaConnect issue. I replied to this issue
> on "Apache Kafka 3.6.0 release" email thread and on
> https://issues.apache.org/jira/browse/KAFKA-15473.
>
> I would like to hear other KafkaConnect experts' opinions on whether
> this issue is really a release blocker.
>
> Thanks,
> Satish.
>
>
>
>
> On Tue, 19 Sept 2023 at 00:27, Greg Harris 
> wrote:
> >
> > Hey all,
> >
> > I noticed this regression in RC0:
> > https://issues.apache.org/jira/browse/KAFKA-15473
> > I've mentioned it in the release thread, and I'm working on a fix.
> >
> > I'm -1 (non-binding) until we determine if this regression is a blocker.
> >
> > Thanks!
> >
> > On Mon, Sep 18, 2023 at 10:56 AM Josep Prat 
> wrote:
> > >
> > > Hi Satish,
> > > Thanks for running the release.
> > >
> > > I ran the following validation steps:
> > > - Built from source with Java 11 and Scala 2.13
> > > - Verified Signatures and hashes of the artifacts generated
> > > - Navigated through Javadoc including links to JDK classes
> > > - Run the unit tests
> > > - Run integration tests
> > > - Run the quickstart in KRaft and Zookeeper mode
> > > - Checked License-binary against libs and matched them
> > >
> > > I +1 this release (non-binding)
> > >
> > > Best,
> > >
> > > On Mon, Sep 18, 2023 at 6:02 PM David Arthur  wrote:
> > >
> > > > Hey Satish, thanks for getting the RC underway!
> > > >
> > > > I noticed that the PR for the 3.6 blog post is merged. This makes
> the blog
> > > > post live on the Kafka website https://kafka.apache.org/blog.html.
> The
> > > > blog
> > > > post (along with other public announcements) is usually the last
> thing we
> > > > do as part of the release. I think we should probably take this down
> until
> > > > we're done with the release, otherwise users stumbling on this post
> could
> > > > get confused. It also contains some broken links.
> > > >
> > > > Thanks!
> > > > David
> > > >
> > > > On Sun, Sep 17, 2023 at 1:31 PM Satish Duggana <
> satish.dugg...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hello Kafka users, developers and client-developers,
> > > > >
> > > > > This is the first candidate for the release of Apache Kafka 3.6.0.
> Some
> > > > of
> > > > > the major features include:
> > > > >
> > > > > * KIP-405 : Kafka Tiered Storage
> > > > > * KIP-868 : KRaft Metadata Transactions
> > > > > * KIP-875: First-class offsets support in Kafka Connect
> > > > > * KIP-898: Modernize Connect plugin discovery
> > > > > * KIP-938: Add more metrics for measuring KRaft performance
> > > > > * KIP-902: Upgrade Zookeeper to 3.8.1
> > > > > * KIP-917: Additional custom metadata for remote log segment
> > > > >
> > > > > Release notes for the 3.6.0 release:
> > > > >
> https://home.apache.org/~satishd/kafka-3.6.0-rc0/RELEASE_NOTES.html
> > > > >
> > > > > *** Please download, test and vote by Wednesday, September 21,
> 12pm PT
> > > > >
> > > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > > https://kafka.apache.org/KEYS
> > > > >
> > > > > * Release artifacts to be voted upon (source and binary):
> > > > > https://home.apache.org/~satishd/kafka-3.6.0-rc0/
> > > > >
> > > > > * Maven artifacts to be voted upon:
> > > > >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > >
> > > > > * Javadoc:
> > > > > https://home.apache.org/~satishd/kafka-3.6.0-rc0/javadoc/
> > > > >
> > > > > * Tag to be voted upon (off 3.6 branch) is the 3.6.0 tag:
> > > > > https://github.com/apache/kafka/releases/tag/3.6.0-rc0
> > > > >
> > > > > * Documentation:
> > > > > https://kafka.apache.org/36/documentation.html
> > > > >
> > > > > * Protocol:
> > > > > https://kafka.apache.org/36/protocol.html
> > > > >
> > > > > * Successful Jenkins builds for the 3.6 branch:
> > > > > There are a few runs of unit/integration tests. You can see the
> latest at
> > > > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/. We will
> > > > > continue
> > > > > running a few more iterations.
> > > > > System tests:
> > > > > We will send an update once we have the results.
> > > > >
> > > > > Thanks,
> > > > > Satish.
> > > > >
> > > >
> > > >
> > > > --
> > > > David Arthur
> > > >
> > >
> > >
> > > --
> > > [image: Aiven] 
> > >
> > > *Josep Prat*
> > > Open Source Engineering Director, *Aiven*
> > > josep.p...@aiven.io   |   +491715557497
> > > aiven.io    |   <
> https://www.facebook.com/aivencloud>
> > >      <
> https://twitter.com/aiven_io>
> > > *Aiven De

Re: [DISCUSS] KIP-975 Docker Image for Apache Kafka

[Viktor Somogyi-Vass]

Hi Ismael,

I'm not trying to advocate against the docker image, I just pointed out
that the current scoping of the KIP may be a bit too generic and thought
that KIP-974 and KIP-975 were aiming for mostly the same thing and can be
discussed under one umbrella. Apologies if this was rooted in a
misunderstanding.

Kirshna,

I think we need to refine the KIP a bit more. I think there are some
interfaces that we need to include in the KIP as Kafka has plugins in
certain cases where users are expected to provide implementation and I
think it's worth discussing this in the KIP as they're kind of interfaces
for users. Here are my questions in order:
1. In what environments do you want the image to be used? As I understand
it would replace the current testing image and serve as a basis for
development, but would it aim at production use cases too (docker-compose,
Kubernetes, etc.)?
2. How do you plan to forward configs to the broker? Do we expect a
populated server.properties file placed in a certain location or should the
docker image create this file based on some input (like env vars)?
3. Certain parts can be pluggable, like metric reporters or remote log
implementations that were just introduced by KIP-405. These manifest in jar
files that must be put on the classpath of Kafka while certain classnames
have to be configured. How do you plan to implement this, how do we
allow users to configure such things?

Thanks,
Viktor




On Thu, Sep 14, 2023 at 4:59 PM Kenneth Eversole
 wrote:

> Hello,
>
> I think this would be a wonderful improvement to the ecosystem. While
> Viktor is correct that most Docker pipelines eventually lead to a
> kubernetes deployment, that should not stop us from creating an
> Official Docker Image. Creating a Docker image would allow us to ensure a
> level of quality and support for people who want to deploy Kafka as a
> container on baremetal machines, it could allow us to create
> a sandbox/developer environment for new contributors and developers to test
> and have a single agreed upon environment that kafka works in for future
> KIPs and would most likely spawn more contributions from people wanting to
> optimize kafka for k8s.
>
>
> I am 100% for this and will gladly help if approved.
>
> Kenneth
>
> On Thu, Sep 14, 2023 at 5:47 AM Ismael Juma  wrote:
>
> > Hi Viktor,
> >
> > I disagree. Docker is a very popular deployment tool and it's not only
> used
> > with Kubernetes.
> >
> > Ismael
> >
> > On Thu, Sep 14, 2023, 1:14 AM Viktor Somogyi-Vass
> >  wrote:
> >
> > > Hi Krishna,
> > >
> > > I think you should merge this KIP and KIP-974
>  as there are
> overlaps as
> > > Federico pointed out on KIP-974
> . I think you
> should keep that one as it
> > > has well defined goals (improve tests) while I feel this one is too
> > > generic. Docker is usually just a tool for either testing or
> Kubernetes,
> > so
> > > they have very well defined use-cases. In the case of Flink for
> instance
> > > the image is used for its kubernetes operator. The use case would
> > determine
> > > a lot of things and I think a generic image would likely not fit the
> > needs
> > > of all use-cases.
> > >
> > > Best,
> > > Viktor
> > >
> > > On Fri, Sep 8, 2023 at 9:58 AM Krishna Agarwal <
> > > krishna0608agar...@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > > Apache Kafka does not have an official docker image currently.
> > > > I want to submit a KIP to publish a docker image for Apache Kafka.
> > > >
> > > > KIP-975 :
> Docker Image for Apache Kafka
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-975%3A+Docker+Image+for+Apache+Kafka
> > > > >
> > > >
> > > > Regards,
> > > > Krishna
> > > >
> > >
> >
>

[GitHub] [kafka-site] satishd opened a new pull request, #547: Added a blog entry for 3.6.0 release

[via GitHub]

satishd opened a new pull request, #547:
URL: https://github.com/apache/kafka-site/pull/547

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [VOTE] 3.6.0 RC0

[Satish Duggana]

Hi Greg,
Thanks for reporting the KafkaConnect issue. I replied to this issue
on "Apache Kafka 3.6.0 release" email thread and on
https://issues.apache.org/jira/browse/KAFKA-15473.

I would like to hear other KafkaConnect experts' opinions on whether
this issue is really a release blocker.

Thanks,
Satish.




On Tue, 19 Sept 2023 at 00:27, Greg Harris  wrote:
>
> Hey all,
>
> I noticed this regression in RC0:
> https://issues.apache.org/jira/browse/KAFKA-15473
> I've mentioned it in the release thread, and I'm working on a fix.
>
> I'm -1 (non-binding) until we determine if this regression is a blocker.
>
> Thanks!
>
> On Mon, Sep 18, 2023 at 10:56 AM Josep Prat  
> wrote:
> >
> > Hi Satish,
> > Thanks for running the release.
> >
> > I ran the following validation steps:
> > - Built from source with Java 11 and Scala 2.13
> > - Verified Signatures and hashes of the artifacts generated
> > - Navigated through Javadoc including links to JDK classes
> > - Run the unit tests
> > - Run integration tests
> > - Run the quickstart in KRaft and Zookeeper mode
> > - Checked License-binary against libs and matched them
> >
> > I +1 this release (non-binding)
> >
> > Best,
> >
> > On Mon, Sep 18, 2023 at 6:02 PM David Arthur  wrote:
> >
> > > Hey Satish, thanks for getting the RC underway!
> > >
> > > I noticed that the PR for the 3.6 blog post is merged. This makes the blog
> > > post live on the Kafka website https://kafka.apache.org/blog.html. The
> > > blog
> > > post (along with other public announcements) is usually the last thing we
> > > do as part of the release. I think we should probably take this down until
> > > we're done with the release, otherwise users stumbling on this post could
> > > get confused. It also contains some broken links.
> > >
> > > Thanks!
> > > David
> > >
> > > On Sun, Sep 17, 2023 at 1:31 PM Satish Duggana 
> > > wrote:
> > >
> > > > Hello Kafka users, developers and client-developers,
> > > >
> > > > This is the first candidate for the release of Apache Kafka 3.6.0. Some
> > > of
> > > > the major features include:
> > > >
> > > > * KIP-405 : Kafka Tiered Storage
> > > > * KIP-868 : KRaft Metadata Transactions
> > > > * KIP-875: First-class offsets support in Kafka Connect
> > > > * KIP-898: Modernize Connect plugin discovery
> > > > * KIP-938: Add more metrics for measuring KRaft performance
> > > > * KIP-902: Upgrade Zookeeper to 3.8.1
> > > > * KIP-917: Additional custom metadata for remote log segment
> > > >
> > > > Release notes for the 3.6.0 release:
> > > > https://home.apache.org/~satishd/kafka-3.6.0-rc0/RELEASE_NOTES.html
> > > >
> > > > *** Please download, test and vote by Wednesday, September 21, 12pm PT
> > > >
> > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > https://kafka.apache.org/KEYS
> > > >
> > > > * Release artifacts to be voted upon (source and binary):
> > > > https://home.apache.org/~satishd/kafka-3.6.0-rc0/
> > > >
> > > > * Maven artifacts to be voted upon:
> > > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > >
> > > > * Javadoc:
> > > > https://home.apache.org/~satishd/kafka-3.6.0-rc0/javadoc/
> > > >
> > > > * Tag to be voted upon (off 3.6 branch) is the 3.6.0 tag:
> > > > https://github.com/apache/kafka/releases/tag/3.6.0-rc0
> > > >
> > > > * Documentation:
> > > > https://kafka.apache.org/36/documentation.html
> > > >
> > > > * Protocol:
> > > > https://kafka.apache.org/36/protocol.html
> > > >
> > > > * Successful Jenkins builds for the 3.6 branch:
> > > > There are a few runs of unit/integration tests. You can see the latest 
> > > > at
> > > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.6/. We will
> > > > continue
> > > > running a few more iterations.
> > > > System tests:
> > > > We will send an update once we have the results.
> > > >
> > > > Thanks,
> > > > Satish.
> > > >
> > >
> > >
> > > --
> > > David Arthur
> > >
> >
> >
> > --
> > [image: Aiven] 
> >
> > *Josep Prat*
> > Open Source Engineering Director, *Aiven*
> > josep.p...@aiven.io   |   +491715557497
> > aiven.io    |   
> >      
> > *Aiven Deutschland GmbH*
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > Amtsgericht Charlottenburg, HRB 209739 B

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

[Nick Telford]

Hi Bruno,

Agreed, I can live with that for now.

In an effort to keep the scope of this KIP from expanding, I'm leaning
towards just providing a configurable default.state.isolation.level and
removing IsolationLevel from the StateStoreContext. This would be
compatible with adding support for query-time IsolationLevels in the
future, whilst providing a way for users to select an isolation level now.

The big problem with this, however, is that if a user selects processing.mode
= "exactly-once(-v2|-beta)", and default.state.isolation.level =
"READ_UNCOMMITTED", we need to guarantee that the data isn't written to
disk until commit() is called, but we also need to permit IQ threads to
read from the ongoing transaction.

A simple solution would be to (temporarily) forbid this combination of
configuration, and have default.state.isolation.level automatically switch
to READ_COMMITTED when processing.mode is anything other than
at-least-once. Do you think this would be acceptable?

In a later KIP, we can add support for query-time isolation levels and
solve this particular problem there, which would relax this restriction.

Regards,
Nick

On Tue, 19 Sept 2023 at 09:30, Bruno Cadonna  wrote:

> Why do we need to add READ_COMMITTED to InMemoryKeyValueStore? I think
> it is perfectly valid to say InMemoryKeyValueStore do not support
> READ_COMMITTED for now, since READ_UNCOMMITTED is the de-facto default
> at the moment.
>
> Best,
> Bruno
>
> On 9/18/23 7:12 PM, Nick Telford wrote:
> > Oh! One other concern I haven't mentioned: if we make IsolationLevel a
> > query-time constraint, then we need to add support for READ_COMMITTED to
> > InMemoryKeyValueStore too, which will require some changes to the
> > implementation.
> >
> > On Mon, 18 Sept 2023 at 17:24, Nick Telford 
> wrote:
> >
> >> Hi everyone,
> >>
> >> I agree that having IsolationLevel be determined at query-time is the
> >> ideal design, but there are a few sticking points:
> >>
> >> 1.
> >> There needs to be some way to communicate the IsolationLevel down to the
> >> RocksDBStore itself, so that the query can respect it. Since stores are
> >> "layered" in functionality (i.e. ChangeLoggingStore, MeteredStore,
> etc.),
> >> we need some way to deliver that information to the bottom layer. For
> IQv2,
> >> we can use the existing State#query() method, but IQv1 has no way to do
> >> this.
> >>
> >> A simple approach, which would potentially open up other options, would
> be
> >> to add something like: ReadOnlyKeyValueStore
> >> readOnlyView(IsolationLevel isolationLevel) to ReadOnlyKeyValueStore
> (and
> >> similar to ReadOnlyWindowStore, ReadOnlySessionStore, etc.).
> >>
> >> 2.
> >> As mentioned above, RocksDB WriteBatches are not thread-safe, which
> causes
> >> a problem if we want to provide READ_UNCOMMITTED Iterators. I also had a
> >> look at RocksDB Transactions[1], but they solve a very different
> problem,
> >> and have the same thread-safety issue.
> >>
> >> One possible approach that I mentioned is chaining WriteBatches: every
> >> time a new Interactive Query is received (i.e. readOnlyView, see above,
> >> is called) we "freeze" the existing WriteBatch, and start a new one for
> new
> >> writes. The Interactive Query queries the "chain" of previous
> WriteBatches
> >> + the underlying database; while the StreamThread starts writing to the
> >> *new* WriteBatch. On-commit, the StreamThread would write *all*
> >> WriteBatches in the chain to the database (that have not yet been
> written).
> >>
> >> WriteBatches would be closed/freed only when they have been both
> >> committed, and all open Interactive Queries on them have been closed.
> This
> >> would require some reference counting.
> >>
> >> Obviously a drawback of this approach is the potential for increased
> >> memory usage: if an Interactive Query is long-lived, for example by
> doing a
> >> full scan over a large database, or even just pausing in the middle of
> an
> >> iteration, then the existing chain of WriteBatches could be kept around
> for
> >> a long time, potentially forever.
> >>
> >> --
> >>
> >> A.
> >> Going off on a tangent, it looks like in addition to supporting
> >> READ_COMMITTED queries, we could go further and support REPEATABLE_READ
> >> queries (i.e. where subsequent reads to the same key in the same
> >> Interactive Query are guaranteed to yield the same value) by making use
> of
> >> RocksDB Snapshots[2]. These are fairly lightweight, so the performance
> >> impact is likely to be negligible, but they do require that the
> Interactive
> >> Query session can be explicitly closed.
> >>
> >> This could be achieved if we made the above readOnlyView interface look
> >> more like:
> >>
> >> interface ReadOnlyKeyValueView implements ReadOnlyKeyValueStore >> V>, AutoCloseable {}
> >>
> >> interface ReadOnlyKeyValueStore {
> >>  ...
> >>  ReadOnlyKeyValueView readOnlyView(IsolationLevel
> isolationLevel);
> >> }
> >>
> >> But this would be a breaking change, as exi

[jira] [Resolved] (KAFKA-15476) Improve checkstyle performance

[Divij Vaidya (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Divij Vaidya resolved KAFKA-15476.
--
Resolution: Fixed

> Improve checkstyle performance
> --
>
> Key: KAFKA-15476
> URL: https://issues.apache.org/jira/browse/KAFKA-15476
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Divij Vaidya
>Priority: Minor
> Fix For: 3.7.0
>
>
> Checkstyle is not using cache during build due to absolute file paths. See 
> details in description at [https://github.com/apache/kafka/pull/14344]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15476) Improve checkstyle performance

[Divij Vaidya (Jira)]

Divij Vaidya created KAFKA-15476:


 Summary: Improve checkstyle performance
 Key: KAFKA-15476
 URL: https://issues.apache.org/jira/browse/KAFKA-15476
 Project: Kafka
  Issue Type: Improvement
Reporter: Divij Vaidya
 Fix For: 3.7.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Apache Kafka 3.6.0 release

[Satish Duggana]

Hi Greg,
Is this API documented that it does not return duplicate entries?

Can we also get an opinion from PMC/Committers who have KafkaConnect
expertise on whether this issue is a release blocker?

If we agree that it is not a release blocker then we can have a
release note clarifying this behaviour and add a reference to the JIRA
that follows up on the possible solutions.

Thanks,
Satish.


On Tue, 19 Sept 2023 at 03:29, Greg Harris  wrote:
>
> Hey Satish,
>
> After investigating further, I believe that this is a regression, but
> mostly a cosmetic one.
> I don't think there is significant risk of breaking clients with this
> change, but it would be confusing for users, so I'd still like to get
> the fix into the next RC.
> I've opened a PR here: https://github.com/apache/kafka/pull/14398 and
> I'll work to get it merged promptly.
>
> Thanks!
>
> On Mon, Sep 18, 2023 at 11:54 AM Greg Harris  wrote:
> >
> > Hi Satish,
> >
> > While validating 3.6.0-rc0, I noticed this regression as compared to
> > 3.5.1: https://issues.apache.org/jira/browse/KAFKA-15473
> >
> > Impact: The `connector-plugins` endpoint lists duplicates which may
> > cause confusion for users, or poor behavior in clients.
> > Using the other REST API endpoints appears unaffected.
> > I'll open a PR for this later today.
> >
> > Thanks,
> > Greg
> >
> > On Thu, Sep 14, 2023 at 11:56 AM Satish Duggana
> >  wrote:
> > >
> > > Thanks Justine for the update. I saw in the morning that these changes
> > > are pushed to trunk and 3.6.
> > >
> > > ~Satish.
> > >
> > > On Thu, 14 Sept 2023 at 21:54, Justine Olshan
> > >  wrote:
> > > >
> > > > Hi Satish,
> > > > We were able to merge
> > > > https://issues.apache.org/jira/browse/KAFKA-15459 yesterday
> > > > and pick to 3.6.
> > > >
> > > > Hopefully nothing more from me on this release.
> > > >
> > > > Thanks,
> > > > Justine
> > > >
> > > > On Wed, Sep 13, 2023 at 9:51 PM Satish Duggana 
> > > > 
> > > > wrote:
> > > >
> > > > > Thanks Luke for the update.
> > > > >
> > > > > ~Satish.
> > > > >
> > > > > On Thu, 14 Sept 2023 at 07:29, Luke Chen  wrote:
> > > > > >
> > > > > > Hi Satish,
> > > > > >
> > > > > > Since this PR:
> > > > > > https://github.com/apache/kafka/pull/14366 only changes the doc, 
> > > > > > I've
> > > > > > backported to 3.6 branch. FYI.
> > > > > >
> > > > > > Thanks.
> > > > > > Luke
> > > > > >
> > > > > > On Thu, Sep 14, 2023 at 12:15 AM Justine Olshan
> > > > > >  wrote:
> > > > > >
> > > > > > > Hey Satish -- yes, you are correct. KAFKA-15459 only affects 3.6.
> > > > > > > PR should be finalized soon.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Justine
> > > > > > >
> > > > > > > On Wed, Sep 13, 2023 at 1:41 AM Federico Valeri 
> > > > > > > 
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Satish, this is a small documentation fix about ZK to KRaft
> > > > > > > > migration, that we would like to backport to 3.5 and 3.6 
> > > > > > > > branches.
> > > > > Are
> > > > > > > > you ok with that?
> > > > > > > >
> > > > > > > > https://github.com/apache/kafka/pull/14366
> > > > > > > >
> > > > > > > > On Wed, Sep 13, 2023 at 3:13 AM Satish Duggana <
> > > > > satish.dugg...@gmail.com
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Thanks David for the quick resolution.
> > > > > > > > >
> > > > > > > > > ~Satish.
> > > > > > > > >
> > > > > > > > > On Tue, 12 Sept 2023 at 22:51, David Arthur
> > > > > > > > >  wrote:
> > > > > > > > > >
> > > > > > > > > > Satish,
> > > > > > > > > >
> > > > > > > > > > KAFKA-15450 is merged to 3.6 (as well as trunk, 3.5, and 
> > > > > > > > > > 3.4)
> > > > > > > > > >
> > > > > > > > > > Thanks!
> > > > > > > > > > David
> > > > > > > > > >
> > > > > > > > > > On Tue, Sep 12, 2023 at 11:44 AM Ismael Juma 
> > > > > > > > > > 
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Justine,
> > > > > > > > > > >
> > > > > > > > > > > Probably best to have the conversation in the JIRA ticket 
> > > > > > > > > > > vs
> > > > > the
> > > > > > > > release
> > > > > > > > > > > thread. Generally, we want to only include low risk bug 
> > > > > > > > > > > fixes
> > > > > that
> > > > > > > > are
> > > > > > > > > > > fully compatible in patch releases.
> > > > > > > > > > >
> > > > > > > > > > > Ismael
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Sep 12, 2023 at 7:16 AM Justine Olshan
> > > > > > > > > > > 
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Thanks Satish. I understand.
> > > > > > > > > > > > Just curious, is this something that could be added to
> > > > > 3.6.1? It
> > > > > > > > would be
> > > > > > > > > > > > nice to say that hanging transactions are fully covered 
> > > > > > > > > > > > in a
> > > > > 3.6
> > > > > > > > release.
> > > > > > > > > > > > I'm not as familiar with the rules around minor 
> > > > > > > > > > > > releases, but
> > > > > > > > adding it
> > > > > > > > > > > > there would give more time to ensure stability.
> 

Re: [DISCUSS] KIP-892: Transactional Semantics for StateStores

[Bruno Cadonna]

Why do we need to add READ_COMMITTED to InMemoryKeyValueStore? I think 
it is perfectly valid to say InMemoryKeyValueStore do not support 
READ_COMMITTED for now, since READ_UNCOMMITTED is the de-facto default 
at the moment.


Best,
Bruno

On 9/18/23 7:12 PM, Nick Telford wrote:

Oh! One other concern I haven't mentioned: if we make IsolationLevel a
query-time constraint, then we need to add support for READ_COMMITTED to
InMemoryKeyValueStore too, which will require some changes to the
implementation.

On Mon, 18 Sept 2023 at 17:24, Nick Telford  wrote:


Hi everyone,

I agree that having IsolationLevel be determined at query-time is the
ideal design, but there are a few sticking points:

1.
There needs to be some way to communicate the IsolationLevel down to the
RocksDBStore itself, so that the query can respect it. Since stores are
"layered" in functionality (i.e. ChangeLoggingStore, MeteredStore, etc.),
we need some way to deliver that information to the bottom layer. For IQv2,
we can use the existing State#query() method, but IQv1 has no way to do
this.

A simple approach, which would potentially open up other options, would be
to add something like: ReadOnlyKeyValueStore
readOnlyView(IsolationLevel isolationLevel) to ReadOnlyKeyValueStore (and
similar to ReadOnlyWindowStore, ReadOnlySessionStore, etc.).

2.
As mentioned above, RocksDB WriteBatches are not thread-safe, which causes
a problem if we want to provide READ_UNCOMMITTED Iterators. I also had a
look at RocksDB Transactions[1], but they solve a very different problem,
and have the same thread-safety issue.

One possible approach that I mentioned is chaining WriteBatches: every
time a new Interactive Query is received (i.e. readOnlyView, see above,
is called) we "freeze" the existing WriteBatch, and start a new one for new
writes. The Interactive Query queries the "chain" of previous WriteBatches
+ the underlying database; while the StreamThread starts writing to the
*new* WriteBatch. On-commit, the StreamThread would write *all*
WriteBatches in the chain to the database (that have not yet been written).

WriteBatches would be closed/freed only when they have been both
committed, and all open Interactive Queries on them have been closed. This
would require some reference counting.

Obviously a drawback of this approach is the potential for increased
memory usage: if an Interactive Query is long-lived, for example by doing a
full scan over a large database, or even just pausing in the middle of an
iteration, then the existing chain of WriteBatches could be kept around for
a long time, potentially forever.

--

A.
Going off on a tangent, it looks like in addition to supporting
READ_COMMITTED queries, we could go further and support REPEATABLE_READ
queries (i.e. where subsequent reads to the same key in the same
Interactive Query are guaranteed to yield the same value) by making use of
RocksDB Snapshots[2]. These are fairly lightweight, so the performance
impact is likely to be negligible, but they do require that the Interactive
Query session can be explicitly closed.

This could be achieved if we made the above readOnlyView interface look
more like:

interface ReadOnlyKeyValueView implements ReadOnlyKeyValueStore, AutoCloseable {}

interface ReadOnlyKeyValueStore {
 ...
 ReadOnlyKeyValueView readOnlyView(IsolationLevel isolationLevel);
}

But this would be a breaking change, as existing IQv1 queries are
guaranteed to never call store.close(), and therefore these would leak
memory under REPEATABLE_READ.

B.
One thing that's notable: MyRocks states that they support READ_COMMITTED
and REPEATABLE_READ, but they make no mention of READ_UNCOMMITTED[3][4].
This could be because doing so is technically difficult/impossible using
the primitives available in RocksDB.

--

Lucas, to address your points:

U1.
It's only "SHOULD" to permit alternative (i.e. non-RocksDB)
implementations of StateStore that do not support atomic writes. Obviously
in those cases, the guarantees Kafka Streams provides/expects would be
relaxed. Do you think we should require all implementations to support
atomic writes?

U2.
Stores can support multiple IsolationLevels. As we've discussed above, the
ideal scenario would be to specify the IsolationLevel at query-time.
Failing that, I think the second-best approach is to define the
IsolationLevel for *all* queries based on the processing.mode, which is
what the default StateStoreContext#isolationLevel() achieves. Would you
prefer an alternative?

While the existing implementation is equivalent to READ_UNCOMMITTED, this
can yield unexpected results/errors under EOS, if a transaction is rolled
back. While this would be a change in behaviour for users, it would look
more like a bug fix than a breaking change. That said, we *could* make it
configurable, and default to the existing behaviour (READ_UNCOMMITTED)
instead of inferring it from the processing.mode?

N1, N2.
These were only primitives to avoid boxing costs, but since thi

Re: Help to the community with review

[Luke Chen]

Hi Taras,

Thank you very much for your offers.
Really appreciate it!

For KIPs review, it's also great if the community can help review them if
available.
I'll also try my best to have a look when available.

Thank you for your contribution!
Luke

On Tue, Sep 19, 2023 at 3:08 PM Taras Ledkov  wrote:

> Hi,
>
> I'n not a committer to Kafka, but I offer my help with review of patches
> too.
> Please, tag me on GitHub (@tledkov).
>
> Unfortunately, no one offers help with KIPs review...
>

Re:Help to the community with review

[Taras Ledkov]

Hi,

I'n not a committer to Kafka, but I offer my help with review of patches too.
Please, tag me on GitHub (@tledkov).

Unfortunately, no one offers help with KIPs review...