[jira] [Created] (KAFKA-17011) SupportedFeatures.MinVersion incorrectly blocks v0

2024-06-20 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-17011: Summary: SupportedFeatures.MinVersion incorrectly blocks v0 Key: KAFKA-17011 URL: https://issues.apache.org/jira/browse/KAFKA-17011 Project: Kafka Issue

Re: DISCUSS KIP-984 Add pluggable compression interface to Kafka

2024-06-17 Thread Colin McCabe
Hi Assane, Pluggable compression codecs in the protocol seem likely to greatly fragment the ecosystem. I don't see a corresponding benefit that offsets this. As Greg says, people who want to use a closed source codec already can, by forking Kafka. Open source clients won't be able to talk to t

Re: [VOTE] KIP-1042: Support for wildcard when creating new acls

2024-06-17 Thread Colin McCabe
My concern is that the extra complexity may actually slow us down. In general people already complain about the speed of ACL matches, and adding another "degree of freedom" seems likely to make things worse. It would be useful to understand how much faster or slower the code is with the propsed

Re: [DISCUSS] Apache Kafka 3.9.0 release

2024-06-17 Thread Colin McCabe
Hi all, Thanks, everyone. Quick update: on the release plan page, I moved feature freeze forward and code freeze by one week to make sure we can hit that. No other dates changed. With regard to 4.0, I was assuming that we'd do it 4 months after 3.9 was released. I think for that release, we sh

[jira] [Created] (KAFKA-16968) Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0

2024-06-14 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-16968: Summary: Make 3.8-IV0 a stable MetadataVersion and create 3.9-IV0 Key: KAFKA-16968 URL: https://issues.apache.org/jira/browse/KAFKA-16968 Project: Kafka

Re: [DISCUSS] Apache Kafka 3.9.0 release

2024-06-14 Thread Colin McCabe
; > I think Colin as RM should double check this list shortly before the 3.9 > release, and it should hopefully be much shorter. > > > -Matthias > > On 6/14/24 12:32 PM, Ismael Juma wrote: >> +1, thanks Colin. >> >> Ismael >> >> On Fri, Jun 14, 2024 at

Re: [DISCUSS] Apache Kafka 3.8.0 release

2024-06-14 Thread Colin McCabe
Progress. >>>>> I'll be checking all the open blockers and check if they are really a >>>>> blocker or can be pushed. >>>>> >>>>> >>>>> Regarding timeline, I'll attempt to generate the first RC on Wednesday &g

[jira] [Resolved] (KAFKA-16946) Utils.getHost/getPort cannot parse SASL_PLAINTEXT://host:port

2024-06-14 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-16946. -- Resolution: Fixed > Utils.getHost/getPort cannot parse SASL_PLAINTEXT://host:p

[DISCUSS] Apache Kafka 3.9.0 release

2024-06-14 Thread Colin McCabe
Hi all, Based on the discussion in the 3.8 release thread, we will need a 3.9 release. I guess I should create a new email thread to discuss this. I'd like to volunteer as release manager for the Apache Kafka 3.9.0 release. There's a release plan here: https://cwiki.apache.org/confluence/displ

Re: [DISCUSS] Apache Kafka 3.8.0 release

2024-06-14 Thread Colin McCabe
Hi all, We have had many delays with releases this year. We should try to get back on schedule. I agree with the idea that was proposed a few times in this thread of drawing a line under 3.8 now, and doing a short 3.9 release. I posted a 3.9 release plan here: https://cwiki.apache.org/confluen

[jira] [Created] (KAFKA-16952) Do not bump broker epoch when re-registering the same incarnation

2024-06-13 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-16952: Summary: Do not bump broker epoch when re-registering the same incarnation Key: KAFKA-16952 URL: https://issues.apache.org/jira/browse/KAFKA-16952 Project: Kafka

Re: [DISCUSS] Apache Kafka 3.8.0 release

2024-06-12 Thread Colin McCabe
Hi Josep, KIP-966 is not needed for feature parity with ZK. Indeed, KIP-966 will not be implemented in ZK at all (and never was). There is a PR related to honoring dynamic configs for unclean.leader.election which we did agree to do for 3.8. It's under review and we hope to get it in soon. We

[jira] [Resolved] (KAFKA-16522) Admin client changes for adding and removing voters

2024-06-05 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-16522. -- Resolution: Fixed Implemented by {{KAFKA-16535: Implement AddVoter, RemoveVoter, UpdateVoter

[jira] [Created] (KAFKA-16757) Fix broker re-registration issues around MV 3.7-IV2

2024-05-13 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-16757: Summary: Fix broker re-registration issues around MV 3.7-IV2 Key: KAFKA-16757 URL: https://issues.apache.org/jira/browse/KAFKA-16757 Project: Kafka Issue

[jira] [Created] (KAFKA-16649) Fix potential deadlock in DynamicBrokerConfig

2024-04-30 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-16649: Summary: Fix potential deadlock in DynamicBrokerConfig Key: KAFKA-16649 URL: https://issues.apache.org/jira/browse/KAFKA-16649 Project: Kafka Issue Type

[jira] [Created] (KAFKA-16624) Don't generate useless PartitionChangeRecord on older MV

2024-04-25 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-16624: Summary: Don't generate useless PartitionChangeRecord on older MV Key: KAFKA-16624 URL: https://issues.apache.org/jira/browse/KAFKA-16624 Project:

[ANNOUNCE] New committer: Igor Soarez

2024-04-24 Thread Colin McCabe
Hi all, The PMC of Apache Kafka is pleased to announce a new Kafka committer, Igor Soarez. Igor has been a Kafka contributor since 2019. In addition to being a regular contributor and reviewer, he has made significant contributions to improving Kafka's JBOD support in KRaft mode. He has also c

[jira] [Created] (KAFKA-16509) CurrentControllerId metric is unreliable in ZK mode

2024-04-10 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-16509: Summary: CurrentControllerId metric is unreliable in ZK mode Key: KAFKA-16509 URL: https://issues.apache.org/jira/browse/KAFKA-16509 Project: Kafka Issue

[jira] [Created] (KAFKA-16475) Create unit test for TopicImageNode

2024-04-04 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-16475: Summary: Create unit test for TopicImageNode Key: KAFKA-16475 URL: https://issues.apache.org/jira/browse/KAFKA-16475 Project: Kafka Issue Type: Improvement

[jira] [Created] (KAFKA-16469) Metadata Schema Checker

2024-04-03 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-16469: Summary: Metadata Schema Checker Key: KAFKA-16469 URL: https://issues.apache.org/jira/browse/KAFKA-16469 Project: Kafka Issue Type: New Feature

[jira] [Resolved] (KAFKA-16428) Fix bug where config change notification znode may not get created during migration

2024-03-27 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-16428. -- Resolution: Fixed > Fix bug where config change notification znode may not get created dur

[jira] [Resolved] (KAFKA-16411) Correctly migrate default client quota entities in KRaft migration

2024-03-27 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-16411. -- Resolution: Fixed > Correctly migrate default client quota entities in KRaft migrat

[jira] [Created] (KAFKA-16435) Add test for KAFKA-16428

2024-03-27 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-16435: Summary: Add test for KAFKA-16428 Key: KAFKA-16435 URL: https://issues.apache.org/jira/browse/KAFKA-16435 Project: Kafka Issue Type: Bug

[jira] [Created] (KAFKA-16428) Fix bug where config change notification znode may not get created during migration

2024-03-26 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-16428: Summary: Fix bug where config change notification znode may not get created during migration Key: KAFKA-16428 URL: https://issues.apache.org/jira/browse/KAFKA-16428

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

2024-03-26 Thread Colin McCabe
Hi all, Thanks for the discussion. Let me respond to a few questions I saw in the thread (hope I didn't miss any!) Feature level 0 is "special" because it effectively means that the feature hasn't been set up. So I could create a foo feature, a bar feature, and a baz feature tomo

Re: [VOTE] 3.6.2 RC1

2024-03-22 Thread Colin McCabe
I have created a PR to fix this here : https://github.com/apache/kafka/pull/15584 best, Colin On Fri, Mar 22, 2024, at 13:28, Colin McCabe wrote: > Sorry but I have to vote -1 > > I tried verifying that the migration quotas bug described in > https://issues.apache.org/jira/browse

[jira] [Created] (KAFKA-16411) Correctly migrate default entities in KRaft migration

2024-03-22 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-16411: Summary: Correctly migrate default entities in KRaft migration Key: KAFKA-16411 URL: https://issues.apache.org/jira/browse/KAFKA-16411 Project: Kafka Issue

Re: [VOTE] 3.6.2 RC1

2024-03-22 Thread Colin McCabe
Sorry but I have to vote -1 I tried verifying that the migration quotas bug described in https://issues.apache.org/jira/browse/KAFKA-16222 was fixed, and it appears to still be an issue with 3.6.2 RC1. The quota on the default resource is still getting translated improperly. I am looking into

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

2024-03-15 Thread Colin McCabe
gt; For transaction version specifically, I have no metadata version >> > dependencies besides requiring 3.3 to write the feature records and use >> the >> > feature tools. I would suspect all new features would have this >> > requirement. >> > >> > >> >

Re: [DISCUSS] KIP-1022 Formatting and Updating Features

2024-03-07 Thread Colin McCabe
res would have this >> requirement. >> >> >> 11. *Basically, if **--release-version is not used, the command will just >> use the latest* >> >> *production version of every feature. Should we apply that logic to both* >> >> *tools?* >> &

[jira] [Created] (KAFKA-16321) Default directory ids to MIGRATING, not UNASSIGNED

2024-03-01 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-16321: Summary: Default directory ids to MIGRATING, not UNASSIGNED Key: KAFKA-16321 URL: https://issues.apache.org/jira/browse/KAFKA-16321 Project: Kafka Issue

[jira] [Resolved] (KAFKA-16216) Reduce batch size for initial metadata load during ZK migration

2024-02-01 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-16216. -- Fix Version/s: 3.7.0 Reviewer: Colin McCabe Assignee: David Arthur (was

[jira] [Created] (KAFKA-16216) Reduce batch size for initial metadata load during ZK migration

2024-02-01 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-16216: Summary: Reduce batch size for initial metadata load during ZK migration Key: KAFKA-16216 URL: https://issues.apache.org/jira/browse/KAFKA-16216 Project: Kafka

Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka

2024-01-29 Thread Colin McCabe
t sure if we should start a full >> fledged discussion on these topics. >> >> > I'm struggling a bit with your phrasing. Are you suggesting that unstable >> MVs would not be able to be in trunk? >> >> Unstable MV should be in trunk. The wording is relat

[jira] [Created] (KAFKA-16180) Full metadata request sometimes fails during zk migration

2024-01-19 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-16180: Summary: Full metadata request sometimes fails during zk migration Key: KAFKA-16180 URL: https://issues.apache.org/jira/browse/KAFKA-16180 Project: Kafka

[jira] [Resolved] (KAFKA-16078) Be more consistent about getting the latest MetadataVersion

2024-01-17 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-16078. -- Fix Version/s: 3.7.0 Reviewer: Colin Patrick McCabe Resolution: Fixed >

[jira] [Resolved] (KAFKA-16131) Repeated UnsupportedVersionException logged when running Kafka 3.7.0-RC2 KRaft cluster with metadata version 3.6

2024-01-17 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-16131. -- Resolution: Fixed > Repeated UnsupportedVersionException logged when running Kafka 3.7.0-

Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka

2024-01-15 Thread Colin McCabe
On Fri, Jan 12, 2024, at 11:32, Artem Livshits wrote: > I think using feature flags (whether we support a framework and tooling for > feature flags or just an ad-hoc XyzEnabled flag) can be an alternative to > this KIP. I think the value of this KIP is that it's trying to propose a > systemic appr

[jira] [Resolved] (KAFKA-16121) Partition reassignments in ZK migration dual write mode stalled until leader epoch incremented

2024-01-15 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-16121. -- Fix Version/s: 3.7.0 Reviewer: Colin McCabe Assignee: David Mao

Re: [VOTE] 3.7.0 RC2

2024-01-15 Thread Colin McCabe
Docs fix discussed in the thread is here: https://github.com/apache/kafka/pull/15193 best, Colin On Sun, Jan 14, 2024, at 23:56, Colin McCabe wrote: > Hi Stanislav, > > Thanks for making the first RC. The fact that it's titled RC2 is > messing with my mind a bit. I hope

Re: [VOTE] 3.7.0 RC2

2024-01-14 Thread Colin McCabe
Hi Stanislav, Thanks for making the first RC. The fact that it's titled RC2 is messing with my mind a bit. I hope this doesn't make people think that we're farther along than we are, heh. On Sun, Jan 14, 2024, at 13:54, Jakub Scholz wrote: > *> Nice catch! It does seem like we should have gated

[jira] [Resolved] (KAFKA-16120) Partition reassignments in ZK migration dual write leaves stray partitions

2024-01-14 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-16120. -- Fix Version/s: 3.7.0 Reviewer: Colin McCabe Assignee: David Mao

[jira] [Created] (KAFKA-16126) Kcontroller dynamic configurations may fail to apply at startup

2024-01-14 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-16126: Summary: Kcontroller dynamic configurations may fail to apply at startup Key: KAFKA-16126 URL: https://issues.apache.org/jira/browse/KAFKA-16126 Project: Kafka

Re: Logging in Kafka

2024-01-10 Thread Colin McCabe
Hi Mickael, Thanks for bringing this up. If we move to log4j2 in 4.0, is there any work that needs to be done in 3.8? That's probably what we should focus on. P.S. My assumption is that if the log4j2 work misses the train, we'll stick with reload4j in 4.0. Hopefully this won't happen. best, C

Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka

2024-01-10 Thread Colin McCabe
inside the controller. Fatal ones. >> >> Thanks for this clarification. I think this concrete risk should be >> highlighted in the KIP and in the "unstable.metadata.versions.enable" >> documentation. >> >> In the test plan, should we also have one system test

Re: [DISCUSS] KIP-853: KRaft Controller Membership Changes

2024-01-09 Thread Colin McCabe
On Tue, Jan 9, 2024, at 17:07, Jason Gustafson wrote: > Hi Jose, > > Thanks for the KIP! A few initial questions below: > > 1. In the user experience section, the user is expected to provide supply > the UUID for each voter. I'm assuming this is the directory.id coming from > KIP-858. I thought it

Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka

2024-01-09 Thread Colin McCabe
example. Nobody would have said that it was production ready in 3.7 ... hence it belonged (and still belongs) in an unstable MV, until that changes (hopefully soon :) ) best, Colin > --Proven > > On Tue, Jan 9, 2024 at 3:26 PM Colin McCabe wrote: > >> Hi Justine, >> >>

Re: [DISCUSS] KIP-853: KRaft Controller Membership Changes

2024-01-09 Thread Colin McCabe
Hi José, Thanks for the revisions. I'm really excited to see this going forward for Kafka 3.8. One important piece of feedback that a lot of people have given me is that they really want auto-formatting in KRaft mode. In other words, they want to start up a process and just have it do the righ

[jira] [Resolved] (KAFKA-16094) BrokerRegistrationRequest.logDirs field must be ignorable

2024-01-09 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-16094. -- Fix Version/s: 3.7.0 Resolution: Fixed > BrokerRegistrationRequest.logDirs field m

Re: Apache Kafka 3.7.0 Release

2024-01-09 Thread Colin McCabe
KAFKA-16094 has been fixed and backported to 3.7. Colin On Mon, Jan 8, 2024, at 14:52, Colin McCabe wrote: > On an unrelated note, I found a blocker bug related to upgrades from > 3.6 (and earlier) to 3.7. > > The JIRA is here: > https://issues.apache.org/jira/browse/KAFK

Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka

2024-01-09 Thread Colin McCabe
gt; > Let's clarify if we are ok with reordering unstable metadata versions :) > > Justine > > On Mon, Jan 8, 2024 at 1:56 PM Colin McCabe wrote: > >> On Mon, Jan 8, 2024, at 13:19, Justine Olshan wrote: >> > Hey all, >> > >> > I was wonde

Re: Kafka trunk test & build stability

2024-01-09 Thread Colin McCabe
Sorry, but to put it bluntly, the current build setup isn't good enough at partial rebuilds that build caching would make sense. All Kafka devs have had the experience of needing to clean the build directory in order to get a valid build. The scala code esspecially seems to have this issue. reg

Re: Apache Kafka 3.7.0 Release

2024-01-08 Thread Colin McCabe
On an unrelated note, I found a blocker bug related to upgrades from 3.6 (and earlier) to 3.7. The JIRA is here: https://issues.apache.org/jira/browse/KAFKA-16094 Fix here: https://github.com/apache/kafka/pull/15153 best, Colin On Mon, Jan 8, 2024, at 14:47, Colin McCabe wrote: >

Re: Apache Kafka 3.7.0 Release

2024-01-08 Thread Colin McCabe
in On Mon, Jan 8, 2024, at 14:09, Ismael Juma wrote: > All modules are published to Sonatype - that's a requirement. You may be > missing the fact that `core` is published as `kafka_2.13` and `kafka_2.12`. > > Ismael > > On Tue, Jan 9, 2024 at 12:00 AM Colin McCabe wrote: &g

[jira] [Created] (KAFKA-16094) 3.7 brokers must be able to register with 3.6 and earlier controllers

2024-01-08 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-16094: Summary: 3.7 brokers must be able to register with 3.6 and earlier controllers Key: KAFKA-16094 URL: https://issues.apache.org/jira/browse/KAFKA-16094 Project: Kafka

Re: [VOTE] KIP-1012: The need for a Kafka 3.8.x release

2024-01-08 Thread Colin McCabe
ot done, we would need another minor version >> >> > - Which are the KIPs/Features the community identified that should be >> >> > present in a 3.x version so they can safely migrate to a potential 4.0 >> >> > version >> >> > - The ti

Re: Apache Kafka 3.7.0 Release

2024-01-08 Thread Colin McCabe
nderstood what they mean by gradle metadata - it's > not the Kafka metadata module. > > Ismael > > On Mon, Jan 8, 2024 at 9:45 PM Colin McCabe wrote: > >> Oops, hit send too soon. I see that #15127 was already merged. So we >> should no longer be publishin

Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka

2024-01-08 Thread Colin McCabe
way, the contents of the 3.7 branch will change up until 3.7.0 is released. Once it gets released, it's never unreleased. We just move on to 3.7.1. Same thing here. best, Colin > > Thanks, > > Justine > > On Mon, Jan 8, 2024 at 12:44 PM Colin McCabe wrote: > >>

Re: [DISCUSS] KIP-1014: Managing Unstable Metadata Versions in Apache Kafka

2024-01-08 Thread Colin McCabe
Hi Proven, Thanks for the KIP. I think there is a need for this capability, for those of us who deploy from trunk (or branches dervied from trunk). With regard to "unstable.metadata.versions.enable": is this going to be a documented configuration, or an internal one? I am guessing we want it to

Re: [DISCUSS] KIP-1015: Limit number of ssl connections in brokers

2024-01-08 Thread Colin McCabe
Hi zw, As you yourself wrote in the rejected alternatives section, the existing listener-specific connection limit already lets administrators limit the number of SSL connections (assuming that one listener is SSL and another is not). I don't understand the objection to just using that capabili

Re: [DISCUSS] KIP-1012: The need for a Kafka 3.8.x release

2024-01-08 Thread Colin McCabe
there is more stuff you feel is needed in 3.x to make migration possible, now is the time to speak up. best, Colin > > [1] https://lists.apache.org/thread/k3bbz7k0hsb4y0b2xn1lh7bjrmjowmcq > > Thanks! > Greg > > On Fri, Jan 5, 2024 at 2:20 PM Colin McCabe wrote: >>

Re: Apache Kafka 3.7.0 Release

2024-01-08 Thread Colin McCabe
Oops, hit send too soon. I see that #15127 was already merged. So we should no longer be publishing :metadata as part of the clients artifacts, right? thanks, Colin On Mon, Jan 8, 2024, at 11:42, Colin McCabe wrote: > Hi Apporv, > > Please remove the metadata module from any

Re: Apache Kafka 3.7.0 Release

2024-01-08 Thread Colin McCabe
721681581 > > > On Fri, Jan 5, 2024 at 10:22 PM Colin McCabe wrote: > >> metadata is an internal gradle module. It is not used by clients. So I >> don't see why you would want to publish it (unless I'm misunderstanding >> something). >> >> best

Re: [VOTE] KIP-1012: The need for a Kafka 3.8.x release

2024-01-08 Thread Colin McCabe
g more on the need to have at least another minor >> > release in 3.x to enable and ease migration to a potential 4.0 version >> > >> > I always mention potential in terms of 4.0 as we don't know what will be >> in >> > there yet, and this KIP

Re: [VOTE] KIP-1012: The need for a Kafka 3.8.x release

2024-01-08 Thread Colin McCabe
27;t know what will be in > there yet, and this KIP's scope is not meant to define this. > > Best, > > On Fri, Jan 5, 2024 at 10:46 PM Ismael Juma wrote: > >> I agree with Colin. Also, 4.0 would be started after 3.8 is branched (not >> after 3.8.0 is released). The

Re: Apache Kafka 3.7.0 Release

2024-01-05 Thread Colin McCabe
metadata is an internal gradle module. It is not used by clients. So I don't see why you would want to publish it (unless I'm misunderstanding something). best, Colin On Fri, Jan 5, 2024, at 10:05, Stanislav Kozlovski wrote: > Thanks for reporting the blockers, folks. Good job finding. > > I ha

Re: [DISCUSS] KIP-1012: The need for a Kafka 3.8.x release

2024-01-05 Thread Colin McCabe
anderufer 3-7, 10117 Berlin >> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen >> Amtsgericht Charlottenburg, HRB 209739 B >> >> On Fri, Jan 5, 2024, 00:24 Colin McCabe wrote: >> >> > Hi all, >> > >> > Since this has been open for a

Re: [VOTE] KIP-1012: The need for a Kafka 3.8.x release

2024-01-05 Thread Colin McCabe
esides the required ones best, Colin On Fri, Jan 5, 2024, at 13:27, Colin McCabe wrote: > Hi, > > Thanks for calling the vote, Josep. > > I re-checked this, and saw something that we missed updating. One thing > we talked about earlier is that KIP-966 is actually not required for >

Re: [DISCUSS] KIP-1012: The need for a Kafka 3.8.x release

2024-01-05 Thread Colin McCabe
gt;> There is still some discussion on the road to 4.0 thread that may be worth >> having here. >> >> On Fri, Jan 5, 2024 at 1:25 AM Josep Prat >> wrote: >> >> > Hi Colin, >> > Sorry for being quiet these last days (PTO). >> > >> > I w

Re: [VOTE] KIP-1012: The need for a Kafka 3.8.x release

2024-01-05 Thread Colin McCabe
Hi, Thanks for calling the vote, Josep. I re-checked this, and saw something that we missed updating. One thing we talked about earlier is that KIP-966 is actually not required for 3.8. What is required is some way of enabling unclean leader election by default in KRaft. (Which could be KIP-96

Re: [DISCUSS] KIP-1012: The need for a Kafka 3.8.x release

2024-01-04 Thread Colin McCabe
a tentative schedule. https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.8.0 Please let me know if these dates make sense -- they're just proposals right now. best, Colin On Thu, Dec 28, 2023, at 20:14, Colin McCabe wrote: > On Thu, Dec 28, 2023, at 18:17, Justine Olsh

Re: [DISCUSS] Road to Kafka 4.0

2024-01-04 Thread Colin McCabe
On Thu, Jan 4, 2024 at 2:15 AM Mickael Maison >> wrote: >> >> > Hi, >> > >> > I've not seen replies about log4j2. >> > The plan was to deprecated the appender (KIP-719) and switch to log4j2 >> > (KIP-653). >> > >> > While r

Re: [DISCUSS] KIP-1012: The need for a Kafka 3.8.x release

2023-12-28 Thread Colin McCabe
se/KAFKA-15975 > Yeah. There is a bunch of docs and quickstart cleanup that we should do. I don't think any of it is a blocker for 3.8 or 4.0, but the new year is a good time to clean things up. best, Colin > Justine > > On Thu, Dec 28, 2023 at 5:12 PM Colin McCabe wrote: >

Re: Apache Kafka 3.7.0 Release

2023-12-28 Thread Colin McCabe
Just to update this thread, everything in KAFKA-14127 is done now. A few tasks got moved to a separate umbrella JIRA. Some folks are going to do more testing, both manual and automated, in the next week or two. I think this will give us a good indicator of stability and what we need to fix. Ri

[jira] [Resolved] (KAFKA-14127) KIP-858: Handle JBOD broker disk failure in KRaft

2023-12-28 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-14127. -- Resolution: Fixed > KIP-858: Handle JBOD broker disk failure in KR

[jira] [Resolved] (KAFKA-15650) Data-loss on leader shutdown right after partition creation?

2023-12-28 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-15650. -- Resolution: Not A Problem > Data-loss on leader shutdown right after partition creat

[jira] [Created] (KAFKA-16061) JBOD follow-ups

2023-12-28 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-16061: Summary: JBOD follow-ups Key: KAFKA-16061 URL: https://issues.apache.org/jira/browse/KAFKA-16061 Project: Kafka Issue Type: Improvement Reporter

Re: [DISCUSS] Road to Kafka 4.0

2023-12-28 Thread Colin McCabe
Hi all, Let's continue this dicsussion on the "[DISCUSS] KIP-1012: The need for a Kafka 3.8.x release" email thread. Colin On Tue, Dec 26, 2023, at 12:50, José Armando García Sancio wrote: > Hi Divij, > > Thanks for the feedback. I agree that having a 3.8 release is > beneficial but some of th

Re: [DISCUSS] KIP-1012: The need for a Kafka 3.8.x release

2023-12-28 Thread Colin McCabe
Hi Josep, Thanks for the KIP. Based on the discussions we had previously, I agree that we need a 3.8. It would be good to link to KIP-833 in the motivation section, since this KIP builds on that one. Also, I think we should mention in KIP-1012 that 3.8 will be a general-purpose release that m

Re: [ANNOUNCE] New Kafka PMC Member: Divij Vaidya

2023-12-28 Thread Colin McCabe
Congratulations, Divij! best, Colin On Thu, Dec 28, 2023, at 11:38, Bruno Cadonna wrote: > Congratulations Divij! Well deserved! > > Best, > Bruno > > On 12/27/23 12:45 PM, Luke Chen wrote: >> Hi, Everyone, >> >> Divij has been a Kafka committer since June, 2023. He has remained very >> active a

Re: [DISCUSS] KIP-1013: Drop broker and tools support for Java 11 in Kafka 4.0 (deprecate in 3.7)

2023-12-26 Thread Colin McCabe
Hi Ismael, +1 from me. Looking at the list of languages features for JDK17, from a developer productivity standpoint, the biggest wins are probably pattern matching and java.util.HexFormat. Also, Java 11 is getting long in the tooth, even though we never adopted it. It was released 6 years ag

[jira] [Created] (KAFKA-15979) Add KIP-1001 CurrentControllerId metric

2023-12-06 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-15979: Summary: Add KIP-1001 CurrentControllerId metric Key: KAFKA-15979 URL: https://issues.apache.org/jira/browse/KAFKA-15979 Project: Kafka Issue Type

[jira] [Created] (KAFKA-15980) Add KIP-1001 CurrentControllerId metric

2023-12-06 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-15980: Summary: Add KIP-1001 CurrentControllerId metric Key: KAFKA-15980 URL: https://issues.apache.org/jira/browse/KAFKA-15980 Project: Kafka Issue Type

Re: [VOTE] KIP-858: Handle JBOD broker disk failure in KRaft

2023-12-04 Thread Colin McCabe
Yes, this should be CLUSTER_ACTION on CLUSTER, to be consistent with our other internal RPCs. best, Colin On Mon, Dec 4, 2023, at 04:28, Viktor Somogyi-Vass wrote: > Hi Igor, > > I'm just reading through your KIP and noticed that the new protocol you > created doesn't say anything about ACLs of

[jira] [Created] (KAFKA-15956) MetadataShell must take the directory lock when reading

2023-12-01 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-15956: Summary: MetadataShell must take the directory lock when reading Key: KAFKA-15956 URL: https://issues.apache.org/jira/browse/KAFKA-15956 Project: Kafka

[jira] [Resolved] (KAFKA-15311) Fix docs about reverting to ZooKeeper mode during KRaft migration

2023-11-29 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-15311. -- Fix Version/s: 3.7.0 Resolution: Fixed > Fix docs about reverting to ZooKeeper m

[jira] [Created] (KAFKA-15922) Add MetadataVersion for JBOD

2023-11-28 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-15922: Summary: Add MetadataVersion for JBOD Key: KAFKA-15922 URL: https://issues.apache.org/jira/browse/KAFKA-15922 Project: Kafka Issue Type: Improvement

[jira] [Resolved] (KAFKA-15860) ControllerRegistration must be written out to the metadata image

2023-11-27 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-15860. -- Fix Version/s: 3.7.0 Resolution: Fixed > ControllerRegistration must be written out

Re: [DISCUSS] Road to Kafka 4.0

2023-11-27 Thread Colin McCabe
ay, once branch 3's time has come, it needs to fade away, just like JDK 6 support or the old Scala producer. best, Colin > BR, > Anton > > Den tors 23 nov. 2023 kl 05:48 skrev Colin McCabe : > >> On Tue, Nov 21, 2023, at 19:30, Luke Chen wrote: >> > Yes, KIP-853

Re: [DISCUSS] Road to Kafka 4.0

2023-11-22 Thread Colin McCabe
preciate the offer. But, on the KRaft side, I still maintain that nothing is missing except JBOD, which we already have a plan for. best, Colin > Thank you. > Luke > > On Wed, Nov 22, 2023 at 2:54 AM Colin McCabe wrote: > >> On Tue, Nov 21, 2023, at 03:47, Josep Prat wro

Re: [VOTE] KIP-896: Remove old client protocol API versions in Kafka 4.0

2023-11-21 Thread Colin McCabe
e/display/KAFKA/KIP-724%3A+Drop+support+for+message+formats+v0+and+v1 > > Ismael > > On Tue, Nov 21, 2023, 12:21 PM Colin McCabe wrote: > >> Hi Ismael, >> >> Can we state somewhere that the message.format.version configuration will >> be gone in 4.0? We only w

Re: [VOTE] KIP-896: Remove old client protocol API versions in Kafka 4.0

2023-11-21 Thread Colin McCabe
Hi Ismael, Can we state somewhere that the message.format.version configuration will be gone in 4.0? We only will support one message format version (for now, at least). If we do want more versions later, I don't think we'll want to configure them via a static config. best, Colin On Tue, Nov

Re: [VOTE] KIP-1001; CurrentControllerId Metric

2023-11-21 Thread Colin McCabe
> wrote: >> >> The KIP makes sense. +1 >> >> On Mon, Nov 20, 2023 at 12:37 PM David Arthur >> wrote: >> >> > Thanks Colin, >> > >> > +1 from me >> > >> > -David >> > >> > On Tue, Nov 14, 2

Re: [DISCUSS] Road to Kafka 4.0

2023-11-21 Thread Colin McCabe
aned up of deprecated classes and methods on top of the usual > work we have. For this reason I think there would be enough time > between releasing 3.8.0 and 4.0.0. > > What do you all think? > > Best, > Josep Prat > > On 2023/11/20 20:03:18 Colin McCabe wrote: >>

[jira] [Created] (KAFKA-15860) ControllerRegistration must be written out to the metadata image

2023-11-20 Thread Colin McCabe (Jira)
Colin McCabe created KAFKA-15860: Summary: ControllerRegistration must be written out to the metadata image Key: KAFKA-15860 URL: https://issues.apache.org/jira/browse/KAFKA-15860 Project: Kafka

Re: [DISCUSS] Road to Kafka 4.0

2023-11-20 Thread Colin McCabe
should have a 3.8.x version before 4.0.x. > > What do other think in this regard? > > Best, > > On 2023/11/14 20:47:10 Colin McCabe wrote: >> On Tue, Nov 14, 2023, at 04:37, Anton Agestam wrote: >> > Hi Colin, >> > >> > Thank you for your thoughtful

[VOTE] KIP-1001; CurrentControllerId Metric

2023-11-14 Thread Colin McCabe
Hi all, I'd like to call a vote for KIP-1001: Add CurrentControllerId metric. Take a look here: https://cwiki.apache.org/confluence/x/egyZE best, Colin

Re: [DISCUSS] Road to Kafka 4.0

2023-11-14 Thread Colin McCabe
s good, but the point I was making was if we could have a release > with both KRaft and ZK supporting this feature to ease the migration out of > ZK. > The problem is, supporting multiple controller implementations is a huge burden. So we don't want to extend the 3.x release past the

Re: [VOTE] KIP-979 Allow independently stop KRaft processes

2023-11-14 Thread Colin McCabe
e > process role provided." > > What do you think? > > Thanks, > Hailey > > On Thu, Nov 9, 2023 at 4:03 PM Colin McCabe wrote: > >> Hi Hailey, >> >> Thanks for the KIP. >> >> It feels clunky to have to pass an absolute path to the configura

[jira] [Resolved] (KAFKA-15532) ZkWriteBehindLag should not be reported by inactive controllers

2023-11-13 Thread Colin McCabe (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe resolved KAFKA-15532. -- Resolution: Fixed > ZkWriteBehindLag should not be reported by inactive controll

<    1   2   3   4   5   6   7   8   9   10   >