On Tue, Nov 21, 2023, at 19:30, Luke Chen wrote: > Yes, KIP-853 and disk failure support are both very important missing > features. For the disk failure support, I don't think this is a > "good-to-have-feature", it should be a "must-have" IMO. We can't announce > the 4.0 release without a good solution for disk failure in KRaft.
Hi Luke, Thanks for the reply. Controller disk failure support is not missing from KRaft. I described how to handle controller disk failures earlier in this thread. I should note here that the broker in ZooKeeper mode also requires manual handling of disk failures. Restarting a broker with the same ID, but an empty disk, breaks the invariants of replication when in ZK mode. Consider: 1. Broker 1 goes down. A ZK state change notification for /brokers fires and goes on the controller queue. 2. Broker 1 comes back up with an empty disk. 3. The controller processes the zk state change notification for /brokers. Since broker 1 is up no action is taken. 4. Now broker 1 is in the ISR for any partitions it was previously, but has no data. If it is or becomes leader for any partitions, irreversable data loss will occur. This problem is more than theoretical. We at Confluent have observed it in production and put in place special workarounds for the ZK clusters we still have. KRaft has never had this problem because brokers are removed from ISRs when a new incarnation of the broker registers. So perhaps ZK mode is not ready for production for Aiven? Since disk failures do in fact require special handling there. (And/or bringing up new nodes with empty disks, which seems to be their main concern.) > > It’s also worth thinking about how Apache Kafka users who depend on JBOD > might look at the risks of not having a 3.8 release. JBOD support on KRaft > is planned to be added in 3.7, and is still in progress so far. So it’s > hard to say it’s a blocker or not. But in practice, even if the feature is > made into 3.7 in time, a lot of new code for this feature is unlikely to be > entirely bug free. We need to maintain the confidence of those users, and > forcing them to migrate through 3.7 where this new code is hardly > battle-tested doesn’t appear to do that. > As Ismael said, if there are JBOD bugs in 3.7, we will do follow-on point releases to address them. > Our goal for 4.0 should be that all the “main” features in KRaft are in > production ready state. To reach the goal, I think having one more release > makes sense. We can have different opinions about what the “main features” > in KRaft are, but we should all agree, JBOD is one of them. The current plan is for JBOD to be production-ready in the 3.7 branch. The other features of KRaft have been in production-ready state since the 3.3 release. (Well, except for delegation tokens and SCRAM, which were implemented in 3.5 and 3.6) > I totally agree with you. We can keep delaying the 4.0 release forever. I'd > also like to draw a line to it. So, in my opinion, the 3.8 release is the > line. No 3.9, 3.10 releases after that. If this is the decision, will your > concern about this infinite loop disappear? Well, the line was drawn in KIP-833. If we redraw it, what is to stop us from redrawing it again and again? > > Final note: Speaking of the missing features, I can always cooperate with > you and all other community contributors to make them happen, like we have > discussed earlier. Just let me know. > Thanks, Luke. I appreciate the offer. But, on the KRaft side, I still maintain that nothing is missing except JBOD, which we already have a plan for. best, Colin > Thank you. > Luke > > On Wed, Nov 22, 2023 at 2:54 AM Colin McCabe <cmcc...@apache.org> wrote: > >> On Tue, Nov 21, 2023, at 03:47, Josep Prat wrote: >> > Hi Colin, >> > >> > I think it's great that Confluent runs KRaft clusters in production, >> > and it means that it is production ready for Confluent and it's users. >> > But luckily for Kafka, the community is bigger than this (self managed >> > in the cloud or in-prem, or customers of other SaaS companies). >> >> Hi Josep, >> >> Confluent is not the only company using or developing KRaft. Most of the >> big organizations developing Kafka are involved. I mentioned Confluent's >> deployments because I wanted to be clear that KRaft mode is not >> experimental or new. Talking about software in production is a good way to >> clear up these misconceptions. >> >> Indeed, KRaft mode is many years old. It started around 2020, and became >> production-ready in AK 3.5 in 2022. ZK mode was deprecated in AK 3.5, which >> was released June 2023. If we release AK 4.0 around April (or maybe a month >> or two later) then that will be almost a full year between deprecation and >> removal of ZK mode. We've talked about this a lot, in KIPs, in Apache blog >> posts, at conferences, and so forth. >> >> > We've heard at least from 1 SaaS company, Aiven (disclaimer, it is my >> > employer) where the current feature set makes it not trivial to >> > migrate. This same issue might happen not only at Aiven but with any >> > user of Kafka who uses immutable infrastructure. >> >> Can you discuss why you feel it is "not trivial to migrate"? From the >> discussion above, the main gap is that we should improve the documentation >> for handling failed disks. >> >> > Another case is for >> > users that have hundreds (or more) of clusters and more than 100k nodes >> > experience node failures multiple times during a single day. In this >> > situation, not having KIP 853 makes these power users unable to join >> > the game as introducing a new error-prone manual (or needed to >> > automate) operation is usually a huge no-go. >> >> We have thousands of KRaft clusters in production and haven't seen these >> problems, as I described above. >> >> best, >> Colin >> >> > >> > But I hear the concerns of delaying 4.0 for another 3 to 4 months. >> > Would it help if we would aim at shortening the timeline for 3.8.0 and >> > start with the 4.0.0 a bit earlier help? >> > Maybe we could work on 3.8.0 almost in parallel with 4.0.0: >> > - Start with 3.8.0 release process >> > - After a small time (let's say a week) create the release branch >> > - Start with 4.0.0 release process as usual >> > - Cherry pick KRaft related issues to 3.8.0 >> > - Release 3.8.0 >> > I suspect 4.0.0 will need a bit more time than usual to ensure the code >> > is cleaned up of deprecated classes and methods on top of the usual >> > work we have. For this reason I think there would be enough time >> > between releasing 3.8.0 and 4.0.0. >> > >> > What do you all think? >> > >> > Best, >> > Josep Prat >> > >> > On 2023/11/20 20:03:18 Colin McCabe wrote: >> >> Hi Josep, >> >> >> >> I think there is some confusion here. Quorum reconfiguration is not >> needed for KRaft to become production ready. Confluent runs thousands of >> KRaft clusters without quorum reconfiguration, and has for years. While >> dynamic quorum reconfiguration is a nice feature, it doesn't block >> anything: not migration, not deployment. As best as I understand it, the >> use-case Aiven has isn't even reconfiguration per se, just wiping a disk. >> There are ways to handle this -- I discussed some earlier in the thread. I >> think it would be productive to continue that discussion -- especially the >> part around documentation and testing of these cases. >> >> >> >> A lot of people have done a lot of work to get Kafka 4.0 ready. I would >> not want to delay that because we want an additional feature. And we will >> always want additional features. So I am concerned we will end up in an >> infinite loop of people asking for "just one more feature" before they >> migrate. >> >> >> >> best, >> >> Colin >> >> >> >> >> >> On Mon, Nov 20, 2023, at 04:15, Josep Prat wrote: >> >> > Hi all, >> >> > >> >> > I wanted to share my opinion regarding this topic. I know some >> >> > discussions happened some time ago (over a year) but I believe it's >> >> > wise to reflect and re-evaluate if those decisions are still valid. >> >> > KRaft, as of Kafka 3.6.x and 3.7.x, has not yet feature parity with >> >> > Zookeeper. By dropping Zookeeper altogether before achieving such >> >> > parity, we are opening the door to leaving a chunk of Apache Kafka >> >> > users without an easy way to upgrade to 4.0. >> >> > In pro of making upgrades as smooth as possible, I propose to have a >> >> > Kafka version where KIP-853 is merged and Zookeeper still is >> supported. >> >> > This will enable community members who can't migrate yet to KRaft to >> do >> >> > so in a safe way (rolling back is something goes wrong). >> Additionally, >> >> > this will give us more confidence on having KRaft replacing >> >> > successfully Zookeeper without any big problems by discovering and >> >> > fixing bugs or by confirming that KRaft works as expected. >> >> > For this I strongly believe we should have a 3.8.x version before >> 4.0.x. >> >> > >> >> > What do other think in this regard? >> >> > >> >> > Best, >> >> > >> >> > On 2023/11/14 20:47:10 Colin McCabe wrote: >> >> >> On Tue, Nov 14, 2023, at 04:37, Anton Agestam wrote: >> >> >> > Hi Colin, >> >> >> > >> >> >> > Thank you for your thoughtful and comprehensive response. >> >> >> > >> >> >> >> KIP-853 is not a blocker for either 3.7 or 4.0. We discussed this >> in >> >> >> >> several KIPs that happened this year and last year. The most >> notable was >> >> >> >> probably KIP-866, which was approved in May 2022. >> >> >> > >> >> >> > I understand this is the case, I'm raising my concern because I was >> >> >> > foreseeing some major pain points as a consequence of this >> decision. Just >> >> >> > to make it clear though: I am not asking for anyone to do work for >> me, and >> >> >> > I understand the limitations of resources available to implement >> features. >> >> >> > What I was asking is rather to consider the implications of >> _removing_ >> >> >> > features before there exists a replacement for them. >> >> >> > >> >> >> > I understand that the timeframe for 3.7 isn't feasible, and >> because of that >> >> >> > I think what I was asking is rather: can we make sure that there >> are more >> >> >> > 3.x releases until controller quorum online resizing is >> implemented? >> >> >> > >> >> >> > From your response, I gather that your stance is that it's >> important to >> >> >> > drop ZK support sooner rather than later and that the necessary >> pieces for >> >> >> > doing so are already in place. >> >> >> >> >> >> Hi Anton, >> >> >> >> >> >> Yes. I'm basically just repeating what we agreed upon in 2022 as >> part of KIP-833. >> >> >> >> >> >> > >> >> >> > --- >> >> >> > >> >> >> > I want to make sure I've understood your suggested sequence for >> controller >> >> >> > node replacement. I hope the mentions of Kubernetes are rather for >> examples >> >> >> > of how to carry things out, rather than saying "this is only >> supported on >> >> >> > Kubernetes"? >> >> >> >> >> >> Apache Kafka is supported in lots of environments, including non-k8s >> ones. I was just pointing out that using k8s means that you control your >> own DNS resolution, which simplifies matters. If you don't control DNS >> there are some extra steps for changing the quorum voters. >> >> >> >> >> >> > >> >> >> > Given we have three existing nodes as such: >> >> >> > >> >> >> > - a.local -> 192.168.0.100 >> >> >> > - b.local -> 192.168.0.101 >> >> >> > - c.local -> 192.168.0.102 >> >> >> > >> >> >> > As well as a candidate node 192.168.0.103 that we want to replace >> for the >> >> >> > role of c.local. >> >> >> > >> >> >> > 1. Shut down controller process on node .102 (to make sure we >> don't "go >> >> >> > back in time"). >> >> >> > 2. rsync state from leader to .103. >> >> >> > 3. Start controller process on .103. >> >> >> > 4. Point the c.local entry at .103. >> >> >> > >> >> >> > I have a few questions about this sequence: >> >> >> > >> >> >> > 1. Would this sequence be safe against leadership changes? >> >> >> > >> >> >> >> >> >> If the leader changes, the new leader should have all of the >> committed entries that the old leader had. >> >> >> >> >> >> > 2. Does it work >> >> >> >> >> >> Probably the biggest issue is dealing with "torn writes" that happen >> because you're copying the current log segment while it's being written to. >> The system should be robust against this. However, we don't regularly do >> this, so there hasn't been a lot of testing. >> >> >> >> >> >> I think Jose had a PR for improving the handling of this which we >> might want to dig up. We'd want the system to auto-truncate the partial >> record at the end of the log, if there is one. >> >> >> >> >> >> > 3. By "state", do we mean `metadata.log.dir`? Something else? >> >> >> >> >> >> Yes, the state of the metadata.log.dir. Keep in mind you will need >> to change the node ID in meta.properties after copying, of course. >> >> >> >> >> >> > 4. What are the effects on cluster availability? (I think this is >> the same >> >> >> > as asking what happens if a or b crashes during the process, or if >> network >> >> >> > partitions occur). >> >> >> >> >> >> Cluster metadata state tends to be pretty small. typically a hundred >> megabytes or so. Therefore, I do not think it will take more than a second >> or two to copy from one node to another. However, if you do experience a >> crash when one node out of three is down, then you will be unavailable >> until you can bring up a second node to regain a majority. >> >> >> >> >> >> > >> >> >> > --- >> >> >> > >> >> >> > If this is considered the official way of handling controller node >> >> >> > replacements, does it make sense to improve documentation in this >> area? Is >> >> >> > there already a plan for this documentation layed out in some >> KIPs? This is >> >> >> > something I'd be happy to contribute to. >> >> >> > >> >> >> >> >> >> Yes, I think we should have official documentation about this. We'd >> be happy to review anything in that area. >> >> >> >> >> >> >> To circle back to KIP-853, I think it stands a good chance of >> making it >> >> >> >> into AK 4.0. >> >> >> > >> >> >> > This sounds good, but the point I was making was if we could have >> a release >> >> >> > with both KRaft and ZK supporting this feature to ease the >> migration out of >> >> >> > ZK. >> >> >> > >> >> >> >> >> >> The problem is, supporting multiple controller implementations is a >> huge burden. So we don't want to extend the 3.x release past the point >> that's needed to complete all the must-dos (SCRAM, delegation tokens, JBOD) >> >> >> >> >> >> best, >> >> >> Colin >> >> >> >> >> >> >> >> >> > BR, >> >> >> > Anton >> >> >> > >> >> >> > Den tors 9 nov. 2023 kl 23:04 skrev Colin McCabe < >> cmcc...@apache.org>: >> >> >> > >> >> >> >> Hi Anton, >> >> >> >> >> >> >> >> It rarely makes sense to scale up and down the number of >> controller nodes >> >> >> >> in the cluster. Only one controller node will be active at any >> given time. >> >> >> >> The main reason to use 5 nodes would be to be able to tolerate 2 >> failures >> >> >> >> instead of 1. >> >> >> >> >> >> >> >> At Confluent, we generally run KRaft with 3 controllers. We have >> not seen >> >> >> >> problems with this setup, even with thousands of clusters. We have >> >> >> >> discussed using 5 node controller clusters on certain very big >> clusters, >> >> >> >> but we haven't done that yet. This is all very similar to ZK, >> where most >> >> >> >> deployments were 3 nodes as well. >> >> >> >> >> >> >> >> KIP-853 is not a blocker for either 3.7 or 4.0. We discussed this >> in >> >> >> >> several KIPs that happened this year and last year. The most >> notable was >> >> >> >> probably KIP-866, which was approved in May 2022. >> >> >> >> >> >> >> >> Many users these days run in a Kubernetes environment where >> Kubernetes >> >> >> >> actually controls the DNS. This makes changing the set of voters >> less >> >> >> >> important than it was historically. >> >> >> >> >> >> >> >> For example, in a world with static DNS, you might have to change >> the >> >> >> >> controller.quorum.voters setting from: >> >> >> >> >> >> >> >> 100@a.local:9073,101@b.local:9073,102@c.local:9073 >> >> >> >> >> >> >> >> to: >> >> >> >> >> >> >> >> 100@a.local:9073,101@b.local:9073,102@d.local:9073 >> >> >> >> >> >> >> >> In a world with k8s controlling the DNS, you simply remap c.local >> to point >> >> >> >> ot the IP address of your new pod for controller 102, and you're >> done. No >> >> >> >> need to update controller.quorum.voters. >> >> >> >> >> >> >> >> Another question is whether you re-create the pod data from >> scratch every >> >> >> >> time you add a new node. If you store the controller data on an >> EBS volume >> >> >> >> (or cloud-specific equivalent), you really only have to detach it >> from the >> >> >> >> previous pod and re-attach it to the new pod. k8s also handles >> this >> >> >> >> automatically, of course. >> >> >> >> >> >> >> >> If you want to reconstruct the full controller pod state each >> time you >> >> >> >> create a new pod (for example, so that you can use only instance >> storage), >> >> >> >> you should be able to rsync that state from the leader. In >> general, the >> >> >> >> invariant that we want to maintain is that the state should not >> "go back in >> >> >> >> time" -- if controller 102 promised to hold all log data up to >> offset X, it >> >> >> >> should come back with committed data at at least that offset. >> >> >> >> >> >> >> >> There are lots of new features we'd like to implement for KRaft, >> and Kafka >> >> >> >> in general. If you have some you really would like to see, I >> think everyone >> >> >> >> in the community would be happy to work with you. The flip side, >> of course, >> >> >> >> is that since there are an unlimited number of features we could >> do, we >> >> >> >> can't really block the release for any one feature. >> >> >> >> >> >> >> >> To circle back to KIP-853, I think it stands a good chance of >> making it >> >> >> >> into AK 4.0. Jose, Alyssa, and some other people have worked on >> it. It >> >> >> >> definitely won't make it into 3.7, since we have only a few weeks >> left >> >> >> >> before that release happens. >> >> >> >> >> >> >> >> best, >> >> >> >> Colin >> >> >> >> >> >> >> >> >> >> >> >> On Thu, Nov 9, 2023, at 00:20, Anton Agestam wrote: >> >> >> >> > Hi Luke, >> >> >> >> > >> >> >> >> > We have been looking into what switching from ZK to KRaft will >> mean for >> >> >> >> > Aiven. >> >> >> >> > >> >> >> >> > We heavily depend on an “immutable infrastructure” model for >> deployments. >> >> >> >> > This means that, when we perform upgrades, we introduce new >> nodes to our >> >> >> >> > clusters, scale the cluster up to incorporate the new nodes, >> and then >> >> >> >> phase >> >> >> >> > the old ones out once all partitions are moved to the new >> generation. >> >> >> >> This >> >> >> >> > allows us, and anyone else using a similar model, to do >> upgrades as well >> >> >> >> as >> >> >> >> > cluster resizing with zero downtime. >> >> >> >> > >> >> >> >> > Reading up on KRaft and the ZK-to-KRaft migration path, this is >> somewhat >> >> >> >> > worrying for us. It seems like, if KIP-853 is not included >> prior to >> >> >> >> > dropping support for ZK, we will essentially have no satisfying >> upgrade >> >> >> >> > path. Even if KIP-853 is included in 4.0, I’m unsure if that >> would allow >> >> >> >> a >> >> >> >> > migration path for us, since a new cluster generation would not >> be able >> >> >> >> to >> >> >> >> > use ZK during the migration step. >> >> >> >> > On the other hand, if KIP-853 was released in a version prior >> to dropping >> >> >> >> > ZK support, because it allows online resizing of KRaft >> clusters, this >> >> >> >> would >> >> >> >> > allow us and others that use an immutable infrastructure >> deployment >> >> >> >> model, >> >> >> >> > to provide a zero downtime migration path. >> >> >> >> > >> >> >> >> > For that reason, we’d like to raise awareness around this issue >> and >> >> >> >> > encourage considering the implementation of KIP-853 or >> equivalent a >> >> >> >> blocker >> >> >> >> > not only for 4.0, but for the last version prior to 4.0. >> >> >> >> > >> >> >> >> > BR, >> >> >> >> > Anton >> >> >> >> > >> >> >> >> > On 2023/10/11 12:17:23 Luke Chen wrote: >> >> >> >> >> Hi all, >> >> >> >> >> >> >> >> >> >> While Kafka 3.6.0 is released, I’d like to start the >> discussion for the >> >> >> >> >> “road to Kafka 4.0”. Based on the plan in KIP-833 >> >> >> >> >> < >> >> >> >> > >> >> >> >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-833%3A+Mark+KRaft+as+Production+Ready#KIP833:MarkKRaftasProductionReady-Kafka3.7 >> >> >> >> >>, >> >> >> >> >> the next release 3.7 will be the final release before moving >> to Kafka >> >> >> >> 4.0 >> >> >> >> >> to remove the Zookeeper from Kafka. Before making this major >> change, I'd >> >> >> >> >> like to get consensus on the "must-have features/fixes for >> Kafka 4.0", >> >> >> >> to >> >> >> >> >> avoid some users being surprised when upgrading to Kafka 4.0. >> The intent >> >> >> >> > is >> >> >> >> >> to have a clear communication about what to expect in the >> following >> >> >> >> > months. >> >> >> >> >> In particular we should be signaling what features and >> configurations >> >> >> >> are >> >> >> >> >> not supported, or at risk (if no one is able to add support or >> fix known >> >> >> >> >> bugs). >> >> >> >> >> >> >> >> >> >> Here is the JIRA tickets list >> >> >> >> >> < >> https://issues.apache.org/jira/issues/?jql=labels%20%3D%204.0-blocker> >> >> >> >> I >> >> >> >> >> labeled for "4.0-blocker". The criteria I labeled as >> “4.0-blocker” are: >> >> >> >> >> 1. The feature is supported in Zookeeper Mode, but not >> supported in >> >> >> >> KRaft >> >> >> >> >> mode, yet (ex: KIP-858: JBOD in KRaft) >> >> >> >> >> 2. Critical bugs in KRaft, (ex: KAFKA-15489 : split brain in >> KRaft >> >> >> >> >> controller quorum) >> >> >> >> >> >> >> >> >> >> If you disagree with my current list, welcome to have >> discussion in the >> >> >> >> >> specific JIRA ticket. Or, if you think there are some tickets >> I missed, >> >> >> >> >> welcome to start a discussion in the JIRA ticket and ping me >> or other >> >> >> >> >> people. After we get the consensus, we can label/unlabel it >> afterwards. >> >> >> >> >> Again, the goal is to have an open communication with the >> community >> >> >> >> about >> >> >> >> >> what will be coming in 4.0. >> >> >> >> >> >> >> >> >> >> Below is the high level category of the list content: >> >> >> >> >> >> >> >> >> >> 1. Recovery from disk failure >> >> >> >> >> KIP-856 >> >> >> >> >> < >> >> >> >> > >> >> >> >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-856:+KRaft+Disk+Failure+Recovery >> >> >> >> >>: >> >> >> >> >> KRaft Disk Failure Recovery >> >> >> >> >> >> >> >> >> >> 2. Prevote to support controllers more than 3 >> >> >> >> >> KIP-650 >> >> >> >> >> < >> >> >> >> > >> >> >> >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-650%3A+Enhance+Kafkaesque+Raft+semantics >> >> >> >> >>: >> >> >> >> >> Enhance Kafkaesque Raft semantics >> >> >> >> >> >> >> >> >> >> 3. JBOD support >> >> >> >> >> KIP-858 >> >> >> >> >> < >> >> >> >> > >> >> >> >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-858%3A+Handle+JBOD+broker+disk+failure+in+KRaft >> >> >> >> >>: >> >> >> >> >> Handle >> >> >> >> >> JBOD broker disk failure in KRaft >> >> >> >> >> >> >> >> >> >> 4. Scale up/down Controllers >> >> >> >> >> KIP-853 >> >> >> >> >> < >> >> >> >> > >> >> >> >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Controller+Membership+Changes >> >> >> >> >>: >> >> >> >> >> KRaft Controller Membership Changes >> >> >> >> >> >> >> >> >> >> 5. Modifying dynamic configurations on the KRaft controller >> >> >> >> >> >> >> >> >> >> 6. Critical bugs in KRaft >> >> >> >> >> >> >> >> >> >> Does this make sense? >> >> >> >> >> Any feedback is welcomed. >> >> >> >> >> >> >> >> >> >> Thank you. >> >> >> >> >> Luke >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>
