Bump to try gathering more voices before taking action. For now, I see two voices as option 2 & 5 (similar to option 2 but not in the migration node but in the release note).
On Fri, Mar 18, 2022 at 7:15 PM Jungtaek Lim <kabhwan.opensou...@gmail.com> wrote: > CORRECTION: in option 2, we enumerate KIPs which may bring incompatibility > with older brokers (not all KIPs). > > On Fri, Mar 18, 2022 at 7:12 PM Jungtaek Lim <kabhwan.opensou...@gmail.com> > wrote: > >> Hi dev, >> >> I would like to initiate the discussion about how to deal with the >> migration guide on upgrading Kafka to 3.1 (from 2.8.1) in upcoming Spark >> 3.3. >> >> We didn't care much about the upgrade of Kafka dependency since our >> belief on Kafka client has been that the new Kafka client version should >> have no compatibility issues with older brokers. Based on semantic >> versioning, upgrading major versions rings an alarm for me. >> >> I haven't gone through changes that happened between versions, but found >> one KIP (KIP-679 >> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-679%3A+Producer+will+enable+the+strongest+delivery+guarantee+by+default>) >> which may not work with older brokers with specific setup. (It's described >> in the "Compatibility, Deprecation, and Migration Plan" section of the KIP). >> >> This may not be problematic for the users who upgrade both client and >> broker altogether, but end users of Spark may be unlikely the case. >> Computation engines are relatively easier to upgrade. Storage systems >> aren't. End users would think the components are independent. >> >> I looked through the notable changes in the Kafka doc, and it does >> mention this KIP, but it just says the default config has changed and >> doesn't mention about the impacts. There is a link to >> KIP, that said, everyone needs to read through the KIP wiki page for >> details. >> >> Based on the context, what would be the best way to notice end users for >> the major version upgrade of Kafka? I can imagine several options >> including... >> >> 1. Explicitly mention that Spark 3.3 upgrades Kafka to 3.1 with linking >> the noticeable changes in the Kafka doc in the migration guide. >> 2. Do 1 & spend more effort to read through all KIPs and check >> "Compatibility, Deprecation, and Migration Plan" section, and enumerate all >> KIPs (or even summarize) in the migration guide. >> 3. Do 2 & actively override the default configs to be compatible with >> older versions if the change of the default configs in Kafka 3.0 is >> backward incompatible. End users should set these configs explicitly to >> override them back. >> 4. Do not care. End users can indicate the upgrade in the release note, >> and we expect end users to actively check the notable changes (& KIPs) from >> Kafka doc. >> 5. Options not described above... >> >> Please take a look and provide your voice on this. >> >> Thanks, >> Jungtaek Lim (HeartSaVioR) >> >> ps. Probably this would be applied to all non-bugfix versions of >> dependency upgrades. We may still want to be pragmatic, e.g. pass-through >> for minor versions, though. >> >
