ijuma commented on pull request #34089: URL: https://github.com/apache/spark/pull/34089#issuecomment-1034384586
I think this discussion has been a bit confusing because it started as a stability concern and then moved to compatibility. So, let's address them separately. On the stability front, 3.0.1 and 3.1.1 should both be stable and likely stabler than older releases. We haven't made any significant architectural changes in 3.x and a bunch of bugs have been fixed. On the compatibility front, there are two potential issues I can think of: 1. Default configuration changes. This [KIP](https://cwiki.apache.org/confluence/display/KAFKA/KIP-679%3A+Producer+will+enable+the+strongest+delivery+guarantee+by+default) has the details on the potential compatibility issues. Some of them are edge cases, but the issue with old kafka clusters may not be so uncommon. There is an easy way to avoid this issue, but it does require the customer to set a config. 2. Binary compatibility issues if an application uses both Spark and Kafka clients directly and they use one of the removed methods. This is somewhat unlikely, but including for completeness. If the expectation for Spark users is that a minor release is a drop-in replacement and _no action_ is expected from users, then I agree that the above poses a problem. The approach when it comes to these things vary from project to project. Since it takes a long time to go from one major to another major, the bar is usually a little lower. For example, Kafka dropped support for Scala 2.11 _and_ added support for ZK 3.5.7 in Apache Kafka 2.5. We did a KIP for the former (Scala 2.11 upgrade), but did not do a KIP for the latter (ZK 3.5.x upgrade) as our analysis indicated that it was a compatible upgrade given how Kafka uses ZK. Even though it was a .7, it did have a bunch of critical bugs in it still, but they would probably not have been found and fixed if we had delayed adoption. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
