HeartSaVioR commented on pull request #34089: URL: https://github.com/apache/spark/pull/34089#issuecomment-1033442550
It sounds to me as the phases are switched. If we were to test the dependency against the stage/production to make sure we don't break anything, testing on stage/production should be performed "before" we push to master branch. It is totally fine if the PR is submitted and stayed for a while on testing phase (it would be nice if we share the progress), but it seems odd to me if we push the change first and say "we will test the change". Let me quote my comment again: > Personally I'm in favor of holding on upgrade for major version till a couple of bugfix versions based on the major version are released. There is around 6 months for Spark 3.3.0 to be released, and we can let early-adopters to experiment with Kafka 3.0.0 (even 3.0.x) clients in the meanwhile. I didn't point out any ticket there. It wasn't about KAFKA-13322. I raised general concern about upgrading major release. Everyone tend to be very cautious about the dependency version upgrade (bugfix versions tend to be OK). Flink uses Kafka 2.8.1 in master branch, which was upgraded from 2.4.1 in Nov 2021 (while Kafka 3.0 was already available) due to Scala 2.12. The latest version of Kafka client listed in Beam is 2.5.1 if I understand correctly. Confluent doesn't even announce Confluent platform backed by Kafka 3.1. If we strongly believe we should take a step on Kafka 3.0 earlier than any others, let's build a strong rationalization on main purpose to upgrade, and ask for Kafka community to release 3.0.1. Otherwise, 2.8.1 seems to be the thing we want to go (it's already 2.8.1.) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
