ramackri opened a new pull request, #689: URL: https://github.com/apache/atlas/pull/689
## Summary - Bump Apache Kafka client and test broker from 2.8.2 to 3.9.1 (Scala 2.13). - Replace ZooKeeper-based embedded notification broker with KRaft via `KafkaClusterTestKit`. - Update docker dev Kafka image to `kafka_2.13-3.9.1`. ## Motivation - Kafka 2.8.x clients are only partially compatible with modern 3.9.x / 4.x brokers. - Embedded dev/test broker no longer requires an in-process ZooKeeper instance. - Aligns Atlas with the 3.9.x bridge release before external cluster KRaft migration. ## Changes | Area | Before | After | |------|--------|-------| | `kafka.version` | 2.8.2 | 3.9.1 | | `kafka.scala.binary.version` | 2.12 | 2.13 | | `EmbeddedKafkaServer` | In-process ZK + `KafkaServer` | `KafkaClusterTestKit` (KRaft) | | Docker `atlas-kafka` image | `kafka_2.12-*` | `kafka_2.13-3.9.1` | | Distro embedded config | ZK-oriented settings | KRaft; `bootstrap.servers` set at runtime | ## Testing - [x] `mvn -pl notification -Dtest=KafkaNotificationTest test` - [x] `mvn -pl webapp -am -Dtest=NotificationHookConsumerKafkaTest -DskipEnunciate=true test` - [x] Manual: Atlas docker with `atlas.notification.embedded=true` — KRaft broker starts; REST `/api/atlas/admin/version` → 200 - [x] Manual: coexist stack with external `atlas-kafka` 3.9.1 — Hive hook, TagSync, audit paths ## Notes - Production deployments should continue using `atlas.notification.embedded=false` and external `bootstrap.servers`. - Jackson bumped to 2.16.2 (required for Kafka 3.9 testkit dependencies). https://issues.apache.org/jira/browse/ATLAS-5336 Made with [Cursor](https://cursor.com) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
