Ramachandran Krishnan created ATLAS-5336:
--------------------------------------------
Summary: Upgrade Kafka to 3.9.1; embedded notification broker
uses KRaft (no ZooKeeper))
Key: ATLAS-5336
URL: https://issues.apache.org/jira/browse/ATLAS-5336
Project: Atlas
Issue Type: Task
Components: atlas-core
Reporter: Ramachandran Krishnan
Assignee: Ramachandran Krishnan
Fix For: 3.0.0
h3. Summary
Upgrade Apache Atlas notification Kafka from *2.8.2* (Scala 2.12) to *3.9.1*
(Scala 2.13), and replace the ZooKeeper-based in-process embedded broker with a
*KRaft* broker via {{{}KafkaClusterTestKit{}}}.
h3. Background / Motivation
* Kafka 2.8.x clients are only partially compatible with modern 3.9.x / 4.x
brokers.
* The legacy {{EmbeddedKafkaServer}} started an in-process {{ZooKeeperServer}}
plus {{KafkaServer}} tied to ZK — unnecessary complexity and a maintenance
burden as Kafka moves to KRaft-only.
* Atlas remains on Spring 5.3; Spring Kafka 3.x / {{EmbeddedKafkaKraftBroker}}
requires Spring 6. Using Kafka's {{KafkaClusterTestKit}} provides the same
KRaft engine without a Spring upgrade.
* Aligns Atlas with Kafka 3.9.x before external cluster KRaft migration and a
future Kafka 4.x path.
h3. Proposed Changes
||Area||Before||After||
|{{kafka.version}} (root {{{}pom.xml{}}})|2.8.2|*3.9.1*|
|{{kafka.scala.binary.version}}|2.12|*2.13*|
|{{EmbeddedKafkaServer}}|In-process ZK +
{{KafkaServer}}|{{KafkaClusterTestKit}} (combined KRaft controller+broker)|
|Embedded ZK for notification|Required at Atlas startup when
embedded=true|*Removed* — no {{ZooKeeperServer}} / {{zookeeper.connect}} in
embedded path|
|Docker {{atlas-kafka}} image|{{kafka_2.12-*}} tarball|{{kafka_2.13-3.9.1}}
({{{}.env{}}}, {{{}Dockerfile.atlas-kafka{}}}, {{{}download-archives.sh{}}})|
|Default distro config|ZK-oriented embedded
settings|{{{}atlas.notification.embedded=true{}}}; {{bootstrap.servers}}
placeholder updated *at runtime* by {{EmbeddedKafkaServer}}|
h3. Core code — {{EmbeddedKafkaServer}}
*Path:*
{{notification/src/main/java/org/apache/atlas/kafka/EmbeddedKafkaServer.java}}
When {{{}atlas.notification.embedded=true{}}}:
# Start KRaft broker via {{KafkaClusterTestKit}} (1 combined controller+broker
node).
# Assign ephemeral {{bootstrap.servers}} (e.g. {{{}localhost:40111{}}}).
# Write live bootstrap URL into {{atlas.kafka.bootstrap.servers}} so
{{KafkaNotification}} clients connect.
When {{{}atlas.notification.embedded=false{}}}: no-op.
*Removed (old 2.8 embedded path):*
* {{{}startZk(){}}}, {{{}ZooKeeperServer{}}}, {{ServerCnxnFactory}}
* Direct {{kafka.server.KafkaServer}} construction tied to ZK
* Port-kill retry loops for fixed ZK/Kafka ports
h3.
h3. Acceptance Criteria
* {{kafka.version}} is 3.9.1 and Scala binary version is 2.13 across build.
* {{EmbeddedKafkaServer}} starts KRaft broker without ZooKeeper when
{{{}atlas.notification.embedded=true{}}}.
* {{atlas.kafka.bootstrap.servers}} is rewritten at runtime to the live
embedded broker address.
* Unit tests pass:
** {{KafkaNotificationTest}} (notification module)
** {{NotificationHookConsumerKafkaTest}} (webapp, with {{{}-am{}}})
* Manual: Atlas docker container with {{atlas.notification.embedded=true}} —
KRaft log lines present, REST {{/api/atlas/admin/version}} returns 200.
* Manual: Coexist stack with external {{atlas-kafka}} 3.9.1 — Hive hook →
Atlas, PII classification → TagSync → Ranger, Hive/Kafka audit paths work.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
