[
https://issues.apache.org/jira/browse/ATLAS-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramachandran Krishnan resolved ATLAS-5336.
------------------------------------------
Resolution: Fixed
> Upgrade Kafka to 3.9.1; embedded notification broker uses KRaft (no
> ZooKeeper))
> --------------------------------------------------------------------------------
>
> Key: ATLAS-5336
> URL: https://issues.apache.org/jira/browse/ATLAS-5336
> Project: Atlas
> Issue Type: Task
> Components: atlas-core
> Reporter: Ramachandran Krishnan
> Assignee: Ramachandran Krishnan
> Priority: Major
> Fix For: 3.0.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> h3. Summary
> Upgrade Apache Atlas notification Kafka from *2.8.2* (Scala 2.12) to *3.9.1*
> (Scala 2.13), and replace the ZooKeeper-based in-process embedded broker with
> a *KRaft* broker via {{{}KafkaClusterTestKit{}}}.
> h3. Background / Motivation
> * Kafka 2.8.x clients are only partially compatible with modern 3.9.x / 4.x
> brokers.
> * The legacy {{EmbeddedKafkaServer}} started an in-process
> {{ZooKeeperServer}} plus {{KafkaServer}} tied to ZK — unnecessary complexity
> and a maintenance burden as Kafka moves to KRaft-only.
> * Atlas remains on Spring 5.3; Spring Kafka 3.x /
> {{EmbeddedKafkaKraftBroker}} requires Spring 6. Using Kafka's
> {{KafkaClusterTestKit}} provides the same KRaft engine without a Spring
> upgrade.
> * Aligns Atlas with Kafka 3.9.x before external cluster KRaft migration and
> a future Kafka 4.x path.
> h3. Proposed Changes
> ||Area||Before||After||
> |{{kafka.version}} (root {{{}pom.xml{}}})|2.8.2|*3.9.1*|
> |{{kafka.scala.binary.version}}|2.12|*2.13*|
> |{{EmbeddedKafkaServer}}|In-process ZK +
> {{KafkaServer}}|{{KafkaClusterTestKit}} (combined KRaft controller+broker)|
> |Embedded ZK for notification|Required at Atlas startup when
> embedded=true|*Removed* — no {{ZooKeeperServer}} / {{zookeeper.connect}} in
> embedded path|
> |Docker {{atlas-kafka}} image|{{kafka_2.12-*}} tarball|{{kafka_2.13-3.9.1}}
> ({{{}.env{}}}, {{{}Dockerfile.atlas-kafka{}}}, {{{}download-archives.sh{}}})|
> |Default distro config|ZK-oriented embedded
> settings|{{{}atlas.notification.embedded=true{}}}; {{bootstrap.servers}}
> placeholder updated *at runtime* by {{EmbeddedKafkaServer}}|
> h3. Core code — {{EmbeddedKafkaServer}}
> *Path:*
> {{notification/src/main/java/org/apache/atlas/kafka/EmbeddedKafkaServer.java}}
> When {{{}atlas.notification.embedded=true{}}}:
> # Start KRaft broker via {{KafkaClusterTestKit}} (1 combined
> controller+broker node).
> # Assign ephemeral {{bootstrap.servers}} (e.g. {{{}localhost:40111{}}}).
> # Write live bootstrap URL into {{atlas.kafka.bootstrap.servers}} so
> {{KafkaNotification}} clients connect.
> When {{{}atlas.notification.embedded=false{}}}: no-op.
> *Removed (old 2.8 embedded path):*
> * {{{}startZk(){}}}, {{{}ZooKeeperServer{}}}, {{ServerCnxnFactory}}
> * Direct {{kafka.server.KafkaServer}} construction tied to ZK
> * Port-kill retry loops for fixed ZK/Kafka ports
> h3.
> h3. Acceptance Criteria
> * {{kafka.version}} is 3.9.1 and Scala binary version is 2.13 across build.
> * {{EmbeddedKafkaServer}} starts KRaft broker without ZooKeeper when
> {{{}atlas.notification.embedded=true{}}}.
> * {{atlas.kafka.bootstrap.servers}} is rewritten at runtime to the live
> embedded broker address.
> * Unit tests pass:
> ** {{KafkaNotificationTest}} (notification module)
> ** {{NotificationHookConsumerKafkaTest}} (webapp, with {{{}-am{}}})
> * Manual: Atlas docker container with {{atlas.notification.embedded=true}} —
> KRaft log lines present, REST {{/api/atlas/admin/version}} returns 200.
> * Manual: Coexist stack with external {{atlas-kafka}} 3.9.1 — Hive hook →
> Atlas, PII classification → TagSync → Ranger, Hive/Kafka audit paths work.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
