[
https://issues.apache.org/jira/browse/ATLAS-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18094488#comment-18094488
]
ASF subversion and git services commented on ATLAS-5336:
--------------------------------------------------------
Commit 4bec890393d06abea98ee135472b4df9f25d0d87 in atlas's branch
refs/heads/master from Ramachandran Krishnan
[ https://gitbox.apache.org/repos/asf?p=atlas.git;h=4bec89039 ]
ATLAS-5336: Upgrade Kafka to 3.9.1; embedded notification broker uses KRaft (no
ZooKeeper) (#689)
* ATLAS-5336: Upgrade Kafka to 3.9.1; embedded notification broker uses KRaft
Bump kafka-clients and embedded test broker from 2.8.2 to 3.9.1 (Scala 2.13),
replace ZooKeeper-based EmbeddedKafkaServer with KafkaClusterTestKit, and
update docker dev Kafka image packaging to kafka_2.13-3.9.1.
* Document jackson-jaxrs 2.12.7 pin for Jersey 1.19 compatibility.
Point readers to ATLAS-5190 for full Jackson unification via Jersey 2.x.
* Remove unused junit-jupiter-api from atlas-notification compile deps.
It leaked to falcon-bridge via atlas-notification and caused Failsafe to
select JUnit Platform, which then failed on transitive junit 3.8.1 from
falcon-common during FalconHookIT discovery.
* Scope Kafka test and unit test deps in notification to test.
Kafka 3.9 TestUtils needs junit-jupiter-api at test runtime; keep it off the
compile classpath so falcon-bridge Failsafe stays on TestNG. Align mockito and
testng with other Atlas modules (test scope).
* Add test-scoped junit-jupiter-api to webapp for Kafka embedded broker tests.
NotificationHookConsumerKafkaTest starts EmbeddedKafkaServer, which uses Kafka
3.9 TestUtils requiring JUnit Jupiter on the test classpath. Test-scoped deps
from atlas-notification do not propagate to webapp.
* Fix flaky ImportTaskListenerImplTest async consumer assertions on CI.
Shut down the import executor in @AfterMethod and wait with CountDownLatch
instead of Thread.sleep so Mockito state does not leak between tests.
* Harden BasicSearchIT and AdminExportImportTestIT against CI flakes.
Scope hive searches to largedb. entities from the imported test dataset
instead of the shared @cl1 cluster suffix, and tolerate Stream Closed
during AdminExportImportTestIT teardown when the embedded server stops.
* Fix IT scoping and Failsafe/TestNG classpath for Kafka 3.9.
Scope BasicSearchIT to default. entities from hive-db-50-tables.zip instead
of the shared @cl1 suffix, skip scoping for fulltext and paginated searches,
keep junit-jupiter on the notification compile classpath for embedded KRaft
runtime, and pin Failsafe to TestNG after junit landed on the webapp test path.
* Bind embedded KRaft broker to configured bootstrap.servers for ITs.
Derive EXTERNAL and CONTROLLER listeners from atlas.kafka.bootstrap.servers so
Jetty and Failsafe JVMs share the same broker port, fixing
NotificationHookConsumerIT topic timeouts on CI.
* Pre-serialize classification request bodies to avoid Codehaus Jackson
conflicts.
HiveHookIT fails when Jersey serializes AtlasClassification because
isPropagate()
and getPropagate() both map to "propagate"; AtlasType.toJson() matches the
entity fix.
* Pin Jackson 2.16 on sqoop-bridge IT classpath for AtlasClient v1.
SqoopHookIT fails with NoClassDefFoundError for StreamConstraintsException
when jackson-databind 2.16 is paired with an older jackson-core from
Hadoop/Hive.
* Pin Jackson 2.16 databind on storm-bridge IT classpath for AtlasClient v1.
StormAtlasHookIT fails with NoSuchFieldError for
READ_UNKNOWN_ENUM_VALUES_USING_DEFAULT_VALUE
when jackson-databind 2.16 is mixed with an older databind from storm-core on
the test classpath.
Co-authored-by: Cursor <[email protected]>
* Skip trino-extractor Failsafe ITs on default mvn verify.
Pinning Failsafe to TestNG for the Kafka upgrade requires testng on the
module test classpath; the extractor only has a stub IT and no test deps yet.
---------
Co-authored-by: ramk <[email protected]>
Co-authored-by: Cursor <[email protected]>
> Upgrade Kafka to 3.9.1; embedded notification broker uses KRaft (no
> ZooKeeper))
> --------------------------------------------------------------------------------
>
> Key: ATLAS-5336
> URL: https://issues.apache.org/jira/browse/ATLAS-5336
> Project: Atlas
> Issue Type: Task
> Components: atlas-core
> Reporter: Ramachandran Krishnan
> Assignee: Ramachandran Krishnan
> Priority: Major
> Fix For: 3.0.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> h3. Summary
> Upgrade Apache Atlas notification Kafka from *2.8.2* (Scala 2.12) to *3.9.1*
> (Scala 2.13), and replace the ZooKeeper-based in-process embedded broker with
> a *KRaft* broker via {{{}KafkaClusterTestKit{}}}.
> h3. Background / Motivation
> * Kafka 2.8.x clients are only partially compatible with modern 3.9.x / 4.x
> brokers.
> * The legacy {{EmbeddedKafkaServer}} started an in-process
> {{ZooKeeperServer}} plus {{KafkaServer}} tied to ZK — unnecessary complexity
> and a maintenance burden as Kafka moves to KRaft-only.
> * Atlas remains on Spring 5.3; Spring Kafka 3.x /
> {{EmbeddedKafkaKraftBroker}} requires Spring 6. Using Kafka's
> {{KafkaClusterTestKit}} provides the same KRaft engine without a Spring
> upgrade.
> * Aligns Atlas with Kafka 3.9.x before external cluster KRaft migration and
> a future Kafka 4.x path.
> h3. Proposed Changes
> ||Area||Before||After||
> |{{kafka.version}} (root {{{}pom.xml{}}})|2.8.2|*3.9.1*|
> |{{kafka.scala.binary.version}}|2.12|*2.13*|
> |{{EmbeddedKafkaServer}}|In-process ZK +
> {{KafkaServer}}|{{KafkaClusterTestKit}} (combined KRaft controller+broker)|
> |Embedded ZK for notification|Required at Atlas startup when
> embedded=true|*Removed* — no {{ZooKeeperServer}} / {{zookeeper.connect}} in
> embedded path|
> |Docker {{atlas-kafka}} image|{{kafka_2.12-*}} tarball|{{kafka_2.13-3.9.1}}
> ({{{}.env{}}}, {{{}Dockerfile.atlas-kafka{}}}, {{{}download-archives.sh{}}})|
> |Default distro config|ZK-oriented embedded
> settings|{{{}atlas.notification.embedded=true{}}}; {{bootstrap.servers}}
> placeholder updated *at runtime* by {{EmbeddedKafkaServer}}|
> h3. Core code — {{EmbeddedKafkaServer}}
> *Path:*
> {{notification/src/main/java/org/apache/atlas/kafka/EmbeddedKafkaServer.java}}
> When {{{}atlas.notification.embedded=true{}}}:
> # Start KRaft broker via {{KafkaClusterTestKit}} (1 combined
> controller+broker node).
> # Assign ephemeral {{bootstrap.servers}} (e.g. {{{}localhost:40111{}}}).
> # Write live bootstrap URL into {{atlas.kafka.bootstrap.servers}} so
> {{KafkaNotification}} clients connect.
> When {{{}atlas.notification.embedded=false{}}}: no-op.
> *Removed (old 2.8 embedded path):*
> * {{{}startZk(){}}}, {{{}ZooKeeperServer{}}}, {{ServerCnxnFactory}}
> * Direct {{kafka.server.KafkaServer}} construction tied to ZK
> * Port-kill retry loops for fixed ZK/Kafka ports
> h3.
> h3. Acceptance Criteria
> * {{kafka.version}} is 3.9.1 and Scala binary version is 2.13 across build.
> * {{EmbeddedKafkaServer}} starts KRaft broker without ZooKeeper when
> {{{}atlas.notification.embedded=true{}}}.
> * {{atlas.kafka.bootstrap.servers}} is rewritten at runtime to the live
> embedded broker address.
> * Unit tests pass:
> ** {{KafkaNotificationTest}} (notification module)
> ** {{NotificationHookConsumerKafkaTest}} (webapp, with {{{}-am{}}})
> * Manual: Atlas docker container with {{atlas.notification.embedded=true}} —
> KRaft log lines present, REST {{/api/atlas/admin/version}} returns 200.
> * Manual: Coexist stack with external {{atlas-kafka}} 3.9.1 — Hive hook →
> Atlas, PII classification → TagSync → Ranger, Hive/Kafka audit paths work.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
