[
https://issues.apache.org/jira/browse/RANGER-5658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramachandran Krishnan updated RANGER-5658:
------------------------------------------
Description:
h3. Summary
Ranger Tag Sync's Atlas Kafka source (ranger.tagsync.source.atlas) consumed
Atlas entity notifications via kafka-clients but still required the legacy
property atlas.kafka.zookeeper.connect at startup. That property was never used
by Atlas KafkaNotification for consumer creation. With Kafka 3.9.x and KRaft
brokers, clients use bootstrap.servers only.
This change removes the ZK requirement and fixes Tag Sync Atlas Kafka setup so
consumption works with both PLAINTEXT and Kerberos (SASL) Atlas Kafka,
including when Ranger runs with is_secure=true but the Atlas metadata bus is
PLAINTEXT.
h3. Problem
* Tag Sync validated atlas.kafka.zookeeper.connect even though Atlas 2.4+
KafkaNotification does not use it
* setup.py wrote Kerberos JAAS for the Atlas Kafka client whenever
is_secure=true, even when atlas.kafka.security.protocol=PLAINTEXT — breaking
PLAINTEXT/KRaft metadata buses
* AtlasTagSource loaded atlas-application.properties from the classpath only;
installer-generated conf under TAGSYNC_CONF_DIR was not reliably used without
-Datlas.conf
h3. Changes
* AtlasTagSource — remove ZK validation; load config via ApplicationProperties;
validate bootstrap.servers + consumer group only
* setup.py — atlas_kafka_uses_kerberos(): emit atlas.jaas.KafkaClient.* only
when security.protocol starts with SASL; always write
atlas.kafka.security.protocol
* ranger-tagsync-services.sh — pass -Datlas.conf=${TAGSYNC_CONF_DIR}
* install.properties / installprop2xml.properties — remove
TAG_SOURCE_ATLAS_KAFKA_ZOOKEEPER_CONNECT
* setup.py — use rangerTagsync_password for initial tagadmin jceks credential
(fixes 401 on tag upload in basic-auth deployments)
* dev-support/ranger-docker/scripts/tagsync/ranger-tagsync-install.properties —
default Atlas REST on, Kafka off; keep Kafka sample settings for
ranger-kafka.rangernw:9092 + SASL when enabled
* dev-support/README-TAGSYNC-ATLAS-KAFKA-CONFIG.md — operator guide (PLAINTEXT
vs SASL, startup, verification)
h3. Supported Atlas Kafka modes
|| Mode || TAG_SOURCE_ATLAS_KAFKA_SECURITY_PROTOCOL || Kerberos JAAS in
atlas-application.properties ||
| PLAINTEXT | PLAINTEXT | No |
| Kerberos | SASL_PLAINTEXT / SASL_SSL | Yes (when is_secure=true +
principal/keytab) |
Atlas Kafka security is independent of Tag Sync → Ranger Admin authentication
(basic auth or Ranger Kerberos for tag upload).
h3. Out of scope
* Tag Sync HA (ranger-tagsync.server.ha.zookeeper.*) — unchanged
* Atlas REST tag source behavior — unchanged except jceks password fix
h3. Upgrade note
Existing installs may still have atlas.kafka.zookeeper.connect in
conf/atlas-application.properties; harmless, can be removed on next config
refresh.
h3. Test plan
* mvn test -pl tagsync -Dtest=AtlasTagSourceConfigTest
* Fresh install: atlas-application.properties has no zookeeper.connect
* PLAINTEXT: is_secure=true + security.protocol=PLAINTEXT → no Kerberos JAAS;
consumer connects
* SASL: security.protocol=SASL_PLAINTEXT + principal/keytab → JAAS written;
consumer connects (e.g. ranger-kafka.rangernw:9092 in ranger-docker)
* Tag Sync starts with -Datlas.conf and consumes ATLAS_ENTITIES from Kafka
3.9.x (KRaft)
* Upgrade: legacy zookeeper.connect present → Tag Sync still starts
* Tag Sync HA smoke test (if HA enabled)
was:
Remove the legacy atlas.kafka.zookeeper.connect configuration requirement from
Ranger Tag Sync's Atlas Kafka source.
Background:
- Tag Sync consumes Atlas entity notifications via kafka-clients
(bootstrap.servers).
- atlas.kafka.zookeeper.connect was never used by the Atlas 2.4
KafkaNotification
consumer path; it was only required by Ranger startup validation.
- With Kafka 3.9.x / KRaft brokers, Kafka clients do not use ZooKeeper.
Changes:
- Drop TAGSYNC_ATLAS_ZOOKEEPER_ENDPOINT validation from AtlasTagSource
- Remove TAG_SOURCE_ATLAS_KAFKA_ZOOKEEPER_CONNECT from install.properties,
installprop2xml.properties, setup.py, and docker sample config
Out of scope:
- Tag Sync HA (ranger-tagsync.server.ha.zookeeper.*) — unchanged; still
requires ZooKeeper when HA is enabled
Upgrade note:
Existing deployments may still have atlas.kafka.zookeeper.connect in
conf/atlas-application.properties; it is harmless and can be removed manually.
New installs no longer generate or require it.
> Tag Sync: remove atlas.kafka.zookeeper.connect and fix Atlas Kafka consumer
> for Kafka 3.9
> -----------------------------------------------------------------------------------------
>
> Key: RANGER-5658
> URL: https://issues.apache.org/jira/browse/RANGER-5658
> Project: Ranger
> Issue Type: Task
> Components: Ranger
> Affects Versions: 3.0.0
> Reporter: Ramachandran Krishnan
> Assignee: Ramachandran Krishnan
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> h3. Summary
> Ranger Tag Sync's Atlas Kafka source (ranger.tagsync.source.atlas) consumed
> Atlas entity notifications via kafka-clients but still required the legacy
> property atlas.kafka.zookeeper.connect at startup. That property was never
> used by Atlas KafkaNotification for consumer creation. With Kafka 3.9.x and
> KRaft brokers, clients use bootstrap.servers only.
> This change removes the ZK requirement and fixes Tag Sync Atlas Kafka setup
> so consumption works with both PLAINTEXT and Kerberos (SASL) Atlas Kafka,
> including when Ranger runs with is_secure=true but the Atlas metadata bus is
> PLAINTEXT.
> h3. Problem
> * Tag Sync validated atlas.kafka.zookeeper.connect even though Atlas 2.4+
> KafkaNotification does not use it
> * setup.py wrote Kerberos JAAS for the Atlas Kafka client whenever
> is_secure=true, even when atlas.kafka.security.protocol=PLAINTEXT — breaking
> PLAINTEXT/KRaft metadata buses
> * AtlasTagSource loaded atlas-application.properties from the classpath only;
> installer-generated conf under TAGSYNC_CONF_DIR was not reliably used without
> -Datlas.conf
> h3. Changes
> * AtlasTagSource — remove ZK validation; load config via
> ApplicationProperties; validate bootstrap.servers + consumer group only
> * setup.py — atlas_kafka_uses_kerberos(): emit atlas.jaas.KafkaClient.* only
> when security.protocol starts with SASL; always write
> atlas.kafka.security.protocol
> * ranger-tagsync-services.sh — pass -Datlas.conf=${TAGSYNC_CONF_DIR}
> * install.properties / installprop2xml.properties — remove
> TAG_SOURCE_ATLAS_KAFKA_ZOOKEEPER_CONNECT
> * setup.py — use rangerTagsync_password for initial tagadmin jceks credential
> (fixes 401 on tag upload in basic-auth deployments)
> * dev-support/ranger-docker/scripts/tagsync/ranger-tagsync-install.properties
> — default Atlas REST on, Kafka off; keep Kafka sample settings for
> ranger-kafka.rangernw:9092 + SASL when enabled
> * dev-support/README-TAGSYNC-ATLAS-KAFKA-CONFIG.md — operator guide
> (PLAINTEXT vs SASL, startup, verification)
> h3. Supported Atlas Kafka modes
> || Mode || TAG_SOURCE_ATLAS_KAFKA_SECURITY_PROTOCOL || Kerberos JAAS in
> atlas-application.properties ||
> | PLAINTEXT | PLAINTEXT | No |
> | Kerberos | SASL_PLAINTEXT / SASL_SSL | Yes (when is_secure=true +
> principal/keytab) |
> Atlas Kafka security is independent of Tag Sync → Ranger Admin authentication
> (basic auth or Ranger Kerberos for tag upload).
> h3. Out of scope
> * Tag Sync HA (ranger-tagsync.server.ha.zookeeper.*) — unchanged
> * Atlas REST tag source behavior — unchanged except jceks password fix
> h3. Upgrade note
> Existing installs may still have atlas.kafka.zookeeper.connect in
> conf/atlas-application.properties; harmless, can be removed on next config
> refresh.
> h3. Test plan
> * mvn test -pl tagsync -Dtest=AtlasTagSourceConfigTest
> * Fresh install: atlas-application.properties has no zookeeper.connect
> * PLAINTEXT: is_secure=true + security.protocol=PLAINTEXT → no Kerberos JAAS;
> consumer connects
> * SASL: security.protocol=SASL_PLAINTEXT + principal/keytab → JAAS written;
> consumer connects (e.g. ranger-kafka.rangernw:9092 in ranger-docker)
> * Tag Sync starts with -Datlas.conf and consumes ATLAS_ENTITIES from Kafka
> 3.9.x (KRaft)
> * Upgrade: legacy zookeeper.connect present → Tag Sync still starts
> * Tag Sync HA smoke test (if HA enabled)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
