This is an automated email from the ASF dual-hosted git repository. ramackri pushed a commit to branch tagsync-remove-atlas-kafka-zookeeper-connect in repository https://gitbox.apache.org/repos/asf/ranger.git
commit 1f8dd4e5da53e7ada8aa5ed9d6235933f04a4b56 Author: ramk <[email protected]> AuthorDate: Mon Jun 29 19:15:45 2026 +0530 Remove obsolete atlas.kafka.zookeeper.connect requirement from Tag Sync. Tag Sync consumes Atlas notifications via kafka-clients (bootstrap.servers only). The legacy zookeeper.connect property was never used by Atlas KafkaNotification but was still required at startup and in installer templates. Co-authored-by: Cursor <[email protected]> --- dev-support/README-TAGSYNC-ATLAS-KAFKA-CONFIG.md | 136 +++++++++++++++++++++ tagsync/conf/templates/installprop2xml.properties | 1 - tagsync/scripts/install.properties | 3 +- tagsync/scripts/setup.py | 3 - .../tagsync/source/atlas/AtlasTagSource.java | 9 +- 5 files changed, 139 insertions(+), 13 deletions(-) diff --git a/dev-support/README-TAGSYNC-ATLAS-KAFKA-CONFIG.md b/dev-support/README-TAGSYNC-ATLAS-KAFKA-CONFIG.md new file mode 100644 index 000000000..463b20571 --- /dev/null +++ b/dev-support/README-TAGSYNC-ATLAS-KAFKA-CONFIG.md @@ -0,0 +1,136 @@ +# Tag Sync — Atlas Kafka configuration (Kafka 3.9+ / KRaft) + +Reference for operators and developers configuring Ranger Tag Sync with the **Atlas Kafka** tag source (`ranger.tagsync.source.atlas`). + +Related code change: remove obsolete `atlas.kafka.zookeeper.connect` requirement from Tag Sync startup and installer templates. + +--- + +## Summary + +Tag Sync consumes Atlas entity notifications using the standard **`kafka-clients`** library (`bootstrap.servers`, consumer group, SASL/SSL as needed). It does **not** connect to Kafka via ZooKeeper. + +The property `atlas.kafka.zookeeper.connect` was legacy configuration carried forward from older Atlas/Kafka deployments. Ranger previously **required** it at startup even though: + +- Atlas 2.4 `KafkaNotification` never uses it for consumer/producer creation +- Kafka 3.x clients (including 3.9.1) talk to brokers over `bootstrap.servers` only +- KRaft-mode Kafka brokers do not expose ZooKeeper to clients + +That requirement has been **removed** from Tag Sync. + +--- + +## Required configuration (Atlas Kafka source) + +Set these in `install.properties` (installer writes `conf/atlas-application.properties`): + +| Install property | Atlas property | Required | +|------------------|----------------|----------| +| `TAG_SOURCE_ATLAS_KAFKA_BOOTSTRAP_SERVERS` | `atlas.kafka.bootstrap.servers` | Yes | +| `TAG_SOURCE_ATLAS_KAFKA_ENTITIES_GROUP_ID` | `atlas.kafka.entities.group.id` | Yes | +| `TAG_SOURCE_ATLAS_KAFKA_SECURITY_PROTOCOL` | `atlas.kafka.security.protocol` | If secured | +| `TAG_SOURCE_ATLAS_KAFKA_SERVICE_NAME` | `atlas.kafka.sasl.kerberos.service.name` | If Kerberos | +| Kerberos principal/keytab | `atlas.jaas.KafkaClient.*` | If Kerberos | + +Example `conf/atlas-application.properties`: + +```properties +atlas.kafka.bootstrap.servers=kafka-broker1:9092,kafka-broker2:9092 +atlas.kafka.entities.group.id=ranger_entities_consumer +atlas.kafka.security.protocol=SASL_PLAINTEXT +atlas.kafka.sasl.kerberos.service.name=kafka +``` + +--- + +## Removed configuration + +| Property | Status | +|----------|--------| +| `atlas.kafka.zookeeper.connect` | **No longer required** — not generated by `setup.py`, not validated by `AtlasTagSource` | + +### Upgrading existing deployments + +If `conf/atlas-application.properties` still contains `atlas.kafka.zookeeper.connect` from an older install, it is harmless (Kafka client ignores unknown keys). You may delete the line during the next config refresh. + +--- + +## ZooKeeper still used elsewhere (do not confuse) + +Tag Sync has **two unrelated** ZooKeeper touch points: + +| Config | Purpose | Needed for KRaft Kafka? | +|--------|---------|-------------------------| +| `atlas.kafka.zookeeper.connect` | Legacy Atlas/Kafka config | **No** (removed) | +| `ranger-tagsync.server.ha.zookeeper.connect` | Tag Sync active/passive leader election | **Only when HA enabled** | + +When `ranger-tagsync.server.ha.enabled=true`, Tag Sync still needs a real ZooKeeper ensemble for HA — independent of how Kafka brokers run (ZooKeeper vs KRaft). + +Default in `ranger-tagsync-site.xml`: `ranger-tagsync.server.ha.enabled=false`. + +--- + +## Alternative: Atlas REST source + +If Tag Sync reads tags over Atlas REST instead of Kafka, none of the Atlas Kafka properties apply: + +```properties +TAG_SOURCE_ATLASREST_ENABLED=true +TAG_SOURCE_ATLASREST_ENDPOINT=http://atlas-host:21000 +``` + +Enable via `ranger.tagsync.source.atlasrest` in `ranger-tagsync-site.xml`. + +--- + +## Code / config files changed + +| File | Change | +|------|--------| +| `tagsync/src/main/java/org/apache/ranger/tagsync/source/atlas/AtlasTagSource.java` | Drop `TAGSYNC_ATLAS_ZOOKEEPER_ENDPOINT` and startup validation | +| `tagsync/scripts/install.properties` | Remove `TAG_SOURCE_ATLAS_KAFKA_ZOOKEEPER_CONNECT` | +| `tagsync/conf/templates/installprop2xml.properties` | Remove install → Atlas mapping | +| `tagsync/scripts/setup.py` | Stop writing `atlas.kafka.zookeeper.connect` | +| `dev-support/ranger-docker/scripts/tagsync/ranger-tagsync-install.properties` | Remove sample ZK property | + +**Unchanged:** Tag Sync HA properties, `tagsync/pom.xml` ZooKeeper dependency (used by `ranger-common-ha` when HA is on). + +--- + +## Jira ticket template + +**Title:** `RANGER-XXXX: Remove obsolete atlas.kafka.zookeeper.connect requirement from Tag Sync` + +**Description:** + +``` +Remove the legacy atlas.kafka.zookeeper.connect configuration requirement from +Ranger Tag Sync's Atlas Kafka source. + +Tag Sync consumes Atlas notifications via kafka-clients (bootstrap.servers). +The zookeeper.connect property was never used by Atlas 2.4 KafkaNotification +for consumer creation; Ranger only validated that it was set. With Kafka 3.9.x +and KRaft brokers, clients do not use ZooKeeper. + +Changes: +- Drop validation from AtlasTagSource +- Remove property from install.properties, installprop2xml.properties, setup.py, + and docker sample config + +Out of scope: Tag Sync HA (ranger-tagsync.server.ha.zookeeper.*) unchanged. +``` + +**Test plan:** + +- [ ] Fresh install: `atlas-application.properties` has no `zookeeper.connect` +- [ ] Tag Sync starts and consumes from Kafka 3.9.x (KRaft) +- [ ] Upgrade: existing installs with old property still start +- [ ] Tag Sync HA smoke test (if HA enabled) + +--- + +## References + +- Tag Sync Atlas source: `tagsync/src/main/java/org/apache/ranger/tagsync/source/atlas/AtlasTagSource.java` +- Atlas notification (external): `org.apache.atlas.kafka.KafkaNotification` in `atlas-notification` 2.4.0 +- Root Kafka client version: `${kafka.version}` in root `pom.xml` (3.9.1) diff --git a/tagsync/conf/templates/installprop2xml.properties b/tagsync/conf/templates/installprop2xml.properties index d6f289a55..aea6537ee 100644 --- a/tagsync/conf/templates/installprop2xml.properties +++ b/tagsync/conf/templates/installprop2xml.properties @@ -24,7 +24,6 @@ TAG_SOURCE_ATLAS_ENABLED = ranger.tagsync.source.atlas TAG_SOURCE_ATLAS_KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR = atlas.kafka.offsets.topic.replication.factor TAG_SOURCE_ATLAS_KAFKA_BOOTSTRAP_SERVERS = atlas.kafka.bootstrap.servers -TAG_SOURCE_ATLAS_KAFKA_ZOOKEEPER_CONNECT = atlas.kafka.zookeeper.connect TAG_SOURCE_ATLAS_KAFKA_ENTITIES_GROUP_ID = atlas.kafka.entities.group.id TAG_SOURCE_ATLAS_KAFKA_SERVICE_NAME = atlas.kafka.sasl.kerberos.service.name diff --git a/tagsync/scripts/install.properties b/tagsync/scripts/install.properties index ae2e555eb..278df0ff8 100644 --- a/tagsync/scripts/install.properties +++ b/tagsync/scripts/install.properties @@ -27,9 +27,8 @@ TAG_DEST_RANGER_SSL_CONFIG_FILENAME = TAG_SOURCE_ATLAS_ENABLED = true -# Endpoint specifications needed by Atlas +# Kafka settings for Atlas notification consumer (bootstrap.servers only; no ZooKeeper) TAG_SOURCE_ATLAS_KAFKA_BOOTSTRAP_SERVERS = localhost:6667 -TAG_SOURCE_ATLAS_KAFKA_ZOOKEEPER_CONNECT = localhost:2181 TAG_SOURCE_ATLAS_KAFKA_ENTITIES_GROUP_ID = ranger_entities_consumer TAG_SOURCE_ATLAS_KAFKA_SERVICE_NAME = kafka diff --git a/tagsync/scripts/setup.py b/tagsync/scripts/setup.py index 1fbf5f181..df98f87c0 100755 --- a/tagsync/scripts/setup.py +++ b/tagsync/scripts/setup.py @@ -68,7 +68,6 @@ initPrefixList = ['S99', 'K00'] TAGSYNC_ATLAS_KAFKA_ENDPOINTS_KEY = 'TAG_SOURCE_ATLAS_KAFKA_BOOTSTRAP_SERVERS' -TAGSYNC_ATLAS_ZOOKEEPER_ENDPOINT_KEY = 'TAG_SOURCE_ATLAS_KAFKA_ZOOKEEPER_CONNECT' TAGSYNC_ATLAS_KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR_KEY = 'TAG_SOURCE_ATLAS_KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR' TAGSYNC_ATLAS_CONSUMER_GROUP_KEY = 'TAG_SOURCE_ATLAS_KAFKA_ENTITIES_GROUP_ID' @@ -231,8 +230,6 @@ def convertInstallPropsToXML(props): newKey = directKeyMap[k] if (k == TAGSYNC_ATLAS_KAFKA_ENDPOINTS_KEY): atlasOutFile.write(newKey + "=" + v + "\n") - elif (k == TAGSYNC_ATLAS_ZOOKEEPER_ENDPOINT_KEY): - atlasOutFile.write(newKey + "=" + v + "\n") elif (k == TAGSYNC_ATLAS_KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR_KEY): atlasOutFile.write(newKey + "=" + v + "\n") elif (k == TAGSYNC_ATLAS_CONSUMER_GROUP_KEY): diff --git a/tagsync/src/main/java/org/apache/ranger/tagsync/source/atlas/AtlasTagSource.java b/tagsync/src/main/java/org/apache/ranger/tagsync/source/atlas/AtlasTagSource.java index eeeccaf4a..382d40418 100644 --- a/tagsync/src/main/java/org/apache/ranger/tagsync/source/atlas/AtlasTagSource.java +++ b/tagsync/src/main/java/org/apache/ranger/tagsync/source/atlas/AtlasTagSource.java @@ -48,9 +48,8 @@ public class AtlasTagSource extends AbstractTagSource { private static final Logger LOG = LoggerFactory.getLogger(AtlasTagSource.class); public static final String TAGSYNC_ATLAS_PROPERTIES_FILE_NAME = "atlas-application.properties"; - public static final String TAGSYNC_ATLAS_KAFKA_ENDPOINTS = "atlas.kafka.bootstrap.servers"; - public static final String TAGSYNC_ATLAS_ZOOKEEPER_ENDPOINT = "atlas.kafka.zookeeper.connect"; - public static final String TAGSYNC_ATLAS_CONSUMER_GROUP = "atlas.kafka.entities.group.id"; + public static final String TAGSYNC_ATLAS_KAFKA_ENDPOINTS = "atlas.kafka.bootstrap.servers"; + public static final String TAGSYNC_ATLAS_CONSUMER_GROUP = "atlas.kafka.entities.group.id"; public static final int MAX_WAIT_TIME_IN_MILLIS = 1000; private int maxBatchSize; @@ -92,10 +91,6 @@ public boolean initialize(Properties properties) { ret = false; LOG.error("Value of property '{}' is not specified!", TAGSYNC_ATLAS_KAFKA_ENDPOINTS); } - if (StringUtils.isBlank(atlasProperties.getProperty(TAGSYNC_ATLAS_ZOOKEEPER_ENDPOINT))) { - ret = false; - LOG.error("Value of property '{}' is not specified!", TAGSYNC_ATLAS_ZOOKEEPER_ENDPOINT); - } if (StringUtils.isBlank(atlasProperties.getProperty(TAGSYNC_ATLAS_CONSUMER_GROUP))) { ret = false; LOG.error("Value of property '{}' is not specified!", TAGSYNC_ATLAS_CONSUMER_GROUP);
