This is an automated email from the ASF dual-hosted git repository. ramackri pushed a commit to branch RANGER-5645-patch in repository https://gitbox.apache.org/repos/asf/ranger.git
commit d05e556380eca8a56bad33de97344e45c533135b Author: ramk <[email protected]> AuthorDate: Sun Jun 14 22:42:50 2026 +0530 RANGER-5645: Add audit-ingestor service-user allowlist for Docker plugins Ship per-repo allowed.users and auth_to_local rules so plugins using the audit-server destination are authorized after Kerberos SPNEGO (fixes HTTP 403). Align create-ranger-services.py with policy.download.auth.users for Ozone, Atlas, Kudu, and NiFi. Add troubleshooting README for ingestor 403 errors. Co-authored-by: Cursor <[email protected]> --- .../README-AUDIT-INGESTOR-SERVICE-ALLOWLIST.md | 197 +++++++++++++++++++++ .../resources/conf/ranger-audit-ingestor-site.xml | 107 +++++++++-- audit-server/scripts/README.md | 2 + .../scripts/admin/create-ranger-services.py | 33 +++- 4 files changed, 320 insertions(+), 19 deletions(-) diff --git a/audit-server/README-AUDIT-INGESTOR-SERVICE-ALLOWLIST.md b/audit-server/README-AUDIT-INGESTOR-SERVICE-ALLOWLIST.md new file mode 100644 index 000000000..7093b044a --- /dev/null +++ b/audit-server/README-AUDIT-INGESTOR-SERVICE-ALLOWLIST.md @@ -0,0 +1,197 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> +# Audit ingestor: plugin 403 (service allowlist) + +When a Ranger plugin uses the **audit-server** destination (`xasecure.audit.destination.auditserver`), it POSTs audit batches to audit-ingestor at `/audit/access?serviceName=<repo>`. Failures often look like authentication problems but are usually **authorization (403)** — Kerberos/SPNEGO succeeded; the mapped service user is not on the ingestor allowlist for that Policy Manager repo. + +Related: [dev-support/ranger-docker/README.md](../dev-support/ranger-docker/README.md), shipped defaults in `audit-ingestor/src/main/resources/conf/ranger-audit-ingestor-site.xml`. + +--- + +## Quick read: 401 vs 403 + +| HTTP | Meaning | Typical cause | +|------|---------|---------------| +| **401** | Authentication failed | Missing/invalid Kerberos ticket, wrong SPNEGO, clock skew, wrong ingestor URL | +| **403** | Authentication OK, **authorization failed** | Caller authenticated but short username not in `ranger.audit.ingestor.service.<repo>.allowed.users` | + +If ingestor logs show `Unauthorized user: user=<name> ... service=<repo>`, Kerberos worked — fix the allowlist or `auth_to_local`, not the keytab. + +--- + +## What's happening + +```text +Plugin (e.g. KMS) audit-ingestor + | | + | POST /audit/access?serviceName=dev_kms + | Authorization: Negotiate (SPNEGO) | + |------------------------------------->| + | | 1. Authenticate Kerberos principal + | | e.g. rangerkms/host@REALM + | | 2. auth_to_local → short name "rangerkms" + | | 3. isAllowedServiceUser(dev_kms, rangerkms) + | | checks allowed.users for dev_kms + | | 4. Not listed → 403 Forbidden + |<-------------------------------------| + | {"message":"User is not authorized to send audit events",...} +``` + +Implementation: `AuditREST.java` → `isAllowedServiceUser()`. Config keys: + +```properties +ranger.audit.ingestor.service.<repoName>.allowed.users=<comma-separated short names> +ranger.audit.ingestor.auth.to.local=<Kerberos RULE lines> +``` + +`<repoName>` must match the Policy Manager **service name** (e.g. `dev_kms`, `dev_hdfs`), not the plugin type alone. + +--- + +## Example: Ranger KMS (403) + +**Plugin log:** + +```text +ERROR o.a.r.a.d.RangerAuditServerDestination - Failed to send audit batch. +HTTP status: 403, Response: {"message":"User is not authorized to send audit events",...} +ERROR o.a.r.a.d.RangerAuditServerDestination - Failed to send batch of 1 events +``` + +**audit-ingestor log:** + +```text +ERROR - Unauthorized user: user=rangerkms is authorized report audit logs for service=dev_kms. +Rejecting audit request. +``` + +**Interpretation:** SPNEGO to ingestor works. `rangerkms` is not allowed for repo `dev_kms` (missing property, wrong short name, or stale runtime config). + +**Fix:** + +```xml +<property> + <name>ranger.audit.ingestor.service.dev_kms.allowed.users</name> + <value>rangerkms</value> +</property> +``` + +Ensure `auth_to_local` includes `rangerkms/*` → `rangerkms`, restart ingestor, confirm the **running** pod/container loads the updated site XML. + +--- + +## Root causes (side-by-side) + +| # | Symptom | Likely cause | Fix | +|---|---------|--------------|-----| +| 1 | 403, ingestor log names user + repo | No `allowed.users` for that repo | Add `ranger.audit.ingestor.service.<repo>.allowed.users` | +| 2 | 403, user name looks wrong (e.g. `kms` vs `rangerkms`) | Allowlist short name ≠ `auth_to_local` output | Align with `policy.download.auth.users` in Ranger Admin | +| 3 | 403 for Ozone OM/SCM/DN only | Allowlist has `ozone` but principal maps to `om`/`scm`/`dn` | Use `ozone,om,scm,dn` for `dev_ozone` | +| 4 | 403 after editing source XML only | Ingestor still using old classpath copy | Patch `conf/` **and** `WEB-INF/classes/conf/`, or rebuild/redeploy image | +| 5 | 403, repo name unexpected | Plugin `serviceName` ≠ Policy Manager service name | Match repo name in plugin audit config to Ranger Admin service | +| 6 | 401 instead of 403 | Kerberos/SPNEGO failure | Keytab, principal, `_HOST`, ingestor URL/FQDN, KDC — not allowlist | + +--- + +## How to fix + +### 1. Identify repo and short username + +From ingestor error line: `user=<shortname> ... service=<repo>`. + +Or from Ranger Admin → service → **Configs** → `policy.download.auth.users` (ingestor allowlist should match). + +### 2. Add or update allowlist + +In `ranger-audit-ingestor-site.xml` (or deployment overlay): + +```xml +<property> + <name>ranger.audit.ingestor.service.dev_<service>.allowed.users</name> + <value><short names, comma-separated></value> +</property> +``` + +### 3. Verify `auth_to_local` + +Property: `ranger.audit.ingestor.auth.to_local`. Service-specific `RULE:` lines must run **before** the generic `user@REALM` fallback so `service/host@REALM` maps to the same short names as the allowlist. + +### 4. Restart audit-ingestor + +Allowlist is loaded at startup. After change, restart the ingestor pod/container. + +### 5. Docker dev stack extras + +| Step | Action | +|------|--------| +| Policy Manager repos | `docker exec ranger python3 /home/ranger/scripts/create-ranger-services.py` (idempotent) | +| New plugin / new repo | Add allowlist entry + `auth_to_local` rule + create service in Admin | + +--- + +## Shipped allowlist reference (Docker dev repos) + +Align each row with `policy.download.auth.users` for that service in Policy Manager. + +| Policy Manager repo | `allowed.users` (short names) | +|---------------------|-------------------------------| +| `dev_hdfs` | `hdfs` | +| `dev_yarn` | `yarn` | +| `dev_hive` | `hive` | +| `dev_hbase` | `hbase` | +| `dev_kafka` | `kafka` | +| `dev_knox` | `knox` | +| `dev_kms` | `rangerkms` | +| `dev_trino` | `trino` | +| `dev_ozone` | `ozone,om,scm,dn` | +| `dev_solr` | `solr` | +| `dev_atlas` | `atlas` | +| `dev_kudu` | `kudu` | +| `dev_nifi` | `nifi` | + +For a **new** repo, add both the property and a matching `auth_to_local` rule; there is no wildcard allowlist. + +--- + +## Verification + +1. Trigger an access event on the plugin (e.g. KMS operation). +2. Plugin log: no `HTTP status: 403` from `RangerAuditServerDestination`. +3. Ingestor log: no `Unauthorized user` for that repo; debug may show `isAllowedServiceUser(...): ret=true`. +4. Downstream (optional): audit count increases in Kafka/Solr/Admin UI. + +--- + +## Product vs Docker scope + +| Scope | Problem | Owner / fix | +|-------|---------|-------------| +| **Product** | Fresh install missing allowlist for enabled plugins | Ship complete defaults in `ranger-audit-ingestor-site.xml`; document in install guide | +| **Docker / ops** | Stack built before allowlist was added; stale WEB-INF copy | Patch/redeploy ingestor; run `create-ranger-services.py` | + +Same 403 mechanism in both cases; difference is whether defaults are shipped vs patched at deploy time. + +--- + +## Code references + +| Piece | Location | +|-------|----------| +| Authorization check | `audit-ingestor/.../AuditREST.java` — `isAllowedServiceUser()` | +| Property prefix | `ranger.audit.ingestor.service.` + `<repo>` + `.allowed.users` | +| Default config | `audit-ingestor/src/main/resources/conf/ranger-audit-ingestor-site.xml` | +| Plugin client | `agents-audit/.../RangerAuditServerDestination.java` | diff --git a/audit-server/audit-ingestor/src/main/resources/conf/ranger-audit-ingestor-site.xml b/audit-server/audit-ingestor/src/main/resources/conf/ranger-audit-ingestor-site.xml index 976c2a0cb..e75f70904 100644 --- a/audit-server/audit-ingestor/src/main/resources/conf/ranger-audit-ingestor-site.xml +++ b/audit-server/audit-ingestor/src/main/resources/conf/ranger-audit-ingestor-site.xml @@ -185,19 +185,79 @@ <property> <name>ranger.audit.ingestor.service.dev_hdfs.allowed.users</name> <value>hdfs</value> - <description>Comma-separated list of allowed users that can send audits for dev_hdfs service</description> + <description>Allowed users for dev_hdfs (Policy Manager service name; from policy.download.auth.users)</description> + </property> + + <property> + <name>ranger.audit.ingestor.service.dev_yarn.allowed.users</name> + <value>yarn</value> + <description>Allowed users for dev_yarn (YARN RM/NM)</description> </property> <property> <name>ranger.audit.ingestor.service.dev_hive.allowed.users</name> <value>hive</value> - <description>Comma-separated list of allowed users that can send audits for dev_hive service.</description> + <description>Allowed users for dev_hive (HiveServer2 + Metastore)</description> </property> <property> <name>ranger.audit.ingestor.service.dev_hbase.allowed.users</name> <value>hbase</value> - <description>Comma-separated list of allowed users that can send audits for dev_hbase service.</description> + <description>Allowed users for dev_hbase (HBase Master + RegionServer)</description> + </property> + + <property> + <name>ranger.audit.ingestor.service.dev_kafka.allowed.users</name> + <value>kafka</value> + <description>Allowed users for dev_kafka</description> + </property> + + <property> + <name>ranger.audit.ingestor.service.dev_knox.allowed.users</name> + <value>knox</value> + <description>Allowed users for dev_knox</description> + </property> + + <property> + <name>ranger.audit.ingestor.service.dev_kms.allowed.users</name> + <value>rangerkms</value> + <description>Allowed users for dev_kms</description> + </property> + + <property> + <name>ranger.audit.ingestor.service.dev_trino.allowed.users</name> + <value>trino</value> + <description>Allowed users for dev_trino</description> + </property> + + <property> + <name>ranger.audit.ingestor.service.dev_ozone.allowed.users</name> + <value>ozone,om,scm,dn</value> + <description>Allowed users for dev_ozone (OM, SCM, DataNode)</description> + </property> + + <property> + <name>ranger.audit.ingestor.service.dev_solr.allowed.users</name> + <value>solr</value> + <description>Allowed users for dev_solr (Solr plugin)</description> + </property> + + <property> + <name>ranger.audit.ingestor.service.dev_atlas.allowed.users</name> + <value>atlas</value> + <description>Allowed users for dev_atlas (Atlas)</description> + </property> + + <property> + <name>ranger.audit.ingestor.service.dev_kudu.allowed.users</name> + <value>kudu</value> + <description>Allowed users for dev_kudu (Kudu master/tserver)</description> + </property> + + <property> + <name>ranger.audit.ingestor.service.dev_nifi.allowed.users</name> + <value>nifi</value> + <description>Allowed users for dev_nifi (NiFi)</description> </property> <!-- AUTH_TO_LOCAL RULES FOR KERBEROS PRINCIPAL MAPPING --> @@ -207,22 +267,39 @@ RULE:[2:$1/$2@$0]([ndj]n/.*@.*|hdfs/.*@.*)s/.*/hdfs/ RULE:[2:$1/$2@$0]([rn]m/.*@.*|yarn/.*@.*)s/.*/yarn/ RULE:[2:$1/$2@$0](jhs/.*@.*)s/.*/mapred/ + RULE:[2:$1/$2@$0](hive/.*@.*)s/.*/hive/ + RULE:[2:$1/$2@$0](hbase/.*@.*)s/.*/hbase/ + RULE:[2:$1/$2@$0](kafka/.*@.*)s/.*/kafka/ + RULE:[2:$1/$2@$0](knox/.*@.*)s/.*/knox/ + RULE:[2:$1/$2@$0](rangerkms/.*@.*)s/.*/rangerkms/ + RULE:[2:$1/$2@$0](trino/.*@.*)s/.*/trino/ + RULE:[2:$1/$2@$0](ozone/.*@.*)s/.*/ozone/ + RULE:[2:$1/$2@$0](om/.*@.*)s/.*/om/ + RULE:[2:$1/$2@$0](scm/.*@.*)s/.*/scm/ + RULE:[2:$1/$2@$0](dn/.*@.*)s/.*/dn/ + RULE:[2:$1/$2@$0](solr/.*@.*)s/.*/solr/ + RULE:[2:$1/$2@$0](atlas/.*@.*)s/.*/atlas/ + RULE:[2:$1/$2@$0](kudu/.*@.*)s/.*/kudu/ + RULE:[2:$1/$2@$0](nifi/.*@.*)s/.*/nifi/ RULE:[1:$1@$0](.*@.*)s/@.*// DEFAULT </value> <description> - Kerberos auth_to_local rules for mapping authenticated principals to service names. - Uses Hadoop KerberosName syntax to convert full Kerberos principals to short usernames. - - IMPORTANT: These rules only apply to Kerberos/SPNEGO authentication. JWT and basic auth - usernames are already in short form and pass through unchanged. - - Default rules provided: - - RULE:[2:$1/$2@$0]([ndj]n/.*@.*|hdfs/.*@.*)s/.*/hdfs/ # nn,dn,jn,hdfs/* -> hdfs - - RULE:[2:$1/$2@$0]([rn]m/.*@.*|yarn/.*@.*)s/.*/yarn/ # rm,nm,yarn/* -> yarn - - RULE:[2:$1/$2@$0](jhs/.*@.*)s/.*/mapred/ # jhs/* -> mapred - - RULE:[1:$1@$0](.*@.*)s/@.*// # user@REALM -> user - - DEFAULT + Kerberos auth_to_local rules: map service/host@REALM to the short username checked + against ranger.audit.ingestor.service.<repo>.allowed.users (Policy Manager + policy.download.auth.users). + + Order matters — service-specific RULE lines run before the user@REALM fallback. + + Hadoop: nn/dn/jn/hdfs/* -> hdfs (dev_hdfs); rm/nm/yarn/* -> yarn (dev_yarn); + jhs/* -> mapred. + Plugins: hive/* -> hive (dev_hive); hbase/* -> hbase (dev_hbase); + kafka/* -> kafka (dev_kafka); knox/* -> knox (dev_knox); + rangerkms/* -> rangerkms (dev_kms); trino/* -> trino (dev_trino); + ozone/* -> ozone; om/* -> om; scm/* -> scm; dn/* -> dn (dev_ozone); + solr/* -> solr (dev_solr); atlas/* -> atlas (dev_atlas); + kudu/* -> kudu (dev_kudu); nifi/* -> nifi (dev_nifi). + Fallback: user@REALM -> user; then DEFAULT. </description> </property> diff --git a/audit-server/scripts/README.md b/audit-server/scripts/README.md index 6bd09a03b..917b0b02e 100644 --- a/audit-server/scripts/README.md +++ b/audit-server/scripts/README.md @@ -30,6 +30,8 @@ The codebase is organized into two main directories: - `audit-ingestor/` - `audit-dispatcher/` (contains a unified dispatcher application that dynamically loads specific dispatcher types) +**Troubleshooting:** Plugin audits rejected with HTTP 403 from ingestor (KMS, Kafka, Ozone, etc.) — see [README-AUDIT-INGESTOR-SERVICE-ALLOWLIST.md](../README-AUDIT-INGESTOR-SERVICE-ALLOWLIST.md). + ## Prerequisites Before running these scripts, ensure you have: diff --git a/dev-support/ranger-docker/scripts/admin/create-ranger-services.py b/dev-support/ranger-docker/scripts/admin/create-ranger-services.py index 68d9b915e..6f85c2371 100644 --- a/dev-support/ranger-docker/scripts/admin/create-ranger-services.py +++ b/dev-support/ranger-docker/scripts/admin/create-ranger-services.py @@ -129,9 +129,9 @@ def service_not_exists(service): 'configs': {'username': 'hdfs', 'password': 'hdfs', 'ozone.om.http-address': 'http://om:9874', 'hadoop.security.authentication': 'simple', - 'policy.download.auth.users': 'ozone', - 'tag.download.auth.users': 'ozone', - 'userstore.download.auth.users': 'ozone', + 'policy.download.auth.users': 'ozone,om,scm,dn', + 'tag.download.auth.users': 'ozone,om,scm,dn', + 'userstore.download.auth.users': 'ozone,om,scm,dn', 'ranger.plugin.ozone.policy.refresh.synchronous':'true'}}) solr = RangerService({'name': 'dev_solr', 'type': 'solr', @@ -148,7 +148,32 @@ def service_not_exists(service): 'ranger.plugin.super.users': 'solr', 'ranger.plugin.solr.policy.refresh.synchronous':'true'}}) -services = [hdfs, yarn, hive, hbase, kafka, knox, kms, trino, ozone, solr] +atlas = RangerService({'name': 'dev_atlas', 'type': 'atlas', + 'configs': {'username': 'atlas', 'password': 'atlas', + 'atlas.rest.address': 'http://atlas:21000', + 'policy.download.auth.users': 'atlas', + 'tag.download.auth.users': 'atlas', + 'userstore.download.auth.users': 'atlas', + 'ranger.plugin.atlas.policy.refresh.synchronous': 'true'}}) + +kudu = RangerService({'name': 'dev_kudu', 'type': 'kudu', + 'configs': {'username': 'kudu', 'password': 'kudu', + 'policy.download.auth.users': 'kudu', + 'tag.download.auth.users': 'kudu', + 'userstore.download.auth.users': 'kudu', + 'ranger.plugin.kudu.policy.refresh.synchronous': 'true'}}) + +nifi = RangerService({'name': 'dev_nifi', 'type': 'nifi', + 'configs': {'username': 'nifi', 'password': 'nifi', + 'nifi.url': 'http://ranger-nifi.rangernw:8080/nifi-api/resources', + 'nifi.authentication': 'NONE', + 'nifi.ssl.use.default.context': 'false', + 'policy.download.auth.users': 'nifi', + 'tag.download.auth.users': 'nifi', + 'userstore.download.auth.users': 'nifi', + 'ranger.plugin.nifi.policy.refresh.synchronous': 'true'}}) + +services = [hdfs, yarn, hive, hbase, kafka, knox, kms, trino, ozone, solr, atlas, kudu, nifi] for service in services: try: if service_not_exists(service):
