This is an automated email from the ASF dual-hosted git repository. ramackri pushed a commit to branch ranger-kafka-dynamic-partition-plan in repository https://gitbox.apache.org/repos/asf/ranger.git
commit 2a1425bf7e32a650e7f33cdf1f10971f5a7ec4c7 Author: ramk <[email protected]> AuthorDate: Sun Jun 21 19:24:19 2026 +0530 Add dynamic partition and auth access E2E harness for audit ingestor. Introduce Docker scripts to onboard plugins via partition-plan REST, verify Kafka partition routing and auth_to_local allowlists per plugin, and document the flow alongside the ranger-audit-e2e-harness plugin matrix. Co-authored-by: Cursor <[email protected]> --- .../README-AUDIT-INGESTOR-ACCESS-CURL-E2E.md | 202 ++++++++++++++ .../README-DYNAMIC-PARTITION-PLUGIN-E2E.md | 110 ++++++++ .../ranger-docker/scripts/audit/audit-stack-lib.sh | 53 ++++ .../scripts/audit/dynamic-auth-to-local-e2e-lib.sh | 299 +++++++++++++++++++++ .../audit/dynamic-partition-plugin-e2e-lib.sh | 258 ++++++++++++++++++ .../audit/verify-dynamic-auth-to-local-e2e.sh | 167 ++++++++++++ .../audit/verify-dynamic-partition-plugin-e2e.sh | 138 ++++++++++ .../scripts/audit/verify-partition-plan-e2e-all.sh | 9 + .../scripts/audit/wait-for-audit-health.sh | 57 ++++ 9 files changed, 1293 insertions(+) diff --git a/audit-server/README-AUDIT-INGESTOR-ACCESS-CURL-E2E.md b/audit-server/README-AUDIT-INGESTOR-ACCESS-CURL-E2E.md new file mode 100644 index 000000000..531aed246 --- /dev/null +++ b/audit-server/README-AUDIT-INGESTOR-ACCESS-CURL-E2E.md @@ -0,0 +1,202 @@ +# Audit ingestor access E2E — curl and dynamic allowlist guide + +End-to-end validation for **dynamic `auth_to_local` rule composition** and **per-plugin service allowlists** on the Ranger audit ingestor. + +## What this tests + +1. **Kerberos SPNEGO** on `POST /api/audit/access` from each plugin container +2. **`auth_to_local`** maps the plugin principal → short name (`hdfs`, `hive`, `rangerkms`, …) +3. **`services[repo].allowedUsers`** in the partition-plan registry authorizes the short name +4. **Dynamic updates** via `PUT /partition-plan` or `POST /partition-plan/onboard-repo` refresh composed rules and allowlist without restart + +## Quick start (Docker Tier 3) + +```bash +cd dev-support/ranger-docker + +# Bring up audit stack (ingestor + Kafka + plugins you need) +docker compose -f docker-compose.ranger.yml \ + -f docker-compose.ranger-kafka.yml \ + -f docker-compose.ranger-audit-server.yml \ + -f docker-compose.ranger-hadoop.yml \ + -f docker-compose.ranger-hive.yml \ + -f docker-compose.ranger-kms.yml up -d + +# Run E2E (enables dynamic mode if needed) +chmod +x scripts/audit/verify-dynamic-auth-to-local-e2e.sh +./scripts/audit/verify-dynamic-auth-to-local-e2e.sh +``` + +Generate curl cookbook only (no Docker required): + +```bash +./scripts/audit/verify-dynamic-auth-to-local-e2e.sh --generate-curl-only +# → dist/audit-e2e/access-ingestor-curl-cookbook.sh +``` + +## Base URLs + +| Endpoint | URL | Auth | +|----------|-----|------| +| Health | `http://localhost:7081/api/audit/health` | None | +| Partition plan | `http://ranger-audit-ingestor.rangernw:7081/api/audit/partition-plan` | SPNEGO (HTTP service keytab) | +| Onboard repo | `…/api/audit/partition-plan/onboard-repo` | SPNEGO | +| **Post audits** | `…/api/audit/access?serviceName=<repo>&appId=<pluginId>` | SPNEGO (plugin keytab) | + +**Important:** SPNEGO curl must use the **FQDN** host (`ranger-audit-ingestor.rangernw`), not `localhost`, when run inside Docker. + +## Three identifiers (do not conflate) + +| Field | Example | Role | +|-------|---------|------| +| **Repo / `serviceName`** | `dev_hdfs`, `dev_kms` | Query param on POST; key in `services[]` | +| **Kerberos short name** | `hdfs`, `rangerkms` | Output of `auth_to_local`; checked in `allowedUsers` | +| **Plugin ID / `appId`** | `hdfs`, `hiveServer2`, `kms` | Kafka partition routing; audit `agentId` | + +HDFS `auth_to_local` maps `nn/dn/jn/hdfs/*` → short name **`hdfs`**. That is **not** the repo name (`dev_hdfs`); it is the allowlist entry. + +## POST /access — curl template + +Run **inside the plugin container** (has keytab + krb5): + +```bash +export KRB5CCNAME=/tmp/krb-$$ +kinit -kt /etc/keytabs/hdfs.keytab hdfs/[email protected] + +curl -s -w '\nHTTP %{http_code}\n' --negotiate -u : -X POST \ + -H 'Content-Type: application/json' \ + 'http://ranger-audit-ingestor.rangernw:7081/api/audit/access?serviceName=dev_hdfs&appId=hdfs' \ + -d '[{ + "repo": "dev_hdfs", + "reqUser": "e2e-audit-user", + "evtTime": 1700000000000, + "access": "read", + "resource": "/e2e/path", + "result": 1, + "agent": "hdfs" + }]' +``` + +### Expected success response (HTTP 200) + +```json +{ + "total": 1, + "timestamp": 1717500000000, + "serviceName": "dev_hdfs", + "appId": "hdfs", + "authenticatedUser": "hdfs" +} +``` + +`authenticatedUser` is the **short name** after `auth_to_local`, not the full Kerberos principal. + +### Error responses + +| HTTP | Meaning | +|------|---------| +| **401** | No SPNEGO / authentication failed | +| **403** | Authenticated, but short name not in `services[serviceName].allowedUsers` | +| **400** | Missing `serviceName` or empty audit batch | + +## Per-plugin matrix (Docker Tier 3) + +| Repo | appId | auth_to_local short name | Container | kinit principal | +|------|-------|--------------------------|-----------|-----------------| +| `dev_hdfs` | `hdfs` | `hdfs` | `ranger-hadoop` | `hdfs/[email protected]` | +| `dev_yarn` | `yarn` | `yarn` | `ranger-hadoop` | `yarn/[email protected]` | +| `dev_hive` | `hiveServer2` | `hive` | `ranger-hive` | `hive/[email protected]` | +| `dev_hbase` | `hbaseMaster` | `hbase` | `ranger-hbase` | `hbase/[email protected]` | +| `dev_kafka` | `kafka` | `kafka` | `ranger-kafka` | `kafka/[email protected]` | +| `dev_knox` | `knox` | `knox` | `ranger-knox` | `knox/[email protected]` | +| `dev_kms` | `kms` | `rangerkms` | `ranger-kms` | `rangerkms/[email protected]` | +| `dev_trino` | `trino` | `trino` | `ranger-trino` | `trino/[email protected]` | +| `dev_solr` | `solr` | `solr` | `ranger-solr` | `solr/[email protected]` | +| `dev_ozone` | `ozone` | `om` | `ozone-om` | `om/[email protected]` | + +## Dynamically add allowlist users + +### Option A — onboard new repo + plugin (recommended) + +```bash +kinit -kt /etc/keytabs/HTTP.keytab HTTP/[email protected] + +curl -s --negotiate -u : -X POST -H 'Content-Type: application/json' \ + 'http://ranger-audit-ingestor.rangernw:7081/api/audit/partition-plan/onboard-repo' \ + -d '{ + "repo": "dev_trino", + "pluginId": "trino", + "partitionCount": 3, + "allowedUsers": ["trino"], + "expectedVersion": 1 + }' +``` + +After install, ingestor logs: + +```text +Applied composed auth_to_local rules for plan version N (M active short names) +``` + +### Option B — update allowlist only (PUT full plan) + +```bash +# GET current plan, set services[dev_kms].allowedUsers, bump expectedVersion +curl -s --negotiate -u : -X PUT -H 'Content-Type: application/json' \ + 'http://ranger-audit-ingestor.rangernw:7081/api/audit/partition-plan' \ + -d '{ + "expectedVersion": 2, + "topicPartitionCount": 48, + "plugins": { "...": "..." }, + "buffer": { "partitions": [45,46,47] }, + "services": { + "dev_kms": { "allowedUsers": ["rangerkms"] } + } + }' +``` + +## E2E script scenarios + +`verify-dynamic-auth-to-local-e2e.sh` runs: + +1. **Per-plugin POST** — each running plugin container posts audits → 200 + correct `authenticatedUser` +2. **Allowlist toggle** — clear `dev_kms` users → 403; restore → 200 +3. **Cross-repo denial** — HDFS principal posting to `dev_kms` → 403 +4. **onboard-repo** — new synthetic repo + buffer plugin; HDFS principal → 200 + +Options: + +```bash +./scripts/audit/verify-dynamic-auth-to-local-e2e.sh --plugins hdfs,hive,kms +./scripts/audit/verify-dynamic-auth-to-local-e2e.sh --no-enable # assume dynamic already on +``` + +## Access log (ingestor) + +Successful POSTs are logged at DEBUG/INFO in the ingestor container: + +```bash +docker logs ranger-audit-ingestor 2>&1 | grep -E 'logAccessAudit|authenticatedUser|Applied composed auth_to_local' +``` + +Example log lines: + +```text +Authenticated user: Principal: hdfs/[email protected], shortName: hdfs +Applied composed auth_to_local rules for plan version 3 (12 active short names) +``` + +Audits accepted on `/access` are produced to Kafka topic **`ranger_audits`** (partition keyed by `appId`). + +## Related docs + +- `README-KAFKA-DYNAMIC-PARTITIONING-GUIDE.md` — unified partition plan + allowlist +- `README-KAFKA-PARTITION-PLAN-E2E-TEST-PLAN.md` — partition routing E2E +- `dev-support/ranger-docker/scripts/audit/verify-partition-plan-e2e-all.sh` — routing-only suite + +## Unit tests (no Docker) + +```bash +mvn test -pl audit-server/audit-ingestor -am \ + -Dtest=AuthToLocalRuleComposerTest -Dsurefire.failIfNoSpecifiedTests=false +``` diff --git a/audit-server/README-DYNAMIC-PARTITION-PLUGIN-E2E.md b/audit-server/README-DYNAMIC-PARTITION-PLUGIN-E2E.md new file mode 100644 index 000000000..16d80f62e --- /dev/null +++ b/audit-server/README-DYNAMIC-PARTITION-PLUGIN-E2E.md @@ -0,0 +1,110 @@ +# Dynamic partition plugin E2E + +End-to-end validation for **dynamic partition allocation**: onboard plugins via **`POST /api/audit/partition-plan/onboard-repo`**, then prove audits land on **assigned Kafka partitions** and pass **auth_to_local + allowlist** checks. + +Built on: + +| Layer | Branch / path | +|-------|----------------| +| Plugin audit pipelines (Hive, Ozone, Kafka, HDFS, HBase, Knox, KMS) | `ranger-audit-e2e-harness` — `dev-support/ranger-docker/scripts/{hive,kafka,...}/` | +| Partition-plan REST + watcher | `ranger-kafka-dynamic-partition-plan` — `scripts/audit/partition-plan-e2e-lib.sh` | +| Access + allowlist curl | `scripts/audit/dynamic-auth-to-local-e2e-lib.sh` | + +## What this tests + +For each plugin that has a **running** Docker container: + +1. **Onboard** — `POST /partition-plan/onboard-repo` with: + - `repo` (Policy Manager service name, e.g. `dev_kms`) + - `pluginId` (Kafka record key / agent id, e.g. `kms`, `hiveServer2`) + - `partitionCount` (dedicated partitions carved from buffer) + - `allowedUsers` (Kerberos short names after `auth_to_local`, from XML catalog rules) + - `expectedVersion` (optimistic concurrency) +2. **Auth** — `POST /api/audit/access` with plugin SPNEGO → **200/202** +3. **Routing** — consume `ranger_audits`, verify record **partition ∈ plan.plugins[pluginId].partitions** +4. **auth_to_local** — recomposed after onboard (same as access E2E) + +`HTTP.keytab` on the ingestor is **unchanged** when onboarding plugins. + +## Plugin matrix (default specs) + +| Repo | pluginId | allowedUsers (short names) | Container | +|------|----------|----------------------------|-----------| +| `dev_hdfs` | `hdfs` | `hdfs` | `ranger-hadoop` | +| `dev_hive` | `hiveServer2` | `hive` | `ranger-hive` | +| `dev_hbase` | `hbaseMaster` | `hbase` | `ranger-hbase` | +| `dev_kafka` | `kafka` | `kafka` | `ranger-kafka` | +| `dev_knox` | `knox` | `knox` | `ranger-knox` | +| `dev_kms` | `kms` | `rangerkms` | `ranger-kms` | +| `dev_ozone` | `ozone` | `om`, `ozone` | `ozone-om` | + +Specs live in `dynamic-partition-plugin-e2e-lib.sh` (`DPP_PLUGIN_ONBOARD_SPECS`). Align with `ranger-audit-ingestor-site.xml` `auth.to.local` catalog and harness allowlists. + +## Quick start (Tier 3 Docker) + +```bash +cd dev-support/ranger-docker + +# Stack: ingestor + Kafka + plugins (same compose as partition-plan E2E) +docker compose -f docker-compose.ranger.yml \ + -f docker-compose.ranger-kafka.yml \ + -f docker-compose.ranger-audit-server.yml \ + -f docker-compose.ranger-hadoop.yml \ + -f docker-compose.ranger-hive.yml \ + -f docker-compose.ranger-kms.yml up -d + +chmod +x scripts/audit/verify-dynamic-partition-plugin-e2e.sh +./scripts/audit/verify-dynamic-partition-plugin-e2e.sh +``` + +Subset of plugins: + +```bash +./scripts/audit/verify-dynamic-partition-plugin-e2e.sh --plugins kms,hiveServer2 +``` + +With harness trigger scripts (from `ranger-audit-e2e-harness` when checked out): + +```bash +./scripts/audit/verify-dynamic-partition-plugin-e2e.sh --with-harness-triggers +``` + +Full suite (core partition-plan + auth + plugin onboard): + +```bash +./scripts/audit/verify-partition-plan-e2e-all.sh --with-auth-access --with-plugin-onboard +``` + +## Greenfield layout (empty `configured.plugins`) + +Sample site XML leaves `kafka.configured.plugins` **empty**: + +- First bootstrap → **buffer-only** plan sized by `kafka.topic.partitions` (default **10**) +- Each `onboard-repo` promotes a plugin from buffer with `partitionCount` (default **2** in E2E specs) +- `auth_to_local` rules compose from union of `services[].allowedUsers` (XML catalog + generated rules) + +## Onboard-repo example (manual) + +```bash +# Inside ingestor container or any host with HTTP.keytab + FQDN +kinit -kt /etc/keytabs/HTTP.keytab HTTP/[email protected] + +curl -s --negotiate -u : -X POST \ + -H 'Content-Type: application/json' \ + 'http://ranger-audit-ingestor.rangernw:7081/api/audit/partition-plan/onboard-repo' \ + -d '{ + "repo": "dev_trino", + "pluginId": "trino", + "partitionCount": 2, + "allowedUsers": ["trino"], + "expectedVersion": 1 + }' +``` + +Then POST audits from the Trino container using `serviceName=dev_trino&appId=trino`. + +## Related docs + +- [README-AUDIT-INGESTOR-ACCESS-CURL-E2E.md](README-AUDIT-INGESTOR-ACCESS-CURL-E2E.md) — SPNEGO, three identifiers, curl cookbook +- [README-KAFKA-DYNAMIC-PARTITIONING-GUIDE.md](README-KAFKA-DYNAMIC-PARTITIONING-GUIDE.md) — operator cutover and REST API +- `ranger-audit-e2e-harness` → `dev-support/ranger-docker/README-AUDIT-E2E.md` — full plugin → Solr pipelines diff --git a/dev-support/ranger-docker/scripts/audit/audit-stack-lib.sh b/dev-support/ranger-docker/scripts/audit/audit-stack-lib.sh new file mode 100755 index 000000000..e483f050a --- /dev/null +++ b/dev-support/ranger-docker/scripts/audit/audit-stack-lib.sh @@ -0,0 +1,53 @@ +#!/bin/bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Shared helpers for Tier 3 audit Docker E2E scripts. + +AUDIT_INGESTOR_CONTAINER="${AUDIT_INGESTOR_CONTAINER:-ranger-audit-ingestor}" +KAFKA_CONTAINER="${KAFKA_CONTAINER:-ranger-kafka}" +RANGER_SOLR_CONTAINER="${RANGER_SOLR_CONTAINER:-ranger-solr}" + +audit_stack_container_running() { + local name="$1" + docker ps --filter "name=^${name}$" --filter status=running --format '{{.Names}}' | grep -qx "${name}" +} + +audit_stack_wait_url() { + local url="$1" + local label="${2:-Service}" + local timeout="${3:-120}" + local deadline=$((SECONDS + timeout)) + + while (( SECONDS < deadline )); do + if curl -sf "${url}" >/dev/null 2>&1; then + echo " ${label} ready: ${url}" + return 0 + fi + sleep 2 + done + + echo "ERROR: ${label} not ready within ${timeout}s: ${url}" >&2 + return 1 +} + +audit_stack_require_container() { + local name="$1" + if ! audit_stack_container_running "${name}"; then + echo "ERROR: container ${name} is not running" >&2 + return 1 + fi + return 0 +} diff --git a/dev-support/ranger-docker/scripts/audit/dynamic-auth-to-local-e2e-lib.sh b/dev-support/ranger-docker/scripts/audit/dynamic-auth-to-local-e2e-lib.sh new file mode 100755 index 000000000..1f31a8096 --- /dev/null +++ b/dev-support/ranger-docker/scripts/audit/dynamic-auth-to-local-e2e-lib.sh @@ -0,0 +1,299 @@ +#!/bin/bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# E2E helpers: dynamic allowlist + auth_to_local + POST /api/audit/access via curl/SPNEGO. +# Requires partition-plan-e2e-lib.sh (pp_*) to be sourced first. + +DAEL_INGESTOR_HOST="${CONTAINER}.rangernw" +DAEL_INGESTOR_HTTP_PORT="${INGESTOR_HTTP_PORT:-7081}" +DAEL_ACCESS_PATH="/api/audit/access" +DAEL_ONBOARD_PATH="/api/audit/partition-plan/onboard-repo" +DAEL_ACCESS_CODE="" +DAEL_ACCESS_BODY="" + +# repo|appId|shortName|source_container|keytab|principal +# shortName = auth_to_local output checked against services[].allowedUsers +# appId = Kafka plugin id (agentId on audit events) +readonly -a DAEL_PLUGIN_PROFILES=( + "dev_hdfs|hdfs|hdfs|ranger-hadoop|/etc/keytabs/hdfs.keytab|hdfs/[email protected]" + "dev_yarn|yarn|yarn|ranger-hadoop|/etc/keytabs/yarn.keytab|yarn/[email protected]" + "dev_hive|hiveServer2|hive|ranger-hive|/etc/keytabs/hive.keytab|hive/[email protected]" + "dev_hbase|hbaseMaster|hbase|ranger-hbase|/etc/keytabs/hbase.keytab|hbase/[email protected]" + "dev_kafka|kafka|kafka|ranger-kafka|/etc/keytabs/kafka.keytab|kafka/[email protected]" + "dev_knox|knox|knox|ranger-knox|/etc/keytabs/knox.keytab|knox/[email protected]" + "dev_kms|kms|rangerkms|ranger-kms|/etc/keytabs/rangerkms.keytab|rangerkms/[email protected]" + "dev_trino|trino|trino|ranger-trino|/etc/keytabs/trino.keytab|trino/[email protected]" + "dev_solr|solr|solr|ranger-solr|/etc/keytabs/solr.keytab|solr/[email protected]" + "dev_ozone|ozone|om|ozone-om|/etc/keytabs/om.keytab|om/[email protected]" +) + +dael_ingestor_access_url() { + echo "http://${DAEL_INGESTOR_HOST}:${DAEL_INGESTOR_HTTP_PORT}${DAEL_ACCESS_PATH}"; +} + +dael_audit_event_json() { + local repo="$1" + local app_id="$2" + local evt_ms + evt_ms="$(python3 -c 'import time; print(int(time.time()*1000))')" + printf '[{"repo":"%s","reqUser":"e2e-audit-user","evtTime":%s,"access":"read","resource":"/e2e/path","result":1,"agent":"%s"}]' \ + "${repo}" "${evt_ms}" "${app_id}" +} + +dael_container_running() { + local name="$1" + docker ps --filter "name=^${name}$" --filter status=running --format '{{.Names}}' | grep -qx "${name}" +} + +dael_access_post_from_container() { + local src_container="$1" + local keytab="$2" + local principal="$3" + local service_name="$4" + local app_id="$5" + local body + local url + local outfile="/tmp/dael-access-body-${src_container}-$$" + local krb_cc="/tmp/dael-access-krb-${src_container}-$$" + local body_file="/tmp/dael-access-req-${src_container}-$$" + + body="$(dael_audit_event_json "${service_name}" "${app_id}")" + url="$(dael_ingestor_access_url)?serviceName=${service_name}&appId=${app_id}" + + if ! dael_container_running "${src_container}"; then + DAEL_ACCESS_CODE="000" + DAEL_ACCESS_BODY="container ${src_container} not running" + return 1 + fi + + if ! docker exec "${src_container}" test -f "${keytab}" 2>/dev/null; then + DAEL_ACCESS_CODE="000" + DAEL_ACCESS_BODY="keytab missing: ${keytab}" + return 1 + fi + + printf '%s' "${body}" | docker exec -i "${src_container}" tee "${body_file}" >/dev/null + DAEL_ACCESS_CODE="$(docker exec "${src_container}" bash -c \ + "export KRB5CCNAME=${krb_cc}; kinit -kt '${keytab}' '${principal}' >/dev/null 2>&1 && \ +curl -s -o ${outfile} -w '%{http_code}' --negotiate -u : -X POST \ + -H 'Content-Type: application/json' -d @${body_file} '${url}'")" + DAEL_ACCESS_BODY="$(docker exec "${src_container}" cat "${outfile}" 2>/dev/null || true)" + docker exec "${src_container}" rm -f "${outfile}" "${body_file}" "${krb_cc}" 2>/dev/null || true +} + +dael_expect_access() { + local label="$1" + local expect_code="$2" + if [[ "${DAEL_ACCESS_CODE}" == "${expect_code}" ]]; then + pp_record_pass "${label} (HTTP ${expect_code})" + return 0 + fi + pp_record_fail "${label} expected HTTP ${expect_code}, got ${DAEL_ACCESS_CODE}: ${DAEL_ACCESS_BODY}" + return 1 +} + +dael_json_authenticated_user() { + printf '%s' "${DAEL_ACCESS_BODY}" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('authenticatedUser',''))" 2>/dev/null || echo "" +} + +dael_plan_put_body_with_allowed_users() { + local plan_json="$1" + local repo="$2" + local users_csv="$3" + printf '%s' "${plan_json}" | python3 -c \ + "import sys,json; plan=json.load(sys.stdin); repo, users=sys.argv[1], [u.strip() for u in sys.argv[2].split(',') if u.strip()]; \ +body={'expectedVersion': plan['version'], 'topicPartitionCount': plan['topicPartitionCount'], 'plugins': plan['plugins'], 'buffer': plan['buffer']}; \ +body['services'] = plan.get('services') or {}; body['services'][repo] = {'allowedUsers': users}; print(json.dumps(body))" "${repo}" "${users_csv}" +} + +dael_put_allowed_users() { + local repo="$1" + local users_csv="$2" + local put_url + local put_body + + put_url="$(pp_ingestor_plan_url "${CONTAINER}")" + pp_ingestor_request "${CONTAINER}" GET "${put_url}" + if [[ "${HTTP_CODE}" != "200" ]]; then + pp_record_fail "GET plan before PUT allowedUsers for ${repo}: ${HTTP_CODE}" + return 1 + fi + put_body="$(dael_plan_put_body_with_allowed_users "${HTTP_BODY}" "${repo}" "${users_csv}")" + pp_ingestor_request "${CONTAINER}" PUT "${put_url}" "${put_body}" + if [[ "${HTTP_CODE}" == "200" ]]; then + return 0 + fi + pp_record_fail "PUT allowedUsers for ${repo} failed: HTTP ${HTTP_CODE} ${HTTP_BODY}" + return 1 +} + +dael_onboard_repo() { + local repo="$1" + local plugin_id="$2" + local short_name="$3" + local partition_count="${4:-2}" + local version="$5" + local url body + + url="http://$(pp_ingestor_host "${CONTAINER}"):${INGESTOR_HTTP_PORT}/api/audit/partition-plan/onboard-repo" + body="$(python3 -c \ + 'import json,sys; repo,plugin,part,user,ver=sys.argv[1:6]; \ +print(json.dumps({"repo":repo,"pluginId":plugin,"partitionCount":int(part),"allowedUsers":[user],"expectedVersion":int(ver)}))' \ + "${repo}" "${plugin_id}" "${partition_count}" "${short_name}" "${version}")" + pp_ingestor_request "${CONTAINER}" POST "${url}" "${body}" + [[ "${HTTP_CODE}" == "200" ]] +} + +dael_wait_auth_to_local_applied() { + local container="$1" + local timeout="${2:-60}" + local deadline=$((SECONDS + timeout)) + while (( SECONDS < deadline )); do + if docker logs "${container}" 2>&1 | tail -200 | grep -q "Applied composed auth_to_local rules"; then + return 0 + fi + sleep 2 + done + return 1 +} + +dael_write_curl_cookbook() { + local out_file="$1" + local access_url onboard_url + access_url="$(dael_ingestor_access_url)" + onboard_url="http://${DAEL_INGESTOR_HOST}:${DAEL_INGESTOR_HTTP_PORT}/api/audit/partition-plan/onboard-repo"; + + { + echo "# Ranger audit ingestor access E2E — curl cookbook" + echo "# Generated: $(date -u +%Y-%m-%dT%H:%M:%SZ)" + echo "# Run inside a container on rangernw with the plugin keytab (SPNEGO Host must be FQDN)." + echo "" + echo "# Health (no auth):" + echo "curl -s http://localhost:7081/api/audit/health"; + echo "" + echo "# Partition plan (HTTP SPNEGO as ingestor HTTP service):" + echo "kinit -kt /etc/keytabs/HTTP.keytab HTTP/[email protected]" + echo "curl -s --negotiate -u : http://ranger-audit-ingestor.rangernw:7081/api/audit/partition-plan"; + echo "" + echo "# Onboard repo + allowlist user (dynamic mode):" + echo "curl -s --negotiate -u : -X POST -H 'Content-Type: application/json' \\" + echo " '${onboard_url}' \\" + echo " -d '{\"repo\":\"dev_trino\",\"pluginId\":\"trino\",\"partitionCount\":3,\"allowedUsers\":[\"trino\"],\"expectedVersion\":1}'" + echo "" + + local profile repo app_id short_name src_container keytab principal body + for profile in "${DAEL_PLUGIN_PROFILES[@]}"; do + IFS='|' read -r repo app_id short_name src_container keytab principal <<< "${profile}" + body="$(dael_audit_event_json "${repo}" "${app_id}")" + echo "# POST audit — ${repo} (${short_name} via auth_to_local, appId=${app_id})" + echo "# Run on: ${src_container}" + echo "kinit -kt ${keytab} ${principal}" + echo "curl -s -w '\\nHTTP %{http_code}\\n' --negotiate -u : -X POST \\" + echo " -H 'Content-Type: application/json' \\" + echo " '${access_url}?serviceName=${repo}&appId=${app_id}' \\" + echo " -d '${body}'" + echo "" + done + } > "${out_file}" + echo " Wrote curl cookbook: ${out_file}" +} + +dael_test_plugin_access() { + local profile="$1" + local repo app_id short_name src_container keytab principal auth_user + + IFS='|' read -r repo app_id short_name src_container keytab principal <<< "${profile}" + + if ! dael_container_running "${src_container}"; then + echo " SKIP ${repo}: ${src_container} not running" + return 0 + fi + + echo "" + echo " Plugin ${app_id} / repo ${repo} (shortName=${short_name})..." + dael_access_post_from_container "${src_container}" "${keytab}" "${principal}" "${repo}" "${app_id}" + if ! dael_expect_access "${repo} POST /access as ${short_name}" "200"; then + return 1 + fi + auth_user="$(dael_json_authenticated_user)" + if [[ "${auth_user}" == "${short_name}" ]]; then + pp_record_pass "${repo} authenticatedUser=${auth_user}" + else + pp_record_fail "${repo} authenticatedUser expected ${short_name}, got ${auth_user}" + return 1 + fi + return 0 +} + +dael_test_dynamic_user_toggle() { + local repo="$1" + local short_name="$2" + local src_container="$3" + local keytab="$4" + local principal="$5" + local app_id="$6" + + echo "" + echo " Dynamic allowlist toggle for ${repo}..." + + if ! dael_put_allowed_users "${repo}" "wronguser-not-allowed"; then + return 1 + fi + if dael_wait_auth_to_local_applied "${CONTAINER}" 30; then + pp_record_pass "auth_to_local recomposed after allowlist changed" + else + pp_record_fail "no auth_to_local recompose log after allowlist changed" + fi + + dael_access_post_from_container "${src_container}" "${keytab}" "${principal}" "${repo}" "${app_id}" + dael_expect_access "${repo} denied when user not in allowlist" "403" || return 1 + + if ! dael_put_allowed_users "${repo}" "${short_name}"; then + return 1 + fi + if dael_wait_auth_to_local_applied "${CONTAINER}" 30; then + pp_record_pass "auth_to_local recomposed after allowlist restored" + else + pp_record_fail "no auth_to_local recompose log after allowlist restored" + fi + + dael_access_post_from_container "${src_container}" "${keytab}" "${principal}" "${repo}" "${app_id}" + dael_expect_access "${repo} allowed after allowlist restored" "200" || return 1 + return 0 +} + +dael_test_cross_repo_denied() { + local src_container="$1" + local keytab="$2" + local principal="$3" + local wrong_repo="$4" + local app_id="$5" + local label="$6" + + dael_access_post_from_container "${src_container}" "${keytab}" "${principal}" "${wrong_repo}" "${app_id}" + dael_expect_access "${label}" "403" +} + +dael_filter_profiles() { + local want="$1" + local profile + for profile in "${DAEL_PLUGIN_PROFILES[@]}"; do + IFS='|' read -r _ app_id _ _ _ _ <<< "${profile}" + if [[ -z "${want}" || ",${want}," == *",${app_id},"* || ",${want}," == *",$(echo "${profile}" | cut -d'|' -f1),"* ]]; then + echo "${profile}" + fi + done +} diff --git a/dev-support/ranger-docker/scripts/audit/dynamic-partition-plugin-e2e-lib.sh b/dev-support/ranger-docker/scripts/audit/dynamic-partition-plugin-e2e-lib.sh new file mode 100755 index 000000000..7efe689b1 --- /dev/null +++ b/dev-support/ranger-docker/scripts/audit/dynamic-partition-plugin-e2e-lib.sh @@ -0,0 +1,258 @@ +#!/bin/bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# E2E helpers: dynamic partition-plan onboard-repo + Kafka partition verification. +# Requires partition-plan-e2e-lib.sh and dynamic-auth-to-local-e2e-lib.sh. + +# repo|pluginId|shortName|partitionCount|source_container|keytab|principal +# partitionCount is the default for onboard-repo when not overridden. +readonly -a DPP_PLUGIN_ONBOARD_SPECS=( + "dev_hdfs|hdfs|hdfs|2|ranger-hadoop|/etc/keytabs/hdfs.keytab|hdfs/[email protected]" + "dev_hive|hiveServer2|hive|2|ranger-hive|/etc/keytabs/hive.keytab|hive/[email protected]" + "dev_hbase|hbaseMaster|hbase|2|ranger-hbase|/etc/keytabs/hbase.keytab|hbase/[email protected]" + "dev_kafka|kafka|kafka|2|ranger-kafka|/etc/keytabs/kafka.keytab|kafka/[email protected]" + "dev_knox|knox|knox|2|ranger-knox|/etc/keytabs/knox.keytab|knox/[email protected]" + "dev_kms|kms|rangerkms|2|ranger-kms|/etc/keytabs/rangerkms.keytab|rangerkms/[email protected]" + "dev_ozone|ozone|om|2|ozone-om|/etc/keytabs/om.keytab|om/[email protected]" +) + +dpp_filter_specs() { + local want="$1" + local spec repo plugin_id + for spec in "${DPP_PLUGIN_ONBOARD_SPECS[@]}"; do + IFS='|' read -r repo plugin_id _ _ _ _ _ <<< "${spec}" + if [[ -z "${want}" || ",${want}," == *",${plugin_id},"* || ",${want}," == *",${repo},"* ]]; then + echo "${spec}" + fi + done +} + +dpp_plan_version() { + local plan_url + plan_url="$(pp_ingestor_plan_url "${CONTAINER}")" + pp_ingestor_request "${CONTAINER}" GET "${plan_url}" + if [[ "${HTTP_CODE}" != "200" ]]; then + echo "" + return 1 + fi + pp_json_field "${HTTP_BODY}" version +} + +dpp_plan_has_plugin() { + local plugin_id="$1" + local plan_json="$2" + printf '%s' "${plan_json}" | python3 -c \ + "import sys,json; d=json.load(sys.stdin); sys.exit(0 if sys.argv[1] in (d.get('plugins') or {}) else 1)" \ + "${plugin_id}" 2>/dev/null +} + +dpp_plan_plugin_partitions() { + local plugin_id="$1" + local plan_json="$2" + printf '%s' "${plan_json}" | python3 -c \ + "import sys,json; d=json.load(sys.stdin); p=(d.get('plugins') or {}).get(sys.argv[1]); \ +print(','.join(str(x) for x in (p or {}).get('partitions') or []))" "${plugin_id}" 2>/dev/null || echo "" +} + +dpp_onboard_repo_multi() { + local repo="$1" + local plugin_id="$2" + local partition_count="$3" + local users_csv="$4" + local version="$5" + local url body + + url="http://$(pp_ingestor_host "${CONTAINER}"):${INGESTOR_HTTP_PORT}/api/audit/partition-plan/onboard-repo" + body="$(python3 -c \ + 'import json,sys; repo,plugin,part,users,ver=sys.argv[1:6]; \ +users=[u.strip() for u in users.split(",") if u.strip()]; \ +print(json.dumps({"repo":repo,"pluginId":plugin,"partitionCount":int(part),"allowedUsers":users,"expectedVersion":int(ver)}))' \ + "${repo}" "${plugin_id}" "${partition_count}" "${users_csv}" "${version}")" + pp_ingestor_request "${CONTAINER}" POST "${url}" "${body}" + [[ "${HTTP_CODE}" == "200" ]] +} + +dpp_ensure_plugin_onboarded() { + local spec="$1" + local repo plugin_id short_name part_count container _keytab _principal + local version plan_url users_csv + + IFS='|' read -r repo plugin_id short_name part_count container _keytab _principal <<< "${spec}" + + if ! dael_container_running "${container}"; then + echo " SKIP onboard ${plugin_id}: ${container} not running" + return 0 + fi + + plan_url="$(pp_ingestor_plan_url "${CONTAINER}")" + pp_ingestor_request "${CONTAINER}" GET "${plan_url}" + if [[ "${HTTP_CODE}" != "200" ]]; then + pp_record_fail "GET plan before onboard ${plugin_id}: ${HTTP_CODE}" + return 1 + fi + + if dpp_plan_has_plugin "${plugin_id}" "${HTTP_BODY}"; then + pp_record_pass "${plugin_id} already in partition plan" + return 0 + fi + + version="$(pp_json_field "${HTTP_BODY}" version)" + users_csv="${short_name}" + if [[ "${repo}" == "dev_ozone" ]]; then + users_csv="om,ozone" + fi + + echo " Onboard ${repo} pluginId=${plugin_id} partitions=${part_count} allowedUsers=[${users_csv}]..." + if ! dpp_onboard_repo_multi "${repo}" "${plugin_id}" "${part_count}" "${users_csv}" "${version}"; then + pp_record_fail "onboard-repo ${repo} failed: HTTP ${HTTP_CODE} ${HTTP_BODY}" + return 1 + fi + + pp_record_pass "onboard-repo ${plugin_id} -> plan v$(pp_json_field "${HTTP_BODY}" version)" + if dael_wait_auth_to_local_applied "${CONTAINER}" 45; then + pp_record_pass "auth_to_local recomposed after onboard ${plugin_id}" + else + pp_record_fail "no auth_to_local recompose log after onboard ${plugin_id}" + fi + return 0 +} + +dpp_audit_event_with_marker() { + local repo="$1" + local app_id="$2" + local marker="$3" + local evt_ms + evt_ms="$(python3 -c 'import time; print(int(time.time()*1000))')" + printf '[{"repo":"%s","reqUser":"e2e-partition","evtTime":%s,"access":"read","resource":"/e2e/%s","result":1,"agent":"%s"}]' \ + "${repo}" "${evt_ms}" "${marker}" "${app_id}" +} + +dpp_post_access_with_marker() { + local src_container="$1" + local keytab="$2" + local principal="$3" + local repo="$4" + local app_id="$5" + local marker="$6" + local body url outfile krb_cc body_file + + body="$(dpp_audit_event_with_marker "${repo}" "${app_id}" "${marker}")" + url="$(dael_ingestor_access_url)?serviceName=${repo}&appId=${app_id}" + outfile="/tmp/dpp-access-body-${src_container}-$$" + krb_cc="/tmp/dpp-access-krb-${src_container}-$$" + body_file="/tmp/dpp-access-req-${src_container}-$$" + + printf '%s' "${body}" | docker exec -i "${src_container}" tee "${body_file}" >/dev/null + DAEL_ACCESS_CODE="$(docker exec "${src_container}" bash -c \ + "export KRB5CCNAME=${krb_cc}; kinit -kt '${keytab}' '${principal}' >/dev/null 2>&1 && \ +curl -s -o ${outfile} -w '%{http_code}' --negotiate -u : -X POST \ + -H 'Content-Type: application/json' -d @${body_file} '${url}'")" + DAEL_ACCESS_BODY="$(docker exec "${src_container}" cat "${outfile}" 2>/dev/null || true)" + docker exec "${src_container}" rm -f "${outfile}" "${body_file}" "${krb_cc}" 2>/dev/null || true +} + +dpp_kafka_find_partition_for_marker() { + local record_key="$1" + local marker="$2" + local max_messages="${3:-800}" + local out + + out="$(pp_kafka_run "timeout 25 /opt/kafka/bin/kafka-console-consumer.sh \ + --bootstrap-server ranger-kafka.rangernw:9092 \ + --topic '${AUDIT_TOPIC}' \ + --from-beginning \ + --max-messages ${max_messages} \ + --property print.partition=true \ + --property print.key=true \ + --timeout-ms 15000")" + + printf '%s' "${out}" | python3 -c \ + "import sys,re; key,marker=sys.argv[1],sys.argv[2]; \ +for line in sys.stdin: \ + if key not in line or marker not in line: continue; \ + m=re.search(r'Partition[:\\s]+(\\d+)', line, re.I); \ + if m: print(m.group(1)); break" \ + "${record_key}" "${marker}" 2>/dev/null || echo "" +} + +dpp_verify_plugin_partition_routing() { + local spec="$1" + local repo plugin_id short_name _part_count container keytab principal + local marker assigned part found + + IFS='|' read -r repo plugin_id short_name _part_count container keytab principal <<< "${spec}" + + if ! dael_container_running "${container}"; then + return 0 + fi + + pp_ingestor_request "${CONTAINER}" GET "$(pp_ingestor_plan_url "${CONTAINER}")" + if [[ "${HTTP_CODE}" != "200" ]]; then + pp_record_fail "GET plan for ${plugin_id} routing check" + return 1 + fi + + assigned="$(dpp_plan_plugin_partitions "${plugin_id}" "${HTTP_BODY}")" + if [[ -z "${assigned}" ]]; then + pp_record_fail "${plugin_id} not in plan — run onboard first" + return 1 + fi + pp_record_pass "${plugin_id} assigned partitions [${assigned}]" + + marker="partition-e2e-${plugin_id}-$(date +%s)" + dpp_post_access_with_marker "${container}" "${keytab}" "${principal}" "${repo}" "${plugin_id}" "${marker}" + if [[ "${DAEL_ACCESS_CODE}" != "200" && "${DAEL_ACCESS_CODE}" != "202" ]]; then + pp_record_fail "${plugin_id} POST /access for partition test: HTTP ${DAEL_ACCESS_CODE} ${DAEL_ACCESS_BODY}" + return 1 + fi + pp_record_pass "${plugin_id} POST /access HTTP ${DAEL_ACCESS_CODE}" + + sleep 3 + found="$(dpp_kafka_find_partition_for_marker "${plugin_id}" "${marker}")" + if [[ -z "${found}" ]]; then + pp_record_fail "${plugin_id} audit not found on Kafka (key=${plugin_id}, marker=${marker}) — check ACLs or dispatcher lag" + return 1 + fi + + if printf '%s' "${assigned}" | python3 -c "import sys; allowed=set(sys.argv[1].split(',')); sys.exit(0 if sys.argv[2] in allowed else 1)" "${assigned}" "${found}"; then + pp_record_pass "${plugin_id} Kafka partition ${found} in assigned [${assigned}]" + return 0 + fi + pp_record_fail "${plugin_id} Kafka partition ${found} NOT in assigned [${assigned}]" + return 1 +} + +dpp_run_harness_plugin_smoke() { + local script="$1" + local label="$2" + if [[ -x "${script}" ]]; then + echo "" + echo " Harness smoke: ${label}..." + if "${script}" 2>/dev/null; then + pp_record_pass "harness ${label} smoke" + else + pp_record_fail "harness ${label} smoke" + fi + fi +} + +dpp_optional_harness_triggers() { + local docker_dir="$1" + dpp_run_harness_plugin_smoke "${docker_dir}/scripts/kms/trigger-kms-plugin-audit-e2e.sh" "KMS" + dpp_run_harness_plugin_smoke "${docker_dir}/scripts/kafka/trigger-kafka-plugin-audit-e2e.sh" "Kafka" + dpp_run_harness_plugin_smoke "${docker_dir}/scripts/knox/trigger-knox-plugin-audit-e2e.sh" "Knox" + dpp_run_harness_plugin_smoke "${docker_dir}/scripts/hbase/trigger-hbase-plugin-audit-e2e.sh" "HBase" +} diff --git a/dev-support/ranger-docker/scripts/audit/verify-dynamic-auth-to-local-e2e.sh b/dev-support/ranger-docker/scripts/audit/verify-dynamic-auth-to-local-e2e.sh new file mode 100755 index 000000000..1a46d70b4 --- /dev/null +++ b/dev-support/ranger-docker/scripts/audit/verify-dynamic-auth-to-local-e2e.sh @@ -0,0 +1,167 @@ +#!/bin/bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# E2E: dynamic service allowlist + auth_to_local + POST /api/audit/access per plugin. +# +# For each configured plugin repo: +# 1. POST /access with plugin Kerberos principal (SPNEGO) → 200 + authenticatedUser +# 2. Toggle allowlist via PUT partition-plan → 403 then 200 +# 3. Cross-repo denial (hdfs principal → dev_kms) → 403 +# +# Writes a curl cookbook: dist/audit-e2e/access-ingestor-curl-cookbook.sh +# +# Prerequisites: Tier 3 Docker (ingestor + Kafka + plugin containers with keytabs). +# +# Usage: +# cd dev-support/ranger-docker +# ./scripts/audit/verify-dynamic-auth-to-local-e2e.sh +# ./scripts/audit/verify-dynamic-auth-to-local-e2e.sh --no-enable --plugins hdfs,hive +# ./scripts/audit/verify-dynamic-auth-to-local-e2e.sh --generate-curl-only + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +cd "${SCRIPT_DIR}" + +# shellcheck source=partition-plan-e2e-lib.sh +source "${SCRIPT_DIR}/scripts/audit/partition-plan-e2e-lib.sh" +# shellcheck source=dynamic-auth-to-local-e2e-lib.sh +source "${SCRIPT_DIR}/scripts/audit/dynamic-auth-to-local-e2e-lib.sh" + +DO_ENABLE=true +GENERATE_ONLY=false +PLUGIN_FILTER="" +TIMEOUT=300 +COOKBOOK="${SCRIPT_DIR}/dist/audit-e2e/access-ingestor-curl-cookbook.sh" + +while [[ $# -gt 0 ]]; do + case "$1" in + --no-enable) DO_ENABLE=false; shift ;; + --generate-curl-only) GENERATE_ONLY=true; shift ;; + --plugins) PLUGIN_FILTER="${2:?}"; shift 2 ;; + --cookbook) COOKBOOK="${2:?}"; shift 2 ;; + --timeout) TIMEOUT="${2:?}"; shift 2 ;; + -h|--help) + sed -n '19,33p' "$0" + exit 0 + ;; + *) + echo "Unknown option: $1" >&2 + exit 1 + ;; + esac +done + +mkdir -p "$(dirname "${COOKBOOK}")" +dael_write_curl_cookbook "${COOKBOOK}" +chmod +x "${COOKBOOK}" 2>/dev/null || true + +if [[ "${GENERATE_ONLY}" == "true" ]]; then + echo "Curl cookbook only (no Docker tests): ${COOKBOOK}" + exit 0 +fi + +echo "=== Dynamic auth_to_local + allowlist E2E ===" +pp_preflight_tier3 "${TIMEOUT}" + +PLAN_URL="$(pp_ingestor_plan_url "${CONTAINER}")" + +if [[ "${DO_ENABLE}" == "true" ]]; then + if ! pp_dynamic_enabled "${CONTAINER}" "${SITE_XMLS[0]}"; then + echo "Enabling dynamic partition plan..." + pp_set_dynamic_enabled "${CONTAINER}" "true" || { pp_record_fail "enable dynamic"; pp_print_results; exit 1; } + pp_wait_health 7081 "Ingestor after enable" "${TIMEOUT}" || { pp_record_fail "health"; pp_print_results; exit 1; } + fi +fi + +if ! pp_dynamic_enabled "${CONTAINER}" "${SITE_XMLS[0]}"; then + pp_record_fail "dynamic.enabled must be true (use default or drop --no-enable)" + pp_print_results + exit 1 +fi + +if ! pp_wait_watcher "${CONTAINER}" "${TIMEOUT}"; then + pp_record_fail "PartitionPlanWatcher not ready" + pp_print_results + exit 1 +fi +pp_record_pass "PartitionPlanWatcher ready" + +pp_ingestor_request "${CONTAINER}" GET "${PLAN_URL}" +if [[ "${HTTP_CODE}" == "200" ]]; then + pp_record_pass "GET partition-plan returns 200" +else + pp_record_fail "GET partition-plan expected 200, got ${HTTP_CODE}" + pp_print_results + exit 1 +fi + +echo "" +echo "=== Per-plugin POST /access (auth_to_local + allowlist) ===" +profile="" +while IFS= read -r profile; do + [[ -n "${profile}" ]] || continue + dael_test_plugin_access "${profile}" || true +done < <(dael_filter_profiles "${PLUGIN_FILTER}") + +echo "" +echo "=== Dynamic allowlist toggle (dev_kms / rangerkms) ===" +if dael_container_running "ranger-kms"; then + dael_test_dynamic_user_toggle \ + "dev_kms" "rangerkms" "ranger-kms" \ + "/etc/keytabs/rangerkms.keytab" "rangerkms/[email protected]" "kms" || true +else + echo " SKIP dev_kms toggle: ranger-kms not running" +fi + +echo "" +echo "=== Cross-repo denial (hdfs principal → dev_kms) ===" +if dael_container_running "ranger-hadoop" && dael_container_running "ranger-kms"; then + dael_test_cross_repo_denied \ + "ranger-hadoop" "/etc/keytabs/hdfs.keytab" "hdfs/[email protected]" \ + "dev_kms" "kms" "hdfs principal rejected for dev_kms" || true +else + echo " SKIP cross-repo: ranger-hadoop or ranger-kms not running" +fi + +echo "" +echo "=== Onboard new repo dynamically (buffer plugin) ===" +pp_ingestor_request "${CONTAINER}" GET "${PLAN_URL}" +version="$(pp_json_field "${HTTP_BODY}" version)" +new_repo="e2e_auth_$(date +%s)" +new_plugin="e2eAuth$(date +%s)" +if dael_onboard_repo "${new_repo}" "${new_plugin}" "hdfs" 2 "${version}"; then + pp_record_pass "onboard-repo ${new_repo} plugin=${new_plugin}" + if dael_wait_auth_to_local_applied "${CONTAINER}" 30; then + pp_record_pass "auth_to_local recomposed after onboard-repo" + else + pp_record_fail "no auth_to_local log after onboard-repo" + fi + if dael_container_running "ranger-hadoop"; then + dael_access_post_from_container \ + "ranger-hadoop" "/etc/keytabs/hdfs.keytab" "hdfs/[email protected]" \ + "${new_repo}" "${new_plugin}" + dael_expect_access "POST /access to onboarded ${new_repo}" "200" || true + fi +else + pp_record_fail "onboard-repo failed: HTTP ${HTTP_CODE} ${HTTP_BODY}" +fi + +echo "" +echo "Curl cookbook: ${COOKBOOK}" +echo "See: audit-server/README-AUDIT-INGESTOR-ACCESS-CURL-E2E.md" + +pp_print_results diff --git a/dev-support/ranger-docker/scripts/audit/verify-dynamic-partition-plugin-e2e.sh b/dev-support/ranger-docker/scripts/audit/verify-dynamic-partition-plugin-e2e.sh new file mode 100755 index 000000000..cd6be5a13 --- /dev/null +++ b/dev-support/ranger-docker/scripts/audit/verify-dynamic-partition-plugin-e2e.sh @@ -0,0 +1,138 @@ +#!/bin/bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# E2E: dynamic partition allocation — onboard-repo REST per plugin + Kafka routing. +# +# Flow (mirrors ranger-audit-e2e-harness plugin matrix): +# 1. Enable dynamic mode (empty configured.plugins → buffer-only bootstrap) +# 2. For each running plugin container: POST /partition-plan/onboard-repo +# (repo, pluginId, partitionCount, allowedUsers from auth_to_local short names) +# 3. POST /api/audit/access from plugin keytab → verify Kafka record partition ∈ plan +# 4. Optional: run harness trigger-* scripts when present (KMS, Kafka, Knox, HBase) +# +# Prerequisites: Tier 3 Docker (see audit-server/README-DYNAMIC-PARTITION-PLUGIN-E2E.md). +# Reference harness branch: ranger-audit-e2e-harness (plugin verify/trigger scripts). +# +# Usage: +# cd dev-support/ranger-docker +# ./scripts/audit/verify-dynamic-partition-plugin-e2e.sh +# ./scripts/audit/verify-dynamic-partition-plugin-e2e.sh --no-enable --plugins hdfs,kms +# ./scripts/audit/verify-dynamic-partition-plugin-e2e.sh --with-harness-triggers + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +cd "${SCRIPT_DIR}" + +# shellcheck source=partition-plan-e2e-lib.sh +source "${SCRIPT_DIR}/scripts/audit/partition-plan-e2e-lib.sh" +# shellcheck source=dynamic-auth-to-local-e2e-lib.sh +source "${SCRIPT_DIR}/scripts/audit/dynamic-auth-to-local-e2e-lib.sh" +# shellcheck source=dynamic-partition-plugin-e2e-lib.sh +source "${SCRIPT_DIR}/scripts/audit/dynamic-partition-plugin-e2e-lib.sh" + +DO_ENABLE=true +PLUGIN_FILTER="" +WITH_HARNESS=false +TIMEOUT=300 +FRESH_PLAN=false + +while [[ $# -gt 0 ]]; do + case "$1" in + --no-enable) DO_ENABLE=false; shift ;; + --plugins) PLUGIN_FILTER="${2:?}"; shift 2 ;; + --with-harness-triggers) WITH_HARNESS=true; shift ;; + --fresh-plan) FRESH_PLAN=true; shift ;; + --timeout) TIMEOUT="${2:?}"; shift 2 ;; + -h|--help) + sed -n '31,36p' "$0" + exit 0 + ;; + *) + echo "Unknown option: $1" >&2 + exit 1 + ;; + esac +done + +echo "=== Dynamic partition plugin onboard + routing E2E ===" +pp_preflight_tier3 "${TIMEOUT}" + +if [[ "${DO_ENABLE}" == "true" ]]; then + if ! pp_dynamic_enabled "${CONTAINER}" "${SITE_XMLS[0]}"; then + echo "Preparing audit topic for greenfield layout..." + pp_prepare_audit_topic_for_greenfield "${CONTAINER}" + if [[ "${FRESH_PLAN}" == "true" ]]; then + pp_delete_plan_topic + fi + echo "Enabling dynamic partition plan..." + pp_set_dynamic_enabled "${CONTAINER}" "true" || { pp_record_fail "enable dynamic"; pp_print_results; exit 1; } + pp_wait_health 7081 "Ingestor after enable" "${TIMEOUT}" || { pp_record_fail "health"; pp_print_results; exit 1; } + fi +fi + +if ! pp_dynamic_enabled "${CONTAINER}" "${SITE_XMLS[0]}"; then + pp_record_fail "dynamic.enabled must be true" + pp_print_results + exit 1 +fi + +if ! pp_wait_watcher "${CONTAINER}" "${TIMEOUT}"; then + pp_record_fail "PartitionPlanWatcher not ready" + pp_print_results + exit 1 +fi +pp_record_pass "PartitionPlanWatcher ready" + +PLAN_URL="$(pp_ingestor_plan_url "${CONTAINER}")" +pp_ingestor_request "${CONTAINER}" GET "${PLAN_URL}" +if [[ "${HTTP_CODE}" == "200" ]]; then + pp_record_pass "GET partition-plan returns 200 (v$(pp_json_field "${HTTP_BODY}" version))" +else + pp_record_fail "GET partition-plan expected 200, got ${HTTP_CODE}" + pp_print_results + exit 1 +fi + +echo "" +echo "=== Phase 1: onboard-repo per running plugin (REST) ===" +spec="" +while IFS= read -r spec; do + [[ -n "${spec}" ]] || continue + dpp_ensure_plugin_onboarded "${spec}" || true +done < <(dpp_filter_specs "${PLUGIN_FILTER}") + +echo "" +echo "=== Phase 2: POST /access + Kafka partition ∈ plan ===" +while IFS= read -r spec; do + [[ -n "${spec}" ]] || continue + IFS='|' read -r repo plugin_id _ _ container _ _ <<< "${spec}" + echo "" + echo "--- ${plugin_id} (${repo}) ---" + dpp_verify_plugin_partition_routing "${spec}" || true +done < <(dpp_filter_specs "${PLUGIN_FILTER}") + +if [[ "${WITH_HARNESS}" == "true" ]]; then + echo "" + echo "=== Phase 3: optional ranger-audit-e2e-harness plugin triggers ===" + dpp_optional_harness_triggers "${SCRIPT_DIR}" +fi + +echo "" +echo "See: audit-server/README-DYNAMIC-PARTITION-PLUGIN-E2E.md" +echo "Curl cookbook: dist/audit-e2e/access-ingestor-curl-cookbook.sh (run verify-dynamic-auth-to-local-e2e.sh --generate-curl-only)" + +pp_print_results diff --git a/dev-support/ranger-docker/scripts/audit/verify-partition-plan-e2e-all.sh b/dev-support/ranger-docker/scripts/audit/verify-partition-plan-e2e-all.sh index 46b7fa18e..e671e06db 100755 --- a/dev-support/ranger-docker/scripts/audit/verify-partition-plan-e2e-all.sh +++ b/dev-support/ranger-docker/scripts/audit/verify-partition-plan-e2e-all.sh @@ -21,6 +21,7 @@ # ./scripts/audit/verify-partition-plan-e2e-all.sh --skip-kafka-down # ./scripts/audit/verify-partition-plan-e2e-all.sh --with-audit-smoke # ./scripts/audit/verify-partition-plan-e2e-all.sh --with-auth-access +# ./scripts/audit/verify-partition-plan-e2e-all.sh --with-plugin-onboard set -euo pipefail @@ -29,6 +30,7 @@ cd "${SCRIPT_DIR}" SKIP_KAFKA_DOWN=false WITH_AUTH_ACCESS=false +WITH_PLUGIN_ONBOARD=false EXTRA_ARGS=() while [[ $# -gt 0 ]]; do @@ -36,6 +38,7 @@ while [[ $# -gt 0 ]]; do --skip-kafka-down) SKIP_KAFKA_DOWN=true; shift ;; --with-audit-smoke) EXTRA_ARGS+=(--with-audit-smoke); shift ;; --with-auth-access) WITH_AUTH_ACCESS=true; shift ;; + --with-plugin-onboard) WITH_PLUGIN_ONBOARD=true; shift ;; --timeout) EXTRA_ARGS+=(--timeout "${2:?}"); shift 2 ;; -h|--help) sed -n '19,23p' "$0" @@ -53,6 +56,7 @@ chmod +x scripts/audit/verify-partition-plan-e2e.sh \ scripts/audit/verify-partition-plan-brownfield-e2e.sh \ scripts/audit/verify-partition-plan-kafka-down-e2e.sh \ scripts/audit/verify-dynamic-auth-to-local-e2e.sh \ + scripts/audit/verify-dynamic-partition-plugin-e2e.sh \ scripts/audit/wait-for-audit-health.sh 2>/dev/null || true run_step() { @@ -90,5 +94,10 @@ if [[ "${WITH_AUTH_ACCESS}" == "true" ]]; then ./scripts/audit/verify-dynamic-auth-to-local-e2e.sh --no-enable fi +if [[ "${WITH_PLUGIN_ONBOARD}" == "true" ]]; then + run_step "Dynamic partition onboard-repo + Kafka routing E2E" \ + ./scripts/audit/verify-dynamic-partition-plugin-e2e.sh --no-enable +fi + echo "" echo "All partition-plan E2E suites passed." diff --git a/dev-support/ranger-docker/scripts/audit/wait-for-audit-health.sh b/dev-support/ranger-docker/scripts/audit/wait-for-audit-health.sh new file mode 100755 index 000000000..a527e9979 --- /dev/null +++ b/dev-support/ranger-docker/scripts/audit/wait-for-audit-health.sh @@ -0,0 +1,57 @@ +#!/bin/bash +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Wait for audit ingestor (and optional Tier 3 dependencies) to become healthy. +# +# Usage: +# ./scripts/audit/wait-for-audit-health.sh +# ./scripts/audit/wait-for-audit-health.sh --tier 3 --timeout 180 + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" + +# shellcheck source=audit-stack-lib.sh +source "${SCRIPT_DIR}/scripts/audit/audit-stack-lib.sh" + +TIER=3 +TIMEOUT=120 +INGESTOR_PORT=7081 + +while [[ $# -gt 0 ]]; do + case "$1" in + --tier) TIER="${2:?}"; shift 2 ;; + --timeout) TIMEOUT="${2:?}"; shift 2 ;; + --port) INGESTOR_PORT="${2:?}"; shift 2 ;; + -h|--help) + sed -n '19,22p' "$0" + exit 0 + ;; + *) + echo "Unknown option: $1" >&2 + exit 1 + ;; + esac +done + +audit_stack_wait_url "http://localhost:${INGESTOR_PORT}/api/audit/health"; "Audit ingestor" "${TIMEOUT}" +audit_stack_require_container "${AUDIT_INGESTOR_CONTAINER}" + +if [[ "${TIER}" -ge 3 ]]; then + audit_stack_require_container "${KAFKA_CONTAINER}" +fi + +echo "Audit stack health check passed (tier=${TIER})."
