[
https://issues.apache.org/jira/browse/RANGER-5655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramachandran Krishnan updated RANGER-5655:
------------------------------------------
Attachment: (was: Dynamic Ingestor Registry Guide (Ranger Audit
Ingestor).pdf)
> Implement dynamic unified ingestor registry for audit-ingestor: runtime Kafka
> partition routing and per-repo service allowlists via compacted topic + REST,
> without ingestor restarts. Feature flag default off.
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: RANGER-5655
> URL: https://issues.apache.org/jira/browse/RANGER-5655
> Project: Ranger
> Issue Type: Improvement
> Components: Ranger
> Reporter: Ramachandran Krishnan
> Assignee: Ramachandran Krishnan
> Priority: Major
> Fix For: 3.0.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Implement a dynamic unified ingestor registry for Ranger audit-ingestor so
> operators can change Kafka partition routing and per-repo service allowlists
> at runtime — without restarting ingestor pods.
> The registry is stored in a Kafka compacted topic
> ({{ranger_audit_partition_plan}}) and managed via REST. All ingestor replicas
> converge on the same versioned plan through {{PartitionPlanWatcher}};
> {{AuditPartitioner}} routes audit records on the hot path from in-memory
> state only.
> *Feature flag (default off):*
> {{ranger.audit.ingestor.kafka.partition.plan.dynamic.enabled=false}}
> *Pull request:* [GitHub PR #1032|https://github.com/apache/ranger/pull/1032]
> ----
> h2. Problem
> Today (static mode), audit-ingestor loads two kinds of configuration from XML
> at startup only:
> || Job || Question || Static behavior ||
> | Service allowlist | May this Kerberos principal POST audits for repo R? |
> {{ranger.audit.ingestor.service.<repo>.allowed.users}} in site XML |
> | Partition routing | After accept, which {{ranger_audits}} partition? |
> {{kafka.configured.plugins}} + per-plugin overrides in site XML |
> Changing either requires editing XML and restarting every ingestor replica.
> Contiguous-range static allocation can also reshuffle later plugins when an
> early plugin's partition count changes.
> *Goals:*
> * Onboard new plugins/repos and scale hot plugins without ingestor restart
> * Append-only partition growth (no reshuffle of existing plugin assignments)
> * One shared source of truth across all ingestor pods
> * No new infra (no Postgres / ZooKeeper for the registry)
> ----
> h2. Solution
> Introduce a unified partition plan document (versioned JSON) in Kafka topic
> {{ranger_audit_partition_plan}} (1 partition, compacted). One document holds:
> * {{plugins}} — dedicated partition IDs per plugin id (Kafka record key /
> agent id)
> * {{buffer}} — partition pool for not-yet-promoted plugins (sticky hash)
> * {{services}} — per-repo {{allowedUsers}} for {{POST /api/audit/access}}
> (optional {{pluginId}} for ownership)
> * {{topicPartitionCount}} — must match live {{ranger_audits}} partition count
> * {{version}} — optimistic locking for REST mutations
> {code}
> Control plane: GET/PATCH /api/audit/partition-plan → Kafka registry topic
> → PartitionPlanWatcher
> Data plane: POST /api/audit/access (Kerberos) → allowlist check
> → optional auto-onboard
>
> → AuditPartitioner → ranger_audits
> {code}
> Solr/HDFS dispatchers are unchanged; they consume all partitions of
> {{ranger_audits}}.
> h3. REST API (v2 — two public endpoints)
> Plan admin endpoints are *unauthenticated* (same as health/status):
> {{permitAll()}} in Spring Security and bypass in
> {{AuditDelegationTokenFilter}}. Audit ingestion remains Kerberos-protected on
> {{POST /api/audit/access}}.
> All mutations use optimistic locking via {{expectedVersion}} (from GET).
> Stale writes return *409* with the current plan body.
> || Method || Endpoint || Purpose ||
> | GET | {{/api/audit/partition-plan}} | Read current plan (503 when dynamic
> mode off) |
> | PATCH | {{/api/audit/partition-plan}} | Partial update via
> {{PartitionPlanReplacement}} JSON body |
> *PATCH body semantics* ({{PartitionPlanReplacement}}):
> * {{expectedVersion}} — required; stale → 409
> * {{plugins}} / {{services}} — merge *new keys only* (promote/onboard entries
> not already in plan)
> * {{pluginScales}} — map of {{pluginId → additionalPartitions}}; append-only
> tail growth
> * {{buffer}} / {{topicPartitionCount}} — optional; omitted fields inherited
> from current plan
> *Removed (superseded by PATCH + auto-onboard):*
> * {{POST /api/audit/partition-plan/plugins}}
> * {{PATCH /api/audit/partition-plan/plugins/\{pluginId\}}}
> h3. Auto-onboard on audit access
> When dynamic mode is enabled and a Kerberos-authenticated {{POST
> /api/audit/access}} includes {{appId}} for a plugin not yet in the plan:
> # Allowlist check passes for {{serviceName}} + authenticated principal
> # Ingestor promotes the plugin from buffer (default partition count from
> config) and upserts {{services[serviceName]}} in one plan version bump
> # Audit batch is then produced using the updated in-memory plan
> No separate onboard REST call is required for first-time plugin discovery.
> Operators can still pre-promote plugins and allowlists via PATCH.
> h3. Allowlist updates
> {{services}} entries in the registry drive {{POST /access}} authorization. In
> dynamic mode, {{AuthToLocalRuleComposer}} recomposes {{auth_to_local}} from
> the union of all {{services[*].allowedUsers}} whenever the plan changes.
> Removing or tightening a repo's allowlist causes *403* on subsequent access
> attempts without pod restart.
> ----
> h2. Key deliverables (PR #1032)
> * Kafka registry — {{KafkaPartitionPlanRegistry}}, bootstrap from XML on
> greenfield, brownfield pre-seed support
> * REST API — {{GET}} / {{PATCH}} {{/api/audit/partition-plan}} with
> {{PartitionPlanReplacement}} and {{expectedVersion}} (409 on stale)
> * Auto-onboard — {{PartitionPlanService.ensurePluginOnboarded()}} on {{POST
> /api/audit/access}} when {{appId}} is unknown
> * Dynamic partitioner — {{AuditPartitioner}} reads {{PartitionPlanHolder}};
> round-robin for promoted plugins; buffer sticky hash for unknown plugins;
> post-scale routing uses {{max(cluster, plan)}} when metadata lags
> * Unified allowlist — {{services}} map in same registry; per-repo POST checks
> use each repo's own allowlist
> * Topic grow — grow {{ranger_audits}} before registry write when
> promoting/scaling
> * Unit tests — partition-plan, allowlist, partitioner, and bootstrap coverage
> in {{audit-common}} + {{audit-ingestor}}
> *Out of scope for PR #1032 (follow-up / lab validation):* Docker E2E harness,
> ops runbooks, brownfield migration guide (tracked separately in dev-support /
> audit-server docs).
> ----
> h2. Configuration
> {code:xml}
> <property>
> <name>ranger.audit.ingestor.kafka.partition.plan.dynamic.enabled</name>
> <value>true</value>
> </property>
> <property>
> <name>ranger.audit.ingestor.kafka.partition.plan.topic</name>
> <value>ranger_audit_partition_plan</value>
> </property>
> {code}
> When dynamic mode is on and the registry topic is empty, the first ingestor
> pod bootstraps an initial plan from existing XML properties
> ({{kafka.configured.plugins}}, buffer/per-plugin counts, service allowlists).
> h3. Example plan JSON
> {code:json}
> {
> "topic": "ranger_audits",
> "version": 12,
> "topicPartitionCount": 48,
> "plugins": {
> "hdfs": { "partitions": [0, 1, 2, 3, 4, 5] },
> "hiveServer2": { "partitions": [6, 7, 8, 9, 10, 11] }
> },
> "buffer": { "partitions": [12, 13, "..."] },
> "services": {
> "dev_hive": { "allowedUsers": ["hive"], "pluginId": "hiveServer2" },
> "dev_ozone": { "allowedUsers": ["om", "ozone"], "pluginId": "ozone" },
> "dev_hdfs": { "allowedUsers": ["hdfs", "nn"], "pluginId": "hdfs" }
> }
> }
> {code}
> h3. Example PATCH (promote + scale)
> {code:json}
> {
> "expectedVersion": 12,
> "plugins": {
> "storm": { "partitions": [12, 13, 14] }
> },
> "services": {
> "dev_storm": { "allowedUsers": ["storm"], "pluginId": "storm" }
> },
> "pluginScales": {
> "hdfs": 2
> }
> }
> {code}
> ----
> h2. Testing
> h3. Unit tests and quality gates
> {code:bash}
> mvn verify -pl audit-server/audit-common,audit-server/audit-ingestor
> -Drat.skip=true
> {code}
> || Gate || Result ||
> | Unit tests | 104 passed (audit-ingestor module) |
> | Checkstyle | Pass |
> | PMD / SpotBugs | Pass (existing audit-ingestor findings unchanged) |
> h3. Validated scenarios (manual + lab)
> || # || Scenario || Pass criteria ||
> | 1 | Static mode (default off) | {{GET /partition-plan}} → 503; health 200;
> plugin audits unchanged |
> | 2 | Dynamic bootstrap | Plan topic created; watcher active; {{GET
> /partition-plan}} → 200 with v≥1 |
> | 3 | Public plan REST | {{GET}} / {{PATCH}} without Kerberos succeed when
> dynamic mode on |
> | 4 | PATCH promote/onboard | New {{plugins}} + {{services}} keys merged in
> one version bump |
> | 5 | PATCH {{pluginScales}} | Tail partitions appended; {{ranger_audits}}
> grown; no reshuffle |
> | 6 | Optimistic locking | Stale {{expectedVersion}} → 409 with current plan
> body |
> | 7 | Auto-onboard on {{/access}} | Unknown {{appId}} + allowlisted principal
> → plan updated, audit accepted |
> | 8 | Multi-pod convergence | All replicas report same version after PATCH on
> one pod |
> | 9 | Routing | Kafka record partition ∈ plugin assignment from plan |
> | 10 | Allowlist enforcement | Allow/deny/cross-repo → 200/403; recomposition
> on plan change |
> ----
> h2. Acceptance criteria
> # With {{dynamic.enabled=false}}, behavior matches existing static XML
> partitioning; {{GET /partition-plan}} returns 503
> # With {{dynamic.enabled=true}}, registry topic created (1 partition,
> compacted); bootstrap plan published on greenfield
> # {{PATCH /api/audit/partition-plan}} promotes/scales/mutates allowlists;
> stale {{expectedVersion}} returns 409
> # Unknown {{appId}} on authenticated {{POST /access}} auto-onboards plugin
> when allowlist permits
> # All ingestor replicas converge to same plan within watcher refresh interval
> # Audits for promoted plugin land only on assigned {{ranger_audits}}
> partitions
> # Service allowlists in registry drive {{/access}} authorization;
> unauthorized principal returns 403
> # Post-scale routing works when Kafka metadata lags plan
> ({{AuditPartitioner}} bound logic)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
