paras200 opened a new pull request, #986:
URL: https://github.com/apache/ranger/pull/986
Adds a new OpenSearch dispatcher module to the Ranger Audit Server that
consumes audit events from Kafka and bulk-indexes them into OpenSearch,
providing an alternative to the Solr-based audit store.
Core dispatcher module
(audit-server/audit-dispatcher/dispatcher-opensearch):
- OpenSearchDispatcherManager — lifecycle manager with retry-based
initialization (exponential backoff, max 5 attempts) and graceful shutdown
- AuditOpenSearchDispatcher — Kafka consumer that batches audit events and
writes them to OpenSearch via the _bulk API using the low-level RestClient
- Supports basic auth and Kerberos/SPNEGO authentication for OpenSearch
connections
- Document ID deduplication — uses audit.eventId as _id in bulk metadata,
falls back to UUID when absent
- Error handling with partition seek-back and retry sleep on batch failures
Shared mapping (audit-server/audit-dispatcher/dispatcher-common):
- AuditEventDocMapper — canonical 27-field event-to-document mapper,
reusable across dispatcher destinations
Configuration & packaging (distro):
- Per-dispatcher logback support (logback-opensearch.xml) in
start-audit-dispatcher.sh
- Assembly descriptor updated to package the opensearch dispatcher module
Docker & E2E infrastructure (dev-support/ranger-docker):
- docker-compose.ranger-audit-dispatcher-opensearch.yml for the dispatcher
container
- KDC healthcheck + ZK depends_on: service_healthy to fix keytab
provisioning race condition
- e2e-audit-opensearch.sh — single-command end-to-end test script (start →
validate → teardown)
- Helper scripts: create-ranger-audit-topic.sh,
create-ranger-audit-index.sh
Cleanup:
- Make ElasticSearchMgr fields private with proper accessors
How was this patch tested?
Unit tests:
- TestAuditOpenSearchDispatcher (6 tests) — validates bulk request
formatting, document field mapping, HTTP error handling, item-level error
detection, UUID generation for missing event IDs
- TestOpenSearchDispatcherManager (5 tests) — validates dispatcher type
filtering, disabled destination handling, fail-fast when dispatcher class
cannot be instantiated
- TestAuditEventDocMapper — validates all 27 fields are correctly mapped
from AuthzAuditEvent to document
End-to-end test (./scripts/audit/e2e-audit-opensearch.sh):
- Full Docker stack: KDC → ZK → Kafka → Ranger Admin → Audit Ingestor →
OpenSearch → OpenSearch Dispatcher
- Posts a SPNEGO-authenticated audit event through the ingestor REST API
- Verifies the document is indexed in OpenSearch with the correct _id
(marker-based assertion)
- Validates all service health endpoints and container states
- Automated teardown on exit (or --no-teardown for debugging)
Pipeline validated: Plugin → Ingestor → Kafka → Dispatcher → OpenSearch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
