[
https://issues.apache.org/jira/browse/RANGER-5680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Kumar updated RANGER-5680:
-----------------------------------
Description:
Merged via: [PR #1167|https://github.com/apache/ranger/pull/1167]
h3. Summary
Consolidated the Ranger docker audit pipeline into
{{{}docker-compose.ranger-audit-service.yml{}}}. *OpenSearch* (official
{{{}opensearchproject/opensearch:3.7{}}}) is the default audit index store;
*Solr* remains available as an alternative. Solr/OpenSearch are only started
via the audit service, not as standalone stacks.
h3. Key changes
* Single audit compose file: ingestor, Kafka, index store (Solr or
OpenSearch), and matching dispatcher. Removed legacy
audit-ingestor/dispatcher/solr/opensearch compose files.
* {{AUDIT_INDEX_STORE}} ({{{}solr{}}} | {{{}opensearch{}}}): selects the index
store; {{ranger.sh}} sets Ranger Admin {{audit_store}} at setup (no manual
{{install.properties}} edit).
* Compose profiles: {{{}audit-store-solr{}}}, {{{}audit-store-opensearch{}}},
{{{}audit-store-hdfs{}}}; activated via {{{}--profile
$\{AUDIT_DESTINATIONS{}}}} on the CLI.
* Optional HDFS fan-out: {{--profile audit-store-hdfs}} with
{{docker-compose.ranger-audit-destination-hdfs.yml}} (avoids {{ranger-hadoop}}
merge conflicts in full hive/hbase stacks).
* Kafka readiness: broker reachable and Ranger plugin policy cache ready
before ingestor/dispatcher start. Dispatchers depend on ingestor healthy (until
RANGER-5752 is resolved).
* CI updated to build/bring up audit services with OpenSearch by default.
h3. Usage
*Core stack:*
{code:bash}
export RANGER_DB_TYPE=postgres
export AUDIT_INDEX_STORE=opensearch
export AUDIT_DESTINATIONS=audit-store-${AUDIT_INDEX_STORE}
docker compose --profile ${AUDIT_DESTINATIONS} \
-f docker-compose.ranger.yml \
-f docker-compose.ranger-audit-service.yml up -d
{code}
*HDFS fan-out* (use separate {{-{-}profile{-}}} flags; commas are not split in
a single {{{}-profile{}}}):
{code:bash}
docker compose \
--profile audit-store-${AUDIT_INDEX_STORE} \
--profile audit-store-hdfs \
-f docker-compose.ranger.yml \
-f docker-compose.ranger-audit-service.yml \
-f docker-compose.ranger-audit-destination-hdfs.yml up -d
{code}
h3. Verification
* CI passing
* Manual bring-up with OpenSearch and Solr
* Audits verified flowing to OpenSearch / Solr / HDFS
was:
Background
In the Ranger docker dev environment, the database flavor is cleanly selected
via the RANGER_DB_TYPE environment variable, which drives three points: the
Dockerfile.ranger multi-stage build (FROM ranger_${RANGER_DB_TYPE}), the
install.properties volume mount, and the compose extends: service:
${RANGER_DB_TYPE}. There is no equivalent mechanism for the audit store.
As a result, Solr is effectively mandatory: ranger-solr is defined inside
docker-compose.ranger.yml and is a hard depends_on of the ranger service, and
is also hard-depended-on by ranger-pdp and ozone-om. Choosing OpenSearch
instead requires hand-editing install.properties, and Solr still starts
regardless.
Scope
- Introduce a RANGER_AUDIT_STORE environment variable (e.g. solr | opensearch),
mirroring the RANGER_DB_TYPE pattern, to select the active audit destination
without editing property files by hand.
- Extract the ranger-solr service definition out of docker-compose.ranger.yml
into its own docker-compose.ranger-solr.yml (mirroring the existing
docker-compose.ranger-opensearch.yml pattern).
- Remove the hard depends_on: ranger-solr from the ranger, ranger-pdp, and
ozone-om service definitions so Ranger can start without Solr.
- Ensure the selected audit-store env var drives which install.properties
block/mount is active, so the docker stack comes up with the correct audit
backend end-to-end.
- Update the docker README to document export RANGER_AUDIT_STORE=opensearch
alongside export RANGER_DB_TYPE=....
Acceptance criteria
- Ranger admin, PDP, and Ozone stacks come up successfully with
RANGER_AUDIT_STORE=opensearch and no Solr container running.
- Existing Solr-based flows continue to work unchanged (RANGER_AUDIT_STORE=solr
remains the default).
- Regression-verified across DB flavors (postgres/mysql/oracle/sqlserver) and
dependent stacks (pdp, ozone, kms, audit-server).
> Restructure audit services in docker compose and use OpenSearch as default
> Index Store
> --------------------------------------------------------------------------------------
>
> Key: RANGER-5680
> URL: https://issues.apache.org/jira/browse/RANGER-5680
> Project: Ranger
> Issue Type: Improvement
> Components: audit
> Reporter: Paras
> Assignee: Abhishek Kumar
> Priority: Major
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> Merged via: [PR #1167|https://github.com/apache/ranger/pull/1167]
> h3. Summary
> Consolidated the Ranger docker audit pipeline into
> {{{}docker-compose.ranger-audit-service.yml{}}}. *OpenSearch* (official
> {{{}opensearchproject/opensearch:3.7{}}}) is the default audit index store;
> *Solr* remains available as an alternative. Solr/OpenSearch are only started
> via the audit service, not as standalone stacks.
> h3. Key changes
> * Single audit compose file: ingestor, Kafka, index store (Solr or
> OpenSearch), and matching dispatcher. Removed legacy
> audit-ingestor/dispatcher/solr/opensearch compose files.
> * {{AUDIT_INDEX_STORE}} ({{{}solr{}}} | {{{}opensearch{}}}): selects the
> index store; {{ranger.sh}} sets Ranger Admin {{audit_store}} at setup (no
> manual {{install.properties}} edit).
> * Compose profiles: {{{}audit-store-solr{}}},
> {{{}audit-store-opensearch{}}}, {{{}audit-store-hdfs{}}}; activated via
> {{{}--profile $\{AUDIT_DESTINATIONS{}}}} on the CLI.
> * Optional HDFS fan-out: {{--profile audit-store-hdfs}} with
> {{docker-compose.ranger-audit-destination-hdfs.yml}} (avoids
> {{ranger-hadoop}} merge conflicts in full hive/hbase stacks).
> * Kafka readiness: broker reachable and Ranger plugin policy cache ready
> before ingestor/dispatcher start. Dispatchers depend on ingestor healthy
> (until RANGER-5752 is resolved).
> * CI updated to build/bring up audit services with OpenSearch by default.
> h3. Usage
> *Core stack:*
> {code:bash}
> export RANGER_DB_TYPE=postgres
> export AUDIT_INDEX_STORE=opensearch
> export AUDIT_DESTINATIONS=audit-store-${AUDIT_INDEX_STORE}
> docker compose --profile ${AUDIT_DESTINATIONS} \
> -f docker-compose.ranger.yml \
> -f docker-compose.ranger-audit-service.yml up -d
> {code}
> *HDFS fan-out* (use separate {{-{-}profile{-}}} flags; commas are not split
> in a single {{{}-profile{}}}):
> {code:bash}
> docker compose \
> --profile audit-store-${AUDIT_INDEX_STORE} \
> --profile audit-store-hdfs \
> -f docker-compose.ranger.yml \
> -f docker-compose.ranger-audit-service.yml \
> -f docker-compose.ranger-audit-destination-hdfs.yml up -d
> {code}
> h3. Verification
> * CI passing
> * Manual bring-up with OpenSearch and Solr
> * Audits verified flowing to OpenSearch / Solr / HDFS
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
