ramackri opened a new pull request, #1212: URL: https://github.com/apache/ranger/pull/1212
## Summary [RANGER-5786](https://issues.apache.org/jira/browse/RANGER-5786) — Reduce noisy Kafka access audits from the broker service user `kafka` on internal topics (`__consumer_offsets`, `ranger_audits`). The Kafka service-def already had a broad `{'users':['kafka'],'isAudited':false}` catch-all, but docker `dev_kafka` bootstrap and many upgraded clusters use customized `ranger.plugin.audit.filters` without explicit internal-topic skip rules. Broker describe traffic on those topics still shows up in Admin audit views (Solr/OpenSearch/HDFS) and drowns out real user access. This PR: - Adds an explicit internal-topic skip rule to the Kafka service-def default - Aligns docker `dev_kafka` bootstrap with that default - Adds Java patch **`PatchForKafkaServiceUserAuditFilters_J10067`** to merge missing RANGER-5786 rules into existing Kafka services on upgrade (append-only; preserves operator customizations) ## Changes | Area | File(s) | What | |------|---------|------| | Service-def default | `agents-common/.../ranger-servicedef-kafka.json` | Insert `{'resources':{'topic':{'values':['__consumer_offsets','ranger_audits']}},'users':['kafka'],'isAudited':false}` before the broad `kafka` user skip | | Docker bootstrap | `dev-support/ranger-docker/scripts/admin/create-ranger-services.py` | Same internal-topic rule for `dev_kafka` | | Upgrade patch | `security-admin/.../PatchForKafkaServiceUserAuditFilters_J10067.java`, `KafkaAuditFilterMergeUtil.java` | Merge-style Java patch (J10067): append missing rules; skip services already compliant | | DB version marker | All 5 `security-admin/db/*/optimized/current/ranger_core_db_*.sql` | Register `J10067` for fresh installs | | Unit tests | `TestKafkaServiceDefAuditFilters.java`, `TestPatchForKafkaServiceUserAuditFilters_J10067.java` | Service-def default + merge/patch logic | | Dev / E2E helpers | `verify-kafka-service-user-audit-filter.sh`, `update-kafka-service-audit-filter.sh` | REST verify + manual merge for existing stacks | | Docs | `dev-support/RANGER-5786-KAFKA-AUDIT-FILTER-PLAN.md` | Problem statement, filter order, upgrade table | ### Upgrade behavior (2.9 → 3.0) | Existing Kafka service state | J10067 action | |------------------------------|---------------| | Audit-filters key missing | Apply embedded 3.0 Kafka service-def default | | Custom filters, no `kafka` skip | Append internal-topic rule + broad `kafka` skip | | 2.9 broad `kafka` skip only | Append internal-topic rule only | | Already compliant | Skip (idempotent) | Patch runs during Admin setup/upgrade via `db_setup.py -javapatch` (same mechanism as other Java patches such as J10050). ## Testing ### Unit tests ```bash mvn -pl agents-common -Dtest=TestKafkaServiceDefAuditFilters test mvn -pl security-admin -Dtest=TestPatchForKafkaServiceUserAuditFilters_J10067 test ``` | Test | Result (local) | |------|----------------| | `TestKafkaServiceDefAuditFilters` | Pass | | `TestPatchForKafkaServiceUserAuditFilters_J10067` | Not run locally — `security-admin` compile blocked by unrelated `RoleDBStore.java` errors on this branch; patch logic covered by merge-util unit tests in the test class | ### Manual testing (docker, `dev-support/ranger-docker`) Stack: `docker compose -f docker-compose.ranger.yml` (+ kafka/audit compose files as needed). | Step | Command | Expected | |------|---------|----------| | Verify config | `./scripts/audit-e2e/verify-kafka-service-user-audit-filter.sh` | PASS — filters include `kafka` skip and internal topics | | Update existing service (optional) | `./scripts/audit-e2e/update-kafka-service-audit-filter.sh` | Appends missing rules via Admin REST if absent | **Manual result (Sep 2026, local docker):** `verify-kafka-service-user-audit-filter.sh` — **PASS** on `dev_kafka` after bootstrap. ### Upgrade-flow testing (J10067 patch) Simulated production upgrade on a live docker Admin + PostgreSQL stack (`ranger` @ `localhost:6080`): 1. Deployed compiled `PatchForKafkaServiceUserAuditFilters_J10067` + `KafkaAuditFilterMergeUtil` into the running admin container 2. Reset `dev_kafka` to pre-upgrade audit filters (custom rules, **no** `kafka` skip) 3. Ran patch main (same entry point as `db_setup.py -javapatch`): ```bash docker exec ranger /opt/java/openjdk/bin/java -Dlogdir=/var/log/ranger -Dlogback.configurationFile=file:/opt/ranger/ranger-3.0.0-SNAPSHOT-admin/ews/webapp/WEB-INF/classes/conf/logback.xml -cp <admin-classpath> org.apache.ranger.patch.PatchForKafkaServiceUserAuditFilters_J10067 ``` | Scenario | Before | After patch | Idempotent re-run | |----------|--------|-------------|-------------------| | Custom filters, no `kafka` skip | 4 rules | 5 rules — internal-topic `kafka` skip appended; originals preserved | `already has rules; skipped` | | 2.9 broad `kafka` skip only | 2 rules | 3 rules — internal-topic rule appended | skipped | Post-patch: `verify-kafka-service-user-audit-filter.sh` — **PASS**. Log excerpt (`/var/log/ranger/ranger_db_patch.log`): ``` Merging kafka skip rules into dev_kafka Kafka service dev_kafka already has rules;skipped # on re-run ``` ## Test plan - [x] `mvn -pl agents-common -Dtest=TestKafkaServiceDefAuditFilters test` - [ ] `mvn -pl security-admin -Dtest=TestPatchForKafkaServiceUserAuditFilters_J10067 test` (CI) - [x] Docker manual verify: `verify-kafka-service-user-audit-filter.sh` - [x] Upgrade-flow: J10067 patch main on pre-upgrade `dev_kafka` (custom + 2.9 scenarios) - [ ] CI: `build-17`, checkstyle/PMD (after push) Made with [Cursor](https://cursor.com) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
