freesinger opened a new pull request, #11343:
URL: https://github.com/apache/gravitino/pull/11343
Switch the default audit formatter to structured JSON so SIEM consumers can
parse audit events reliably, while preserving compatibility through the
existing formatter configuration. Also add coverage for JSON serialization,
redaction, and default formatter wiring.
<!--
1. Title: [#<issue>] <type>(<scope>): <subject>
Examples:
- "[#123] feat(operator): Support xxx"
- "[#233] fix: Check null before access result in xxx"
- "[MINOR] refactor: Fix typo in variable name"
- "[MINOR] docs: Fix typo in README"
- "[#255] test: Fix flaky test NameOfTheTest"
Reference: https://www.conventionalcommits.org/en/v1.0.0/
2. If the PR is unfinished, please mark this PR as draft.
-->
### What changes were proposed in this pull request?
This PR adds a structured JSON formatter for audit logs and switches it to
be the default audit formatter.
The main changes are:
1. Add `JsonAuditFormatter` to serialize each audit log entry as one JSON
object per line.
2. Include all core audit fields in the JSON output, including structured
`customInfo`.
3. Format `timestamp` in ISO 8601 with millisecond precision and an explicit
timezone offset.
4. Redact sensitive values before serialization for the following keys:
- `Authorization`
- `Cookie`
- `X-Amz-Security-Token`
- `s3.access-key-id`
- `jdbc-password`
5. Emit `resultCount` as a top-level JSON field for `ListEvent` when the
count is available.
6. Change the default audit formatter from `SimpleFormatterV2` to
`JsonAuditFormatter` through the existing formatter configuration.
7. Update the Helm config template and server configuration documentation to
reflect the new default formatter.
8. Add unit tests for JSON serialization, redaction, null identifier
handling, list event count output, and default formatter wiring.
### Why are the changes needed?
The existing audit formatters mainly emit tab-separated text, which is
harder for SIEM systems and other downstream log processors to consume reliably.
This change is needed because:
1. Structured JSON is easier for SIEM systems to parse and index than TSV
output.
2. `AuditLog.customInfo()` should be preserved in a structured way instead
of being difficult to parse downstream.
3. Audit logs may include HTTP headers or credential-related properties, so
sensitive values must be masked before being written.
4. The default formatter should move to a more production-friendly
structured format while still allowing operators to switch back through
`gravitino.audit.formatter.className` if they need legacy behavior.
Impl: #11171
### Does this PR introduce _any_ user-facing change?
Yes.
1. The default audit log output format changes from TSV-style text to
structured JSON.
2. The default value of `gravitino.audit.formatter.className` is now
`org.apache.gravitino.audit.JsonAuditFormatter`.
3. Audit logs now expose `customInfo` as structured JSON content instead of
relying on TSV-compatible string formatting.
4. Sensitive values for specific headers and properties are redacted in the
JSON audit output.
5. Users can still switch back to legacy formatter implementations through
the existing formatter configuration.
### How was this patch tested?
The patch was tested with targeted unit tests covering the new formatter and
the default formatter wiring.
Executed test command:
```bash
./gradlew :core:test --tests
org.apache.gravitino.audit.TestJsonAuditFormatter --tests
org.apache.gravitino.audit.TestAuditManager --tests
org.apache.gravitino.audit.TestFileAuditWriter
```
The tests cover:
1. Core JSON field serialization.
2. ISO 8601 timestamp serialization with millisecond precision.
3. Sensitive field redaction.
4. `ListEvent` `resultCount` serialization.
5. Null identifier handling.
6. Default audit formatter wiring through `AuditLogManager`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
