capistrant opened a new pull request, #19975:
URL: https://github.com/apache/druid/pull/19975
### Description
#18802 added a compaction policy that prioritizes the most fragmented
intervals for compaction. This policy also adds minimum fragmentation
thresholds that allow an operator to willingly choose to skip compaction on
intervals who do not have certain amounts of uncompacted volume (by uncompacted
segment count and by aggregate uncompacted bytes).
These policy excluded intervals were not being reported in the compaction
progress, making it hard to understand overall compaction state for datasources
when using this policy. This PR aims to close that gap by explicitly reporting
on policy excluded intervals in the compaction stats reporting summary. That
way datasource owners and/or cluster operators can see if compaction policy
thresholds are preventing any of their data from reaching the configured
compaction state for their interval.
#### New Metrics
|`segment/policyExcluded/bytes`|Total bytes of this datasource that do not
match the auto compaction config but whose intervals were filtered out by the
compaction candidate search policy. Relaxing the policy thresholds makes these
intervals eligible for compaction again.|`dataSource`|Varies|
|`segment/policyExcluded/count`|Total number of segments of this datasource
that do not match the auto compaction config but whose intervals were filtered
out by the compaction candidate search policy.|`dataSource`|Varies|
|`interval/policyExcluded/count`|Total number of intervals of this
datasource that need compaction but were filtered out by the compaction
candidate search policy.|`dataSource`|Varies|
#### Updated web console visualization
These new statistics are folded into the web console reporting for
compaction progress
#### Release note
Properly report statistics on intervals who require compaction to be
compliant with their datasources configured compaction state, but are being
excluded by the configured cluster compaction policy (for example, intervals
with under 500 segments and require compaction but the most fragmented first
policy is blocking compaction by having a threshold of 1000 uncompacted
segments before executing a compaction run). This gives a true compaction
progress report to cluster operators and datasource owners.
##### Key changed/added classes in this PR
* `CompactionStatus`
* `CompactionConfigBasedJobTemplate`
<hr>
This PR has:
- [ ] been self-reviewed.
- [x] added documentation for new or modified features or behaviors.
- [x] a release note entry in the PR description.
- [ ] added Javadocs for most classes and all non-trivial methods. Linked
related entities via Javadoc links.
- [ ] added comments explaining the "why" and the intent of the code
wherever would not be obvious for an unfamiliar reader.
- [x] added unit tests or modified existing tests to cover new code paths,
ensuring the threshold for [code
coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md)
is met.
- [ ] added integration tests.
- [ ] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
