capistrant opened a new pull request, #19633: URL: https://github.com/apache/druid/pull/19633
### Description Cascading Reindexing introduced deletion rules that allow for deleting rows that match a filter after some period of time. This has many applications, but one in specific is the ability to do row granular deletions automatically for data compliance needs. For example, if your Druid deploy contains customer data with a column `customer_id` identifying what customer a row of data is associated with, you may have different compliance requirements for different customers. For instance, keep customer `foo` data for 30 days and customer `bar` data for 365 days. With cascading reindexing deletion rules you have a rule for each of these requirements that will properly lifecycle your data and keep you in compliance. Currently there is pain when it comes to compaction candidacy. An operator can put minimum un-compacted thresholds on intervals to avoid over-compacting when the gain from such compactions is negligible. From a pure cost perspective, this is great. Don't spend compute dollars for negligible space savings or performance gains. However, this breaks down from a compliance perspective when considering my leading example for customer data retention. The only option an operator has today if they need to meet compliance is to drop those thresholds down to essentially 0 and run compaction overly-aggressive even if the compaction does not apply any missing deletion rules. So they are eating cost to cover a limited set of cases. This PR introduces a new policy config for compaction that an operator can opt into to force compaction to run on an interval if any of the candidate segments are out of sync with applying deletion rules. This allows keeping the thresholds for compaction at the desired level knowing that that decision will not put you into compliance issues when it comes to data lifecycle. Compliance is just one reason someone may want to enable this config. It is the most compelling that comes to mind because it can be a legal requirement for operators to meet, and the current option to lower thresholds to be compliant is sub-optimal. `forcePendingDeletionCompaction` is the new policy config that defaults to false, but anyone wishing to force pending deletion rules to be applied regardless of interval compaction stats, can simply set it to true and leave their thresholds for non-deletion required candidates in place. #### Release note <!-- Give your best effort to summarize your changes in a couple of sentences aimed toward Druid users. If your change doesn't have end user impact, you can skip this section. For tips about how to write a good release note, see [Release notes](https://github.com/apache/druid/blob/master/CONTRIBUTING.md#release-notes). --> Operators using Cascading Reindexing compaction supervisors with deletion rules can now use compaction policy configuration to opt into their deletion rules being applied to candidate intervals who are not fully compliant regardless of their interval level compaction stats. If you opt in, intervals that required deletions to be applied but did not meet compaction stat thresholds specified in the policy will now be compacted in order to ensure all required deletion rules are applied. Critical for operators who use deletion rules for data compliance needs. Note that you must be using `MostFragmentedIntervalFirst` policy to have this config take effect. <hr> ##### Key changed/added classes in this PR * `CompactionConfigBasedJobTemplate` * `ReindexingDeletionRuleOptimizer` * `CompactionCandidateSearchPolicy` * `MostFragmentedIntervalFirstPolicy` <hr> <!-- Check the items by putting "x" in the brackets for the done things. Not all of these items apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items from the checklist below are strictly necessary, but it would be very helpful if you at least self-review the PR. --> This PR has: - [ ] been self-reviewed. - [ ] added documentation for new or modified features or behaviors. - [ ] a release note entry in the PR description. - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links. - [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md) - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met. - [x] added integration tests. - [ ] been tested in a test Druid cluster. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
