capistrant opened a new issue, #19092: URL: https://github.com/apache/druid/issues/19092
#18939 added a new template that can be used with Compaction supervisors. It is experimental in nature, and will be experimental in Druid 37, but I'd like to scope out work needed to get it as close to production ready as possible for Druid 37 so that we can pull the experimental tag in Druid 38 ### Drop Native Compaction Support. This was recently completed in #19078. Rationale here is that MSQ engine is the desired future of compaction supervisors. Supporting two engines is extra overhead, can lead to confusion, and slows development of more powerful features. ### Refactor Rules Concept This warrants a sub issue perhaps. But with #19061 adding CLUSTERED BY support on expressions, it has exposed some gaps in the initial rule structure that is coupled closely with compaction state. The original proposal for #18939 was even more coupled than the version that got merged. In the merged version we have `ReindexingDataSchemaRule` which is more general and swallows up compaction state concepts for dims, metrics, query gran, etc. We are thinking we need to go further with rule refactoring. One thought is to create a rule that encompasses partitioning ideas like segment gran, partitions spec, etc. The tuning config rule stuff that is pretty static in nearly all cases would be moved outside of the rules and into the top level config for the template. This would be shared by all generated tasks. ### Documentation As the design has been in flux, we have not committed to any druid docs yet. Once the above two sections are finalized a new set of docs about this template + integration with the existing automatic compaction docs is needed. This will be an experimental feature in d37. The docs should make it easy to see the value of this template + guide an operator to onboard to it and evaluate it ### UI in the router https://github.com/apache/druid/pull/19051 is open for adding visualization of the timeline of search intervals and compaction configs generated by a ruleset for an existing supervisor I also think it would be cool to add a supervisor builder UI where you can visualize the timeline as you create/edit a supervisor spec instead of only when it already exists. ### More robust rule provider Right now the only rule provider is `inline` which works and is nice for many smaller use cases, but it becomes hard to reason about quickly. A rule provider in core that sources rules from something like druid-catalog feels like a more production ready long term approach -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
