vtlim commented on code in PR #16681: URL: https://github.com/apache/druid/pull/16681#discussion_r1802001397
########## docs/data-management/automatic-compaction.md: ########## @@ -221,6 +223,133 @@ The following auto-compaction configuration compacts updates the `wikipedia` seg } ``` +## Auto-compaction using compaction supervisors + +:::info Experimental +Compaction supervisors are experimental. For production use, we recommend [Coordinator-based auto-compaction](#auto-compaction-using-coordinator-duties). +::: + +You can run automatic compaction using compaction supervisors on the Overlord rather than Coordinator duties. Compaction supervisors provide the following benefits over Coordinator duties: + +* Can use the supervisor framework to get information about the auto-compaction, such as status or state +* More easily suspend or resume compaction for a datasource +* Can use either the native compaction engine or the [MSQ task engine](#use-msq-for-auto-compaction) +* More reactive and submits tasks as soon as a compaction slot is available +* Tracked compaction task status to avoid re-compacting an interval repeatedly + + +To use compaction supervisors, set the following properties in your Overlord runtime properties: + * `druid.supervisor.compaction.enabled` to `true` so that compaction tasks can be run as a supervisor task + * `druid.supervisor.compaction.engine` to `msq` to specify the MSQ task engine as the compaction engine or to `native` to use the native engine. + +Compaction uses the same syntax as Coordinator-based auto-compaction with some differences. Specifically, you submit a supervisor spec with the `type` set to `autocompact` and the auto-compaction config in the `spec` to configure auto-compaction. + +For information about the syntax, see [automatic-compaction syntax](#auto-compaction-syntax). + +### Manage compaction supervisors with the web console + +To submit a supervisor spec for MSQ task engine automatic compaction, perform the following steps: + +1. In the web console, go to the **Supervisors** tab. +1. Click **...** > **Submit JSON supervisor**. +1. In the dialog, include the following: + - The type of supervisor spec by setting `"type": "autocompact"` + - The compaction configuration by adding it to the `spec` field Review Comment: Should this be included in the code block below? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
