maytasm commented on a change in pull request #12190:
URL: https://github.com/apache/druid/pull/12190#discussion_r791523166
##########
File path: docs/operations/clean-metadata-store.md
##########
@@ -119,7 +145,23 @@ Indexer task log cleanup on the Overlord uses the
following configuration:
For more detail, see [Task logging](../configuration/index.md#task-logging).
-## Automated metadata cleanup example configuration
+
+## Disable automated metadata cleanup
+
+Druid enables automated metadata cleanup by default. To disable this feature,
set the following properties in the `coordinator/runtime.properties` file:
+
+```
+druid.coordinator.kill.on=false
Review comment:
Might be good to add a comment beside each one indicating which
table/entity is the config for
##########
File path: docs/operations/clean-metadata-store.md
##########
@@ -23,20 +23,32 @@ description: "Defines a strategy to maintain Druid metadata
store performance by
~ specific language governing permissions and limitations
~ under the License.
-->
-When you delete some entities from Apache Druid, records related to the entity
may remain in the metadata store including:
-- segments records
-- audit records
-- supervisor records
-- rule records
-- compaction configuration records
-- datasource records created by supervisors
+Apache Druid relies on [metadata storage](../dependencies/metadata-storage.md)
to track information on data storage, operations, and system configuration.
+The metadata store includes the following:
-If you have a high datasource churn rate, meaning you frequently create and
delete many short-lived datasources or other related entities like compaction
configuration or rules, the leftover records can start to fill your metadata
store and cause performance issues.
+- Segment records
+- Audit records
+- Supervisor records
+- Rule records
+- Compaction configuration records
+- Datasource records created by supervisors
+- Indexer task logs
+
+When you delete some entities from Apache Druid, records related to the entity
may remain in the metadata store.
+If you have a high datasource churn rate, meaning you frequently create and
delete many short-lived datasources or other related entities like compaction
configuration or rules, the leftover records can fill your metadata store and
cause performance issues.
+To maintain metadata store performance, you can configure Apache Druid to
automatically remove records associated with deleted entities from the metadata
store.
+
+By default, Druid automatically cleans up metadata older than 90 days.
+This applies to all metadata entities in this topic except compaction
configuration records, which do not have a retention period, and indexer task
logs, for which cleanup is disabled by default.
Review comment:
Might be worth pointing out the behavior for compaction configuration
record here since the durationToRetain does not applies.
##########
File path: docs/dependencies/metadata-storage.md
##########
@@ -23,50 +23,70 @@ title: "Metadata storage"
-->
-The Metadata Storage is an external dependency of Apache Druid. Druid uses it
to store
-various metadata about the system, but not to store the actual data. There are
-a number of tables used for various purposes described below.
+Apache Druid relies on an external dependency for metadata storage.
+Druid uses the metadata store to house various metadata about the system, but
not to store the actual data.
+The metadata store retains all metadata essential for a Druid cluster to work.
+
+The metadata store includes the following:
+- Segments records
+- Rule records
+- Runtime configuration objects
Review comment:
What are Runtime configuration objects?
##########
File path: docs/operations/clean-metadata-store.md
##########
@@ -23,20 +23,32 @@ description: "Defines a strategy to maintain Druid metadata
store performance by
~ specific language governing permissions and limitations
~ under the License.
-->
-When you delete some entities from Apache Druid, records related to the entity
may remain in the metadata store including:
-- segments records
-- audit records
-- supervisor records
-- rule records
-- compaction configuration records
-- datasource records created by supervisors
+Apache Druid relies on [metadata storage](../dependencies/metadata-storage.md)
to track information on data storage, operations, and system configuration.
+The metadata store includes the following:
-If you have a high datasource churn rate, meaning you frequently create and
delete many short-lived datasources or other related entities like compaction
configuration or rules, the leftover records can start to fill your metadata
store and cause performance issues.
+- Segment records
+- Audit records
+- Supervisor records
+- Rule records
+- Compaction configuration records
+- Datasource records created by supervisors
+- Indexer task logs
+
+When you delete some entities from Apache Druid, records related to the entity
may remain in the metadata store.
+If you have a high datasource churn rate, meaning you frequently create and
delete many short-lived datasources or other related entities like compaction
configuration or rules, the leftover records can fill your metadata store and
cause performance issues.
+To maintain metadata store performance, you can configure Apache Druid to
automatically remove records associated with deleted entities from the metadata
store.
+
+By default, Druid automatically cleans up metadata older than 90 days.
+This applies to all metadata entities in this topic except compaction
configuration records, which do not have a retention period, and indexer task
logs, for which cleanup is disabled by default.
Review comment:
The behavior is that at every cycle of the metadata management tasks,
any compaction configuration of datasources that is inactive (datasource with
no used and unused segments) will be deleted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
