suneet-s commented on a change in pull request #10935: URL: https://github.com/apache/druid/pull/10935#discussion_r592063449
########## File path: docs/ingestion/compaction.md ########## @@ -0,0 +1,210 @@ +--- +id: compaction +title: "Compaction" +description: "Defines compaction and automatic compaction (auto-compaction or autocompaction) as a strategy for segment optimization. Use cases for compaction. Describes compaction task configuration." +--- + +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> +Query performance in Apache Druid depends on optimally sized segments. Compaction is one strategy you can use to optimize segment size for your Druid database. Compaction tasks read an existing set of segments for a given time interval and combine the data into a new "compacted" set of segments. The compacted segments are generally larger, but there are fewer of them. Here compaction increases performance because fewer segments require less the per-segment processing and the memory overhead for ingestion and for querying paths. + +As a general strategy, compaction is effective when you have data arriving out of chronological order resulting in lots of small segments. This often happens, for example, if you are appending data using `appendToExisting` for [native batch](./native_batch.md) ingestion. Conversely, if you are rewriting your data with each ingestion task, you don't need to use compaction. + +In some cases you can use compaction to reduce segment size. For example, if a misconfigured ingestion task creates oversized segments, you can create a compaction task to split the segment files into smaller, more optimally sized ones. + +See [Segment optimization](../operations/segment-optimization.md) for guidance to determine if compaction will help in your environment. + + +## Types of segment compaction Review comment: why did you choose to call this `segment compaction` ? Don't have strong feelings right now, but just curious about the thought behind it ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
