kfaraz commented on code in PR #15218: URL: https://github.com/apache/druid/pull/15218#discussion_r1372603881
########## docs/data-management/automatic-compaction.md: ########## @@ -203,6 +203,106 @@ The following auto-compaction configuration compacts updates the `wikipedia` seg } ``` +## Concurrent append and replace + +:::info +Concurrent append and replace is an [experimental feature](../development/experimental.md) and is not currently available for SQL-based ingestion. +::: + +This feature allows you to safely replace the existing data in an interval of a datasource while new data is being appended to that interval. One of the most common applications of this is appending new data (using say streaming ingestion) to an interval while compaction of that interval is already in progress. + +To set up concurrent append and replace, you need to ensure that your ingestion jobs have the appropriate lock types: + +You can enable concurrent append and replace by ensuring the following: +- The append task (with `appendToExisting` set to `true`) has `taskLockType` set to `APPEND` in the task context. +- The replace task (with `appendToExisting` set to `false`) has `taskLockType` set to `REPLACE` in the task context. +- The segment granularity of the append task is equal to or finer than the segment granularity of the replace task. + +:::info + +When using concurrent append and replace, keep the following in mind: + +- When the job with an `APPEND` task has a coarser granularity than the job with a `REPLACE` task, you shouldn't use concurrent append and replace. For example, if the `APPEND` job has a yearly granularity and the `REPLACE` job has a monthly granularity. The job that finishes second will fail. Review Comment: ```suggestion - Concurrent append and replace fails if the task with `APPEND` lock uses a coarser segment granularity than the task with the `REPLACE` lock. For example, if the `APPEND` task uses a segment granularity of YEAR and the `REPLACE` task uses a segment granularity of MONTH, you should not use concurrent append and replace. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
