vtlim commented on code in PR #15760: URL: https://github.com/apache/druid/pull/15760#discussion_r1466983008
########## docs/data-management/automatic-compaction.md: ########## @@ -134,13 +134,28 @@ For more details on each of the specs in an auto-compaction configuration, see [ ### Avoid conflicts with ingestion Compaction tasks may be interrupted when they interfere with ingestion. For example, this occurs when an ingestion task needs to write data to a segment for a time interval locked for compaction. If there are continuous failures that prevent compaction from making progress, consider one of the following strategies: + +* Enable [concurrent append and replace tasks](#enable-concurrent-append-and-replace) on your datasource and on the ingestion tasks. * Set `skipOffsetFromLatest` to reduce the chance of conflicts between ingestion and compaction. See more details in this section below. Review Comment: ```suggestion * Set `skipOffsetFromLatest` to reduce the chance of conflicts between ingestion and compaction. See more details in [Skip latest segments from compaction](#skip-latest-segments-from-compaction). ``` ########## docs/ingestion/concurrent-append-replace.md: ########## @@ -0,0 +1,145 @@ +--- +id: concurrent-append-replace +title: Concurrent append and replace +--- + +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +:::info +Concurrent append and replace is an [experimental feature](../development/experimental.md) available for JSON-based batch and streaming. It is not currently available for SQL-based ingestion. +::: + +This feature allows you to safely replace the existing data in an interval of a datasource while new data is being appended to that interval. One of the most common applications of this is appending new data (using say streaming ingestion) to an interval while compaction of that interval is already in progress. + +To set up concurrent append and replace, you need to use the context flag `useConcurrentLocks`. Druid will then determine the correct lock type for you, either append or replace. Although can set the type of lock manually, we don't recommend it. + +## Update the compaction settings + +If want to append data to a datasource while compaction is running, you need to enable concurrent append and replace for the datasource by updating the compaction settings. + +### Update the compaction settings with the UI + +In the **Compaction config** for a datasource, enable **Allow concurrent compactions (experimental)**. + +### Update the compaction settings with the API + +Add the `taskContext` like you would any other automatic compaction setting through the API: + +```shell +curl --location --request POST 'http://localhost:8081/druid/coordinator/v1/config/compaction' \ +--header 'Content-Type: application/json' \ +--data-raw '{ + "dataSource": "YOUR_DATASOURCE", + "taskContext": { + "useConcurrentLocks": "true" + } +}' +``` + +## Add a task lock to your ingestion job + +You also need to configure the ingestion job to allow concurrent tasks. + +You can provide the context parameter through the API like any other parameter for ingestion job or through the UI. + +### Add a task lock using the Druid console + +As part of the **Load data** wizard for classic batch (JSON-based ingestion) and streaming ingestion, enable the following config on the **Publish** step: **Allow concurrent tasks (experimental)**. + +### Add the task lock through the API + +Add the following JSON snippet to your supervisor or ingestion spec if you're using the API: + +```json +"context": { + "useConcurrentLocks": true +} +``` + + +## Task lock types + +We recommend that you use the `useConcurrentLocks` context parameter so that Druid automatically determines the task lock types for you. If, for some reason, you need to manually set the task lock types explicitly, you can read more about them in this section. + +<details><summary>Click here to read more about the lock types.</summary> + +When setting task lock types manually, you need to ensure the following: + +- The append task (with `appendToExisting` set to `true`) has `taskLockType` set to `APPEND` in the task context. +- The replace task (with `appendToExisting` set to `false`) has `taskLockType` set to `REPLACE` in the task context. +- The segment granularity of the append task is equal to or finer than the segment granularity of the replace task. + +Additionally, keep the following in mind: + +- Concurrent append and replace fails if the task with `APPEND` lock uses a coarser segment granularity than the task with the `REPLACE` lock. For example, if the `APPEND` task uses a segment granularity of YEAR and the `REPLACE` task uses a segment granularity of MONTH, you should not use concurrent append and replace. + +- Only a single task can hold a `REPLACE` lock on a given interval of a datasource. + +- Multiple tasks can hold `APPEND` locks on a given interval of a datasource and append data to that interval simultaneously. + +#### Add a task lock type to your ingestion job + +Next, you need to configure the task lock type for your ingestion job: Review Comment: Not sure where "next" comes in, maybe artifact of refactoring ```suggestion You configure the task lock type for your ingestion job as follows: ``` ########## docs/ingestion/concurrent-append-replace.md: ########## @@ -0,0 +1,145 @@ +--- +id: concurrent-append-replace +title: Concurrent append and replace +--- + +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +:::info +Concurrent append and replace is an [experimental feature](../development/experimental.md) available for JSON-based batch and streaming. It is not currently available for SQL-based ingestion. +::: + +This feature allows you to safely replace the existing data in an interval of a datasource while new data is being appended to that interval. One of the most common applications of this is appending new data (using say streaming ingestion) to an interval while compaction of that interval is already in progress. + +To set up concurrent append and replace, you need to use the context flag `useConcurrentLocks`. Druid will then determine the correct lock type for you, either append or replace. Although can set the type of lock manually, we don't recommend it. Review Comment: Another thing I'm not certain on is what part of the feature is concurrent append and replace. Do you need to set `useConcurrentLocks` to enable concurrent and replace (what's currently documented), or is it just a flag to help manage concurrent append and replace? The second sentence kind of makes it sound like you can still use concurrent append/replace but with manual task locks. ########## docs/ingestion/concurrent-append-replace.md: ########## @@ -0,0 +1,145 @@ +--- +id: concurrent-append-replace +title: Concurrent append and replace +--- + +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +:::info +Concurrent append and replace is an [experimental feature](../development/experimental.md) available for JSON-based batch and streaming. It is not currently available for SQL-based ingestion. +::: + +This feature allows you to safely replace the existing data in an interval of a datasource while new data is being appended to that interval. One of the most common applications of this is appending new data (using say streaming ingestion) to an interval while compaction of that interval is already in progress. + +To set up concurrent append and replace, you need to use the context flag `useConcurrentLocks`. Druid will then determine the correct lock type for you, either append or replace. Although can set the type of lock manually, we don't recommend it. Review Comment: ```suggestion To set up concurrent append and replace, use the context flag `useConcurrentLocks`. Druid will then determine the correct lock type for you, either append or replace. Although you can set the type of lock manually, we don't recommend it. ``` ########## docs/data-management/automatic-compaction.md: ########## @@ -134,13 +134,28 @@ For more details on each of the specs in an auto-compaction configuration, see [ ### Avoid conflicts with ingestion Compaction tasks may be interrupted when they interfere with ingestion. For example, this occurs when an ingestion task needs to write data to a segment for a time interval locked for compaction. If there are continuous failures that prevent compaction from making progress, consider one of the following strategies: + +* Enable [concurrent append and replace tasks](#enable-concurrent-append-and-replace) on your datasource and on the ingestion tasks. * Set `skipOffsetFromLatest` to reduce the chance of conflicts between ingestion and compaction. See more details in this section below. * Increase the priority value of compaction tasks relative to ingestion tasks. Only recommended for advanced users. This approach can cause ingestion jobs to fail or lag. To change the priority of compaction tasks, set `taskPriority` to the desired priority value in the auto-compaction configuration. For details on the priority values of different task types, see [Lock priority](../ingestion/tasks.md#lock-priority). The Coordinator compacts segments from newest to oldest. In the auto-compaction configuration, you can set a time period, relative to the end time of the most recent segment, for segments that should not be compacted. Assign this value to `skipOffsetFromLatest`. Note that this offset is not relative to the current time but to the latest segment time. For example, if you want to skip over segments from five days prior to the end time of the most recent segment, assign `"skipOffsetFromLatest": "P5D"`. To set `skipOffsetFromLatest`, consider how frequently you expect the stream to receive late arriving data. If your stream only occasionally receives late arriving data, the auto-compaction system robustly compacts your data even though data is ingested outside the `skipOffsetFromLatest` window. For most realtime streaming ingestion use cases, it is reasonable to set `skipOffsetFromLatest` to a few hours or a day. +### Enable concurrent append and replace + +You can use concurrent append and replace to safely replace the existing data in an interval of a datasource while new data is being appended to that interval even during compaction. + +To do this, you need to update your datasource to allow concurrent append and replace tasks: + +* If you're using the API, include the following `taskContext` in your API call: `"useConcurrentLocks": "true"` Review Comment: ```suggestion * If you're using the API, include the following `taskContext` property in your API call: `"useConcurrentLocks": "true"` ``` ########## docs/ingestion/concurrent-append-replace.md: ########## @@ -0,0 +1,145 @@ +--- +id: concurrent-append-replace +title: Concurrent append and replace +--- + +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +:::info +Concurrent append and replace is an [experimental feature](../development/experimental.md) available for JSON-based batch and streaming. It is not currently available for SQL-based ingestion. +::: + +This feature allows you to safely replace the existing data in an interval of a datasource while new data is being appended to that interval. One of the most common applications of this is appending new data (using say streaming ingestion) to an interval while compaction of that interval is already in progress. Review Comment: ```suggestion Concurrent append and replace safely replaces the existing data in an interval of a datasource while new data is being appended to that interval. One of the most common applications of this feature is appending new data (using say streaming ingestion) to an interval while compaction of that interval is already in progress. ``` * Suggest starting with feature name instead of relying on admonition to introduce it * Add a noun after demonstrative pronoun per [style guide](https://developers.google.com/style/pronouns#ambiguous-pronoun-references) ########## docs/data-management/automatic-compaction.md: ########## @@ -134,13 +134,28 @@ For more details on each of the specs in an auto-compaction configuration, see [ ### Avoid conflicts with ingestion Compaction tasks may be interrupted when they interfere with ingestion. For example, this occurs when an ingestion task needs to write data to a segment for a time interval locked for compaction. If there are continuous failures that prevent compaction from making progress, consider one of the following strategies: + +* Enable [concurrent append and replace tasks](#enable-concurrent-append-and-replace) on your datasource and on the ingestion tasks. * Set `skipOffsetFromLatest` to reduce the chance of conflicts between ingestion and compaction. See more details in this section below. * Increase the priority value of compaction tasks relative to ingestion tasks. Only recommended for advanced users. This approach can cause ingestion jobs to fail or lag. To change the priority of compaction tasks, set `taskPriority` to the desired priority value in the auto-compaction configuration. For details on the priority values of different task types, see [Lock priority](../ingestion/tasks.md#lock-priority). Review Comment: ```suggestion ### Skip latest segments from compaction ``` ########## docs/ingestion/concurrent-append-replace.md: ########## @@ -0,0 +1,145 @@ +--- +id: concurrent-append-replace +title: Concurrent append and replace +--- + +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +:::info +Concurrent append and replace is an [experimental feature](../development/experimental.md) available for JSON-based batch and streaming. It is not currently available for SQL-based ingestion. +::: + +This feature allows you to safely replace the existing data in an interval of a datasource while new data is being appended to that interval. One of the most common applications of this is appending new data (using say streaming ingestion) to an interval while compaction of that interval is already in progress. + +To set up concurrent append and replace, you need to use the context flag `useConcurrentLocks`. Druid will then determine the correct lock type for you, either append or replace. Although can set the type of lock manually, we don't recommend it. + +## Update the compaction settings + +If want to append data to a datasource while compaction is running, you need to enable concurrent append and replace for the datasource by updating the compaction settings. Review Comment: ```suggestion If you want to append data to a datasource while compaction is running, you need to enable concurrent append and replace for the datasource by updating the compaction settings. ``` ########## docs/ingestion/concurrent-append-replace.md: ########## @@ -0,0 +1,145 @@ +--- +id: concurrent-append-replace +title: Concurrent append and replace +--- + +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +:::info +Concurrent append and replace is an [experimental feature](../development/experimental.md) available for JSON-based batch and streaming. It is not currently available for SQL-based ingestion. +::: + +This feature allows you to safely replace the existing data in an interval of a datasource while new data is being appended to that interval. One of the most common applications of this is appending new data (using say streaming ingestion) to an interval while compaction of that interval is already in progress. + +To set up concurrent append and replace, you need to use the context flag `useConcurrentLocks`. Druid will then determine the correct lock type for you, either append or replace. Although can set the type of lock manually, we don't recommend it. + +## Update the compaction settings + +If want to append data to a datasource while compaction is running, you need to enable concurrent append and replace for the datasource by updating the compaction settings. + +### Update the compaction settings with the UI + +In the **Compaction config** for a datasource, enable **Allow concurrent compactions (experimental)**. Review Comment: ```suggestion In the **Compaction config** for a datasource, enable **Allow concurrent compactions (experimental)**. For details on accessing the compaction config in the UI, see [Enable automatic compaction with the web console](automatic-compaction.md#web-console). ``` ########## docs/data-management/automatic-compaction.md: ########## @@ -134,13 +134,28 @@ For more details on each of the specs in an auto-compaction configuration, see [ ### Avoid conflicts with ingestion Review Comment: What do you think about promoting this to H2, and having the following structure? * Configure automatic compaction * Set frequency of compaction runs * Avoid conflicts with ingestion * Enable concurrent append and replace * Skip latest segments from compaction ########## docs/ingestion/concurrent-append-replace.md: ########## @@ -0,0 +1,145 @@ +--- +id: concurrent-append-replace +title: Concurrent append and replace +--- + +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +:::info +Concurrent append and replace is an [experimental feature](../development/experimental.md) available for JSON-based batch and streaming. It is not currently available for SQL-based ingestion. +::: + +This feature allows you to safely replace the existing data in an interval of a datasource while new data is being appended to that interval. One of the most common applications of this is appending new data (using say streaming ingestion) to an interval while compaction of that interval is already in progress. + +To set up concurrent append and replace, you need to use the context flag `useConcurrentLocks`. Druid will then determine the correct lock type for you, either append or replace. Although can set the type of lock manually, we don't recommend it. + +## Update the compaction settings + +If want to append data to a datasource while compaction is running, you need to enable concurrent append and replace for the datasource by updating the compaction settings. + +### Update the compaction settings with the UI + +In the **Compaction config** for a datasource, enable **Allow concurrent compactions (experimental)**. + +### Update the compaction settings with the API + +Add the `taskContext` like you would any other automatic compaction setting through the API: + +```shell +curl --location --request POST 'http://localhost:8081/druid/coordinator/v1/config/compaction' \ +--header 'Content-Type: application/json' \ +--data-raw '{ + "dataSource": "YOUR_DATASOURCE", + "taskContext": { + "useConcurrentLocks": "true" + } +}' +``` + +## Add a task lock to your ingestion job + +You also need to configure the ingestion job to allow concurrent tasks. + +You can provide the context parameter through the API like any other parameter for ingestion job or through the UI. Review Comment: ```suggestion You can provide the context parameter like any other parameter for ingestion jobs through the API or the UI. ``` ########## docs/ingestion/concurrent-append-replace.md: ########## @@ -0,0 +1,145 @@ +--- +id: concurrent-append-replace +title: Concurrent append and replace +--- + +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +:::info +Concurrent append and replace is an [experimental feature](../development/experimental.md) available for JSON-based batch and streaming. It is not currently available for SQL-based ingestion. +::: + +This feature allows you to safely replace the existing data in an interval of a datasource while new data is being appended to that interval. One of the most common applications of this is appending new data (using say streaming ingestion) to an interval while compaction of that interval is already in progress. + +To set up concurrent append and replace, you need to use the context flag `useConcurrentLocks`. Druid will then determine the correct lock type for you, either append or replace. Although can set the type of lock manually, we don't recommend it. Review Comment: Would it be worth including a description of how the lock types fit into this feature? For example, >Druid uses lock types to manage data consistency when two tasks are working on the same segment. The concurrent append and replace feature automatically determines the correct lock type to use when ingesting and replacing data at the same time. Something like the above, but not sure if my understanding is correct. ########## docs/ingestion/concurrent-append-replace.md: ########## @@ -0,0 +1,145 @@ +--- +id: concurrent-append-replace +title: Concurrent append and replace +--- + +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +:::info +Concurrent append and replace is an [experimental feature](../development/experimental.md) available for JSON-based batch and streaming. It is not currently available for SQL-based ingestion. +::: + +This feature allows you to safely replace the existing data in an interval of a datasource while new data is being appended to that interval. One of the most common applications of this is appending new data (using say streaming ingestion) to an interval while compaction of that interval is already in progress. + +To set up concurrent append and replace, you need to use the context flag `useConcurrentLocks`. Druid will then determine the correct lock type for you, either append or replace. Although can set the type of lock manually, we don't recommend it. + +## Update the compaction settings + +If want to append data to a datasource while compaction is running, you need to enable concurrent append and replace for the datasource by updating the compaction settings. + +### Update the compaction settings with the UI + +In the **Compaction config** for a datasource, enable **Allow concurrent compactions (experimental)**. + +### Update the compaction settings with the API + +Add the `taskContext` like you would any other automatic compaction setting through the API: + +```shell +curl --location --request POST 'http://localhost:8081/druid/coordinator/v1/config/compaction' \ +--header 'Content-Type: application/json' \ +--data-raw '{ + "dataSource": "YOUR_DATASOURCE", + "taskContext": { + "useConcurrentLocks": "true" + } +}' +``` + +## Add a task lock to your ingestion job + +You also need to configure the ingestion job to allow concurrent tasks. + +You can provide the context parameter through the API like any other parameter for ingestion job or through the UI. + +### Add a task lock using the Druid console + +As part of the **Load data** wizard for classic batch (JSON-based ingestion) and streaming ingestion, enable the following config on the **Publish** step: **Allow concurrent tasks (experimental)**. + +### Add the task lock through the API + +Add the following JSON snippet to your supervisor or ingestion spec if you're using the API: + +```json +"context": { + "useConcurrentLocks": true +} +``` + + +## Task lock types + +We recommend that you use the `useConcurrentLocks` context parameter so that Druid automatically determines the task lock types for you. If, for some reason, you need to manually set the task lock types explicitly, you can read more about them in this section. + +<details><summary>Click here to read more about the lock types.</summary> + +When setting task lock types manually, you need to ensure the following: + +- The append task (with `appendToExisting` set to `true`) has `taskLockType` set to `APPEND` in the task context. +- The replace task (with `appendToExisting` set to `false`) has `taskLockType` set to `REPLACE` in the task context. +- The segment granularity of the append task is equal to or finer than the segment granularity of the replace task. + +Additionally, keep the following in mind: + +- Concurrent append and replace fails if the task with `APPEND` lock uses a coarser segment granularity than the task with the `REPLACE` lock. For example, if the `APPEND` task uses a segment granularity of YEAR and the `REPLACE` task uses a segment granularity of MONTH, you should not use concurrent append and replace. + +- Only a single task can hold a `REPLACE` lock on a given interval of a datasource. + +- Multiple tasks can hold `APPEND` locks on a given interval of a datasource and append data to that interval simultaneously. + +#### Add a task lock type to your ingestion job + +Next, you need to configure the task lock type for your ingestion job: + +- For streaming jobs, the context parameter goes in your supervisor spec, and the lock type is always `APPEND` Review Comment: ```suggestion - For streaming jobs, the `taskLockType` context parameter goes in your supervisor spec, and the lock type is always `APPEND`. ``` ########## docs/ingestion/concurrent-append-replace.md: ########## @@ -0,0 +1,145 @@ +--- +id: concurrent-append-replace +title: Concurrent append and replace +--- + +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +:::info +Concurrent append and replace is an [experimental feature](../development/experimental.md) available for JSON-based batch and streaming. It is not currently available for SQL-based ingestion. +::: + +This feature allows you to safely replace the existing data in an interval of a datasource while new data is being appended to that interval. One of the most common applications of this is appending new data (using say streaming ingestion) to an interval while compaction of that interval is already in progress. + +To set up concurrent append and replace, you need to use the context flag `useConcurrentLocks`. Druid will then determine the correct lock type for you, either append or replace. Although can set the type of lock manually, we don't recommend it. + +## Update the compaction settings + +If want to append data to a datasource while compaction is running, you need to enable concurrent append and replace for the datasource by updating the compaction settings. + +### Update the compaction settings with the UI + +In the **Compaction config** for a datasource, enable **Allow concurrent compactions (experimental)**. + +### Update the compaction settings with the API + +Add the `taskContext` like you would any other automatic compaction setting through the API: + +```shell +curl --location --request POST 'http://localhost:8081/druid/coordinator/v1/config/compaction' \ +--header 'Content-Type: application/json' \ +--data-raw '{ + "dataSource": "YOUR_DATASOURCE", + "taskContext": { + "useConcurrentLocks": "true" + } +}' +``` + +## Add a task lock to your ingestion job + +You also need to configure the ingestion job to allow concurrent tasks. + +You can provide the context parameter through the API like any other parameter for ingestion job or through the UI. + +### Add a task lock using the Druid console + +As part of the **Load data** wizard for classic batch (JSON-based ingestion) and streaming ingestion, enable the following config on the **Publish** step: **Allow concurrent tasks (experimental)**. + +### Add the task lock through the API + +Add the following JSON snippet to your supervisor or ingestion spec if you're using the API: + +```json +"context": { + "useConcurrentLocks": true +} +``` + + +## Task lock types + +We recommend that you use the `useConcurrentLocks` context parameter so that Druid automatically determines the task lock types for you. If, for some reason, you need to manually set the task lock types explicitly, you can read more about them in this section. + +<details><summary>Click here to read more about the lock types.</summary> + +When setting task lock types manually, you need to ensure the following: + +- The append task (with `appendToExisting` set to `true`) has `taskLockType` set to `APPEND` in the task context. +- The replace task (with `appendToExisting` set to `false`) has `taskLockType` set to `REPLACE` in the task context. +- The segment granularity of the append task is equal to or finer than the segment granularity of the replace task. + +Additionally, keep the following in mind: + +- Concurrent append and replace fails if the task with `APPEND` lock uses a coarser segment granularity than the task with the `REPLACE` lock. For example, if the `APPEND` task uses a segment granularity of YEAR and the `REPLACE` task uses a segment granularity of MONTH, you should not use concurrent append and replace. Review Comment: ```suggestion - Concurrent append and replace fails if the task with the `APPEND` lock uses a coarser segment granularity than the task with the `REPLACE` lock. For example, if the `APPEND` task uses a segment granularity of YEAR and the `REPLACE` task uses a segment granularity of MONTH, you should not use concurrent append and replace. ``` ########## docs/ingestion/concurrent-append-replace.md: ########## @@ -0,0 +1,145 @@ +--- +id: concurrent-append-replace +title: Concurrent append and replace +--- + +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +:::info +Concurrent append and replace is an [experimental feature](../development/experimental.md) available for JSON-based batch and streaming. It is not currently available for SQL-based ingestion. +::: + +This feature allows you to safely replace the existing data in an interval of a datasource while new data is being appended to that interval. One of the most common applications of this is appending new data (using say streaming ingestion) to an interval while compaction of that interval is already in progress. + +To set up concurrent append and replace, you need to use the context flag `useConcurrentLocks`. Druid will then determine the correct lock type for you, either append or replace. Although can set the type of lock manually, we don't recommend it. + +## Update the compaction settings + +If want to append data to a datasource while compaction is running, you need to enable concurrent append and replace for the datasource by updating the compaction settings. + +### Update the compaction settings with the UI + +In the **Compaction config** for a datasource, enable **Allow concurrent compactions (experimental)**. + +### Update the compaction settings with the API + +Add the `taskContext` like you would any other automatic compaction setting through the API: + +```shell +curl --location --request POST 'http://localhost:8081/druid/coordinator/v1/config/compaction' \ +--header 'Content-Type: application/json' \ +--data-raw '{ + "dataSource": "YOUR_DATASOURCE", + "taskContext": { + "useConcurrentLocks": "true" + } +}' +``` + +## Add a task lock to your ingestion job + +You also need to configure the ingestion job to allow concurrent tasks. + +You can provide the context parameter through the API like any other parameter for ingestion job or through the UI. + +### Add a task lock using the Druid console + +As part of the **Load data** wizard for classic batch (JSON-based ingestion) and streaming ingestion, enable the following config on the **Publish** step: **Allow concurrent tasks (experimental)**. + +### Add the task lock through the API + +Add the following JSON snippet to your supervisor or ingestion spec if you're using the API: + +```json +"context": { + "useConcurrentLocks": true +} Review Comment: ```suggestion } ``` ########## docs/ingestion/concurrent-append-replace.md: ########## @@ -0,0 +1,145 @@ +--- +id: concurrent-append-replace +title: Concurrent append and replace +--- + +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +:::info +Concurrent append and replace is an [experimental feature](../development/experimental.md) available for JSON-based batch and streaming. It is not currently available for SQL-based ingestion. +::: + +This feature allows you to safely replace the existing data in an interval of a datasource while new data is being appended to that interval. One of the most common applications of this is appending new data (using say streaming ingestion) to an interval while compaction of that interval is already in progress. + +To set up concurrent append and replace, you need to use the context flag `useConcurrentLocks`. Druid will then determine the correct lock type for you, either append or replace. Although can set the type of lock manually, we don't recommend it. + +## Update the compaction settings + +If want to append data to a datasource while compaction is running, you need to enable concurrent append and replace for the datasource by updating the compaction settings. + +### Update the compaction settings with the UI + +In the **Compaction config** for a datasource, enable **Allow concurrent compactions (experimental)**. + +### Update the compaction settings with the API + +Add the `taskContext` like you would any other automatic compaction setting through the API: + +```shell +curl --location --request POST 'http://localhost:8081/druid/coordinator/v1/config/compaction' \ +--header 'Content-Type: application/json' \ +--data-raw '{ + "dataSource": "YOUR_DATASOURCE", + "taskContext": { + "useConcurrentLocks": "true" + } +}' +``` + +## Add a task lock to your ingestion job + +You also need to configure the ingestion job to allow concurrent tasks. + +You can provide the context parameter through the API like any other parameter for ingestion job or through the UI. + +### Add a task lock using the Druid console + +As part of the **Load data** wizard for classic batch (JSON-based ingestion) and streaming ingestion, enable the following config on the **Publish** step: **Allow concurrent tasks (experimental)**. + +### Add the task lock through the API + +Add the following JSON snippet to your supervisor or ingestion spec if you're using the API: + +```json +"context": { + "useConcurrentLocks": true +} +``` + + +## Task lock types + +We recommend that you use the `useConcurrentLocks` context parameter so that Druid automatically determines the task lock types for you. If, for some reason, you need to manually set the task lock types explicitly, you can read more about them in this section. + +<details><summary>Click here to read more about the lock types.</summary> + +When setting task lock types manually, you need to ensure the following: + +- The append task (with `appendToExisting` set to `true`) has `taskLockType` set to `APPEND` in the task context. +- The replace task (with `appendToExisting` set to `false`) has `taskLockType` set to `REPLACE` in the task context. +- The segment granularity of the append task is equal to or finer than the segment granularity of the replace task. + +Additionally, keep the following in mind: + +- Concurrent append and replace fails if the task with `APPEND` lock uses a coarser segment granularity than the task with the `REPLACE` lock. For example, if the `APPEND` task uses a segment granularity of YEAR and the `REPLACE` task uses a segment granularity of MONTH, you should not use concurrent append and replace. + +- Only a single task can hold a `REPLACE` lock on a given interval of a datasource. + +- Multiple tasks can hold `APPEND` locks on a given interval of a datasource and append data to that interval simultaneously. + +#### Add a task lock type to your ingestion job + +Next, you need to configure the task lock type for your ingestion job: + +- For streaming jobs, the context parameter goes in your supervisor spec, and the lock type is always `APPEND` +- For legacy JSON-based batch ingestion, the context parameter goes in your ingestion spec, and the lock type can be either `APPEND` or `REPLACE`. Review Comment: ```suggestion - For classic JSON-based batch ingestion, the `taskLockType` context parameter goes in your ingestion spec, and the lock type can be either `APPEND` or `REPLACE`. ``` ########## docs/ingestion/native-batch.md: ########## @@ -95,6 +95,9 @@ The `maxNumConcurrentSubTasks` in the `tuningConfig` determines the number of co By default, JSON-based batch ingestion replaces all data in the intervals in your `granularitySpec` for any segment that it writes to. If you want to add to the segment instead, set the `appendToExisting` flag in the `ioConfig`. JSON-based batch ingestion only replaces data in segments where it actively adds data. If there are segments in the intervals for your `granularitySpec` that don't have data from a task, they remain unchanged. If any existing segments partially overlap with the intervals in the `granularitySpec`, the portion of those segments outside the interval for the new spec remain visible. +You can also perform concurrent appends and replaces. For more information, see [Concurrent append and replace](./concurrent-append-replace.md) Review Comment: ```suggestion You can also perform concurrent append and replace tasks. For more information, see [Concurrent append and replace](./concurrent-append-replace.md) ``` ########## docs/ingestion/concurrent-append-replace.md: ########## @@ -0,0 +1,145 @@ +--- +id: concurrent-append-replace +title: Concurrent append and replace +--- + +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +:::info +Concurrent append and replace is an [experimental feature](../development/experimental.md) available for JSON-based batch and streaming. It is not currently available for SQL-based ingestion. +::: + +This feature allows you to safely replace the existing data in an interval of a datasource while new data is being appended to that interval. One of the most common applications of this is appending new data (using say streaming ingestion) to an interval while compaction of that interval is already in progress. + +To set up concurrent append and replace, you need to use the context flag `useConcurrentLocks`. Druid will then determine the correct lock type for you, either append or replace. Although can set the type of lock manually, we don't recommend it. + +## Update the compaction settings + +If want to append data to a datasource while compaction is running, you need to enable concurrent append and replace for the datasource by updating the compaction settings. + +### Update the compaction settings with the UI + +In the **Compaction config** for a datasource, enable **Allow concurrent compactions (experimental)**. + +### Update the compaction settings with the API + +Add the `taskContext` like you would any other automatic compaction setting through the API: + +```shell +curl --location --request POST 'http://localhost:8081/druid/coordinator/v1/config/compaction' \ +--header 'Content-Type: application/json' \ +--data-raw '{ + "dataSource": "YOUR_DATASOURCE", + "taskContext": { + "useConcurrentLocks": "true" + } +}' +``` + +## Add a task lock to your ingestion job + +You also need to configure the ingestion job to allow concurrent tasks. + +You can provide the context parameter through the API like any other parameter for ingestion job or through the UI. + +### Add a task lock using the Druid console + +As part of the **Load data** wizard for classic batch (JSON-based ingestion) and streaming ingestion, enable the following config on the **Publish** step: **Allow concurrent tasks (experimental)**. + +### Add the task lock through the API + +Add the following JSON snippet to your supervisor or ingestion spec if you're using the API: + +```json +"context": { + "useConcurrentLocks": true +} +``` + + +## Task lock types + +We recommend that you use the `useConcurrentLocks` context parameter so that Druid automatically determines the task lock types for you. If, for some reason, you need to manually set the task lock types explicitly, you can read more about them in this section. + +<details><summary>Click here to read more about the lock types.</summary> + +When setting task lock types manually, you need to ensure the following: + +- The append task (with `appendToExisting` set to `true`) has `taskLockType` set to `APPEND` in the task context. +- The replace task (with `appendToExisting` set to `false`) has `taskLockType` set to `REPLACE` in the task context. +- The segment granularity of the append task is equal to or finer than the segment granularity of the replace task. + +Additionally, keep the following in mind: + +- Concurrent append and replace fails if the task with `APPEND` lock uses a coarser segment granularity than the task with the `REPLACE` lock. For example, if the `APPEND` task uses a segment granularity of YEAR and the `REPLACE` task uses a segment granularity of MONTH, you should not use concurrent append and replace. + +- Only a single task can hold a `REPLACE` lock on a given interval of a datasource. + +- Multiple tasks can hold `APPEND` locks on a given interval of a datasource and append data to that interval simultaneously. + +#### Add a task lock type to your ingestion job + +Next, you need to configure the task lock type for your ingestion job: + +- For streaming jobs, the context parameter goes in your supervisor spec, and the lock type is always `APPEND` +- For legacy JSON-based batch ingestion, the context parameter goes in your ingestion spec, and the lock type can be either `APPEND` or `REPLACE`. + +You can provide the context parameter through the API like any other parameter for ingestion job or through the UI. + +##### Add a task lock using the Druid console + +As part of the **Load data** wizard for classic batch (JSON-based ingestion) and streaming ingestion, you can configure the task lock type for the ingestion during the **Publish** step: + +- If you set **Append to existing** to **True**, you can then set **Allow concurrent append tasks (experimental)** to **True**. +- If you set **Append to existing** to **False**, you can then set **Allow concurrent replace tasks (experimental)** to **True**. + +##### Add the task lock type through the API + +Add the following JSON snippet to your supervisor or ingestion spec if you're using the API: + +```json +"context": { + "taskLockType": LOCK_TYPE +} +``` + +The `LOCK_TYPE` depends on what you're trying to accomplish. Review Comment: This content provides some intro to what APPEND and REPLACE task locks are for and would make more sense towards the start of the `<details>` section ########## docs/ingestion/concurrent-append-replace.md: ########## @@ -0,0 +1,145 @@ +--- +id: concurrent-append-replace +title: Concurrent append and replace +--- + +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +:::info +Concurrent append and replace is an [experimental feature](../development/experimental.md) available for JSON-based batch and streaming. It is not currently available for SQL-based ingestion. +::: + +This feature allows you to safely replace the existing data in an interval of a datasource while new data is being appended to that interval. One of the most common applications of this is appending new data (using say streaming ingestion) to an interval while compaction of that interval is already in progress. + +To set up concurrent append and replace, you need to use the context flag `useConcurrentLocks`. Druid will then determine the correct lock type for you, either append or replace. Although can set the type of lock manually, we don't recommend it. + +## Update the compaction settings + +If want to append data to a datasource while compaction is running, you need to enable concurrent append and replace for the datasource by updating the compaction settings. + +### Update the compaction settings with the UI + +In the **Compaction config** for a datasource, enable **Allow concurrent compactions (experimental)**. + +### Update the compaction settings with the API + +Add the `taskContext` like you would any other automatic compaction setting through the API: + +```shell +curl --location --request POST 'http://localhost:8081/druid/coordinator/v1/config/compaction' \ +--header 'Content-Type: application/json' \ +--data-raw '{ + "dataSource": "YOUR_DATASOURCE", + "taskContext": { + "useConcurrentLocks": "true" + } +}' +``` + +## Add a task lock to your ingestion job + +You also need to configure the ingestion job to allow concurrent tasks. + +You can provide the context parameter through the API like any other parameter for ingestion job or through the UI. + +### Add a task lock using the Druid console + +As part of the **Load data** wizard for classic batch (JSON-based ingestion) and streaming ingestion, enable the following config on the **Publish** step: **Allow concurrent tasks (experimental)**. + +### Add the task lock through the API + +Add the following JSON snippet to your supervisor or ingestion spec if you're using the API: + +```json +"context": { + "useConcurrentLocks": true +} +``` + + +## Task lock types + +We recommend that you use the `useConcurrentLocks` context parameter so that Druid automatically determines the task lock types for you. If, for some reason, you need to manually set the task lock types explicitly, you can read more about them in this section. + +<details><summary>Click here to read more about the lock types.</summary> + +When setting task lock types manually, you need to ensure the following: + +- The append task (with `appendToExisting` set to `true`) has `taskLockType` set to `APPEND` in the task context. +- The replace task (with `appendToExisting` set to `false`) has `taskLockType` set to `REPLACE` in the task context. +- The segment granularity of the append task is equal to or finer than the segment granularity of the replace task. + +Additionally, keep the following in mind: + +- Concurrent append and replace fails if the task with `APPEND` lock uses a coarser segment granularity than the task with the `REPLACE` lock. For example, if the `APPEND` task uses a segment granularity of YEAR and the `REPLACE` task uses a segment granularity of MONTH, you should not use concurrent append and replace. + +- Only a single task can hold a `REPLACE` lock on a given interval of a datasource. + +- Multiple tasks can hold `APPEND` locks on a given interval of a datasource and append data to that interval simultaneously. + +#### Add a task lock type to your ingestion job + +Next, you need to configure the task lock type for your ingestion job: + +- For streaming jobs, the context parameter goes in your supervisor spec, and the lock type is always `APPEND` +- For legacy JSON-based batch ingestion, the context parameter goes in your ingestion spec, and the lock type can be either `APPEND` or `REPLACE`. + +You can provide the context parameter through the API like any other parameter for ingestion job or through the UI. + +##### Add a task lock using the Druid console + +As part of the **Load data** wizard for classic batch (JSON-based ingestion) and streaming ingestion, you can configure the task lock type for the ingestion during the **Publish** step: + +- If you set **Append to existing** to **True**, you can then set **Allow concurrent append tasks (experimental)** to **True**. +- If you set **Append to existing** to **False**, you can then set **Allow concurrent replace tasks (experimental)** to **True**. + +##### Add the task lock type through the API + +Add the following JSON snippet to your supervisor or ingestion spec if you're using the API: + +```json +"context": { + "taskLockType": LOCK_TYPE +} +``` + +The `LOCK_TYPE` depends on what you're trying to accomplish. + +Set `taskLockType` to `APPEND` if either of the following are true: + +- Dynamic partitioning with append to existing is set to `true` +- The ingestion job is a streaming ingestion job + +If you have multiple ingestion jobs that append all targeting the same datasource and want them to run simultaneously, you need to also include the following context parameter: + +```json +"useSharedLock": "true" +``` + +Keep in mind that `taskLockType` takes precedence over `useSharedLock`. Do not use it with `REPLACE` task locks. Review Comment: "it" referring to `useSharedLock`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
