This is an automated email from the ASF dual-hosted git repository.
gian pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git
The following commit(s) were added to refs/heads/master by this push:
new a09d6f8b7bc deprecate segment lock gran & NumberedOverwriteShardSpec
(#19050)
a09d6f8b7bc is described below
commit a09d6f8b7bc80e8b032396e5a85a1e3c86432cb0
Author: Cece Mei <[email protected]>
AuthorDate: Tue Feb 24 23:53:56 2026 -0800
deprecate segment lock gran & NumberedOverwriteShardSpec (#19050)
---
docs/configuration/index.md | 2 +-
docs/data-management/compaction.md | 2 +-
docs/design/storage.md | 2 +-
docs/ingestion/tasks.md | 2 +-
.../java/org/apache/druid/indexing/common/LockGranularity.java | 4 ++++
.../druid/timeline/partition/NumberedOverwriteShardSpec.java | 7 +++++--
6 files changed, 13 insertions(+), 6 deletions(-)
diff --git a/docs/configuration/index.md b/docs/configuration/index.md
index a57d09ea9b3..f1e681d4fff 100644
--- a/docs/configuration/index.md
+++ b/docs/configuration/index.md
@@ -997,7 +997,7 @@ These Overlord static configurations can be defined in the
`overlord/runtime.pro
|`druid.indexer.server.maxConcurrentActions`|Maximum number of concurrent
action requests (such as getting locks, creating segments, fetching segments
etc) that the Overlord will process simultaneously. This prevents thread
exhaustion while preserving access to health check endpoints. Set to `0` to
disable quality of service filtering entirely. If not specified, defaults to
`max(1, max(serverHttpNumThreads - 4, serverHttpNumThreads * 0.8))`.|`max(1,
max(serverHttpNumThreads - 4, serverHt [...]
|`druid.indexer.storage.type`|Indicates whether incoming tasks should be
stored locally (in heap) or in metadata storage. One of `local` or `metadata`.
`local` is mainly for internal testing while `metadata` is recommended in
production because storing incoming tasks in metadata storage allows for tasks
to be resumed if the Overlord should fail.|`local`|
|`druid.indexer.storage.recentlyFinishedThreshold`|Duration of time to store
task results. Default is 24 hours. If you have hundreds of tasks running in a
day, consider increasing this threshold.|`PT24H`|
-|`druid.indexer.tasklock.forceTimeChunkLock`|**Setting this to false is still
experimental**<br/> If set, all tasks are enforced to use time chunk lock. If
not set, each task automatically chooses a lock type to use. This configuration
can be overwritten by setting `forceTimeChunkLock` in the [task
context](../ingestion/tasks.md#context-parameters). See [Task lock
system](../ingestion/tasks.md#task-lock-system) for more details about locking
in tasks.|true|
+|`druid.indexer.tasklock.forceTimeChunkLock`|If set to true, all tasks are
enforced to use time chunk lock. If not set, each task automatically chooses a
lock type to use, and may select the deprecated segment lock. This
configuration can be overwritten by setting `forceTimeChunkLock` in the [task
context](../ingestion/tasks.md#context-parameters). See [Task lock
system](../ingestion/tasks.md#task-lock-system) for more details about locking
in tasks.|true|
|`druid.indexer.tasklock.batchSegmentAllocation`| If set to true, Druid
performs segment allocate actions in batches to improve throughput and reduce
the average `task/action/run/time`. See [batching `segmentAllocate`
actions](../ingestion/tasks.md#batching-segmentallocate-actions) for
details.|true|
|`druid.indexer.tasklock.batchAllocationWaitTime`|Number of milliseconds after
Druid adds the first segment allocate action to a batch, until it executes the
batch. Allows the batch to add more requests and improve the average segment
allocation run time. This configuration takes effect only if
`batchSegmentAllocation` is enabled.|0|
|`druid.indexer.tasklock.batchAllocationNumThreads`|Number of worker threads
to use for batch segment allocation. This represents the maximum number of
allocation batches that can be processed in parallel for distinct datasources.
Batches for a single datasource are always processed sequentially. This
configuration takes effect only if `batchSegmentAllocation` is enabled.|5|
diff --git a/docs/data-management/compaction.md
b/docs/data-management/compaction.md
index 51bf7ee8642..05076fbeace 100644
--- a/docs/data-management/compaction.md
+++ b/docs/data-management/compaction.md
@@ -61,7 +61,7 @@ See [Setting up a manual compaction
task](./manual-compaction.md#setting-up-manu
## Data handling with compaction
-During compaction, Druid overwrites the original set of segments with the
compacted set. Druid also locks the segments for the time interval being
compacted to ensure data consistency. By default, compaction tasks do not
modify the underlying data. You can configure the compaction task to change the
query granularity or add or remove dimensions in the compaction task. This
means that the only changes to query results should be the result of
intentional, not automatic, changes.
+During compaction, Druid overwrites the original set of segments with the
compacted set. Druid also locks the time interval being compacted to ensure
data consistency. By default, compaction tasks do not modify the underlying
data. You can configure the compaction task to change the query granularity or
add or remove dimensions in the compaction task. This means that the only
changes to query results should be the result of intentional, not automatic,
changes.
You can set `dropExisting` in `ioConfig` to "true" in the compaction task to
configure Druid to replace all existing segments fully contained by the
interval. See the suggestion for reindexing with finer granularity under
[Implementation
considerations](../ingestion/native-batch.md#implementation-considerations) for
an example.
:::info
diff --git a/docs/design/storage.md b/docs/design/storage.md
index 365819639ef..1854f549f37 100644
--- a/docs/design/storage.md
+++ b/docs/design/storage.md
@@ -134,7 +134,7 @@ Note that atomic replacement happens for each time chunk
individually. If a batc
Typically, atomic replacement in Druid is based on a core set concept that
works in conjunction with segment versions.
When a time chunk is overwritten, a new core set of segments is created with a
higher version number. The core set must all be available before the Broker
will use them instead of the older set. There can also only be one core set per
version per time chunk. Druid will also only use a single version at a time per
time chunk. Together, these properties provide Druid's atomic replacement
guarantees.
-Druid also supports an experimental segment locking mode that is activated by
setting
+Druid also supports a deprecated segment locking mode that is activated by
setting
[`forceTimeChunkLock`](../ingestion/tasks.md#context-parameters) to false in
the context of an ingestion task. In this case, Druid creates an atomic update
group using the existing version for the time chunk, instead of creating a new
core set with a new version number. There can be multiple atomic update groups
with the same version number per time chunk. Each one replaces a specific set
of earlier segments in the same time chunk and with the same version number.
Druid will query the la [...]
If segments become unavailable due to multiple Historicals going offline
simultaneously (beyond your replication factor), then Druid queries will
include only the segments that are still available. In the background, Druid
will reload these unavailable segments on other Historicals as quickly as
possible, at which point they will be included in queries again.
diff --git a/docs/ingestion/tasks.md b/docs/ingestion/tasks.md
index 2dc82b7dc49..d22e6e8f7c2 100644
--- a/docs/ingestion/tasks.md
+++ b/docs/ingestion/tasks.md
@@ -378,7 +378,7 @@ The reason for this is because a Kafka indexing task always
appends new segments
The segments created with the segment locking have the _same_ major version
and a _higher_ minor version.
:::info
- The segment locking is still experimental. It could have unknown bugs which
potentially lead to incorrect query results.
+ The segment locking has been deprecated. It could have unknown bugs which
potentially lead to incorrect query results.
:::
To enable segment locking, you may need to set `forceTimeChunkLock` to `false`
in the [task context](#context).
diff --git
a/indexing-service/src/main/java/org/apache/druid/indexing/common/LockGranularity.java
b/indexing-service/src/main/java/org/apache/druid/indexing/common/LockGranularity.java
index 37cdecc9132..4ff96587891 100644
---
a/indexing-service/src/main/java/org/apache/druid/indexing/common/LockGranularity.java
+++
b/indexing-service/src/main/java/org/apache/druid/indexing/common/LockGranularity.java
@@ -25,5 +25,9 @@ package org.apache.druid.indexing.common;
public enum LockGranularity
{
TIME_CHUNK,
+ /**
+ * @deprecated use TIME_CHUNK instead.
+ */
+ @Deprecated
SEGMENT
}
diff --git
a/processing/src/main/java/org/apache/druid/timeline/partition/NumberedOverwriteShardSpec.java
b/processing/src/main/java/org/apache/druid/timeline/partition/NumberedOverwriteShardSpec.java
index 563865d70de..c5ffd1c9583 100644
---
a/processing/src/main/java/org/apache/druid/timeline/partition/NumberedOverwriteShardSpec.java
+++
b/processing/src/main/java/org/apache/druid/timeline/partition/NumberedOverwriteShardSpec.java
@@ -32,16 +32,18 @@ import java.util.Map;
import java.util.Objects;
/**
+ * @deprecated segment lock is deprecated, NumberedOverwriteShardSpec should
be only used for backward compatibility.
+ * <p>
* This shardSpec is used only for the segments created by overwriting tasks
with segment lock enabled.
* When the segment lock is used, there is a concept of atomic update group
which is a set of segments atomically
* becoming queryable together in Brokers. It is a similar concept to the core
partition set (explained
* {@link NumberedShardSpec}), but different in a sense that there is only one
core partition set per time chunk
* while there could be multiple atomic update groups in one time chunk.
- *
+ * <p>
* The atomic update group has the root partition range and the minor version
to determine the visibility between
* atomic update groups; the group of the highest minor version in the same
root partition range becomes queryable
* when they have the same major version ({@link DataSegment#getVersion()}).
- *
+ * <p>
* Note that this shardSpec is used only when you overwrite existing segments
with segment lock enabled.
* If the task doesn't overwrite segments, it will use NumberedShardSpec
instead even when segment lock is used.
* Similar to NumberedShardSpec, the size of the atomic update group is
determined when the task publishes segments
@@ -51,6 +53,7 @@ import java.util.Objects;
*
* @see AtomicUpdateGroup
*/
+@Deprecated
public class NumberedOverwriteShardSpec implements OverwriteShardSpec
{
private final int partitionId;
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
