This is an automated email from the ASF dual-hosted git repository.
abhishek pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git
The following commit(s) were added to refs/heads/master by this push:
new bf96ddf5ba Update index.md (#12390)
bf96ddf5ba is described below
commit bf96ddf5bac50e5e9ae778eb28ee300fea6fc62f
Author: mark-imply <[email protected]>
AuthorDate: Fri Apr 8 06:31:54 2022 -0600
Update index.md (#12390)
Added guidance on when to increase
druid.indexer.storage.recentlyFinishedThreshold.
---
docs/configuration/index.md | 50 ++++++++++++++++++++++-----------------------
1 file changed, 25 insertions(+), 25 deletions(-)
diff --git a/docs/configuration/index.md b/docs/configuration/index.md
index 2999e9c10b..21d09a6fc2 100644
--- a/docs/configuration/index.md
+++ b/docs/configuration/index.md
@@ -182,7 +182,7 @@ and `druid.tlsPort` properties on each process. Please see
`Configuration` secti
#### Jetty Server TLS Configuration
-Druid uses Jetty as an embedded web server. To learn more about TLS/SSL,
certificates, and related concepts in Jetty, including explanations of the
configuration settings below, see "Configuring SSL/TLS KeyStores" in the [Jetty
Operations Guide](https://www.eclipse.org/jetty/documentation.php).
+Druid uses Jetty as an embedded web server. To learn more about TLS/SSL,
certificates, and related concepts in Jetty, including explanations of the
configuration settings below, see "Configuring SSL/TLS KeyStores" in the [Jetty
Operations Guide](https://www.eclipse.org/jetty/documentation.php).
For information about TLS/SSL support in Java in general, see the [Java Secure
Socket Extension (JSSE) Reference
Guide](http://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html).
The [Java Cryptography Architecture
@@ -691,11 +691,11 @@ For example, consider the following error response:
If `druid.server.http.errorResponseTransform.allowedRegex` is set to `[]`,
Druid transforms the query error response to the following:
```
{"error":"Plan validation
failed","errorMessage":null,"errorClass":null,"host":null}
-```
+```
On the other hand, if `druid.server.http.errorResponseTransform.allowedRegex`
is set to `[".*CalciteContextException.*"]` then Druid transforms the query
error response to the following:
```
{"error":"Plan validation
failed","errorMessage":"org.apache.calcite.runtime.CalciteContextException:
From line 1, column 15 to line 1, column 38: Object 'nonexistent-datasource'
not found","errorClass":null,"host":null}
-```
+```
### Overlord Discovery
@@ -823,18 +823,18 @@ These Coordinator static configurations can be defined in
the `coordinator/runti
|Property|Description|Required?|Default|
|--------|-----------|---------|-------|
|`druid.coordinator.period.metadataStoreManagementPeriod`|How often to run
metadata management tasks in [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601)
duration format. |No | `PT1H`|
-|`druid.coordinator.kill.supervisor.on`| Boolean value for whether to enable
automatic deletion of terminated supervisors. If set to true, Coordinator will
periodically remove terminated supervisors from the supervisor table in
metadata storage.| No | True|
+|`druid.coordinator.kill.supervisor.on`| Boolean value for whether to enable
automatic deletion of terminated supervisors. If set to true, Coordinator will
periodically remove terminated supervisors from the supervisor table in
metadata storage.| No | True|
|`druid.coordinator.kill.supervisor.period`| How often to do automatic
deletion of terminated supervisor in [ISO
8601](https://en.wikipedia.org/wiki/ISO_8601) duration format. Value must be
equal to or greater than
`druid.coordinator.period.metadataStoreManagementPeriod`. Only applies if
`druid.coordinator.kill.supervisor.on` is set to "True".| No| `P1D`|
|`druid.coordinator.kill.supervisor.durationToRetain`| Duration of terminated
supervisor to be retained from created time in [ISO
8601](https://en.wikipedia.org/wiki/ISO_8601) duration format. Only applies if
`druid.coordinator.kill.supervisor.on` is set to "True".| Yes if
`druid.coordinator.kill.supervisor.on` is set to "True".| `P90D`|
-|`druid.coordinator.kill.audit.on`| Boolean value for whether to enable
automatic deletion of audit logs. If set to true, Coordinator will periodically
remove audit logs from the audit table entries in metadata storage.| No | True|
+|`druid.coordinator.kill.audit.on`| Boolean value for whether to enable
automatic deletion of audit logs. If set to true, Coordinator will periodically
remove audit logs from the audit table entries in metadata storage.| No | True|
|`druid.coordinator.kill.audit.period`| How often to do automatic deletion of
audit logs in [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) duration
format. Value must be equal to or greater than
`druid.coordinator.period.metadataStoreManagementPeriod`. Only applies if
`druid.coordinator.kill.audit.on` is set to "True".| No| `P1D`|
|`druid.coordinator.kill.audit.durationToRetain`| Duration of audit logs to be
retained from created time in [ISO
8601](https://en.wikipedia.org/wiki/ISO_8601) duration format. Only applies if
`druid.coordinator.kill.audit.on` is set to "True".| Yes if
`druid.coordinator.kill.audit.on` is set to "True".| `P90D`|
-|`druid.coordinator.kill.compaction.on`| Boolean value for whether to enable
automatic deletion of compaction configurations. If set to true, Coordinator
will periodically remove compaction configuration of inactive datasource
(datasource with no used and unused segments) from the config table in metadata
storage. | No | False|
+|`druid.coordinator.kill.compaction.on`| Boolean value for whether to enable
automatic deletion of compaction configurations. If set to true, Coordinator
will periodically remove compaction configuration of inactive datasource
(datasource with no used and unused segments) from the config table in metadata
storage. | No | False|
|`druid.coordinator.kill.compaction.period`| How often to do automatic
deletion of compaction configurations in [ISO
8601](https://en.wikipedia.org/wiki/ISO_8601) duration format. Value must be
equal to or greater than
`druid.coordinator.period.metadataStoreManagementPeriod`. Only applies if
`druid.coordinator.kill.compaction.on` is set to "True".| No| `P1D`|
-|`druid.coordinator.kill.rule.on`| Boolean value for whether to enable
automatic deletion of rules. If set to true, Coordinator will periodically
remove rules of inactive datasource (datasource with no used and unused
segments) from the rule table in metadata storage.| No | True|
+|`druid.coordinator.kill.rule.on`| Boolean value for whether to enable
automatic deletion of rules. If set to true, Coordinator will periodically
remove rules of inactive datasource (datasource with no used and unused
segments) from the rule table in metadata storage.| No | True|
|`druid.coordinator.kill.rule.period`| How often to do automatic deletion of
rules in [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) duration format.
Value must be equal to or greater than
`druid.coordinator.period.metadataStoreManagementPeriod`. Only applies if
`druid.coordinator.kill.rule.on` is set to "True".| No| `P1D`|
|`druid.coordinator.kill.rule.durationToRetain`| Duration of rules to be
retained from created time in [ISO
8601](https://en.wikipedia.org/wiki/ISO_8601) duration format. Only applies if
`druid.coordinator.kill.rule.on` is set to "True".| Yes if
`druid.coordinator.kill.rule.on` is set to "True".| `P90D`|
-|`druid.coordinator.kill.datasource.on`| Boolean value for whether to enable
automatic deletion of datasource metadata (Note: datasource metadata only
exists for datasource created from supervisor). If set to true, Coordinator
will periodically remove datasource metadata of terminated supervisor from the
datasource table in metadata storage. | No | True|
+|`druid.coordinator.kill.datasource.on`| Boolean value for whether to enable
automatic deletion of datasource metadata (Note: datasource metadata only
exists for datasource created from supervisor). If set to true, Coordinator
will periodically remove datasource metadata of terminated supervisor from the
datasource table in metadata storage. | No | True|
|`druid.coordinator.kill.datasource.period`| How often to do automatic
deletion of datasource metadata in [ISO
8601](https://en.wikipedia.org/wiki/ISO_8601) duration format. Value must be
equal to or greater than
`druid.coordinator.period.metadataStoreManagementPeriod`. Only applies if
`druid.coordinator.kill.datasource.on` is set to "True".| No| `P1D`|
|`druid.coordinator.kill.datasource.durationToRetain`| Duration of datasource
metadata to be retained from created time in [ISO
8601](https://en.wikipedia.org/wiki/ISO_8601) duration format. Only applies if
`druid.coordinator.kill.datasource.on` is set to "True".| Yes if
`druid.coordinator.kill.datasource.on` is set to "True".| `P90D`|
@@ -1064,7 +1064,7 @@ These Overlord static configurations can be defined in
the `overlord/runtime.pro
|--------|-----------|-------|
|`druid.indexer.runner.type`|Choices "local" or "remote". Indicates whether
tasks should be run locally or in a distributed environment. Experimental task
runner "httpRemote" is also available which is same as "remote" but uses HTTP
to interact with Middle Managers instead of Zookeeper.|local|
|`druid.indexer.storage.type`|Choices are "local" or "metadata". Indicates
whether incoming tasks should be stored locally (in heap) or in metadata
storage. "local" is mainly for internal testing while "metadata" is recommended
in production because storing incoming tasks in metadata storage allows for
tasks to be resumed if the Overlord should fail.|local|
-|`druid.indexer.storage.recentlyFinishedThreshold`|A duration of time to store
task results.|PT24H|
+|`druid.indexer.storage.recentlyFinishedThreshold`|Duration of time to store
task results. Default is 24 hours. If you have hundreds of tasks running in a
day, consider increasing this threshold.|PT24H|
|`druid.indexer.tasklock.forceTimeChunkLock`|_**Setting this to false is still
experimental**_<br/> If set, all tasks are enforced to use time chunk lock. If
not set, each task automatically chooses a lock type to use. This configuration
can be overwritten by setting `forceTimeChunkLock` in the [task
context](../ingestion/tasks.md#context). See [Task Locking &
Priority](../ingestion/tasks.md#context) for more details about locking in
tasks.|true|
|`druid.indexer.task.default.context`|Default task context that is applied to
all tasks submitted to the Overlord. Any default in this config does not
override neither the context values the user provides nor
`druid.indexer.tasklock.forceTimeChunkLock`.|empty context|
|`druid.indexer.queue.maxSize`|Maximum number of active tasks at one
time.|Integer.MAX_VALUE|
@@ -1489,7 +1489,7 @@ then the value from the configuration below is used:
|--------|-----------|-------|
|`druid.worker.version`|Version identifier for the Indexer.|0|
|`druid.worker.capacity`|Maximum number of tasks the Indexer can
accept.|Number of available processors - 1|
-|`druid.worker.globalIngestionHeapLimitBytes`|Total amount of heap available
for ingestion processing. This is applied by automatically setting the
`maxBytesInMemory` property on tasks.|60% of configured JVM heap|
+|`druid.worker.globalIngestionHeapLimitBytes`|Total amount of heap available
for ingestion processing. This is applied by automatically setting the
`maxBytesInMemory` property on tasks.|60% of configured JVM heap|
|`druid.worker.numConcurrentMerges`|Maximum number of segment persist or merge
operations that can run concurrently across all tasks.|`druid.worker.capacity`
/ 2, rounded down|
|`druid.indexer.task.baseDir`|Base temporary working
directory.|`System.getProperty("java.io.tmpdir")`|
|`druid.indexer.task.baseTaskDir`|Base temporary working directory for
tasks.|`${druid.indexer.task.baseDir}/persistent/tasks`|
@@ -1732,7 +1732,7 @@ In this mode, queries are never assigned a lane, and the
concurrent query count
###### 'High/Low' laning strategy
This laning strategy splits queries with a `priority` below zero into a `low`
query lane, automatically. Queries with priority of zero (the default) or above
are considered 'interactive'. The limit on `low` queries can be set to some
desired percentage of the total capacity (or HTTP thread pool size), reserving
capacity for interactive queries. Queries in the `low` lane are _not_
guaranteed their capacity, which may be consumed by interactive queries, but
may use up to this limit if tota [...]
-If the `low` lane is specified in the [query
context](../querying/query-context.md) `lane` parameter, this will override the
computed lane.
+If the `low` lane is specified in the [query
context](../querying/query-context.md) `lane` parameter, this will override the
computed lane.
This strategy can be enabled by setting
`druid.query.scheduler.laning.strategy=hilo`.
@@ -2004,32 +2004,32 @@ This section describes configurations that control
behavior of Druid's query typ
### Overriding default query context values
-Any [Query Context General
Parameter](../querying/query-context.md#general-parameters) default value can
be
-overridden by setting runtime property in the format of
`druid.query.default.context.{query_context_key}`.
-`druid.query.default.context.{query_context_key}` runtime property prefix
applies to all current and future
-query context keys, the same as how query context parameter passed with the
query works. Note that the runtime property
+Any [Query Context General
Parameter](../querying/query-context.md#general-parameters) default value can be
+overridden by setting runtime property in the format of
`druid.query.default.context.{query_context_key}`.
+`druid.query.default.context.{query_context_key}` runtime property prefix
applies to all current and future
+query context keys, the same as how query context parameter passed with the
query works. Note that the runtime property
value can be overridden if value for the same key is explicitly specify in the
query contexts.
-The precedence chain for query context values is as follows:
+The precedence chain for query context values is as follows:
-hard-coded default value in Druid code <- runtime property not prefixed with
`druid.query.default.context`
+hard-coded default value in Druid code <- runtime property not prefixed with
`druid.query.default.context`
<- runtime property prefixed with `druid.query.default.context` <- context
parameter in the query
-Note that not all query context key has a runtime property not prefixed with
`druid.query.default.context` that can
-override the hard-coded default value. For example, `maxQueuedBytes` has
`druid.broker.http.maxQueuedBytes`
-but `joinFilterRewriteMaxSize` does not. Hence, the only way of overriding
`joinFilterRewriteMaxSize` hard-coded default
-value is with runtime property
`druid.query.default.context.joinFilterRewriteMaxSize`.
+Note that not all query context key has a runtime property not prefixed with
`druid.query.default.context` that can
+override the hard-coded default value. For example, `maxQueuedBytes` has
`druid.broker.http.maxQueuedBytes`
+but `joinFilterRewriteMaxSize` does not. Hence, the only way of overriding
`joinFilterRewriteMaxSize` hard-coded default
+value is with runtime property
`druid.query.default.context.joinFilterRewriteMaxSize`.
To further elaborate on the previous example:
If neither `druid.broker.http.maxQueuedBytes` or
`druid.query.default.context.maxQueuedBytes` is set and
the query does not have `maxQueuedBytes` in the context, then the hard-coded
value in Druid code is use.
-If runtime property only contains `druid.broker.http.maxQueuedBytes=x` and
query does not have `maxQueuedBytes` in the
-context, then the value of the property, `x`, is use. However, if query does
have `maxQueuedBytes` in the context,
+If runtime property only contains `druid.broker.http.maxQueuedBytes=x` and
query does not have `maxQueuedBytes` in the
+context, then the value of the property, `x`, is use. However, if query does
have `maxQueuedBytes` in the context,
then that value is use instead.
If runtime property only contains
`druid.query.default.context.maxQueuedBytes=y` OR runtime property contains both
-`druid.broker.http.maxQueuedBytes=x` and
`druid.query.default.context.maxQueuedBytes=y`, then the value of
-`druid.query.default.context.maxQueuedBytes`, `y`, is use (given that query
does not have `maxQueuedBytes` in the
+`druid.broker.http.maxQueuedBytes=x` and
`druid.query.default.context.maxQueuedBytes=y`, then the value of
+`druid.query.default.context.maxQueuedBytes`, `y`, is use (given that query
does not have `maxQueuedBytes` in the
context). If query does have `maxQueuedBytes` in the context, then that value
is use instead.
### TopN query config
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
