jon-wei commented on a change in pull request #6127: Unified configuration doc
page
URL: https://github.com/apache/incubator-druid/pull/6127#discussion_r208771280
##########
File path: docs/content/configuration/configuration-reference.md
##########
@@ -0,0 +1,1512 @@
+---
+layout: doc_page
+---
+
+# Configuration Reference
+
+This page documents all of the configuration properties for each Druid service
type.
+
+## Table of Contents
+ * [Common configurations](#common-configurations)
+ * [Extensions](#extensions)
+ * [Modules](#modules)
+ * [Zookeeper](#zookeper)
+ * [Exhibitor](#exhibitor)
+ * [TLS](#tls)
+ * [Authentication & Authorization](#authentication-and-authorization)
+ * [Startup Logging](#startup-logging)
+ * [Request Logging](#request-logging)
+ * [Enabling Metrics](#enabling-metrics)
+ * [Emitting Metrics](#emitting-metrics)
+ * [Metadata Storage](#metadata-storage)
+ * [Deep Storage](#deep-storage)
+ * [Task Logging](#task-logging)
+ * [Indexing Service Discovery](#indexing-service-discovery)
+ * [Coordinator Discovery](#coordinator-discovery)
+ * [Announcing Segments](#announcing-segments)
+ * [JavaScript](#javascript)
+ * [Double Column Storage](#double-column-storage)
+ * [Coordinator](#coordinator)
+ * [Static Configuration](#static-configuration)
+ * [Node Config](#coordinator-node-config)
+ * [Coordinator Operation](#coordinator-operation)
+ * [Segment Management](#segment-management)
+ * [Metadata Retrieval](#metadata-retrieval)
+ * [Dynamic Configuration](#dynamic-configuration)
+ * [Lookups](#lookups-dynamic-configuration)
+ * [Compaction](#compaction-dynamic-configuration)
+ * [Overlord](#overlord)
+ * [Node Config](#overlord-node-config)
+ * [Static Configuration](#overlord-static-configuration)
+ * [Dynamic Configuration](#overlord-dynamic-configuration)
+ * [Worker Select Strategy](#worker-select-strategy)
+ * [Autoscaler](#autoscaler)
+ * [MiddleManager & Peons](#middle-manager-and-peons)
+ * [Node Config](#middlemanager-node-config)
+ * [MiddleManger Configuration](#middlemanager-configuration)
+ * [Peon Processing](#peon-processing)
+ * [Peon Query Configuration](#peon-query-configuration)
+ * [Caching](#peon-caching)
+ * [Additional Peon Configuration](#additional-peon-configuration)
+ * [Broker](#broker)
+ * [Node Config](#broker-node-configs)
+ * [Query Configuration](#broker-query-configuration)
+ * [SQL](#sql)
+ * [Caching](#broker-caching)
+ * [Segment Discovery](#segment-discovery)
+ * [Historical](#historical)
+ * [Node Configuration](#historical-node-config)
+ * [General Configuration](#historical-general-configuration)
+ * [Query Configs](#historical-query-configs)
+ * [Caching](#historical-caching)
+ * [Caching](#cache-configuration)
+ * [General Query Configuration](#general-query-configuration)
+
+## Common Configurations
+
+The properties under this section are common configurations that should be
shared across all Druid services in a cluster.
+
+### JVM Configuration Best Practices
+
+There are four JVM parameters that we set on all of our processes:
+
+1. `-Duser.timezone=UTC` This sets the default timezone of the JVM to UTC. We
always set this and do not test with other default timezones, so local
timezones might work, but they also might uncover weird and interesting bugs.
To issue queries in a non-UTC timezone, see [query
granularities](../querying/granularities.html#period-granularities)
+2. `-Dfile.encoding=UTF-8` This is similar to timezone, we test assuming
UTF-8. Local encodings might work, but they also might result in weird and
interesting bugs.
+3. `-Djava.io.tmpdir=<a path>` Various parts of the system that interact with
the file system do it via temporary files, and these files can get somewhat
large. Many production systems are set up to have small (but fast) `/tmp`
directories, which can be problematic with Druid so we recommend pointing the
JVM’s tmp directory to something with a little more meat.
+4. `-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager` This
allows log4j2 to handle logs for non-log4j2 components (like jetty) which use
standard java logging.
+
+### Extensions
+
+Many of Druid's external dependencies can be plugged in as modules. Extensions
can be provided using the following configs:
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.extensions.directory`|The root extension directory where user can put
extensions related files. Druid will load extensions stored under this
directory.|`extensions` (This is a relative path to Druid's working directory)|
+|`druid.extensions.hadoopDependenciesDir`|The root hadoop dependencies
directory where user can put hadoop related dependencies files. Druid will load
the dependencies based on the hadoop coordinate specified in the hadoop index
task.|`hadoop-dependencies` (This is a relative path to Druid's working
directory|
+|`druid.extensions.loadList`|A JSON array of extensions to load from extension
directories by Druid. If it is not specified, its value will be `null` and
Druid will load all the extensions under `druid.extensions.directory`. If its
value is empty list `[]`, then no extensions will be loaded at all. It is also
allowed to specify absolute path of other custom extensions not stored in the
common extensions directory.|null|
+|`druid.extensions.searchCurrentClassloader`|This is a boolean flag that
determines if Druid will search the main classloader for extensions. It
defaults to true but can be turned off if you have reason to not automatically
add all modules on the classpath.|true|
+|`druid.extensions.useExtensionClassloaderFirst`|This is a boolean flag that
determines if Druid extensions should prefer loading classes from their own
jars rather than jars bundled with Druid. If false, extensions must be
compatible with classes provided by any jars bundled with Druid. If true,
extensions may depend on conflicting versions.|false|
+|`druid.extensions.hadoopContainerDruidClasspath`|Hadoop Indexing launches
hadoop jobs and this configuration provides way to explicitly set the user
classpath for the hadoop job. By default this is computed automatically by
druid based on the druid process classpath and set of extensions. However,
sometimes you might want to be explicit to resolve dependency conflicts between
druid and hadoop.|null|
+|`druid.extensions.addExtensionsToHadoopContainer`|Only applicable if
`druid.extensions.hadoopContainerDruidClasspath` is provided. If set to true,
then extensions specified in the loadList are added to hadoop container
classpath. Note that when `druid.extensions.hadoopContainerDruidClasspath` is
not provided then extensions are always added to hadoop container
classpath.|false|
+
+### Modules
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.modules.excludeList`|A JSON array of canonical class names (e. g.
`"io.druid.somepackage.SomeModule"`) of module classes which shouldn't be
loaded, even if they are found in extensions specified by
`druid.extensions.loadList`, or in the list of core modules specified to be
loaded on a particular Druid node type. Useful when some useful extension
contains some module, which shouldn't be loaded on some Druid node type because
some dependencies of that module couldn't be satisfied.|[]|
+
+### Zookeeper
+We recommend just setting the base ZK path and the ZK service host, but all ZK
paths that Druid uses can be overwritten to absolute paths.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.zk.paths.base`|Base Zookeeper path.|`/druid`|
+|`druid.zk.service.host`|The ZooKeeper hosts to connect to. This is a REQUIRED
property and therefore a host address must be supplied.|none|
+|`druid.zk.service.user`|The username to authenticate with ZooKeeper. This is
an optional property.|none|
+|`druid.zk.service.pwd`|The [Password
Provider](../operations/password-provider.html) or the string password to
authenticate with ZooKeeper. This is an optional property.|none|
+|`druid.zk.service.authScheme`|digest is the only authentication scheme
supported. |digest|
+
+#### Zookeeper Behavior
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.zk.service.sessionTimeoutMs`|ZooKeeper session timeout, in
milliseconds.|`30000`|
+|`druid.zk.service.compress`|Boolean flag for whether or not created Znodes
should be compressed.|`true`|
+|`druid.zk.service.acl`|Boolean flag for whether or not to enable ACL security
for ZooKeeper. If ACL is enabled, zNode creators will have all
permissions.|`false`|
+
+#### Path Configuration
+Druid interacts with ZK through a set of standard path configurations. We
recommend just setting the base ZK path, but all ZK paths that Druid uses can
be overwritten to absolute paths.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.zk.paths.base`|Base Zookeeper path.|`/druid`|
+|`druid.zk.paths.propertiesPath`|Zookeeper properties
path.|`${druid.zk.paths.base}/properties`|
+|`druid.zk.paths.announcementsPath`|Druid node announcement
path.|`${druid.zk.paths.base}/announcements`|
+|`druid.zk.paths.liveSegmentsPath`|Current path for where Druid nodes announce
their segments.|`${druid.zk.paths.base}/segments`|
+|`druid.zk.paths.loadQueuePath`|Entries here cause historical nodes to load
and drop segments.|`${druid.zk.paths.base}/loadQueue`|
+|`druid.zk.paths.coordinatorPath`|Used by the coordinator for leader
election.|`${druid.zk.paths.base}/coordinator`|
+|`druid.zk.paths.servedSegmentsPath`|@Deprecated. Legacy path for where Druid
nodes announce their segments.|`${druid.zk.paths.base}/servedSegments`|
+
+The indexing service also uses its own set of paths. These configs can be
included in the common configuration.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.zk.paths.indexer.base`|Base zookeeper path for
|`${druid.zk.paths.base}/indexer`|
+|`druid.zk.paths.indexer.announcementsPath`|Middle managers announce
themselves here.|`${druid.zk.paths.indexer.base}/announcements`|
+|`druid.zk.paths.indexer.tasksPath`|Used to assign tasks to middle
managers.|`${druid.zk.paths.indexer.base}/tasks`|
+|`druid.zk.paths.indexer.statusPath`|Parent path for announcement of task
statuses.|`${druid.zk.paths.indexer.base}/status`|
+
+If `druid.zk.paths.base` and `druid.zk.paths.indexer.base` are both set, and
none of the other `druid.zk.paths.*` or `druid.zk.paths.indexer.*` values are
set, then the other properties will be evaluated relative to their respective
`base`.
+For example, if `druid.zk.paths.base` is set to `/druid1` and
`druid.zk.paths.indexer.base` is set to `/druid2` then
`druid.zk.paths.announcementsPath` will default to `/druid1/announcements`
while `druid.zk.paths.indexer.announcementsPath` will default to
`/druid2/announcements`.
+
+The following path is used for service discovery. It is **not** affected by
`druid.zk.paths.base` and **must** be specified separately.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.discovery.curator.path`|Services announce themselves under this
ZooKeeper path.|`/druid/discovery`|
+
+### Exhibitor
+
+[Exhibitor](https://github.com/Netflix/exhibitor/wiki) is a supervisor system
for ZooKeeper.
+Exhibitor can dynamically scale-up/down the cluster of ZooKeeper servers.
+Druid can update self-owned list of ZooKeeper servers through Exhibitor
without restarting.
+That is, it allows Druid to keep the connections of Exhibitor-supervised
ZooKeeper servers.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.exhibitor.service.hosts`|A JSON array which contains the hostnames of
Exhibitor instances. Please specify this property if you want to use
Exhibitor-supervised cluster.|none|
+|`druid.exhibitor.service.port`|The REST port used to connect to
Exhibitor.|`8080`|
+|`druid.exhibitor.service.restUriPath`|The path of the REST call used to get
the server set.|`/exhibitor/v1/cluster/list`|
+|`druid.exhibitor.service.useSsl`|Boolean flag for whether or not to use https
protocol.|`false`|
+|`druid.exhibitor.service.pollingMs`|How ofter to poll the exhibitors for the
list|`10000`|
+
+Note that `druid.zk.service.host` is used as a backup in case an Exhibitor
instance can't be contacted and therefore should still be set.
+
+### TLS
+
+#### General Configuration
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.enablePlaintextPort`|Enable/Disable HTTP connector.|`true`|
+|`druid.enableTlsPort`|Enable/Disable HTTPS connector.|`false`|
+
+Although not recommended but both HTTP and HTTPS connectors can be enabled at
a time and respective ports are configurable using `druid.plaintextPort`
+and `druid.tlsPort` properties on each node. Please see `Configuration`
section of individual nodes to check the valid and default values for these
ports.
+
+#### Jetty Server TLS Configuration
+
+Druid uses Jetty as an embedded web server. To get familiar with TLS/SSL in
general and related concepts like Certificates etc.
+reading this [Jetty
documentation](http://www.eclipse.org/jetty/documentation/9.3.x/configuring-ssl.html)
might be helpful.
+To get more in depth knowledge of TLS/SSL support in Java in general, please
refer to this
[guide](http://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html).
+The documentation
[here](http://www.eclipse.org/jetty/documentation/9.3.x/configuring-ssl.html#configuring-sslcontextfactory)
+can help in understanding TLS/SSL configurations listed below. This
[document](http://docs.oracle.com/javase/8/docs/technotes/guides/security/StandardNames.html)
lists all the possible
+values for the below mentioned configs among others provided by Java
implementation.
+
+|Property|Description|Default|Required|
+|--------|-----------|-------|--------|
+|`druid.server.https.keyStorePath`|The file path or URL of the TLS/SSL Key
store.|none|yes|
+|`druid.server.https.keyStoreType`|The type of the key store.|none|yes|
+|`druid.server.https.certAlias`|Alias of TLS/SSL certificate for the
connector.|none|yes|
+|`druid.server.https.keyStorePassword`|The [Password
Provider](../operations/password-provider.html) or String password for the Key
Store.|none|yes|
+
+Following table contains non-mandatory advanced configuration options, use
caution.
+
+|Property|Description|Default|Required|
+|--------|-----------|-------|--------|
+|`druid.server.https.keyManagerFactoryAlgorithm`|Algorithm to use for creating
KeyManager, more details
[here](https://docs.oracle.com/javase/7/docs/technotes/guides/security/jsse/JSSERefGuide.html#KeyManager).|`javax.net.ssl.KeyManagerFactory.getDefaultAlgorithm()`|no|
+|`druid.server.https.keyManagerPassword`|The [Password
Provider](../operations/password-provider.html) or String password for the Key
Manager.|none|no|
+|`druid.server.https.includeCipherSuites`|List of cipher suite names to
include. You can either use the exact cipher suite name or a regular
expression.|Jetty's default include cipher list|no|
+|`druid.server.https.excludeCipherSuites`|List of cipher suite names to
exclude. You can either use the exact cipher suite name or a regular
expression.|Jetty's default exclude cipher list|no|
+|`druid.server.https.includeProtocols`|List of exact protocols names to
include.|Jetty's default include protocol list|no|
+|`druid.server.https.excludeProtocols`|List of exact protocols names to
exclude.|Jetty's default exclude protocol list|no|
+
+
+#### Internal Client TLS Configuration (requires `simple-client-sslcontext`
extension)
+
+These properties apply to the SSLContext that will be provided to the internal
HTTP client that Druid services use to communicate with each other. These
properties require the `simple-client-sslcontext` extension to be loaded.
Without it, Druid services will be unable to communicate with each other when
TLS is enabled.
+
+|Property|Description|Default|Required|
+|--------|-----------|-------|--------|
+|`druid.client.https.protocol`|SSL protocol to use.|`TLSv1.2`|no|
+|`druid.client.https.trustStoreType`|The type of the key store where trusted
root certificates are stored.|`java.security.KeyStore.getDefaultType()`|no|
+|`druid.client.https.trustStorePath`|The file path or URL of the TLS/SSL Key
store where trusted root certificates are stored.|none|yes|
+|`druid.client.https.trustStoreAlgorithm`|Algorithm to be used by TrustManager
to validate certificate
chains|`javax.net.ssl.TrustManagerFactory.getDefaultAlgorithm()`|no|
+|`druid.client.https.trustStorePassword`|The [Password
Provider](../../operations/password-provider.html) or String password for the
Trust Store.|none|yes|
+
+This
[document](http://docs.oracle.com/javase/8/docs/technotes/guides/security/StandardNames.html)
lists all the possible
+values for the above mentioned configs among others provided by Java
implementation.
+
+### Authentication and Authorization
+
+|Property|Type|Description|Default|Required|
+|--------|-----------|--------|--------|--------|
+|`druid.auth.authenticatorChain`|JSON List of Strings|List of Authenticator
type names|["allowAll"]|no|
+|`druid.escalator.type`|String|Type of the Escalator that should be used for
internal Druid communications. This Escalator must use an authentication scheme
that is supported by an Authenticator in
`druid.auth.authenticationChain`.|"noop"|no|
+|`druid.auth.authorizers`|JSON List of Strings|List of Authorizer type names
|["allowAll"]|no|
+|`druid.auth.unsecuredPaths`| List of Strings|List of paths for which security
checks will not be performed. All requests to these paths will be
allowed.|[]|no|
+|`druid.auth.allowUnauthenticatedHttpOptions`|Boolean|If true, skip
authentication checks for HTTP OPTIONS requests. This is needed for certain use
cases, such as supporting CORS pre-flight requests. Note that disabling
authentication checks for OPTIONS requests will allow unauthenticated users to
determine what Druid endpoints are valid (by checking if the OPTIONS request
returns a 200 instead of 404), so enabling this option may reveal information
about server configuration, including information about what extensions are
loaded (if those extensions add endpoints).|false|no|
+
+For more information, please see [Authentication and
Authorization](../design/auth.html).
+
+For configuration options for specific auth extensions, please refer to the
extension documentation.
+
+### Startup Logging
+
+All nodes can log debugging information on startup.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.startup.logging.logProperties`|Log all properties on startup (from
common.runtime.properties, runtime.properties, and the JVM command line).|false|
+|`druid.startup.logging.maskProperties`|Masks sensitive properties (passwords,
for example) containing theses words.|["password"]|
+
+Note that some sensitive information may be logged if these settings are
enabled.
+
+### Request Logging
+
+All nodes that can serve queries can also log the query requests they see.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.request.logging.type`|Choices: noop, file, emitter, slf4j, filtered,
composing. How to log every query request.|noop|
+
+Note that, you can enable sending all the HTTP requests to log by setting
"io.druid.jetty.RequestLog" to DEBUG level. See
[Logging](../configuration/logging.html)
+
+#### File Request Logging
+
+Daily request logs are stored on disk.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.request.logging.dir`|Historical, Realtime and Broker nodes maintain
request logs of all of the requests they get (interacton is via POST, so normal
request logs don’t generally capture information about the actual query), this
specifies the directory to store the request logs in|none|
+
+#### Emitter Request Logging
+
+Every request is emitted to some external location.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.request.logging.feed`|Feed name for requests.|none|
+
+#### SLF4J Request Logging
+
+Every request is logged via SLF4J. Queries are serialized into JSON in the log
message regardless of the SJF4J format specification. They will be logged under
the class `io.druid.server.log.LoggingRequestLogger`.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.request.logging.setMDC`|If MDC entries should be set in the log entry.
Your logging setup still has to be configured to handle MDC to format this
data|false|
+|`druid.request.logging.setContextMDC`|If the druid query `context` should be
added to the MDC entries. Has no effect unless `setMDC` is `true`|false|
+
+MDC fields populated with `setMDC`:
+
+|MDC field|Description|
+|---------|-----------|
+|`queryId` |The query ID|
+|`dataSource`|The datasource the query was against|
+|`queryType` |The type of the query|
+|`hasFilters`|If the query has any filters|
+|`remoteAddr`|The remote address of the requesting client|
+|`duration` |The duration of the query interval|
+|`resultOrdering`|The ordering of results|
+|`descending`|If the query is a descending query|
+
+#### Filtered Request Logging
+Filtered Request Logger filters requests based on a configurable query/time
threshold. Only request logs where query/time is above the threshold are
emitted.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.request.logging.queryTimeThresholdMs`|Threshold value for query/time
in milliseconds.|0 i.e no filtering|
+|`druid.request.logging.delegate.type`|Type of delegate request logger to log
requests.|none|
+
+#### Composite Request Logging
+Composite Request Logger emits request logs to multiple request loggers.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.request.logging.loggerProviders`|List of request loggers for emitting
request logs.|none|
+
+### Enabling Metrics
+
+Druid nodes periodically emit metrics and different metrics monitors can be
included. Each node can overwrite the default list of monitors.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.monitoring.emissionPeriod`|How often metrics are emitted.|PT1m|
+|`druid.monitoring.monitors`|Sets list of Druid monitors used by a node. See
below for names and more information. For example, you can specify monitors for
a Broker with
`druid.monitoring.monitors=["io.druid.java.util.metrics.SysMonitor","io.druid.java.util.metrics.JvmMonitor"]`.|none
(no monitors)|
+
+The following monitors are available:
+
+|Name|Description|
+|----|-----------|
+|`io.druid.client.cache.CacheMonitor`|Emits metrics (to logs) about the
segment results cache for Historical and Broker nodes. Reports typical cache
statistics include hits, misses, rates, and size (bytes and number of entries),
as well as timeouts and and errors.|
+|`io.druid.java.util.metrics.SysMonitor`|This uses the [SIGAR
library](http://www.hyperic.com/products/sigar) to report on various system
activities and statuses.|
+|`io.druid.server.metrics.HistoricalMetricsMonitor`|Reports statistics on
Historical nodes.|
+|`io.druid.java.util.metrics.JvmMonitor`|Reports various JVM-related
statistics.|
+|`io.druid.java.util.metrics.JvmCpuMonitor`|Reports statistics of CPU
consumption by the JVM.|
+|`io.druid.java.util.metrics.CpuAcctDeltaMonitor`|Reports consumed CPU as per
the cpuacct cgroup.|
+|`io.druid.java.util.metrics.JvmThreadsMonitor`|Reports Thread statistics in
the JVM, like numbers of total, daemon, started, died threads.|
+|`io.druid.segment.realtime.RealtimeMetricsMonitor`|Reports statistics on
Realtime nodes.|
+|`io.druid.server.metrics.EventReceiverFirehoseMonitor`|Reports how many
events have been queued in the EventReceiverFirehose.|
+|`io.druid.server.metrics.QueryCountStatsMonitor`|Reports how many queries
have been successful/failed/interrupted.|
+|`io.druid.server.emitter.HttpEmittingMonitor`|Reports internal metrics of
`http` or `parametrized` emitter (see below). Must not be used with another
emitter type. See the description of the metrics here:
https://github.com/druid-io/druid/pull/4973.|
+
+
+### Emitting Metrics
+
+The Druid servers [emit various metrics](../operations/metrics.html) and
alerts via something we call an Emitter. There are three emitter
implementations included with the code, a "noop" emitter, one that just logs to
log4j ("logging", which is used by default if no emitter is specified) and one
that does POSTs of JSON events to a server ("http"). The properties for using
the logging emitter are described below.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.emitter`|Setting this value to "noop", "logging", "http" or
"parametrized" will initialize one of the emitter modules. value "composing"
can be used to initialize multiple emitter modules. |noop|
+
+#### Logging Emitter Module
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.emitter.logging.loggerClass`|Choices: HttpPostEmitter, LoggingEmitter,
NoopServiceEmitter, ServiceEmitter. The class used for logging.|LoggingEmitter|
+|`druid.emitter.logging.logLevel`|Choices: debug, info, warn, error. The log
level at which message are logged.|info|
+
+#### Http Emitter Module
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.emitter.http.flushMillis`|How often the internal message buffer is
flushed (data is sent).|60000|
+|`druid.emitter.http.flushCount`|How many messages the internal message buffer
can hold before flushing (sending).|500|
+|`druid.emitter.http.basicAuthentication`|Login and password for
authentification in "login:password" form, e. g.
`druid.emitter.http.basicAuthentication=admin:adminpassword`|not specified = no
authentification|
+|`druid.emitter.http.flushTimeOut`|The timeout after which an event should be
sent to the endpoint, even if internal buffers are not filled, in
milliseconds.|not specified = no timeout|
+|`druid.emitter.http.batchingStrategy`|The strategy of how the batch is
formatted. "ARRAY" means `[event1,event2]`, "NEWLINES" means `event1\nevent2`,
ONLY_EVENTS means `event1event2`.|ARRAY|
+|`druid.emitter.http.maxBatchSize`|The maximum batch size, in bytes.|the
minimum of (10% of JVM heap size divided by 2) or (5191680 (i. e. 5 MB))|
+|`druid.emitter.http.batchQueueSizeLimit`|The maximum number of batches in
emitter queue, if there are problems with emitting.|the maximum of (2) or (10%
of the JVM heap size divided by 5MB)|
+|`druid.emitter.http.minHttpTimeoutMillis`|If the speed of filling batches
imposes timeout smaller than that, not even trying to send batch to endpoint,
because it will likely fail, not being able to send the data that fast.
Configure this depending based on emitter/successfulSending/minTimeMs metric.
Reasonable values are 10ms..100ms.|0|
+|`druid.emitter.http.recipientBaseUrl`|The base URL to emit messages to. Druid
will POST JSON to be consumed at the HTTP endpoint specified by this
property.|none, required config|
+
+#### Http Emitter Module TLS Overrides
+
+When emitting events to a TLS-enabled receiver, the Http Emitter will by
default use an SSLContext obtained via the process described at [Druid's
internal communication over
TLS](../operations/tls-support.html#druids-internal-communication-over-tls),
i.e., the same SSLContext that would be used for internal communications
between Druid nodes.
+
+In some use cases it may be desirable to have the Http Emitter use its own
separate truststore configuration. For example, there may be organizational
policies that prevent the TLS-enabled metrics receiver's certificate from being
added to the same truststore used by Druid's internal HTTP client.
+
+The following properties allow the Http Emitter to use its own truststore
configuration when building its SSLContext.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.emitter.http.ssl.useDefaultJavaContext`|If set to true, the
HttpEmitter will use `SSLContext.getDefault()`, the default Java SSLContext,
and all other properties below are ignored.|false|
+|`druid.emitter.http.ssl.trustStorePath`|The file path or URL of the TLS/SSL
Key store where trusted root certificates are stored. If this is unspecified,
the Http Emitter will use the same SSLContext as Druid's internal HTTP client,
as described in the beginning of this section, and all other properties below
are ignored.|null|
+|`druid.emitter.http.ssl.trustStoreType`|The type of the key store where
trusted root certificates are stored.|`java.security.KeyStore.getDefaultType()`|
+|`druid.emitter.http.ssl.trustStoreAlgorithm`|Algorithm to be used by
TrustManager to validate certificate
chains|`javax.net.ssl.TrustManagerFactory.getDefaultAlgorithm()`|
+|`druid.emitter.http.ssl.trustStorePassword`|The [Password
Provider](../../operations/password-provider.html) or String password for the
Trust Store.|none|
+|`druid.emitter.http.ssl.protocol`|TLS protocol to use.|"TLSv1.2"|
+
+#### Parametrized Http Emitter Module
+
+`druid.emitter.parametrized.httpEmitting.*` configs correspond to the configs
of Http Emitter Modules, see above.
+Except `recipientBaseUrl`. E. g.
`druid.emitter.parametrized.httpEmitting.flushMillis`,
+`druid.emitter.parametrized.httpEmitting.flushCount`,
`druid.emitter.parametrized.httpEmitting.ssl.trustStorePath`, etc.
+
+The additional configs are:
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.emitter.parametrized.recipientBaseUrlPattern`|The URL pattern to send
an event to, based on the event's feed. E. g. `http://foo.bar/{feed}`, that
will send event to `http://foo.bar/metrics` if the event's feed is
"metrics".|none, required config|
+
+#### Composing Emitter Module
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.emitter.composing.emitters`|List of emitter modules to load e.g.
["logging","http"].|[]|
+
+#### Graphite Emitter
+
+To use graphite as emitter set `druid.emitter=graphite`. For configuration
details please follow this
[link](../development/extensions-contrib/graphite.html).
+
+
+### Metadata Storage
+
+These properties specify the jdbc connection and other configuration around
the metadata storage. The only processes that connect to the metadata storage
with these properties are the [Coordinator](../design/coordinator.html),
[Indexing service](../design/indexing-service.html) and [Realtime
Nodes](../design/realtime.html).
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.metadata.storage.type`|The type of metadata storage to use. Choose
from "mysql", "postgresql", or "derby".|derby|
+|`druid.metadata.storage.connector.connectURI`|The jdbc uri for the database
to connect to|none|
+|`druid.metadata.storage.connector.user`|The username to connect with.|none|
+|`druid.metadata.storage.connector.password`|The [Password
Provider](../operations/password-provider.html) or String password used to
connect with.|none|
+|`druid.metadata.storage.connector.createTables`|If Druid requires a table and
it doesn't exist, create it?|true|
+|`druid.metadata.storage.tables.base`|The base name for tables.|druid|
+|`druid.metadata.storage.tables.segments`|The table to use to look for
segments.|druid_segments|
+|`druid.metadata.storage.tables.rules`|The table to use to look for segment
load/drop rules.|druid_rules|
+|`druid.metadata.storage.tables.config`|The table to use to look for
configs.|druid_config|
+|`druid.metadata.storage.tables.tasks`|Used by the indexing service to store
tasks.|druid_tasks|
+|`druid.metadata.storage.tables.taskLog`|Used by the indexing service to store
task logs.|druid_taskLog|
+|`druid.metadata.storage.tables.taskLock`|Used by the indexing service to
store task locks.|druid_taskLock|
+|`druid.metadata.storage.tables.supervisors`|Used by the indexing service to
store supervisor configurations.|druid_supervisors|
+|`druid.metadata.storage.tables.audit`|The table to use for audit history of
configuration changes e.g. Coordinator rules.|druid_audit|
+
+### Deep Storage
+
+The configurations concern how to push and pull
[Segments](../design/segments.html) from deep storage.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.storage.type`|Choices:local, noop, s3, hdfs, c*. The type of deep
storage to use.|local|
+
+#### Local Deep Storage
+
+Local deep storage uses the local filesystem.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.storage.storageDirectory`|Directory on disk to use as deep
storage.|/tmp/druid/localStorage|
+
+#### Noop Deep Storage
+
+This deep storage doesn't do anything. There are no configs.
+
+#### S3 Deep Storage
+
+This deep storage is used to interface with Amazon's S3.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.s3.accessKey`|The access key to use to access S3.|none|
+|`druid.s3.secretKey`|The secret key to use to access S3.|none|
+|`druid.storage.bucket`|S3 bucket name.|none|
+|`druid.storage.baseKey`|S3 object key prefix for storage.|none|
+|`druid.storage.disableAcl`|Boolean flag for ACL.|false|
+|`druid.storage.archiveBucket`|S3 bucket name for archiving when running the
indexing-service *archive task*.|none|
+|`druid.storage.archiveBaseKey`|S3 object key prefix for archiving.|none|
+
+#### HDFS Deep Storage
+
+This deep storage is used to interface with HDFS.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.storage.storageDirectory`|HDFS directory to use as deep storage.|none|
+
+#### Cassandra Deep Storage
+
+This deep storage is used to interface with Cassandra.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.storage.host`|Cassandra host.|none|
+|`druid.storage.keyspace`|Cassandra key space.|none|
+
+
+### Task Logging
+
+If you are running the indexing service in remote mode, the task logs must be
stored in S3, Azure Blob Store, Google Cloud Storage or HDFS.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.indexer.logs.type`|Choices:noop, s3, azure, google, hdfs, file. Where
to store task logs|file|
+
+You can also configure the Overlord to automatically retain the task logs only
for last x milliseconds by configuring following additional properties.
+Caution: Automatic log file deletion typically works based on log file
modification timestamp on the backing store, so large clock skews between druid
nodes and backing store nodes might result in un-intended behavior.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.indexer.logs.kill.enabled`|Boolean value for whether to enable
deletion of old task logs. |false|
+|`druid.indexer.logs.kill.durationToRetain`| Required if kill is enabled. In
milliseconds, task logs to be retained created in last x milliseconds. |None|
+|`druid.indexer.logs.kill.initialDelay`| Optional. Number of milliseconds
after overlord start when first auto kill is run. |random value less than
300000 (5 mins)|
+|`druid.indexer.logs.kill.delay`|Optional. Number of milliseconds of delay
between successive executions of auto kill run. |21600000 (6 hours)|
+
+#### File Task Logs
+
+Store task logs in the local filesystem.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.indexer.logs.directory`|Local filesystem path.|log|
+
+#### S3 Task Logs
+
+Store task logs in S3.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.indexer.logs.s3Bucket`|S3 bucket name.|none|
+|`druid.indexer.logs.s3Prefix`|S3 key prefix.|none|
+
+#### Azure Blob Store Task Logs
+Store task logs in Azure Blob Store.
+
+Note: this uses the same storage account as the deep storage module for azure.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.indexer.logs.container`|The Azure Blob Store container to write logs
to|none|
+|`druid.indexer.logs.prefix`|The path to prepend to logs|none|
+
+#### Google Cloud Storage Task Logs
+Store task logs in Google Cloud Storage.
+
+Note: this uses the same storage settings as the deep storage module for
google.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.indexer.logs.bucket`|The Google Cloud Storage bucket to write logs
to|none|
+|`druid.indexer.logs.prefix`|The path to prepend to logs|none|
+
+#### HDFS Task Logs
+
+Store task logs in HDFS.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.indexer.logs.directory`|The directory to store logs.|none|
+
+### Indexing Service Discovery
+
+This config is used to find the [Indexing
Service](../design/indexing-service.html) using Curator service discovery. Only
required if you are actually running an indexing service.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.selectors.indexing.serviceName`|The druid.service name of the indexing
service Overlord node. To start the Overlord with a different name, set it with
this property. |druid/overlord|
+
+
+### Coordinator Discovery
+
+This config is used to find the [Coordinator](../design/coordinator.html)
using Curator service discovery. This config is used by the realtime indexing
nodes to get information about the segments loaded in the cluster.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.selectors.coordinator.serviceName`|The druid.service name of the
coordinator node. To start the Coordinator with a different name, set it with
this property. |druid/coordinator|
+
+
+### Announcing Segments
+
+You can configure how to announce and unannounce Znodes in ZooKeeper (using
Curator). For normal operations you do not need to override any of these
configs.
+
+##### Batch Data Segment Announcer
+
+In current Druid, multiple data segments may be announced under the same Znode.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.announcer.segmentsPerNode`|Each Znode contains info for up to this
many segments.|50|
+|`druid.announcer.maxBytesPerNode`|Max byte size for Znode.|524288|
+|`druid.announcer.skipDimensionsAndMetrics`|Skip Dimensions and Metrics list
from segment announcements. NOTE: Enabling this will also remove the dimensions
and metrics list from coordinator and broker endpoints.|false|
+|`druid.announcer.skipLoadSpec`|Skip segment LoadSpec from segment
announcements. NOTE: Enabling this will also remove the loadspec from
coordinator and broker endpoints.|false|
+
+### JavaScript
+
+Druid supports dynamic runtime extension through JavaScript functions. This
functionality can be configured through
+the following properties.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.javascript.enabled`|Set to "true" to enable JavaScript functionality.
This affects the JavaScript parser, filter, extractionFn, aggregator,
post-aggregator, router strategy, and worker selection strategy.|false|
+
+<div class="note info">
+JavaScript-based functionality is disabled by default. Please refer to the
Druid <a href="../development/javascript.html">JavaScript programming guide</a>
for guidelines about using Druid's JavaScript functionality, including
instructions on how to enable it.
+</div>
+
+### Double Column storage
+
+Prior to version 0.13.0 Druid's storage layer used a 32-bit float
representation to store columns created by the
+doubleSum, doubleMin, and doubleMax aggregators at indexing time.
+Starting from version 0.13.0 the default will be 64-bit floats for Double
columns.
+Using 64-bit representation for double column will lead to avoid precesion
loss at the cost of doubling the storage size of such columns.
+To keep the old format set the system-wide property
`druid.indexing.doubleStorage=float`.
+You can also use floatSum, floatMin and floatMax to use 32-bit float
representation.
+Support for 64-bit floating point columns was released in Druid 0.11.0, so if
you use this feature then older versions of Druid will not be able to read your
data segments.
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.indexing.doubleStorage`|Set to "float" to use 32-bit double
representation for double columns.|double|
+
+## Coordinator
+
+For general Coordinator Node information, see
[here](../design/coordinator.html).
+
+### Static Configuration
+
+The coordinator node uses several of the global configs in
[Configuration](../configuration/index.html) and has the following set of
configurations as well:
+
+#### Coordinator Node Config
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.host`|The host for the current node. This is used to advertise the
current processes location as reachable from another node and should generally
be specified such that `http://${druid.host}/` could actually talk to this
process|InetAddress.getLocalHost().getCanonicalHostName()|
+|`druid.plaintextPort`|This is the port to actually listen on; unless port
mapping is used, this will be the same port as is on `druid.host`|8081|
+|`druid.tlsPort`|TLS port for HTTPS connector, if
[druid.enableTlsPort](../operations/tls-support.html) is set then this config
will be used. If `druid.host` contains port then that port will be ignored.
This should be a non-negative Integer.|8281|
+|`druid.service`|The name of the service. This is used as a dimension when
emitting metrics and alerts to differentiate between the various
services|druid/coordinator|
+
+#### Coordinator Operation
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.coordinator.period`|The run period for the coordinator. The
coordinator’s operates by maintaining the current state of the world in memory
and periodically looking at the set of segments available and segments being
served to make decisions about whether any changes need to be made to the data
topology. This property sets the delay between each of these runs.|PT60S|
+|`druid.coordinator.period.indexingPeriod`|How often to send
compact/merge/conversion tasks to the indexing service. It's recommended to be
longer than `druid.manager.segments.pollDuration`|PT1800S (30 mins)|
+|`druid.coordinator.startDelay`|The operation of the Coordinator works on the
assumption that it has an up-to-date view of the state of the world when it
runs, the current ZK interaction code, however, is written in a way that
doesn’t allow the Coordinator to know for a fact that it’s done loading the
current state of the world. This delay is a hack to give it enough time to
believe that it has all the data.|PT300S|
+|`druid.coordinator.merge.on`|Boolean flag for whether or not the coordinator
should try and merge small segments into a more optimal segment size.|false|
+|`druid.coordinator.conversion.on`|Boolean flag for converting old segment
indexing versions to the latest segment indexing version.|false|
+|`druid.coordinator.load.timeout`|The timeout duration for when the
coordinator assigns a segment to a historical node.|PT15M|
+|`druid.coordinator.kill.pendingSegments.on`|Boolean flag for whether or not
the coordinator clean up old entries in the `pendingSegments` table of metadata
store. If set to true, coordinator will check the created time of most recently
complete task. If it doesn't exist, it finds the created time of the earlist
running/pending/waiting tasks. Once the created time is found, then for all
dataSources not in the `killPendingSegmentsSkipList` (see [Dynamic
configuration](#dynamic-configuration)), coordinator will ask the overlord to
clean up the entries 1 day or more older than the found created time in the
`pendingSegments` table. This will be done periodically based on
`druid.coordinator.period` specified.|false|
+|`druid.coordinator.kill.on`|Boolean flag for whether or not the coordinator
should submit kill task for unused segments, that is, hard delete them from
metadata store and deep storage. If set to true, then for all whitelisted
dataSources (or optionally all), coordinator will submit tasks periodically
based on `period` specified. These kill tasks will delete all segments except
for the last `durationToRetain` period. Whitelist or All can be set via dynamic
configuration `killAllDataSources` and `killDataSourceWhitelist` described
later.|false|
+|`druid.coordinator.kill.period`|How often to send kill tasks to the indexing
service. Value must be greater than `druid.coordinator.period.indexingPeriod`.
Only applies if kill is turned on.|P1D (1 Day)|
+|`druid.coordinator.kill.durationToRetain`| Do not kill segments in last
`durationToRetain`, must be greater or equal to 0. Only applies and MUST be
specified if kill is turned on. Note that default value is invalid.|PT-1S (-1
seconds)|
+|`druid.coordinator.kill.maxSegments`|Kill at most n segments per kill task
submission, must be greater than 0. Only applies and MUST be specified if kill
is turned on. Note that default value is invalid.|0|
+|`druid.coordinator.balancer.strategy`|Specify the type of balancing strategy
that the coordinator should use to distribute segments among the historicals.
`cachingCost` is logically equivalent to `cost` but is more CPU-efficient on
large clusters and will replace `cost` in the future versions, users are
invited to try it. Use `diskNormalized` to distribute segments among nodes so
that the disks fill up uniformly and use `random` to randomly pick nodes to
distribute segments.|`cost`|
+|`druid.coordinator.loadqueuepeon.repeatDelay`|The start and repeat delay for
the loadqueuepeon , which manages the load and drop of segments.|PT0.050S (50
ms)|
+|`druid.coordinator.asOverlord.enabled`|Boolean value for whether this
coordinator node should act like an overlord as well. This configuration allows
users to simplify a druid cluster by not having to deploy any standalone
overlord nodes. If set to true, then overlord console is available at
`http://coordinator-host:port/console.html` and be sure to set
`druid.coordinator.asOverlord.overlordService` also. See next.|false|
+|`druid.coordinator.asOverlord.overlordService`| Required, if
`druid.coordinator.asOverlord.enabled` is `true`. This must be same value as
`druid.service` on standalone Overlord nodes and
`druid.selectors.indexing.serviceName` on Middle Managers.|NULL|
+
+#### Segment Management
+|Property|Possible Values|Description|Default|
+|--------|---------------|-----------|-------|
+|`druid.announcer.type`|batch or http|Segment discovery method to use. "http"
enables discovering segments using HTTP instead of zookeeper.|batch|
+|`druid.coordinator.loadqueuepeon.type`|curator or http|Whether to use "http"
or "curator" implementation to assign segment loads/drops to historical|curator|
+
+##### Additional config when "http" loadqueuepeon is used
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.coordinator.loadqueuepeon.http.batchSize`|Number of segment load/drop
requests to batch in one HTTP request. Note that it must be smaller than
`druid.segmentCache.numLoadingThreads` config on historical node.|1|
+
+#### Metadata Retrieval
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.manager.config.pollDuration`|How often the manager polls the config
table for updates.|PT1m|
+|`druid.manager.segments.pollDuration`|The duration between polls the
Coordinator does for updates to the set of active segments. Generally defines
the amount of lag time it can take for the coordinator to notice new
segments.|PT1M|
+|`druid.manager.rules.pollDuration`|The duration between polls the Coordinator
does for updates to the set of active rules. Generally defines the amount of
lag time it can take for the coordinator to notice rules.|PT1M|
+|`druid.manager.rules.defaultTier`|The default tier from which default rules
will be loaded from.|_default|
+|`druid.manager.rules.alertThreshold`|The duration after a failed poll upon
which an alert should be emitted.|PT10M|
+
+### Dynamic Configuration
+
+The coordinator has dynamic configuration to change certain behaviour on the
fly. The coordinator uses a JSON spec object from the Druid [metadata
storage](../dependencies/metadata-storage.html) config table. This object is
detailed below:
+
+It is recommended that you use the Coordinator Console to configure these
parameters. However, if you need to do it via HTTP, the JSON object can be
submitted to the coordinator via a POST request at:
+
+```
+http://<COORDINATOR_IP>:<PORT>/druid/coordinator/v1/config
+```
+
+Optional Header Parameters for auditing the config change can also be
specified.
+
+|Header Param Name| Description | Default |
+|----------|-------------|---------|
+|`X-Druid-Author`| author making the config change|""|
+|`X-Druid-Comment`| comment describing the change being done|""|
+
+A sample coordinator dynamic config JSON object is shown below:
+
+```json
+{
+ "millisToWaitBeforeDeleting": 900000,
+ "mergeBytesLimit": 100000000,
+ "mergeSegmentsLimit" : 1000,
+ "maxSegmentsToMove": 5,
+ "replicantLifetime": 15,
+ "replicationThrottleLimit": 10,
+ "emitBalancingStats": false,
+ "killDataSourceWhitelist": ["wikipedia", "testDatasource"]
+}
+```
+
+Issuing a GET request at the same URL will return the spec that is currently
in place. A description of the config setup spec is shown below.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`balancerComputeThreads`|The number of threads used in a coordinator
balancing run.|1|
+|`millisToWaitBeforeDeleting`|How long does the coordinator need to be active
before it can start removing (marking unused) segments in metadata
storage.|900000 (15 mins)|
+|`mergeBytesLimit`|The maximum total uncompressed size in bytes of segments to
merge.|524288000L|
+|`mergeSegmentsLimit`|The maximum number of segments that can be in a single
[append task](../ingestion/tasks.html).|100|
+|`maxSegmentsToMove`|The maximum number of segments that can be moved at any
given time.|5|
+|`replicantLifetime`|The maximum number of coordinator runs for a segment to
be replicated before we start alerting.|15|
+|`replicationThrottleLimit`|The maximum number of segments that can be
replicated at one time.|10|
+|`emitBalancingStats`|Boolean flag for whether or not we should emit balancing
stats. This is an expensive operation.|false|
+|`killDataSourceWhitelist`|List of dataSources for which kill tasks are sent
if property `druid.coordinator.kill.on` is true. This can be a list of
comma-separated dataSources or a JSON array.|none|
+|`killAllDataSources`|Send kill tasks for ALL dataSources if property
`druid.coordinator.kill.on` is true. If this is set to true then
`killDataSourceWhitelist` must not be specified or be empty list.|false|
+|`killPendingSegmentsSkipList`|List of dataSources for which pendingSegments
are _NOT_ cleaned up if property `druid.coordinator.kill.pendingSegments.on` is
true. This can be a list of comma-separated dataSources or a JSON array.|none|
+|`maxSegmentsInNodeLoadingQueue`|The maximum number of segments that could be
queued for loading to any given server. This parameter could be used to speed
up segments loading process, especially if there are "slow" nodes in the
cluster (with low loading speed) or if too much segments scheduled to be
replicated to some particular node (faster loading could be preferred to better
segments distribution). Desired value depends on segments loading speed,
acceptable replication time and number of nodes. Value 1000 could be a start
point for a rather big cluster. Default value is 0 (loading queue is unbounded)
|0|
+
+To view the audit history of coordinator dynamic config issue a GET request to
the URL -
+
+```
+http://<COORDINATOR_IP>:<PORT>/druid/coordinator/v1/config/history?interval=<interval>
+```
+
+default value of interval can be specified by setting
`druid.audit.manager.auditHistoryMillis` (1 week if not configured) in
coordinator runtime.properties
+
+To view last <n> entries of the audit history of coordinator dynamic config
issue a GET request to the URL -
+
+```
+http://<COORDINATOR_IP>:<PORT>/druid/coordinator/v1/config/history?count=<n>
+```
+
+#### Lookups Dynamic Configuration (EXPERIMENTAL)<a
id="lookups-dynamic-configuration"></a>
+These configuration options control the behavior of the Lookup dynamic
configuration described in the [lookups page](../querying/lookups.html)
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.manager.lookups.hostDeleteTimeout`|How long to wait for a `DELETE`
request to a particular node before considering the `DELETE` a failure|PT1s|
+|`druid.manager.lookups.hostUpdateTimeout`|How long to wait for a `POST`
request to a particular node before considering the `POST` a failure|PT10s|
+|`druid.manager.lookups.deleteAllTimeout`|How long to wait for all `DELETE`
requests to finish before considering the delete attempt a failure|PT10s|
+|`druid.manager.lookups.updateAllTimeout`|How long to wait for all `POST`
requests to finish before considering the attempt a failure|PT60s|
+|`druid.manager.lookups.threadPoolSize`|How many nodes can be managed
concurrently (concurrent POST and DELETE requests). Requests this limit will
wait in a queue until a slot becomes available.|10|
+|`druid.manager.lookups.period`|How many milliseconds between checks for
configuration changes|30_000|
+
+#### Compaction Dynamic Configuration
+
+Compaction configurations can also be set or updated dynamically without
restarting coordinators. For segment compaction,
+please see [Compacting
Segments](../design/coordinator.html#compacting-segments).
+
+A description of the compaction config is:
+
+|Property|Description|Required|
+|--------|-----------|--------|
+|`dataSource`|dataSource name to be compacted.|yes|
+|`taskPriority`|[Priority](../ingestion/tasks.html#task-priorities) of compact
task.|no (default = 25)|
+|`targetCompactionSizeBytes`|The target segment size of compaction. The actual
size of a compact segment might be slightly larger or smaller than this
value.|no (default = 838860800)|
+|`numTargetCompactionSegments`|Max number of segments to compact together.|no
(default = 150)|
+|`skipOffsetFromLatest`|The offset for searching segments to be compacted.
Strongly recommended to set for realtime dataSources. |no (default = "P1D")|
+|`tuningConfig`|Tuning config for compact tasks. See below [Compact Task
TuningConfig](#compact-task-tuningconfig).|no|
+|`taskContext`|[Task context](../ingestion/tasks.html#task-context) for
compact tasks.|no|
+
+An example of compaction config is:
+
+```json
+{
+ "dataSource": "wikiticker",
+ "targetCompactionSizeBytes": 800000000,
+ "skipOffsetFromLatest": "P1D"
+}
+```
+
+For realtime dataSources, it's recommended to set `skipOffsetFromLatest` to
some sufficiently large values to avoid frequent compact task failures.
+
+## Overlord
+
+### Overlord Node Configs
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.host`|The host for the current node. This is used to advertise the
current processes location as reachable from another node and should generally
be specified such that `http://${druid.host}/` could actually talk to this
process|InetAddress.getLocalHost().getCanonicalHostName()|
+|`druid.plaintextPort`|This is the port to actually listen on; unless port
mapping is used, this will be the same port as is on `druid.host`|8090|
+|`druid.tlsPort`|TLS port for HTTPS connector, if
[druid.enableTlsPort](../operations/tls-support.html) is set then this config
will be used. If `druid.host` contains port then that port will be ignored.
This should be a non-negative Integer.|8290|
+|`druid.service`|The name of the service. This is used as a dimension when
emitting metrics and alerts to differentiate between the various
services|druid/overlord|
+
+### Overlord Static Configuration
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.indexer.runner.type`|Choices "local" or "remote". Indicates whether
tasks should be run locally or in a distributed environment. Experimental task
runner "httpRemote" is also available which is same as "remote" but uses HTTP
to interact with Middle Manaters instead of Zookeeper.|local|
+|`druid.indexer.storage.type`|Choices are "local" or "metadata". Indicates
whether incoming tasks should be stored locally (in heap) or in metadata
storage. Storing incoming tasks in metadata storage allows for tasks to be
resumed if the overlord should fail.|local|
+|`druid.indexer.storage.recentlyFinishedThreshold`|A duration of time to store
task results.|PT24H|
+|`druid.indexer.queue.maxSize`|Maximum number of active tasks at one
time.|Integer.MAX_VALUE|
+|`druid.indexer.queue.startDelay`|Sleep this long before starting overlord
queue management. This can be useful to give a cluster time to re-orient itself
after e.g. a widespread network issue.|PT1M|
+|`druid.indexer.queue.restartDelay`|Sleep this long when overlord queue
management throws an exception before trying again.|PT30S|
+|`druid.indexer.queue.storageSyncRate`|Sync overlord state this often with an
underlying task persistence mechanism.|PT1M|
+
+The following configs only apply if the overlord is running in remote mode:
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.indexer.runner.taskAssignmentTimeout`|How long to wait after a task as
been assigned to a middle manager before throwing an error.|PT5M|
+|`druid.indexer.runner.minWorkerVersion`|The minimum middle manager version to
send tasks to. |"0"|
+|`druid.indexer.runner.compressZnodes`|Indicates whether or not the overlord
should expect middle managers to compress Znodes.|true|
+|`druid.indexer.runner.maxZnodeBytes`|The maximum size Znode in bytes that can
be created in Zookeeper.|524288|
+|`druid.indexer.runner.taskCleanupTimeout`|How long to wait before failing a
task after a middle manager is disconnected from Zookeeper.|PT15M|
+|`druid.indexer.runner.taskShutdownLinkTimeout`|How long to wait on a shutdown
request to a middle manager before timing out|PT1M|
+|`druid.indexer.runner.pendingTasksRunnerNumThreads`|Number of threads to
allocate pending-tasks to workers, must be at least 1.|1|
+|`druid.indexer.runner.maxRetriesBeforeBlacklist`|Number of consecutive times
the middle manager can fail tasks, before the worker is blacklisted, must be
at least 1|5|
+|`druid.indexer.runner.workerBlackListBackoffTime`|How long to wait before a
task is whitelisted again. This value should be greater that the value set for
taskBlackListCleanupPeriod.|PT15M|
+|`druid.indexer.runner.workerBlackListCleanupPeriod`|A duration after which
the cleanup thread will startup to clean blacklisted workers.|PT5M|
+|`druid.indexer.runner.maxPercentageBlacklistWorkers`|The maximum percentage
of workers to blacklist, this must be between 0 and 100.|20|
+
+There are additional configs for autoscaling (if it is enabled):
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.indexer.autoscale.strategy`|Choices are "noop" or "ec2". Sets the
strategy to run when autoscaling is required.|noop|
+|`druid.indexer.autoscale.doAutoscale`|If set to "true" autoscaling will be
enabled.|false|
+|`druid.indexer.autoscale.provisionPeriod`|How often to check whether or not
new middle managers should be added.|PT1M|
+|`druid.indexer.autoscale.terminatePeriod`|How often to check when middle
managers should be removed.|PT5M|
+|`druid.indexer.autoscale.originTime`|The starting reference timestamp that
the terminate period increments upon.|2012-01-01T00:55:00.000Z|
+|`druid.indexer.autoscale.workerIdleTimeout`|How long can a worker be idle
(not a run task) before it can be considered for termination.|PT90M|
+|`druid.indexer.autoscale.maxScalingDuration`|How long the overlord will wait
around for a middle manager to show up before giving up.|PT15M|
+|`druid.indexer.autoscale.numEventsToTrack`|The number of autoscaling related
events (node creation and termination) to track.|10|
+|`druid.indexer.autoscale.pendingTaskTimeout`|How long a task can be in
"pending" state before the overlord tries to scale up.|PT30S|
+|`druid.indexer.autoscale.workerVersion`|If set, will only create nodes of set
version during autoscaling. Overrides dynamic configuration. |null|
+|`druid.indexer.autoscale.workerPort`|The port that middle managers will run
on.|8080|
+
+### Overlord Dynamic Configuration
+
+The overlord can dynamically change worker behavior.
+
+The JSON object can be submitted to the overlord via a POST request at:
+
+```
+http://<OVERLORD_IP>:<port>/druid/indexer/v1/worker
+```
+
+Optional Header Parameters for auditing the config change can also be
specified.
+
+|Header Param Name| Description | Default |
+|----------|-------------|---------|
+|`X-Druid-Author`| author making the config change|""|
+|`X-Druid-Comment`| comment describing the change being done|""|
+
+A sample worker config spec is shown below:
+
+```json
+{
+ "selectStrategy": {
+ "type": "fillCapacity",
+ "affinityConfig": {
+ "affinity": {
+ "datasource1": ["host1:port", "host2:port"],
+ "datasource2": ["host3:port"]
+ }
+ }
+ },
+ "autoScaler": {
+ "type": "ec2",
+ "minNumWorkers": 2,
+ "maxNumWorkers": 12,
+ "envConfig": {
+ "availabilityZone": "us-east-1a",
+ "nodeData": {
+ "amiId": "${AMI}",
+ "instanceType": "c3.8xlarge",
+ "minInstances": 1,
+ "maxInstances": 1,
+ "securityGroupIds": ["${IDs}"],
+ "keyName": ${KEY_NAME}
Review comment:
Added quotes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
