jihoonson commented on a change in pull request #6127: Unified configuration doc page URL: https://github.com/apache/incubator-druid/pull/6127#discussion_r209061604
########## File path: docs/content/configuration/index.md ########## @@ -343,55 +474,69 @@ This deep storage is used to interface with Cassandra. |`druid.storage.host`|Cassandra host.|none| |`druid.storage.keyspace`|Cassandra key space.|none| -### Caching -You can enable caching of results at the broker, historical, or realtime level using following configurations. +### Task Logging -|Property|Possible Values|Description|Default| -|--------|---------------|-----------|-------| -|`druid.cache.type`|`local`, `memcached`, `hybrid`, `caffeine`|The type of cache to use for queries.|`caffeine`| -|<code>druid.(broker|historical|realtime).cache.unCacheable</code>|All druid query types|All query types to not cache.|["groupBy", "select"]| -|<code>druid.(broker|historical|realtime).cache.useCache</code>|true, false|Whether to use cache for getting query results.|false| -|<code>druid.(broker|historical|realtime).cache.populateCache</code>|true, false|Whether to populate cache.|false| -|<code>druid.(broker|historical|realtime).cache.maxEntrySize</code>|positive integer or -1|Maximum size of an individual cache entry (processed results for one segment), in bytes, or -1 for unlimited.|`-1`| +If you are running the indexing service in remote mode, the task logs must be stored in S3, Azure Blob Store, Google Cloud Storage or HDFS. -#### Local Cache +|Property|Description|Default| +|--------|-----------|-------| +|`druid.indexer.logs.type`|Choices:noop, s3, azure, google, hdfs, file. Where to store task logs|file| -<div class="note caution"> -DEPRECATED: Use caffeine instead -</div> +You can also configure the Overlord to automatically retain the task logs only for last x milliseconds by configuring following additional properties. +Caution: Automatic log file deletion typically works based on log file modification timestamp on the backing store, so large clock skews between druid nodes and backing store nodes might result in un-intended behavior. |Property|Description|Default| |--------|-----------|-------| -|`druid.cache.sizeInBytes`|Maximum cache size in bytes. You must set this if you enabled populateCache/useCache, or else cache size of zero wouldn't really cache anything.|0| -|`druid.cache.initialSize`|Initial size of the hashtable backing the cache.|500000| -|`druid.cache.logEvictionCount`|If non-zero, log cache eviction every `logEvictionCount` items.|0| +|`druid.indexer.logs.kill.enabled`|Boolean value for whether to enable deletion of old task logs. |false| +|`druid.indexer.logs.kill.durationToRetain`| Required if kill is enabled. In milliseconds, task logs to be retained created in last x milliseconds. |None| +|`druid.indexer.logs.kill.initialDelay`| Optional. Number of milliseconds after overlord start when first auto kill is run. |random value less than 300000 (5 mins)| +|`druid.indexer.logs.kill.delay`|Optional. Number of milliseconds of delay between successive executions of auto kill run. |21600000 (6 hours)| + +#### File Task Logs -#### Memcached +Store task logs in the local filesystem. |Property|Description|Default| |--------|-----------|-------| -|`druid.cache.expiration`|Memcached [expiration time](https://code.google.com/p/memcached/wiki/NewCommands#Standard_Protocol).|2592000 (30 days)| -|`druid.cache.timeout`|Maximum time in milliseconds to wait for a response from Memcached.|500| -|`druid.cache.hosts`|Comma separated list of Memcached hosts `<host:port>`.|none| -|`druid.cache.maxObjectSize`|Maximum object size in bytes for a Memcached object.|52428800 (50 MB)| -|`druid.cache.memcachedPrefix`|Key prefix for all keys in Memcached.|druid| +|`druid.indexer.logs.directory`|Local filesystem path.|log| -#### Caffeine Cache +#### S3 Task Logs -A highly performant local cache implementation for Druid based on [Caffeine](https://github.com/ben-manes/caffeine). Requires a JRE8u60 or higher if using `COMMON_FJP`. +Store task logs in S3. Review comment: Same for other task log storage types. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
