This is an automated email from the ASF dual-hosted git repository.
cwylie pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/druid.git
The following commit(s) were added to refs/heads/master by this push:
new 6089a16 Docs - update dynamic config provider topic (#11795)
6089a16 is described below
commit 6089a168ea65867a0d2f8e2a83069ee34afda9dd
Author: Charles Smith <[email protected]>
AuthorDate: Thu Oct 14 17:51:32 2021 -0700
Docs - update dynamic config provider topic (#11795)
* update dynamic config provider
* update topic
* add examples for dynamic config provider:
* Update docs/development/extensions-core/kafka-ingestion.md
Co-authored-by: Katya Macedo <[email protected]>
* Update docs/development/extensions-core/kafka-ingestion.md
Co-authored-by: Katya Macedo <[email protected]>
* Update docs/development/extensions-core/kafka-ingestion.md
Co-authored-by: Katya Macedo <[email protected]>
* Update docs/operations/dynamic-config-provider.md
Co-authored-by: Katya Macedo <[email protected]>
* Update docs/operations/dynamic-config-provider.md
Co-authored-by: Katya Macedo <[email protected]>
* Update docs/operations/dynamic-config-provider.md
Co-authored-by: Katya Macedo <[email protected]>
* Update docs/operations/dynamic-config-provider.md
Co-authored-by: Katya Macedo <[email protected]>
* Update docs/development/extensions-core/kafka-ingestion.md
Co-authored-by: Katya Macedo <[email protected]>
* Update docs/operations/dynamic-config-provider.md
Co-authored-by: Clint Wylie <[email protected]>
* Update docs/operations/dynamic-config-provider.md
Co-authored-by: Clint Wylie <[email protected]>
* Update kafka-ingestion.md
Co-authored-by: Katya Macedo <[email protected]>
Co-authored-by: Clint Wylie <[email protected]>
---
.../development/extensions-core/kafka-ingestion.md | 34 ++++++++++++---
docs/operations/dynamic-config-provider.md | 51 ++++++++++++++++++----
website/.spelling | 1 +
3 files changed, 70 insertions(+), 16 deletions(-)
diff --git a/docs/development/extensions-core/kafka-ingestion.md
b/docs/development/extensions-core/kafka-ingestion.md
index 0355389..ad3c9c0 100644
--- a/docs/development/extensions-core/kafka-ingestion.md
+++ b/docs/development/extensions-core/kafka-ingestion.md
@@ -221,15 +221,35 @@ The following example demonstrates supervisor spec with
`lagBased` autoScaler en
#### More on consumerProperties
-This must contain a property `bootstrap.servers` with a list of Kafka brokers
in the form: `<BROKER_1>:<PORT_1>,<BROKER_2>:<PORT_2>,...`.
-By default, `isolation.level` is set to `read_committed`. It should be set to
`read_uncommitted` if you don't want Druid to consume only committed
transactions or working with older versions of Kafka servers with no
transactions support.
+Consumer properties must contain a property `bootstrap.servers` with a list of
Kafka brokers in the form: `<BROKER_1>:<PORT_1>,<BROKER_2>:<PORT_2>,...`.
+By default, `isolation.level` is set to `read_committed`. If you use older
versions of Kafka servers without transactions support or don't want Druid to
consume only committed transactions, set `isolation.level` to
`read_uncommitted`.
-There are few cases that require fetching few/all of consumer properties at
runtime e.g. when `bootstrap.servers` is not known upfront or not static, to
enable SSL connections users might have to provide passwords for `keystore`,
`truststore` and `key` secretly.
-For such consumer properties, user can implement a
[DynamicConfigProvider](../../operations/dynamic-config-provider.md) to supply
them at runtime, by adding
-`druid.dynamic.config.provider`=`{"type":
"<registered_dynamic_config_provider_name>", ...}`
-in consumerProperties map.
+In some cases, you may need to fetch consumer properties at runtime. For
example, when `bootstrap.servers` is not known upfront, or is not static. To
enable SSL connections, you must provide passwords for `keystore`, `truststore`
and `key` secretly. You can provide configurations at runtime with a dynamic
config provider implementation like the environment variable config provider
that comes with Druid. For more information, see
[DynamicConfigProvider](../../operations/dynamic-config-pro [...]
-Note: SSL connections may also be supplied using the deprecated [Password
Provider](../../operations/password-provider.md) interface to define the
`keystore`, `truststore`, and `key`. This functionality might be removed in a
future release.
+For example, if you are using SASL and SSL with Kafka, set the following
environment variables for the Druid user on the machines running the Overlord
and the Peon services:
+
+```
+export
KAFKA_JAAS_CONFIG="org.apache.kafka.common.security.plain.PlainLoginModule
required username='admin_user' password='admin_password';"
+export SSL_KEY_PASSWORD=mysecretkeypassword
+export SSL_KEYSTORE_PASSWORD=mysecretkeystorepassword
+export SSL_TRUSTSTORE_PASSWORD=mysecrettruststorepassword
+```
+
+```
+ "druid.dynamic.config.provider": {
+ "type": "environment",
+ "variables": {
+ "sasl.jaas.config": "KAFKA_JAAS_CONFIG"
+ "ssl.key.password": "SSL_KEY_PASSWORD",
+ "ssl.keystore.password": "SSL_KEYSTORE_PASSWORD",
+ "ssl.truststore.password": "SSL_TRUSTSTORE_PASSWORD"
+ }
+ }
+ }
+```
+Verify that you've changed the values for all configurations to match your own
environment. You can use the environment variable config provider syntax in
the **Consumer properties** field on the **Connect tab** in the **Load Data**
UI in the Druid console. When connecting to Kafka, Druid replaces the
environment variables with their corresponding values.
+
+Note: You can provide SSL connections with [Password
Provider](../../operations/password-provider.md) interface to define the
`keystore`, `truststore`, and `key`, but this feature is deprecated.
#### Specifying data format
diff --git a/docs/operations/dynamic-config-provider.md
b/docs/operations/dynamic-config-provider.md
index 45b61d5..0b34338 100644
--- a/docs/operations/dynamic-config-provider.md
+++ b/docs/operations/dynamic-config-provider.md
@@ -22,25 +22,58 @@ title: "Dynamic Config Providers"
~ under the License.
-->
-Druid's core mechanism of supplying multiple related set of
credentials/secrets/configurations via Druid extension mechanism. Currently, it
is only supported for providing Kafka Consumer configuration in [Kafka
Ingestion](../development/extensions-core/kafka-ingestion.md).
+Druid relies on dynamic config providers to supply multiple related sets of
credentials, secrets, and configurations within a Druid extension. Dynamic
config providers are intended to eventually replace
[PasswordProvider](./password-provider.md).
-Eventually this will replace [PasswordProvider](./password-provider.md)
+By default, Druid includes an environment variable dynamic config provider
that supports Kafka consumer configuration in [Kafka
ingestion](../development/extensions-core/kafka-ingestion.md).
+- Kafka consumer configuration in [Kafka
ingestion](../development/extensions-core/kafka-ingestion.md)
+To develop a custom extension of the `DynamicConfigProvider` interface that is
registered at Druid process startup, see [Adding a new DynamicConfigProvider
implementation](../development/modules.md#adding-a-new-dynamicconfigprovider-implementation).
-Users can create custom extension of the `DynamicConfigProvider` interface
that is registered at Druid process startup.
+## Environment variable dynamic config provider
-For more information, see [Adding a new DynamicConfigProvider
implementation](../development/modules.md#adding-a-new-dynamicconfigprovider-implementation).
+You can use the environment variable dynamic config provider
(`EnvironmentVariableDynamicConfigProvider`) to store passwords or other
sensitive information using system environment variables instead of plain text
configuration.
-## Environment variable dynamic config provider
+The environment variable dynamic config provider uses the following syntax:
-`EnvironmentVariableDynamicConfigProvider` can be used to avoid exposing
credentials or other secret information in the configuration files using
environment variables. An example to use this `configProvider` is:
```json
-druid.some.config.dynamicConfigProvider={"type":
"environment","variables":{"secret1": "SECRET1_VAR","secret2": "SECRET2_VAR"}}
+druid.dynamic.config.provider={"type": "environment","variables":{"secret1":
"SECRET1_VAR","secret2": "SECRET2_VAR"}}
```
-The values are described below.
|Field|Type|Description|Required|
|-----|----|-----------|--------|
|`type`|String|dynamic config provider type|Yes: `environment`|
-|`variables`|Map|environment variables to get information from|Yes|
+|`variables`|Map|environment variables that store the configuration
information|Yes|
+
+When using the environment variable config provider, consider the following:
+- If you manually specify a configuration key-value pair and use the dynamic
config provider for the same key, Druid uses the value from the dynamic config
provider.
+- For use in a supervisor spec, environment variables must be available to the
system user that runs the Overlord service and that runs the Peon service.
+
+The following example shows how to configure environment variables to store
the SSL key and truststore passwords for Kafka.
+
+On the Overlord and Peon machines, set the following environment variables for
the system user that runs the Druid services:
+```
+export SSL_KEY_PASSWORD=mysecretkeypassword
+export SSL_KEYSTORE_PASSWORD=mysecretkeystorepassword
+export SSL_TRUSTSTORE_PASSWORD=mysecrettruststorepassword
+```
+
+When you define the consumer properties in the supervisor spec, use the
dynamic config provider to refer to the environment variables:
+```
+...
+ "consumerProperties": {
+ "bootstrap.servers": "localhost:9092",
+ "ssl.keystore.location": "/opt/kafka/config/kafka01.keystore.jks"
+ "ssl.truststore.location": "/opt/kafka/config/kafka.truststore.jks"
+ "druid.dynamic.config.provider": {
+ "type": "environment",
+ "variables": {
+ "ssl.key.password": "SSL_KEY_PASSWORD",
+ "ssl.keystore.password": "SSL_KEYSTORE_PASSWORD",
+ "ssl.truststore.password": "SSL_TRUSTSTORE_PASSWORD"
+ }
+ }
+ },
+...
+```
+When connecting to Kafka, Druid replaces the environment variables with their
corresponding values.
\ No newline at end of file
diff --git a/website/.spelling b/website/.spelling
index 705cc77..182bf1e 100644
--- a/website/.spelling
+++ b/website/.spelling
@@ -781,6 +781,7 @@ PT30M
PT30S
PT5S
PT80S
+SASL
SegmentWriteOutMediumFactory
UNABLE_TO_CONNECT_TO_STREAM
UNHEALTHY_SUPERVISOR
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
