sthetland commented on a change in pull request #11463: URL: https://github.com/apache/druid/pull/11463#discussion_r682706555
########## File path: docs/tutorials/docker.md ########## @@ -21,67 +21,108 @@ title: "Docker" ~ specific language governing permissions and limitations ~ under the License. --> - + In this quickstart, we will download the Apache Druid image from [Docker Hub](https://hub.docker.com/r/apache/druid) and set it up on a single machine using [Docker](https://www.docker.com/get-started) and [Docker Compose](https://docs.docker.com/compose/). The cluster will be ready to load data after completing this initial setup. -Before beginning the quickstart, it is helpful to read the [general Druid overview](../design/index.md) and the [ingestion overview](../ingestion/index.md), as the tutorials will refer to concepts discussed on those pages. Additionally, familiarity with Docker is recommended. +Before beginning the quickstart, it is helpful to read the [general Druid overview](../design/index.md) and the [ingestion overview](../ingestion/index.md), as the tutorials will refer to concepts discussed on those pages. Additionally, familiarity with [Docker](https://www.docker.com/get-started) is recommended. + +This tutorial assumes you will download the required files from GitHub. The files are also available in a Druid installation and in the Druid sources. ## Prerequisites -* Docker +* [Docker](https://www.docker.com/get-started) ## Getting started -The Druid source code contains [an example `docker-compose.yml`](https://github.com/apache/druid/blob/{{DRUIDVERSION}}/distribution/docker/docker-compose.yml) which can pull an image from Docker Hub and is suited to be used as an example environment and to experiment with Docker based Druid configuration and deployments. +Create a directory to hold the Druid Docker files. + +The Druid source code contains [an example `docker-compose.yml`](https://github.com/apache/druid/blob/{{DRUIDVERSION}}/distribution/docker/docker-compose.yml) which pulls an image from Docker Hub and is suited to be used as an example environment and to experiment with Docker based Druid configuration and deployments. [Download](https://raw.githubusercontent.com/apache/druid/{{DRUIDVERSION}}/distribution/docker/docker-compose.yml) this file to the directory created above. ### Compose file -The example `docker-compose.yml` will create a container for each Druid service, as well as ZooKeeper and a PostgreSQL container as the metadata store. +The example `docker-compose.yml` will create a container for each Druid service, as well as ZooKeeper and a PostgreSQL container as the metadata store. -It will also create a named volumes `druid_shared`, which is mounted as `opt/shared` in container, as deep storage to keep and share segments and task logs among Druid services. +It will also create a named volume `druid_shared` as deep storage to keep and share segments and task logs among Druid services. The volume is mounted as `opt/shared` in the container. -The Druid containers are configured via an [environment file](https://github.com/apache/druid/blob/{{DRUIDVERSION}}/distribution/docker/environment). +### Environment file + +The Druid `docker-compose.yml` example utilizes an [environment file](https://docs.docker.com/compose/environment-variables/#the-env_file-configuration-option) to specify the complete Druid configuration, including the environment variables listed below. This file is named `environment` by default, and must be in the same directory as the `docker-compose.yml` file. [Download](https://raw.githubusercontent.com/apache/druid/{{DRUIDVERSION}}/distribution/docker/environment) the example `environment` file to the directory created above. The options in this file work well for trying Druid and for using the tutorial. + +The single-file approach is inadequate for a production system. Instead we suggest using either `DRUID_COMMON_CONFIG` and `DRUID_CONFIG_${service}` or specially tailored, service-specific environment files. ### Configuration -Configuration of the Druid Docker container is done via environment variables, which may additionally specify paths to [the standard Druid configuration files](../configuration/index.md) +Configuration of the Druid Docker container is done via environment variables set within the container. Docker Compose passes the values from the `environment file` into the container. The variables may additionally specify paths to [the standard Druid configuration files](../configuration/index.md) which must be available within the container. + +The default values are fine for the Quickstart. Production systems will want to modify the defaults. -Special environment variables: +Basic configuration: + +* `DRUID_MAXDIRECTMEMORYSIZE` -- set Java max direct memory size. Default is 6 GB. +* `DRUID_XMX` -- set Java `Xmx`, the maximum heap size. Default is 1 GB. + +Production configuration: -* `JAVA_OPTS` -- set java options -* `DRUID_LOG4J` -- set the entire `log4j.xml` verbatim -* `DRUID_LOG_LEVEL` -- override the default log level in default log4j -* `DRUID_XMX` -- set Java `Xmx` -* `DRUID_XMS` -- set Java `Xms` -* `DRUID_MAXNEWSIZE` -- set Java max new size -* `DRUID_NEWSIZE` -- set Java new size -* `DRUID_MAXDIRECTMEMORYSIZE` -- set Java max direct memory size * `DRUID_CONFIG_COMMON` -- full path to a file for druid 'common' properties Review comment: ```suggestion * `DRUID_CONFIG_COMMON` -- full path to a file for Druid common properties ``` ########## File path: docs/tutorials/docker.md ########## @@ -21,67 +21,108 @@ title: "Docker" ~ specific language governing permissions and limitations ~ under the License. --> - + In this quickstart, we will download the Apache Druid image from [Docker Hub](https://hub.docker.com/r/apache/druid) and set it up on a single machine using [Docker](https://www.docker.com/get-started) and [Docker Compose](https://docs.docker.com/compose/). The cluster will be ready to load data after completing this initial setup. -Before beginning the quickstart, it is helpful to read the [general Druid overview](../design/index.md) and the [ingestion overview](../ingestion/index.md), as the tutorials will refer to concepts discussed on those pages. Additionally, familiarity with Docker is recommended. +Before beginning the quickstart, it is helpful to read the [general Druid overview](../design/index.md) and the [ingestion overview](../ingestion/index.md), as the tutorials will refer to concepts discussed on those pages. Additionally, familiarity with [Docker](https://www.docker.com/get-started) is recommended. + +This tutorial assumes you will download the required files from GitHub. The files are also available in a Druid installation and in the Druid sources. ## Prerequisites -* Docker +* [Docker](https://www.docker.com/get-started) ## Getting started -The Druid source code contains [an example `docker-compose.yml`](https://github.com/apache/druid/blob/{{DRUIDVERSION}}/distribution/docker/docker-compose.yml) which can pull an image from Docker Hub and is suited to be used as an example environment and to experiment with Docker based Druid configuration and deployments. +Create a directory to hold the Druid Docker files. + +The Druid source code contains [an example `docker-compose.yml`](https://github.com/apache/druid/blob/{{DRUIDVERSION}}/distribution/docker/docker-compose.yml) which pulls an image from Docker Hub and is suited to be used as an example environment and to experiment with Docker based Druid configuration and deployments. [Download](https://raw.githubusercontent.com/apache/druid/{{DRUIDVERSION}}/distribution/docker/docker-compose.yml) this file to the directory created above. ### Compose file -The example `docker-compose.yml` will create a container for each Druid service, as well as ZooKeeper and a PostgreSQL container as the metadata store. +The example `docker-compose.yml` will create a container for each Druid service, as well as ZooKeeper and a PostgreSQL container as the metadata store. -It will also create a named volumes `druid_shared`, which is mounted as `opt/shared` in container, as deep storage to keep and share segments and task logs among Druid services. +It will also create a named volume `druid_shared` as deep storage to keep and share segments and task logs among Druid services. The volume is mounted as `opt/shared` in the container. -The Druid containers are configured via an [environment file](https://github.com/apache/druid/blob/{{DRUIDVERSION}}/distribution/docker/environment). +### Environment file + +The Druid `docker-compose.yml` example utilizes an [environment file](https://docs.docker.com/compose/environment-variables/#the-env_file-configuration-option) to specify the complete Druid configuration, including the environment variables listed below. This file is named `environment` by default, and must be in the same directory as the `docker-compose.yml` file. [Download](https://raw.githubusercontent.com/apache/druid/{{DRUIDVERSION}}/distribution/docker/environment) the example `environment` file to the directory created above. The options in this file work well for trying Druid and for using the tutorial. + +The single-file approach is inadequate for a production system. Instead we suggest using either `DRUID_COMMON_CONFIG` and `DRUID_CONFIG_${service}` or specially tailored, service-specific environment files. ### Configuration -Configuration of the Druid Docker container is done via environment variables, which may additionally specify paths to [the standard Druid configuration files](../configuration/index.md) +Configuration of the Druid Docker container is done via environment variables set within the container. Docker Compose passes the values from the `environment file` into the container. The variables may additionally specify paths to [the standard Druid configuration files](../configuration/index.md) which must be available within the container. + +The default values are fine for the Quickstart. Production systems will want to modify the defaults. -Special environment variables: +Basic configuration: + +* `DRUID_MAXDIRECTMEMORYSIZE` -- set Java max direct memory size. Default is 6 GB. +* `DRUID_XMX` -- set Java `Xmx`, the maximum heap size. Default is 1 GB. + +Production configuration: -* `JAVA_OPTS` -- set java options -* `DRUID_LOG4J` -- set the entire `log4j.xml` verbatim -* `DRUID_LOG_LEVEL` -- override the default log level in default log4j -* `DRUID_XMX` -- set Java `Xmx` -* `DRUID_XMS` -- set Java `Xms` -* `DRUID_MAXNEWSIZE` -- set Java max new size -* `DRUID_NEWSIZE` -- set Java new size -* `DRUID_MAXDIRECTMEMORYSIZE` -- set Java max direct memory size * `DRUID_CONFIG_COMMON` -- full path to a file for druid 'common' properties * `DRUID_CONFIG_${service}` -- full path to a file for druid 'service' properties +* `JAVA_OPTS` -- set Java options + +Logging configuration: + +* `DRUID_LOG4J` -- set the entire [`log4j.xml` configuration file](https://logging.apache.org/log4j/2.x/manual/configuration.html#XML) verbatim. ([Example](https://github.com/apache/druid/blob/{{DRUIDVERSION}}/distribution/docker/environment#L52)) +* `DRUID_LOG_LEVEL` -- override the default [Log4j log level](https://en.wikipedia.org/wiki/Log4j#Log4j_log_levels) + +Advanced memory configuration: + +* `DRUID_XMS` -- set Java [`Xms`](https://docs.oracle.com/cd/E19900-01/819-4742/abeik/index.html), the initial heap size. Default is 1 GB. +* `DRUID_MAXNEWSIZE` -- set [Java max new size](https://docs.oracle.com/cd/E19900-01/819-4742/abeik/index.html) +* `DRUID_NEWSIZE` -- set [Java new size](https://docs.oracle.com/cd/E19900-01/819-4742/abeik/index.html) -In addition to the special environment variables, the script which launches Druid in the container will also attempt to use any environment variable starting with the `druid_` prefix as a command-line configuration. For example, an environment variable - -```druid_metadata_storage_type=postgresql``` - -would be translated into +In addition to the special environment variables, the script which launches Druid in the container will use any environment variable starting with the `druid_` prefix as command-line configuration. For example, an environment variable -```-Ddruid.metadata.storage.type=postgresql``` - -for the Druid process in the container. +`druid_metadata_storage_type=postgresql` + +is translated into the following option in the Java launch command for the Druid process in the container: + +`-Ddruid.metadata.storage.type=postgresql` + +Note that Druid uses port 8888 for the console. This port is also used by Jupyter and other tools. To avoid conflicts, you can change the port in the [ports](https://github.com/apache/druid/blob/0.21.1/distribution/docker/docker-compose.yml#L125) section of the `docker-compose.yml` file. Example to expose the Console on port 9999 of the host: Review comment: ```suggestion Note that Druid uses port 8888 for the console. This port is also used by Jupyter and other tools. To avoid conflicts, you can change the port in the [`ports`](https://github.com/apache/druid/blob/0.21.1/distribution/docker/docker-compose.yml#L125) section of the `docker-compose.yml` file. For example, to expose the console on port 9999 of the host: ``` ########## File path: docs/tutorials/docker.md ########## @@ -21,67 +21,108 @@ title: "Docker" ~ specific language governing permissions and limitations ~ under the License. --> - + In this quickstart, we will download the Apache Druid image from [Docker Hub](https://hub.docker.com/r/apache/druid) and set it up on a single machine using [Docker](https://www.docker.com/get-started) and [Docker Compose](https://docs.docker.com/compose/). The cluster will be ready to load data after completing this initial setup. -Before beginning the quickstart, it is helpful to read the [general Druid overview](../design/index.md) and the [ingestion overview](../ingestion/index.md), as the tutorials will refer to concepts discussed on those pages. Additionally, familiarity with Docker is recommended. +Before beginning the quickstart, it is helpful to read the [general Druid overview](../design/index.md) and the [ingestion overview](../ingestion/index.md), as the tutorials will refer to concepts discussed on those pages. Additionally, familiarity with [Docker](https://www.docker.com/get-started) is recommended. + +This tutorial assumes you will download the required files from GitHub. The files are also available in a Druid installation and in the Druid sources. ## Prerequisites -* Docker +* [Docker](https://www.docker.com/get-started) ## Getting started -The Druid source code contains [an example `docker-compose.yml`](https://github.com/apache/druid/blob/{{DRUIDVERSION}}/distribution/docker/docker-compose.yml) which can pull an image from Docker Hub and is suited to be used as an example environment and to experiment with Docker based Druid configuration and deployments. +Create a directory to hold the Druid Docker files. + +The Druid source code contains [an example `docker-compose.yml`](https://github.com/apache/druid/blob/{{DRUIDVERSION}}/distribution/docker/docker-compose.yml) which pulls an image from Docker Hub and is suited to be used as an example environment and to experiment with Docker based Druid configuration and deployments. [Download](https://raw.githubusercontent.com/apache/druid/{{DRUIDVERSION}}/distribution/docker/docker-compose.yml) this file to the directory created above. ### Compose file -The example `docker-compose.yml` will create a container for each Druid service, as well as ZooKeeper and a PostgreSQL container as the metadata store. +The example `docker-compose.yml` will create a container for each Druid service, as well as ZooKeeper and a PostgreSQL container as the metadata store. -It will also create a named volumes `druid_shared`, which is mounted as `opt/shared` in container, as deep storage to keep and share segments and task logs among Druid services. +It will also create a named volume `druid_shared` as deep storage to keep and share segments and task logs among Druid services. The volume is mounted as `opt/shared` in the container. -The Druid containers are configured via an [environment file](https://github.com/apache/druid/blob/{{DRUIDVERSION}}/distribution/docker/environment). +### Environment file + +The Druid `docker-compose.yml` example utilizes an [environment file](https://docs.docker.com/compose/environment-variables/#the-env_file-configuration-option) to specify the complete Druid configuration, including the environment variables listed below. This file is named `environment` by default, and must be in the same directory as the `docker-compose.yml` file. [Download](https://raw.githubusercontent.com/apache/druid/{{DRUIDVERSION}}/distribution/docker/environment) the example `environment` file to the directory created above. The options in this file work well for trying Druid and for using the tutorial. + +The single-file approach is inadequate for a production system. Instead we suggest using either `DRUID_COMMON_CONFIG` and `DRUID_CONFIG_${service}` or specially tailored, service-specific environment files. ### Configuration -Configuration of the Druid Docker container is done via environment variables, which may additionally specify paths to [the standard Druid configuration files](../configuration/index.md) +Configuration of the Druid Docker container is done via environment variables set within the container. Docker Compose passes the values from the `environment file` into the container. The variables may additionally specify paths to [the standard Druid configuration files](../configuration/index.md) which must be available within the container. + +The default values are fine for the Quickstart. Production systems will want to modify the defaults. -Special environment variables: +Basic configuration: + +* `DRUID_MAXDIRECTMEMORYSIZE` -- set Java max direct memory size. Default is 6 GB. Review comment: About "6 GB"... a recently merged PR (https://github.com/apache/druid/pull/11333) updated the docs to distinguish between decimal format and binary format for all size notations. Just to confirm, looks like these values should indeed be in Decimal format based on the docker config, if I'm reading the config file correctly? ########## File path: docs/tutorials/docker.md ########## @@ -21,67 +21,108 @@ title: "Docker" ~ specific language governing permissions and limitations ~ under the License. --> - + In this quickstart, we will download the Apache Druid image from [Docker Hub](https://hub.docker.com/r/apache/druid) and set it up on a single machine using [Docker](https://www.docker.com/get-started) and [Docker Compose](https://docs.docker.com/compose/). The cluster will be ready to load data after completing this initial setup. -Before beginning the quickstart, it is helpful to read the [general Druid overview](../design/index.md) and the [ingestion overview](../ingestion/index.md), as the tutorials will refer to concepts discussed on those pages. Additionally, familiarity with Docker is recommended. +Before beginning the quickstart, it is helpful to read the [general Druid overview](../design/index.md) and the [ingestion overview](../ingestion/index.md), as the tutorials will refer to concepts discussed on those pages. Additionally, familiarity with [Docker](https://www.docker.com/get-started) is recommended. + +This tutorial assumes you will download the required files from GitHub. The files are also available in a Druid installation and in the Druid sources. ## Prerequisites -* Docker +* [Docker](https://www.docker.com/get-started) ## Getting started -The Druid source code contains [an example `docker-compose.yml`](https://github.com/apache/druid/blob/{{DRUIDVERSION}}/distribution/docker/docker-compose.yml) which can pull an image from Docker Hub and is suited to be used as an example environment and to experiment with Docker based Druid configuration and deployments. +Create a directory to hold the Druid Docker files. + +The Druid source code contains [an example `docker-compose.yml`](https://github.com/apache/druid/blob/{{DRUIDVERSION}}/distribution/docker/docker-compose.yml) which pulls an image from Docker Hub and is suited to be used as an example environment and to experiment with Docker based Druid configuration and deployments. [Download](https://raw.githubusercontent.com/apache/druid/{{DRUIDVERSION}}/distribution/docker/docker-compose.yml) this file to the directory created above. ### Compose file -The example `docker-compose.yml` will create a container for each Druid service, as well as ZooKeeper and a PostgreSQL container as the metadata store. +The example `docker-compose.yml` will create a container for each Druid service, as well as ZooKeeper and a PostgreSQL container as the metadata store. -It will also create a named volumes `druid_shared`, which is mounted as `opt/shared` in container, as deep storage to keep and share segments and task logs among Druid services. +It will also create a named volume `druid_shared` as deep storage to keep and share segments and task logs among Druid services. The volume is mounted as `opt/shared` in the container. -The Druid containers are configured via an [environment file](https://github.com/apache/druid/blob/{{DRUIDVERSION}}/distribution/docker/environment). +### Environment file + +The Druid `docker-compose.yml` example utilizes an [environment file](https://docs.docker.com/compose/environment-variables/#the-env_file-configuration-option) to specify the complete Druid configuration, including the environment variables listed below. This file is named `environment` by default, and must be in the same directory as the `docker-compose.yml` file. [Download](https://raw.githubusercontent.com/apache/druid/{{DRUIDVERSION}}/distribution/docker/environment) the example `environment` file to the directory created above. The options in this file work well for trying Druid and for using the tutorial. + +The single-file approach is inadequate for a production system. Instead we suggest using either `DRUID_COMMON_CONFIG` and `DRUID_CONFIG_${service}` or specially tailored, service-specific environment files. ### Configuration -Configuration of the Druid Docker container is done via environment variables, which may additionally specify paths to [the standard Druid configuration files](../configuration/index.md) +Configuration of the Druid Docker container is done via environment variables set within the container. Docker Compose passes the values from the `environment file` into the container. The variables may additionally specify paths to [the standard Druid configuration files](../configuration/index.md) which must be available within the container. + +The default values are fine for the Quickstart. Production systems will want to modify the defaults. -Special environment variables: +Basic configuration: + +* `DRUID_MAXDIRECTMEMORYSIZE` -- set Java max direct memory size. Default is 6 GB. +* `DRUID_XMX` -- set Java `Xmx`, the maximum heap size. Default is 1 GB. + +Production configuration: -* `JAVA_OPTS` -- set java options -* `DRUID_LOG4J` -- set the entire `log4j.xml` verbatim -* `DRUID_LOG_LEVEL` -- override the default log level in default log4j -* `DRUID_XMX` -- set Java `Xmx` -* `DRUID_XMS` -- set Java `Xms` -* `DRUID_MAXNEWSIZE` -- set Java max new size -* `DRUID_NEWSIZE` -- set Java new size -* `DRUID_MAXDIRECTMEMORYSIZE` -- set Java max direct memory size * `DRUID_CONFIG_COMMON` -- full path to a file for druid 'common' properties * `DRUID_CONFIG_${service}` -- full path to a file for druid 'service' properties +* `JAVA_OPTS` -- set Java options + +Logging configuration: + +* `DRUID_LOG4J` -- set the entire [`log4j.xml` configuration file](https://logging.apache.org/log4j/2.x/manual/configuration.html#XML) verbatim. ([Example](https://github.com/apache/druid/blob/{{DRUIDVERSION}}/distribution/docker/environment#L52)) +* `DRUID_LOG_LEVEL` -- override the default [Log4j log level](https://en.wikipedia.org/wiki/Log4j#Log4j_log_levels) + +Advanced memory configuration: + +* `DRUID_XMS` -- set Java [`Xms`](https://docs.oracle.com/cd/E19900-01/819-4742/abeik/index.html), the initial heap size. Default is 1 GB. +* `DRUID_MAXNEWSIZE` -- set [Java max new size](https://docs.oracle.com/cd/E19900-01/819-4742/abeik/index.html) +* `DRUID_NEWSIZE` -- set [Java new size](https://docs.oracle.com/cd/E19900-01/819-4742/abeik/index.html) -In addition to the special environment variables, the script which launches Druid in the container will also attempt to use any environment variable starting with the `druid_` prefix as a command-line configuration. For example, an environment variable - -```druid_metadata_storage_type=postgresql``` - -would be translated into +In addition to the special environment variables, the script which launches Druid in the container will use any environment variable starting with the `druid_` prefix as command-line configuration. For example, an environment variable -```-Ddruid.metadata.storage.type=postgresql``` - -for the Druid process in the container. +`druid_metadata_storage_type=postgresql` + +is translated into the following option in the Java launch command for the Druid process in the container: + +`-Ddruid.metadata.storage.type=postgresql` + +Note that Druid uses port 8888 for the console. This port is also used by Jupyter and other tools. To avoid conflicts, you can change the port in the [ports](https://github.com/apache/druid/blob/0.21.1/distribution/docker/docker-compose.yml#L125) section of the `docker-compose.yml` file. Example to expose the Console on port 9999 of the host: Review comment: Should the version `0.21.1` be replaced with the `{{DRUIDVERSION}}` in these github links? ```suggestion Note that Druid uses port 8888 for the console. This port is also used by Jupyter and other tools. To avoid conflicts, you can change the port in the [ports](https://github.com/apache/druid/blob/{{DRUIDVERSION}}/distribution/docker/docker-compose.yml#L125) section of the `docker-compose.yml` file. Example to expose the Console on port 9999 of the host: ``` ########## File path: docs/tutorials/docker.md ########## @@ -21,67 +21,108 @@ title: "Docker" ~ specific language governing permissions and limitations ~ under the License. --> - + In this quickstart, we will download the Apache Druid image from [Docker Hub](https://hub.docker.com/r/apache/druid) and set it up on a single machine using [Docker](https://www.docker.com/get-started) and [Docker Compose](https://docs.docker.com/compose/). The cluster will be ready to load data after completing this initial setup. -Before beginning the quickstart, it is helpful to read the [general Druid overview](../design/index.md) and the [ingestion overview](../ingestion/index.md), as the tutorials will refer to concepts discussed on those pages. Additionally, familiarity with Docker is recommended. +Before beginning the quickstart, it is helpful to read the [general Druid overview](../design/index.md) and the [ingestion overview](../ingestion/index.md), as the tutorials will refer to concepts discussed on those pages. Additionally, familiarity with [Docker](https://www.docker.com/get-started) is recommended. + +This tutorial assumes you will download the required files from GitHub. The files are also available in a Druid installation and in the Druid sources. ## Prerequisites -* Docker +* [Docker](https://www.docker.com/get-started) ## Getting started -The Druid source code contains [an example `docker-compose.yml`](https://github.com/apache/druid/blob/{{DRUIDVERSION}}/distribution/docker/docker-compose.yml) which can pull an image from Docker Hub and is suited to be used as an example environment and to experiment with Docker based Druid configuration and deployments. +Create a directory to hold the Druid Docker files. + +The Druid source code contains [an example `docker-compose.yml`](https://github.com/apache/druid/blob/{{DRUIDVERSION}}/distribution/docker/docker-compose.yml) which pulls an image from Docker Hub and is suited to be used as an example environment and to experiment with Docker based Druid configuration and deployments. [Download](https://raw.githubusercontent.com/apache/druid/{{DRUIDVERSION}}/distribution/docker/docker-compose.yml) this file to the directory created above. ### Compose file -The example `docker-compose.yml` will create a container for each Druid service, as well as ZooKeeper and a PostgreSQL container as the metadata store. +The example `docker-compose.yml` will create a container for each Druid service, as well as ZooKeeper and a PostgreSQL container as the metadata store. -It will also create a named volumes `druid_shared`, which is mounted as `opt/shared` in container, as deep storage to keep and share segments and task logs among Druid services. +It will also create a named volume `druid_shared` as deep storage to keep and share segments and task logs among Druid services. The volume is mounted as `opt/shared` in the container. -The Druid containers are configured via an [environment file](https://github.com/apache/druid/blob/{{DRUIDVERSION}}/distribution/docker/environment). +### Environment file + +The Druid `docker-compose.yml` example utilizes an [environment file](https://docs.docker.com/compose/environment-variables/#the-env_file-configuration-option) to specify the complete Druid configuration, including the environment variables listed below. This file is named `environment` by default, and must be in the same directory as the `docker-compose.yml` file. [Download](https://raw.githubusercontent.com/apache/druid/{{DRUIDVERSION}}/distribution/docker/environment) the example `environment` file to the directory created above. The options in this file work well for trying Druid and for using the tutorial. + +The single-file approach is inadequate for a production system. Instead we suggest using either `DRUID_COMMON_CONFIG` and `DRUID_CONFIG_${service}` or specially tailored, service-specific environment files. ### Configuration -Configuration of the Druid Docker container is done via environment variables, which may additionally specify paths to [the standard Druid configuration files](../configuration/index.md) +Configuration of the Druid Docker container is done via environment variables set within the container. Docker Compose passes the values from the `environment file` into the container. The variables may additionally specify paths to [the standard Druid configuration files](../configuration/index.md) which must be available within the container. + +The default values are fine for the Quickstart. Production systems will want to modify the defaults. -Special environment variables: +Basic configuration: + +* `DRUID_MAXDIRECTMEMORYSIZE` -- set Java max direct memory size. Default is 6 GB. +* `DRUID_XMX` -- set Java `Xmx`, the maximum heap size. Default is 1 GB. + +Production configuration: -* `JAVA_OPTS` -- set java options -* `DRUID_LOG4J` -- set the entire `log4j.xml` verbatim -* `DRUID_LOG_LEVEL` -- override the default log level in default log4j -* `DRUID_XMX` -- set Java `Xmx` -* `DRUID_XMS` -- set Java `Xms` -* `DRUID_MAXNEWSIZE` -- set Java max new size -* `DRUID_NEWSIZE` -- set Java new size -* `DRUID_MAXDIRECTMEMORYSIZE` -- set Java max direct memory size * `DRUID_CONFIG_COMMON` -- full path to a file for druid 'common' properties * `DRUID_CONFIG_${service}` -- full path to a file for druid 'service' properties Review comment: ```suggestion * `DRUID_CONFIG_${service}` -- full path to a file for Druid service properties ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
