kfaraz commented on code in PR #12368: URL: https://github.com/apache/druid/pull/12368#discussion_r854760613
########## docker-tests/docs/docker.md: ########## @@ -0,0 +1,250 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +# Docker Test Image for Druid + +Integration tests need a Druid cluster. While some tests support using +Kubernetes for the Quickstart cluster, most need a cluster with some +test-specfic configuration. We use Docker Compose to create that cluster, +based on a test-oriented Docker image built by the `test-image` project. Review Comment: Typo? ```suggestion based on a test-oriented Docker image built by the `test-image` profile. ``` or ```suggestion based on a test-oriented Docker image built by the `it-image` project. ``` ########## docker-tests/README.md: ########## @@ -0,0 +1,103 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +# Revised Integration Tests + +This directory builds a Docker image for Druid, +then usees that image, along with test configuration to run tests. +This version greatly evolves the integration tests from the earlier +form. See the last section for details. + +## Run the Tests Review Comment: This can be inside a quickstart section, which has the commands/steps for these items: - Run all tests - Build docker image - Start docker containers - Run a single test (from console or IDE) - Stop containers ########## docker-tests/docs/debugging.md: ########## @@ -0,0 +1,183 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +# Debugging the Druid Image and Integration Tests + +The integration setup has as a primary goal the ability to quickly debug +the Druid image and any individual tests. A first step is to move the +image build into a separate project. A second step is to ensure each +test can run in JUnit in an IDE against a cluster you start by hand. + +This section discusses how to use the various debugging features. + +See: + +* [Docker Configuration](docker.md) for information on debugging + docker builds. + +## General Debug Process + +Ease of debugging is a key goal of the revised structure. + +* Rebuild the Docker image only when the Druid code changes. + Do a normal distribution build, then build a new image. +* Reuse the same image over and over if you only change tests + (such as when adding a new test.) +* Reuse the same `shared` directory when the test does not + make permanent changes. +* Change Druid configuration by changing the Docker compose + files, no need to rebuild the image. +* Work primarily in the IDE when debugging tests. +* To add more logging, change the `log4j2.xml` file in the shared + directory to increase the logging level. +* Remote debug Druid services if needed. + +## Exploring the Test Cluster + +When run in Docker Compose, the endpoints known to Druid nodes differ from +those needed by a client sitting outside the cluster. We could provide an +explicit mapping. Better is to use the +[Router](https://druid.apache.org/docs/latest/design/router.html#router-as-management-proxy) +to proxy requests. Fortunately, the Druid Console already does this. + +## Docker Build Output + +Modern Docker seems to hide the output of commands, which is a hassle to debug +a build. Oddly, the details appear for a failed build, but not for success. +Use the followig to see at least some output: + +```bash +export DOCKER_BUILDKIT=0 +``` + +Once the base container is built, you can run it, log in and poke around. First +identify the name. See the last line of the container build: + +```text +Successfully tagged org.apache.druid/test:<version> +``` + +Or ask Docker: + +```bash +docker images +``` + +## Debug the Docker Image + +You can log into the Docker image and poke around to see what's what: + + +```bash +docker run --rm -it --entrypoint bash org.apache.druid/test:<version> +``` + +Quite a few environment variables are provided by Docker and the setup scripts +to see them, within the container, use: + +```bash +env +``` + +## Debug an Integration Test + +To debug an integration test, you need a Docker image with the latest Druid. +To get that, you need a full Druid build. So, we break the debugging process +down into steps that depend on the state of your code. Assume `DRUID_DEV` +points to your Druid development area. + +### On Each Druid Build + +If you need to rebuild Druid (because you fixed something), do: + +* Do a distribution build of Druid: + +```bash +cd $DRUID_DEV +mvn clean package -P dist,skip-static-checks,skip-tests -Dmaven.javadoc.skip=true -T1.0C +``` + +* Build the test image. + +```bash +cd $DRUID_DEV/docker/test-image +mvn -P test-image install +``` + +### Start the Test Cluster + +* Pick a test "group" to use. Each is in a separate Maven project. + +```bash +cd $DRUID_DEV/docker-tests/<test group> +``` + +* Start a test cluster configured for this test. + +```bash +./start-cluster.sh +``` + +* To run a test from the command line: + +```bash +<command needed> Review Comment: pending TODO? ########## docker-tests/it-high-availability/src/test/java/org/apache/druid/testsEx/leadership/ITHighAvailabilityTest.java: ########## @@ -0,0 +1,242 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +package org.apache.druid.testsEx.leadership; + +import com.google.inject.Inject; +import org.apache.druid.cli.CliCustomNodeRole; +import org.apache.druid.common.config.NullHandling; +import org.apache.druid.discovery.DiscoveryDruidNode; +import org.apache.druid.discovery.DruidNodeDiscovery; +import org.apache.druid.discovery.DruidNodeDiscoveryProvider; +import org.apache.druid.discovery.NodeRole; +import org.apache.druid.java.util.common.StringUtils; +import org.apache.druid.java.util.common.logger.Logger; +import org.apache.druid.java.util.http.client.HttpClient; +import org.apache.druid.testing.IntegrationTestingConfig; +import org.apache.druid.testing.guice.TestClient; +import org.apache.druid.testing.utils.SqlTestQueryHelper; Review Comment: There seem to be some imports from the original `integration-tests`. Do we want to retain these as is? I guess this is why there is a dependency on `druid-integration-tests` in the pom.xml for this test group. ########## docker-tests/docs/docker.md: ########## @@ -0,0 +1,250 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +# Docker Test Image for Druid + +Integration tests need a Druid cluster. While some tests support using +Kubernetes for the Quickstart cluster, most need a cluster with some +test-specfic configuration. We use Docker Compose to create that cluster, Review Comment: Typo: ```suggestion test-specific configuration. We use Docker Compose to create that cluster, ``` ########## docker-tests/docs/docker.md: ########## @@ -0,0 +1,250 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +# Docker Test Image for Druid + +Integration tests need a Druid cluster. While some tests support using +Kubernetes for the Quickstart cluster, most need a cluster with some +test-specfic configuration. We use Docker Compose to create that cluster, +based on a test-oriented Docker image built by the `test-image` project. +The image contains the Druid distribution, +unpacked, along with the MySQL and MariaDB client libaries and +and the Kafka protobuf dependency. Docker Compose is +used to pass configuration specific to each service. + +In addition to the Druid image, we use "official" images for dependencies such +as ZooKeeper, MySQL and Kafka. + +The image here is distinct from the +["retail" image](https://druid.apache.org/docs/latest/tutorials/docker.html) +used for getting started. + +## Build Process + +Assuming `DRUID_DEV` points to your Druid build directory, +to build the image (only): + +```bash +cd $DRUID_DEV/docker-tests/test-image +mvn -P test-image install +``` + +Building of the image occurs in four steps: + +* The Maven `pom.xml` file gathers versions and other information from the build. + It also uses the normal Maven dependency mechanism to download the MySQL, + MariaDB and + Kafka client libraries, then copies them to the `target/docker` directory. + It then invokes the `build-image.sh` script. +* `build-image.sh` adds the Druid build tarball from `distribution/target`, + copies the contents of `test-image/docker` to `target/docker` and + then invokes the `docker build` command. +* `docker build` uses `target/docker` as the context, and thus + uses the `Dockerfile` to build the image. The `Dockerfile` copies artifacts into + the image, then defers to the `test-setup.sh` script. +* The `test-setup.sh` script is copied into the image and run. This script does + the work of installing Druid. + +The resulting image is named `org.apache.druid/test:<version>`. + +### Clean + +A normal `mvn clean` won't remove the Docker image because that is often not +what you want. Instead, do: + +```bash +mvn clean -P druid-image +``` + +You can also remove the image using Docker or the Docker desktop. + +### `target/docker` + +Docker requires that all build resources be within the current directory. We don't want +to change the source directory: in Maven, only the target directories should contain +build artifacts. So, we build up a `target/docker` directory in `pom.xml` file and the +and `build-image.sh` script: + +```text +/target/docker +|- Dockerfile (from docker/) +|- scripts (from docker/) +|- apache-druid-<version>-bin.tar.gz (from distribution, by build-image.sh) +|- MySQL client (done by pom.xml) +|- MariaDB client (done by pom.xml) +|- Kafka protobuf client (done by pom.xml) +``` + +Then, we invoke the `docker build` to build our test image. The `Dockerfile` copies +files into the image. Actual setup is done by the `test-setup.sh` script copied +into the image. + +Many Dockerfiles issue Linux commands inline. In some cases, this can speed up +subsequent builds because Docker can reuse layers. However, such Dockerfiles are +tedious to debug. It is far easier to do the detailed setup in a script within +the image. With this approach, you can debug the script by loading it into +the image, but don't run it in the Dockerfile. Instead, launch the image with +a `bash` shell and run the script by hand to debug. Since our build process +is quick, we don't lose much by reusing layers. + +### Manual Image Rebuilds + +You can quick rebuild the image if you've previously run a Maven image build. +Assume `DRUID_DEV` points to your Druid development root. Start with a +Maven build: + +```bash +cd $DRUID_DEV/docker/test-image +mvn -P image-image install Review Comment: ```suggestion mvn -P test-image install ``` ########## docker-tests/docs/tests.md: ########## @@ -0,0 +1,165 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +# Test Structure + +The structure of integration tests is heavily influenced by the existing +integration test structure. In that previous structure: + +* Each test group ran as separate Maven build. +* Each would build an image, start a cluster, run the test, and shut down the cluster. +* Tests were created using [TestNG](https://testng.org/doc/), a long-obsolete + test framework. +* A `IntegrationTestingConfig` is created from system properties (passed in from + Maven via `-D<key>=<value>` options). +* A TestNG test runner uses a part of the Druid Guice configuration to inject + test objects into the tests. +* The test then runs. + +To minimize test changes, we try to keep much of the "interface" while changing +the "implementation". Basically: + +* The same Docker image is used for all tests. +* Each test defines its own test cluster using Docker Compose. +* Maven runs tests one by one, starting and stopping the test-specific cluster for each. +* A `ClusterConfig` object defines the test configuration and creates the + `IntegrationTestingConfig` object. +* An instance of `Initializer` sets up Guice for each test and injects the + test objects. +* Tests run as JUnit tests. + +The remainer of this section describes the test internals. + +## Test Configuration + +See [Test Configuration](test-config.md) for details on the `docker.yaml` file +that you create for each test module to tell the tests about the cluster you +have defined. + +Test configuration allows inheritance so, as in Docker Compose, we define +standard bits in one place, just providing test-specific information in each +tests `docker.yaml` file. + +The test code assumes that the test configuration file is in `src/test/resources/yaml/docker.yaml` +(or, specifically that it is on the class path at `/yaml/docker.yaml`) +and loads it automatically into a `ClusterConfig` instance. + +The `ClusterConfig` instance provides the backward-compatible +`IntegrationTestingConfig` instance. + +New tests may want to work with `CluserConfig` directly as the older interface +is a bit of a muddle in several areas. + +## Initialization + +We want the new JUnit form of the integration tests to be as simple as possible +to debug. Rather than use a JUnit test suite as a replacement for the TestNG +test suite, we instead make each test class +independent. To do this, we insert code that initializes Guice from the test +configuration: + +```java +public class MyTest Review Comment: I guess this part needs changing as I see that `DruidTestRunner` seems to be handling this now. ########## docker-tests/docs/debugging.md: ########## @@ -0,0 +1,183 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +# Debugging the Druid Image and Integration Tests + +The integration setup has as a primary goal the ability to quickly debug +the Druid image and any individual tests. A first step is to move the +image build into a separate project. A second step is to ensure each +test can run in JUnit in an IDE against a cluster you start by hand. + +This section discusses how to use the various debugging features. + +See: + +* [Docker Configuration](docker.md) for information on debugging + docker builds. + +## General Debug Process + +Ease of debugging is a key goal of the revised structure. + +* Rebuild the Docker image only when the Druid code changes. + Do a normal distribution build, then build a new image. +* Reuse the same image over and over if you only change tests + (such as when adding a new test.) +* Reuse the same `shared` directory when the test does not + make permanent changes. +* Change Druid configuration by changing the Docker compose + files, no need to rebuild the image. +* Work primarily in the IDE when debugging tests. +* To add more logging, change the `log4j2.xml` file in the shared + directory to increase the logging level. +* Remote debug Druid services if needed. + +## Exploring the Test Cluster + +When run in Docker Compose, the endpoints known to Druid nodes differ from +those needed by a client sitting outside the cluster. We could provide an +explicit mapping. Better is to use the +[Router](https://druid.apache.org/docs/latest/design/router.html#router-as-management-proxy) +to proxy requests. Fortunately, the Druid Console already does this. + +## Docker Build Output + +Modern Docker seems to hide the output of commands, which is a hassle to debug +a build. Oddly, the details appear for a failed build, but not for success. +Use the followig to see at least some output: + +```bash +export DOCKER_BUILDKIT=0 +``` + +Once the base container is built, you can run it, log in and poke around. First +identify the name. See the last line of the container build: + +```text +Successfully tagged org.apache.druid/test:<version> +``` + +Or ask Docker: + +```bash +docker images +``` + +## Debug the Docker Image + +You can log into the Docker image and poke around to see what's what: + + +```bash +docker run --rm -it --entrypoint bash org.apache.druid/test:<version> +``` + +Quite a few environment variables are provided by Docker and the setup scripts +to see them, within the container, use: + +```bash +env +``` + +## Debug an Integration Test + +To debug an integration test, you need a Docker image with the latest Druid. +To get that, you need a full Druid build. So, we break the debugging process +down into steps that depend on the state of your code. Assume `DRUID_DEV` +points to your Druid development area. + +### On Each Druid Build + +If you need to rebuild Druid (because you fixed something), do: + +* Do a distribution build of Druid: + +```bash +cd $DRUID_DEV +mvn clean package -P dist,skip-static-checks,skip-tests -Dmaven.javadoc.skip=true -T1.0C +``` + +* Build the test image. + +```bash +cd $DRUID_DEV/docker/test-image +mvn -P test-image install +``` + +### Start the Test Cluster + +* Pick a test "group" to use. Each is in a separate Maven project. + +```bash +cd $DRUID_DEV/docker-tests/<test group> +``` + +* Start a test cluster configured for this test. + +```bash +./start-cluster.sh +``` + +* To run a test from the command line: + +```bash +<command needed> +``` + +### Debug the Test + +* To run from your IDE, find the test to run and run it as a JUnit test. + +Depending on the test, you may be able to run the test over and over against the +same cluster. (In fact, you should try to design your tests so that this is true: +clean up after each run.) + +The tests are just plain old JUnit tests that happen to reach out to the +test cluster and/or Docker to do their work. You can set breakpoints and debug +in the usual way. + +### Stop the Test Cluster + +* When done, stop the cluster. + +```bash +./stop-cluster.sh Review Comment: ```suggestion ./cluster.sh down ``` ########## docker-tests/docs/tests.md: ########## @@ -0,0 +1,165 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +# Test Structure + +The structure of integration tests is heavily influenced by the existing +integration test structure. In that previous structure: + +* Each test group ran as separate Maven build. +* Each would build an image, start a cluster, run the test, and shut down the cluster. +* Tests were created using [TestNG](https://testng.org/doc/), a long-obsolete + test framework. +* A `IntegrationTestingConfig` is created from system properties (passed in from + Maven via `-D<key>=<value>` options). +* A TestNG test runner uses a part of the Druid Guice configuration to inject + test objects into the tests. +* The test then runs. + +To minimize test changes, we try to keep much of the "interface" while changing +the "implementation". Basically: + +* The same Docker image is used for all tests. +* Each test defines its own test cluster using Docker Compose. +* Maven runs tests one by one, starting and stopping the test-specific cluster for each. +* A `ClusterConfig` object defines the test configuration and creates the + `IntegrationTestingConfig` object. +* An instance of `Initializer` sets up Guice for each test and injects the + test objects. +* Tests run as JUnit tests. + +The remainer of this section describes the test internals. Review Comment: Typo: ```suggestion The remainder of this section describes the test internals. ``` ########## docker-tests/docs/docker.md: ########## @@ -0,0 +1,250 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +# Docker Test Image for Druid + +Integration tests need a Druid cluster. While some tests support using +Kubernetes for the Quickstart cluster, most need a cluster with some +test-specfic configuration. We use Docker Compose to create that cluster, +based on a test-oriented Docker image built by the `test-image` project. +The image contains the Druid distribution, +unpacked, along with the MySQL and MariaDB client libaries and +and the Kafka protobuf dependency. Docker Compose is +used to pass configuration specific to each service. + +In addition to the Druid image, we use "official" images for dependencies such +as ZooKeeper, MySQL and Kafka. + +The image here is distinct from the +["retail" image](https://druid.apache.org/docs/latest/tutorials/docker.html) +used for getting started. + +## Build Process + +Assuming `DRUID_DEV` points to your Druid build directory, +to build the image (only): + +```bash +cd $DRUID_DEV/docker-tests/test-image Review Comment: ```suggestion cd $DRUID_DEV/docker-tests/it-image ``` ########## docker-tests/it-batch-index/cluster.sh: ########## @@ -0,0 +1,63 @@ +#! /bin/bash + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +#-------------------------------------------------------------------- + +# Starts the test-specific test cluster using Docker compose using +# versions and other settings gathered when building the images. + +SCRIPT_DIR=$(cd $(dirname $0) && pwd) Review Comment: As I see it, the contents of `cluster.sh` would be the same for every test group. Only the cluster type/config changes. Is it possible to avoid the duplication of the `cluster.sh`? ########## docker-tests/docs/docker.md: ########## @@ -0,0 +1,250 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +# Docker Test Image for Druid + +Integration tests need a Druid cluster. While some tests support using +Kubernetes for the Quickstart cluster, most need a cluster with some +test-specfic configuration. We use Docker Compose to create that cluster, +based on a test-oriented Docker image built by the `test-image` project. +The image contains the Druid distribution, +unpacked, along with the MySQL and MariaDB client libaries and +and the Kafka protobuf dependency. Docker Compose is +used to pass configuration specific to each service. + +In addition to the Druid image, we use "official" images for dependencies such +as ZooKeeper, MySQL and Kafka. + +The image here is distinct from the Review Comment: Maybe add a comment about how this image is different. ########## docker-tests/docs/debugging.md: ########## @@ -0,0 +1,183 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +# Debugging the Druid Image and Integration Tests + +The integration setup has as a primary goal the ability to quickly debug +the Druid image and any individual tests. A first step is to move the +image build into a separate project. A second step is to ensure each +test can run in JUnit in an IDE against a cluster you start by hand. + +This section discusses how to use the various debugging features. + +See: + +* [Docker Configuration](docker.md) for information on debugging + docker builds. + +## General Debug Process + +Ease of debugging is a key goal of the revised structure. + +* Rebuild the Docker image only when the Druid code changes. + Do a normal distribution build, then build a new image. +* Reuse the same image over and over if you only change tests + (such as when adding a new test.) +* Reuse the same `shared` directory when the test does not + make permanent changes. +* Change Druid configuration by changing the Docker compose + files, no need to rebuild the image. +* Work primarily in the IDE when debugging tests. +* To add more logging, change the `log4j2.xml` file in the shared + directory to increase the logging level. +* Remote debug Druid services if needed. + +## Exploring the Test Cluster + +When run in Docker Compose, the endpoints known to Druid nodes differ from +those needed by a client sitting outside the cluster. We could provide an +explicit mapping. Better is to use the +[Router](https://druid.apache.org/docs/latest/design/router.html#router-as-management-proxy) +to proxy requests. Fortunately, the Druid Console already does this. + +## Docker Build Output + +Modern Docker seems to hide the output of commands, which is a hassle to debug +a build. Oddly, the details appear for a failed build, but not for success. +Use the followig to see at least some output: + +```bash +export DOCKER_BUILDKIT=0 +``` + +Once the base container is built, you can run it, log in and poke around. First +identify the name. See the last line of the container build: + +```text +Successfully tagged org.apache.druid/test:<version> +``` + +Or ask Docker: + +```bash +docker images +``` + +## Debug the Docker Image + +You can log into the Docker image and poke around to see what's what: + + +```bash +docker run --rm -it --entrypoint bash org.apache.druid/test:<version> +``` + +Quite a few environment variables are provided by Docker and the setup scripts +to see them, within the container, use: + +```bash +env +``` + +## Debug an Integration Test + +To debug an integration test, you need a Docker image with the latest Druid. +To get that, you need a full Druid build. So, we break the debugging process +down into steps that depend on the state of your code. Assume `DRUID_DEV` +points to your Druid development area. + +### On Each Druid Build + +If you need to rebuild Druid (because you fixed something), do: + +* Do a distribution build of Druid: + +```bash +cd $DRUID_DEV +mvn clean package -P dist,skip-static-checks,skip-tests -Dmaven.javadoc.skip=true -T1.0C +``` + +* Build the test image. + +```bash +cd $DRUID_DEV/docker/test-image +mvn -P test-image install +``` + +### Start the Test Cluster + +* Pick a test "group" to use. Each is in a separate Maven project. + +```bash +cd $DRUID_DEV/docker-tests/<test group> +``` + +* Start a test cluster configured for this test. + +```bash +./start-cluster.sh Review Comment: I think this is now changed to `./cluster.sh` ########## docker-tests/docs/docker.md: ########## @@ -0,0 +1,250 @@ +<!-- + ~ Licensed to the Apache Software Foundation (ASF) under one + ~ or more contributor license agreements. See the NOTICE file + ~ distributed with this work for additional information + ~ regarding copyright ownership. The ASF licenses this file + ~ to you under the Apache License, Version 2.0 (the + ~ "License"); you may not use this file except in compliance + ~ with the License. You may obtain a copy of the License at + ~ + ~ http://www.apache.org/licenses/LICENSE-2.0 + ~ + ~ Unless required by applicable law or agreed to in writing, + ~ software distributed under the License is distributed on an + ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + ~ KIND, either express or implied. See the License for the + ~ specific language governing permissions and limitations + ~ under the License. + --> + +# Docker Test Image for Druid + +Integration tests need a Druid cluster. While some tests support using +Kubernetes for the Quickstart cluster, most need a cluster with some +test-specfic configuration. We use Docker Compose to create that cluster, +based on a test-oriented Docker image built by the `test-image` project. +The image contains the Druid distribution, +unpacked, along with the MySQL and MariaDB client libaries and +and the Kafka protobuf dependency. Docker Compose is +used to pass configuration specific to each service. + +In addition to the Druid image, we use "official" images for dependencies such +as ZooKeeper, MySQL and Kafka. + +The image here is distinct from the +["retail" image](https://druid.apache.org/docs/latest/tutorials/docker.html) +used for getting started. + +## Build Process + +Assuming `DRUID_DEV` points to your Druid build directory, +to build the image (only): + +```bash +cd $DRUID_DEV/docker-tests/test-image +mvn -P test-image install +``` + +Building of the image occurs in four steps: + +* The Maven `pom.xml` file gathers versions and other information from the build. + It also uses the normal Maven dependency mechanism to download the MySQL, + MariaDB and + Kafka client libraries, then copies them to the `target/docker` directory. + It then invokes the `build-image.sh` script. +* `build-image.sh` adds the Druid build tarball from `distribution/target`, + copies the contents of `test-image/docker` to `target/docker` and + then invokes the `docker build` command. +* `docker build` uses `target/docker` as the context, and thus + uses the `Dockerfile` to build the image. The `Dockerfile` copies artifacts into + the image, then defers to the `test-setup.sh` script. +* The `test-setup.sh` script is copied into the image and run. This script does + the work of installing Druid. + +The resulting image is named `org.apache.druid/test:<version>`. + +### Clean + +A normal `mvn clean` won't remove the Docker image because that is often not +what you want. Instead, do: + +```bash +mvn clean -P druid-image +``` + +You can also remove the image using Docker or the Docker desktop. + +### `target/docker` + +Docker requires that all build resources be within the current directory. We don't want +to change the source directory: in Maven, only the target directories should contain +build artifacts. So, we build up a `target/docker` directory in `pom.xml` file and the +and `build-image.sh` script: + +```text +/target/docker +|- Dockerfile (from docker/) +|- scripts (from docker/) +|- apache-druid-<version>-bin.tar.gz (from distribution, by build-image.sh) +|- MySQL client (done by pom.xml) +|- MariaDB client (done by pom.xml) +|- Kafka protobuf client (done by pom.xml) +``` + +Then, we invoke the `docker build` to build our test image. The `Dockerfile` copies +files into the image. Actual setup is done by the `test-setup.sh` script copied +into the image. + +Many Dockerfiles issue Linux commands inline. In some cases, this can speed up +subsequent builds because Docker can reuse layers. However, such Dockerfiles are +tedious to debug. It is far easier to do the detailed setup in a script within +the image. With this approach, you can debug the script by loading it into +the image, but don't run it in the Dockerfile. Instead, launch the image with +a `bash` shell and run the script by hand to debug. Since our build process +is quick, we don't lose much by reusing layers. + +### Manual Image Rebuilds + +You can quick rebuild the image if you've previously run a Maven image build. +Assume `DRUID_DEV` points to your Druid development root. Start with a +Maven build: + +```bash +cd $DRUID_DEV/docker/test-image Review Comment: Same fix required in other places in this file. ```suggestion cd $DRUID_DEV/docker/it-image ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
