paul-rogers opened a new pull request, #13877: URL: https://github.com/apache/druid/pull/13877
The "new" IT framework provides a convenient way to package and run integration tests (ITs), but only for core modules. We have a use case to run an IT for a `contrib` extension: the proposed gRPC query extension. This PR provides the IT framework functionality to allow non-core ITs. The `it.sh` script takes an option third parameter to specify the relative path to the Maven module that holds the test: ```bash ./it.sh test MyCategory extensions-contrib/my-extension-it ``` Because of the Maven build order, the ITs for an extension must be in a Maven module separate from the extension itself. In this case, `my-extension` is the extension, while `my-extension-it` are the ITs for that extension. The layout in the extension mimics the existing `cases` module. Provide a Maven file that runs the tests using FailSafe. Provide a `cluster` directory structured as explained in the IT docs. #### Optional Extensions Path To test an extension, we first have to load it onto the IT cluster. One way to do that would be to add it to the standard IT test image. However, that's an awkward choice for a community extension. Another choice would be to build a separate image for each community extension, which would also be awkward. The solution provided here is to add an extensions search path to Druid configuration. If `druid.extensions.path` is set, it names additional directories to search for extensions. Example: ```text druid.extensions.path=["/shared/extensions"] ``` The `path` is empty by default. If present, the "effective path" is a list consisting of first the `directory` setting, followed by the `path` items. This feature is used in an IT to add the contrib extension to an `extensions` folder in the "shared" directory before starting the IT cluster. The setup work is done in the `setup.sh` script (see below) just before cluster start. The `path` allows us to specify an effective path of `$DRUID_HOME/extensions,/shared/extensions`. With this, the contrib extension is found as normal, if the extension is listed in `loadList`. The `launch.sh` script in the IT container looks for `/shared/extensions`. If found, it does the work to define the `path` property. Tests _do not_ need to add this property to the (generated) `docker-compose.yaml` file. This change makes `directory` optional. However, it defaults to `extensions`. For completeness, this PR allows a value of a blank string, which means "ignore `directory`, just use `path`." The new `path` variable is completely transparent to the user. Current behavior does not change. The only users who might need this feature are those running under K8s who would benefit from providing extensions as file mounts. #### Revision to IT Structure As part of this work, several changes were made to the structure of the existing IT directory. * The Python "template" for Docker Compose moves from `templates/Category.py` to `cluster/Category/docker-compose.py`. * A new 'cluster/Category/verify.sh` script peforms pre-test checks. The awkward code that was in `.it.sh` to check for required environment variables moved into `verify.sh`. * A new `cluster/Category/setup.sh` script does any additional setup needed before launching an IT cluster. Use this to add additional files to the `shared` directory, for example. Using the script is _far_ easier than trying to script in Maven. * The cluster startup and shutdown steps were removed from `cases/pom.xml`. We were not using them: we now use `it.sh` to do this work. * The `it.sh` and `cluster.sh` scripts are now aware of two modules: the "base" `cases` module, and the "test" module. Before now, those were the same. Not they can differ. #### Prevent Snapshot Downloading Also added configuration to the root `pom.xml` file to prevent an attempt to download snapshots from upstream. This most often happens when dependencies are missing and Maven cannot find a required Druid module, and instead tries to ask upstream repos. Bad Things would happen if it actually found such jars. #### Release note Added the `druid.extensions.path` config setting. This setting is optional and is normally not set. Primarily for testing. See `configuration/index.md` for details. <hr> This PR has: - [X] been self-reviewed. - [X] added internal documentation for new or modified features or behaviors. - [X] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader. - [X] added integration tests. - [X] been tested in an integration test Druid cluster. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
