Serge Huber created UNOMI-921:
---------------------------------
Summary: Replace elasticsearch-maven-plugin with Docker-based
Elasticsearch Instance in integration tests
Key: UNOMI-921
URL: https://issues.apache.org/jira/browse/UNOMI-921
Project: Apache Unomi
Issue Type: Bug
Components: unomi(-core)
Affects Versions: unomi-3.0.0, unomi-3.1.0
Reporter: Serge Huber
Fix For: unomi-3.1.0
h2. Summary
Replace the {{elasticsearch-maven-plugin}} with a Docker-based Elasticsearch
instance creation in the integration tests, following the same pattern used for
OpenSearch in {{itests/pom.xml}}.
h2. Description
Currently, the integration tests use {{elasticsearch-maven-plugin}}
(com.github.alexcojocaru) to download and run Elasticsearch directly during the
Maven build lifecycle. This approach has several drawbacks:
* Requires downloading Elasticsearch binaries during build
* Creates a problematic {{default_template}} that overrides user templates in
ES 8/9 (requiring a workaround in {{BaseIT.java}})
* Inconsistent with the OpenSearch approach which uses Docker
* More complex lifecycle management
The OpenSearch profile already uses {{docker-maven-plugin}} (io.fabric8)
successfully. We should align the Elasticsearch setup to use the same
Docker-based approach for consistency and to eliminate the template workaround.
h2. Current State
h3. Elasticsearch Profile ({{itests/pom.xml}})
* Uses {{elasticsearch-maven-plugin}} version 6.29
* Downloads and runs Elasticsearch directly (not containerized)
* HTTP port: 9400
* Transport port: 9500
* Elasticsearch version: 9.1.3 (from {{${elasticsearch.test.version}}})
* Java heap: 4GB ({{-Xms4g -Xmx4g}}) - *Note: Will be increased to 8GB to match
OpenSearch*
* Configuration includes:
** {{xpack.ml.enabled=false}}
** {{path.repo=${project.build.directory}/snapshots_repository}}
** {{cluster.routing.allocation.disk.threshold_enabled=false}}
** CORS settings
h3. Workaround in {{BaseIT.java}}
* {{fixDefaultTemplateIfNeeded()}} method (lines 1431-1520)
* Deletes {{default_template}} created by elasticsearch-maven-plugin
* Called early in test lifecycle before Unomi starts
h2. Desired State
h3. Elasticsearch Profile ({{itests/pom.xml}})
* Use {{docker-maven-plugin}} (io.fabric8) - already available in root pom.xml
* Run Elasticsearch in Docker container
* Use Docker image:
{{docker.elastic.co/elasticsearch/elasticsearch:${elasticsearch.test.version}}}
* Maintain same port configuration (9400 for HTTP)
* Configure similar environment variables and settings
* Proper container lifecycle management (stop/remove before start)
h3. Remove Workaround
* Remove {{fixDefaultTemplateIfNeeded()}} method from {{BaseIT.java}}
* Remove call to {{fixDefaultTemplateIfNeeded()}} in {{checkSearchEngine()}}
method
* Remove related comments about elasticsearch-maven-plugin template issue
h2. Acceptance Criteria
* (/) Elasticsearch integration tests run using Docker container instead of
elasticsearch-maven-plugin
* (/) All existing integration tests pass with Docker-based Elasticsearch
* (/) Container lifecycle matches OpenSearch exactly (stop/remove before start,
stop but don't remove after tests for inspection)
* (/) Port configuration remains the same (9400) to avoid breaking existing
tests
* (/) Workaround code in {{BaseIT.java}} is removed
* (/) Configuration matches OpenSearch pattern for consistency
* (/) Heap size aligned to 8GB (matching OpenSearch and ES 9 recommendations)
* (/) Environment variables aligned with OpenSearch where applicable
(replicas=0, heap size)
* (/) {{elasticsearch-maven-plugin}} is removed from {{itests/pom.xml}}
* (/) {{elasticsearch-maven-plugin.version}} property can be removed from root
{{pom.xml}} (if not used elsewhere)
* (/) Documentation/comments are updated to reflect Docker-based approach
* (/) Container lifecycle matches OpenSearch (stopped but not removed after
tests for inspection)
* (/) {{Migrate16xToCurrentVersionIT.java}} uses dynamic port resolution
instead of hardcoded port
h2. Technical Details
h3. Docker Image
* Official Elasticsearch Docker image:
{{docker.elastic.co/elasticsearch/elasticsearch:9.1.3}}
* Alternative: {{elasticsearch:9.1.3}} (if using Docker Hub)
h3. Port Mapping
* Container port 9200 → Host port 9400 (to maintain compatibility with existing
tests)
h3. Environment Variables
* {{discovery.type=single-node}} (for single-node cluster)
* {{ES_JAVA_OPTS=-Xms8g -Xmx8g
-Dcluster.default.index.settings.number_of_replicas=0}} (aligned with
OpenSearch: 8GB heap for consistency, replicas=0 for single-node test setup)
* {{xpack.ml.enabled=false}}
* {{path.repo=/tmp/snapshots_repository}} (mapped to
{{${project.build.directory}/snapshots_repository}})
* {{cluster.routing.allocation.disk.threshold_enabled=false}}
* CORS settings (if needed via environment variables or config file)
*Note: Heap size increased from 4GB to 8GB to match OpenSearch configuration
and align with ES 9 recommendations (50% of available RAM, up to 32GB). For
integration tests, 8GB provides better consistency with OpenSearch setup.*
h3. Volume Mapping
* {{${project.build.directory}/snapshots_repository:/tmp/snapshots_repository}}
(for snapshot repository)
h3. Container Lifecycle
* *Pre-integration-test phase* (in order):
** First execution: {{stop}} + {{remove}} goals to clean up any existing
container
** Second execution: {{start}} goal to start new container
** Use HTTP wait condition to ensure Elasticsearch is ready before tests run
* *Post-integration-test phase*:
** {{stop}} goal to stop container (but *do not remove* - allows for post-test
inspection/analysis)
** Container remains stopped but exists for debugging purposes
*Note: This matches the OpenSearch lifecycle exactly - container is stopped but
not removed after tests, enabling after-the-run analysis.*
h2. Implementation Steps
# *Update {{itests/pom.xml}} - Elasticsearch Profile*
** Remove {{elasticsearch-maven-plugin}} configuration (lines 293-340)
** Add {{docker-maven-plugin}} configuration similar to OpenSearch profile
** Configure Elasticsearch Docker image with appropriate settings
** Set up port mapping (9400:9200)
** Configure environment variables aligned with OpenSearch:
*** {{ES_JAVA_OPTS=-Xms8g -Xmx8g
-Dcluster.default.index.settings.number_of_replicas=0}} (8GB heap, replicas=0)
*** {{discovery.type=single-node}}
*** {{xpack.ml.enabled=false}}
*** {{path.repo=/tmp/snapshots_repository}}
*** {{cluster.routing.allocation.disk.threshold_enabled=false}}
** Configure volume mounts (snapshots repository)
** Add container lifecycle management matching OpenSearch:
*** Pre-integration-test: First execution with {{stop}} + {{remove}} goals,
then second execution with {{start}} goal
*** Post-integration-test: {{stop}} goal only (container remains for
inspection, not removed)
# *Update {{BaseIT.java}}*
** Remove {{fixDefaultTemplateIfNeeded()}} method (lines 1431-1543)
** Remove call to {{fixDefaultTemplateIfNeeded()}} from {{checkSearchEngine()}}
method (line 216)
** Remove related comments about elasticsearch-maven-plugin template issue
(lines 212-214, 1425-1429, 1485-1489)
** Verify {{getSearchPort()}} method works correctly with Docker (currently
uses {{System.getProperty("elasticsearch.port", "9400")}} - should work as-is
since port remains 9400)
** Verify ES configuration in {{config()}} method (lines 575-578) works with
Docker setup
# *Update Root {{pom.xml}} (if applicable)*
** Check if {{elasticsearch-maven-plugin.version}} property (line 160) is used
elsewhere
** Remove property if only used in itests
# *Fix {{Migrate16xToCurrentVersionIT.java}}*
** Replace hardcoded {{ES_BASE_URL = "http://localhost:9400"}} (line 56) with
dynamic port using {{getSearchPort()}} method
** Update all references to {{ES_BASE_URL}} to use dynamic port resolution
** This test directly connects to ES for migration testing and snapshot
operations, so it must use the correct port
# *Update {{build.sh}} (optional, for future implementation)*
** Add Docker availability check in {{check_requirements()}} function when
integration tests are enabled
** Check for Docker command availability
** Check if Docker daemon is running
** Provide helpful error messages with installation instructions if Docker is
missing
** *Note: This is not part of the initial implementation, but should be
considered for future enhancement*
# *Testing*
** Run integration tests with Elasticsearch profile: {{mvn clean verify
-Pelasticsearch}}
** Verify all tests pass, especially:
*** {{Migrate16xToCurrentVersionIT}} (uses direct ES connections)
*** Tests that use {{getSearchPort()}} method
*** Tests that rely on ES configuration from {{BaseIT.config()}}
** Verify container starts and stops correctly
** Verify no {{default_template}} workaround is needed
** Compare behavior with OpenSearch Docker setup
h2. Reference Implementation
The OpenSearch profile in {{itests/pom.xml}} (lines 344-455) provides a
complete reference:
* Uses {{docker-maven-plugin}} (io.fabric8)
* Container lifecycle management:
** Pre-integration-test: First execution stops and removes existing container,
second execution starts new container
** Post-integration-test: Stops container but does not remove it (allows for
post-test inspection/analysis)
* HTTP wait condition for readiness
* Volume mounting for snapshots repository
* Environment variable configuration
h2. OpenSearch Configuration Analysis
h3. Environment Variables Comparison
*Current OpenSearch Configuration:*
* {{discovery.type=single-node}}
* {{OPENSEARCH_JAVA_OPTS=-Xms8g -Xmx8g
-Dcluster.default.index.settings.number_of_replicas=0}}
* {{path.repo=/tmp/snapshots_repository}}
* {{plugins.security.disabled=true}} (OpenSearch-specific)
* {{OPENSEARCH_INITIAL_ADMIN_PASSWORD=Unomi.1ntegrat10n.Tests}}
(OpenSearch-specific)
*Recommended Elasticsearch Configuration (aligned with OpenSearch where
applicable):*
* {{discovery.type=single-node}} ✓ (same)
* {{ES_JAVA_OPTS=-Xms8g -Xmx8g
-Dcluster.default.index.settings.number_of_replicas=0}} ✓ (aligned: 8GB heap,
replicas=0)
* {{path.repo=/tmp/snapshots_repository}} ✓ (same)
* {{xpack.ml.enabled=false}} (ES-specific, equivalent to disabling ML features)
* {{cluster.routing.allocation.disk.threshold_enabled=false}} (ES-specific
setting)
* CORS settings (ES-specific configuration)
*Note: OpenSearch-specific settings ({{plugins.security.disabled}},
{{OPENSEARCH_INITIAL_ADMIN_PASSWORD}}) are not applicable to Elasticsearch,
which uses a different security model.*
h3. Heap Size Alignment
* *Current Elasticsearch*: 4GB heap ({{-Xms4g -Xmx4g}})
* *OpenSearch*: 8GB heap ({{-Xms8g -Xmx8g}})
* *ES 9 Recommendation*: 50% of available RAM, up to 32GB maximum
* *Decision*: Align Elasticsearch to 8GB to match OpenSearch for consistency
and better performance in integration tests
h3. OpenSearch Dashboard
*OpenSearch Dashboard is NOT deployed in integration tests.* The dashboard
({{opensearch-dashboards}}) is only present in:
* {{docker/src/main/docker/docker-compose-os.yml}} - Full deployment setup
* {{docker/src/main/docker/docker-compose-build-os.yml}} - Build environment
setup
The integration tests in {{itests/pom.xml}} only deploy the OpenSearch server
container, not the dashboard. Therefore, Elasticsearch should also NOT deploy
Kibana (Elasticsearch's equivalent dashboard) in the integration tests to
maintain consistency.
h2. Benefits
# *Consistency*: Both Elasticsearch and OpenSearch use the same Docker-based
approach
# *Simplification*: Removes workaround code for template issue
# *Reliability*: Docker containers are more isolated and predictable
# *Maintainability*: Easier to update Elasticsearch versions (just change image
tag)
# *Resource Management*: Better container lifecycle management
# *CI/CD*: Docker-based approach is more standard in modern CI/CD pipelines
h2. Potential Issues & Considerations
# *Docker Requirement*: Tests will require Docker to be available (same as
OpenSearch tests)
# *Port Conflicts*: Ensure port 9400 is available (same as current setup)
# *Elasticsearch License*: Ensure Docker image license is compatible (official
images may have different licensing)
# *Startup Time*: Docker container startup may have different timing than
direct execution
# *Network Configuration*: Ensure container networking allows connection from
host
# *Heap Size Increase*: Changing from 4GB to 8GB may require more system
resources, but aligns with OpenSearch and ES 9 best practices
# *Dashboard Deployment*: OpenSearch does NOT deploy dashboard in integration
tests (only in docker-compose files), so Elasticsearch should also NOT deploy
Kibana to maintain consistency
h2. Integration Test Impact Analysis
h3. Files Requiring Updates
* *{{BaseIT.java}}*:
** {{fixDefaultTemplateIfNeeded()}} method (lines 1431-1543) - *Remove*
(workaround no longer needed)
** {{checkSearchEngine()}} method (line 216) - *Remove* call to
{{fixDefaultTemplateIfNeeded()}}
** {{getSearchPort()}} method (lines 1294-1304) - *Verify* works correctly
(uses {{System.getProperty("elasticsearch.port", "9400")}})
** ES configuration in {{config()}} method (lines 575-578) - *Verify* works
with Docker setup
* *{{Migrate16xToCurrentVersionIT.java}}*:
** Hardcoded {{ES_BASE_URL = "http://localhost:9400"}} (line 56) - *Replace*
with dynamic port using {{getSearchPort()}}
** All references to {{ES_BASE_URL}} throughout the file - *Update* to use
dynamic port resolution
** This test directly connects to ES for migration testing and snapshot
operations, so port must be correct
h3. Tests That Should Work Without Changes
Most integration tests use the {{PersistenceService}} abstraction layer and
should work without modification:
* All tests extending {{BaseIT}} that use {{persistenceService}}
* Tests that rely on Unomi services rather than direct ES connections
* Tests that use {{getSearchPort()}} method (already dynamic)
h3. Potential Issues
* *Port Hardcoding*: {{Migrate16xToCurrentVersionIT}} has hardcoded port that
must be fixed
* *Startup Timing*: Docker container startup may have different timing than
direct execution - HTTP wait condition should handle this
* *Network Access*: Ensure Docker container networking allows connection from
host (default bridge network should work)
h2. Related Files
* {{itests/pom.xml}} - Main configuration file
* {{itests/src/test/java/org/apache/unomi/itests/BaseIT.java}} - Test base
class with workaround
*
{{itests/src/test/java/org/apache/unomi/itests/migration/Migrate16xToCurrentVersionIT.java}}
- Migration test with hardcoded port
* {{build.sh}} - Build script (needs Docker check)
* {{pom.xml}} - Root POM with plugin version properties
h2. Dependencies
* {{docker-maven-plugin}} version 0.46.0 (already in root pom.xml)
* Docker daemon must be running
* Elasticsearch Docker image must be pullable
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
