[
https://issues.apache.org/jira/browse/UNOMI-921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Serge Huber reassigned UNOMI-921:
---------------------------------
Assignee: Serge Huber
> Replace elasticsearch-maven-plugin with Docker-based Elasticsearch Instance
> in integration tests
> ------------------------------------------------------------------------------------------------
>
> Key: UNOMI-921
> URL: https://issues.apache.org/jira/browse/UNOMI-921
> Project: Apache Unomi
> Issue Type: Bug
> Components: unomi(-core)
> Affects Versions: unomi-3.0.0, unomi-3.1.0
> Reporter: Serge Huber
> Assignee: Serge Huber
> Priority: Major
> Fix For: unomi-3.1.0
>
>
> h2. Summary
> Replace the {{elasticsearch-maven-plugin}} with a Docker-based Elasticsearch
> instance creation in the integration tests, following the same pattern used
> for OpenSearch in {{itests/pom.xml}}.
> h2. Description
> Currently, the integration tests use {{elasticsearch-maven-plugin}}
> (com.github.alexcojocaru) to download and run Elasticsearch directly during
> the Maven build lifecycle. This approach has several drawbacks:
> * Requires downloading Elasticsearch binaries during build
> * Creates a problematic {{default_template}} that overrides user templates in
> ES 8/9 (requiring a workaround in {{BaseIT.java}})
> * Inconsistent with the OpenSearch approach which uses Docker
> * More complex lifecycle management
> The OpenSearch profile already uses {{docker-maven-plugin}} (io.fabric8)
> successfully. We should align the Elasticsearch setup to use the same
> Docker-based approach for consistency and to eliminate the template
> workaround.
> h2. Current State
> h3. Elasticsearch Profile ({{itests/pom.xml}})
> * Uses {{elasticsearch-maven-plugin}} version 6.29
> * Downloads and runs Elasticsearch directly (not containerized)
> * HTTP port: 9400
> * Transport port: 9500
> * Elasticsearch version: 9.1.3 (from {{${elasticsearch.test.version}}})
> * Java heap: 4GB ({{-Xms4g -Xmx4g}}) - *Note: Will be increased to 8GB to
> match OpenSearch*
> * Configuration includes:
> ** {{xpack.ml.enabled=false}}
> ** {{path.repo=${project.build.directory}/snapshots_repository}}
> ** {{cluster.routing.allocation.disk.threshold_enabled=false}}
> ** CORS settings
> h3. Workaround in {{BaseIT.java}}
> * {{fixDefaultTemplateIfNeeded()}} method (lines 1431-1520)
> * Deletes {{default_template}} created by elasticsearch-maven-plugin
> * Called early in test lifecycle before Unomi starts
> h2. Desired State
> h3. Elasticsearch Profile ({{itests/pom.xml}})
> * Use {{docker-maven-plugin}} (io.fabric8) - already available in root pom.xml
> * Run Elasticsearch in Docker container
> * Use Docker image:
> {{docker.elastic.co/elasticsearch/elasticsearch:${elasticsearch.test.version}}}
> * Maintain same port configuration (9400 for HTTP)
> * Configure similar environment variables and settings
> * Proper container lifecycle management (stop/remove before start)
> h3. Remove Workaround
> * Remove {{fixDefaultTemplateIfNeeded()}} method from {{BaseIT.java}}
> * Remove call to {{fixDefaultTemplateIfNeeded()}} in {{checkSearchEngine()}}
> method
> * Remove related comments about elasticsearch-maven-plugin template issue
> h2. Acceptance Criteria
> * (/) Elasticsearch integration tests run using Docker container instead of
> elasticsearch-maven-plugin
> * (/) All existing integration tests pass with Docker-based Elasticsearch
> * (/) Container lifecycle matches OpenSearch exactly (stop/remove before
> start, stop but don't remove after tests for inspection)
> * (/) Port configuration remains the same (9400) to avoid breaking existing
> tests
> * (/) Workaround code in {{BaseIT.java}} is removed
> * (/) Configuration matches OpenSearch pattern for consistency
> * (/) Heap size aligned to 8GB (matching OpenSearch and ES 9 recommendations)
> * (/) Environment variables aligned with OpenSearch where applicable
> (replicas=0, heap size)
> * (/) {{elasticsearch-maven-plugin}} is removed from {{itests/pom.xml}}
> * (/) {{elasticsearch-maven-plugin.version}} property can be removed from
> root {{pom.xml}} (if not used elsewhere)
> * (/) Documentation/comments are updated to reflect Docker-based approach
> * (/) Container lifecycle matches OpenSearch (stopped but not removed after
> tests for inspection)
> * (/) {{Migrate16xToCurrentVersionIT.java}} uses dynamic port resolution
> instead of hardcoded port
> h2. Technical Details
> h3. Docker Image
> * Official Elasticsearch Docker image:
> {{docker.elastic.co/elasticsearch/elasticsearch:9.1.3}}
> * Alternative: {{elasticsearch:9.1.3}} (if using Docker Hub)
> h3. Port Mapping
> * Container port 9200 → Host port 9400 (to maintain compatibility with
> existing tests)
> h3. Environment Variables
> * {{discovery.type=single-node}} (for single-node cluster)
> * {{ES_JAVA_OPTS=-Xms8g -Xmx8g
> -Dcluster.default.index.settings.number_of_replicas=0}} (aligned with
> OpenSearch: 8GB heap for consistency, replicas=0 for single-node test setup)
> * {{xpack.ml.enabled=false}}
> * {{path.repo=/tmp/snapshots_repository}} (mapped to
> {{${project.build.directory}/snapshots_repository}})
> * {{cluster.routing.allocation.disk.threshold_enabled=false}}
> * CORS settings (if needed via environment variables or config file)
> *Note: Heap size increased from 4GB to 8GB to match OpenSearch configuration
> and align with ES 9 recommendations (50% of available RAM, up to 32GB). For
> integration tests, 8GB provides better consistency with OpenSearch setup.*
> h3. Volume Mapping
> *
> {{${project.build.directory}/snapshots_repository:/tmp/snapshots_repository}}
> (for snapshot repository)
> h3. Container Lifecycle
> * *Pre-integration-test phase* (in order):
> ** First execution: {{stop}} + {{remove}} goals to clean up any existing
> container
> ** Second execution: {{start}} goal to start new container
> ** Use HTTP wait condition to ensure Elasticsearch is ready before tests run
> * *Post-integration-test phase*:
> ** {{stop}} goal to stop container (but *do not remove* - allows for
> post-test inspection/analysis)
> ** Container remains stopped but exists for debugging purposes
> *Note: This matches the OpenSearch lifecycle exactly - container is stopped
> but not removed after tests, enabling after-the-run analysis.*
> h2. Implementation Steps
> # *Update {{itests/pom.xml}} - Elasticsearch Profile*
> ** Remove {{elasticsearch-maven-plugin}} configuration (lines 293-340)
> ** Add {{docker-maven-plugin}} configuration similar to OpenSearch profile
> ** Configure Elasticsearch Docker image with appropriate settings
> ** Set up port mapping (9400:9200)
> ** Configure environment variables aligned with OpenSearch:
> *** {{ES_JAVA_OPTS=-Xms8g -Xmx8g
> -Dcluster.default.index.settings.number_of_replicas=0}} (8GB heap, replicas=0)
> *** {{discovery.type=single-node}}
> *** {{xpack.ml.enabled=false}}
> *** {{path.repo=/tmp/snapshots_repository}}
> *** {{cluster.routing.allocation.disk.threshold_enabled=false}}
> ** Configure volume mounts (snapshots repository)
> ** Add container lifecycle management matching OpenSearch:
> *** Pre-integration-test: First execution with {{stop}} + {{remove}} goals,
> then second execution with {{start}} goal
> *** Post-integration-test: {{stop}} goal only (container remains for
> inspection, not removed)
> # *Update {{BaseIT.java}}*
> ** Remove {{fixDefaultTemplateIfNeeded()}} method (lines 1431-1543)
> ** Remove call to {{fixDefaultTemplateIfNeeded()}} from
> {{checkSearchEngine()}} method (line 216)
> ** Remove related comments about elasticsearch-maven-plugin template issue
> (lines 212-214, 1425-1429, 1485-1489)
> ** Verify {{getSearchPort()}} method works correctly with Docker (currently
> uses {{System.getProperty("elasticsearch.port", "9400")}} - should work as-is
> since port remains 9400)
> ** Verify ES configuration in {{config()}} method (lines 575-578) works with
> Docker setup
> # *Update Root {{pom.xml}} (if applicable)*
> ** Check if {{elasticsearch-maven-plugin.version}} property (line 160) is
> used elsewhere
> ** Remove property if only used in itests
> # *Fix {{Migrate16xToCurrentVersionIT.java}}*
> ** Replace hardcoded {{ES_BASE_URL = "http://localhost:9400"}} (line 56) with
> dynamic port using {{getSearchPort()}} method
> ** Update all references to {{ES_BASE_URL}} to use dynamic port resolution
> ** This test directly connects to ES for migration testing and snapshot
> operations, so it must use the correct port
> # *Update {{build.sh}} (optional, for future implementation)*
> ** Add Docker availability check in {{check_requirements()}} function when
> integration tests are enabled
> ** Check for Docker command availability
> ** Check if Docker daemon is running
> ** Provide helpful error messages with installation instructions if Docker is
> missing
> ** *Note: This is not part of the initial implementation, but should be
> considered for future enhancement*
> # *Testing*
> ** Run integration tests with Elasticsearch profile: {{mvn clean verify
> -Pelasticsearch}}
> ** Verify all tests pass, especially:
> *** {{Migrate16xToCurrentVersionIT}} (uses direct ES connections)
> *** Tests that use {{getSearchPort()}} method
> *** Tests that rely on ES configuration from {{BaseIT.config()}}
> ** Verify container starts and stops correctly
> ** Verify no {{default_template}} workaround is needed
> ** Compare behavior with OpenSearch Docker setup
> h2. Reference Implementation
> The OpenSearch profile in {{itests/pom.xml}} (lines 344-455) provides a
> complete reference:
> * Uses {{docker-maven-plugin}} (io.fabric8)
> * Container lifecycle management:
> ** Pre-integration-test: First execution stops and removes existing
> container, second execution starts new container
> ** Post-integration-test: Stops container but does not remove it (allows for
> post-test inspection/analysis)
> * HTTP wait condition for readiness
> * Volume mounting for snapshots repository
> * Environment variable configuration
> h2. OpenSearch Configuration Analysis
> h3. Environment Variables Comparison
> *Current OpenSearch Configuration:*
> * {{discovery.type=single-node}}
> * {{OPENSEARCH_JAVA_OPTS=-Xms8g -Xmx8g
> -Dcluster.default.index.settings.number_of_replicas=0}}
> * {{path.repo=/tmp/snapshots_repository}}
> * {{plugins.security.disabled=true}} (OpenSearch-specific)
> * {{OPENSEARCH_INITIAL_ADMIN_PASSWORD=Unomi.1ntegrat10n.Tests}}
> (OpenSearch-specific)
> *Recommended Elasticsearch Configuration (aligned with OpenSearch where
> applicable):*
> * {{discovery.type=single-node}} ✓ (same)
> * {{ES_JAVA_OPTS=-Xms8g -Xmx8g
> -Dcluster.default.index.settings.number_of_replicas=0}} ✓ (aligned: 8GB heap,
> replicas=0)
> * {{path.repo=/tmp/snapshots_repository}} ✓ (same)
> * {{xpack.ml.enabled=false}} (ES-specific, equivalent to disabling ML
> features)
> * {{cluster.routing.allocation.disk.threshold_enabled=false}} (ES-specific
> setting)
> * CORS settings (ES-specific configuration)
> *Note: OpenSearch-specific settings ({{plugins.security.disabled}},
> {{OPENSEARCH_INITIAL_ADMIN_PASSWORD}}) are not applicable to Elasticsearch,
> which uses a different security model.*
> h3. Heap Size Alignment
> * *Current Elasticsearch*: 4GB heap ({{-Xms4g -Xmx4g}})
> * *OpenSearch*: 8GB heap ({{-Xms8g -Xmx8g}})
> * *ES 9 Recommendation*: 50% of available RAM, up to 32GB maximum
> * *Decision*: Align Elasticsearch to 8GB to match OpenSearch for consistency
> and better performance in integration tests
> h3. OpenSearch Dashboard
> *OpenSearch Dashboard is NOT deployed in integration tests.* The dashboard
> ({{opensearch-dashboards}}) is only present in:
> * {{docker/src/main/docker/docker-compose-os.yml}} - Full deployment setup
> * {{docker/src/main/docker/docker-compose-build-os.yml}} - Build environment
> setup
> The integration tests in {{itests/pom.xml}} only deploy the OpenSearch server
> container, not the dashboard. Therefore, Elasticsearch should also NOT deploy
> Kibana (Elasticsearch's equivalent dashboard) in the integration tests to
> maintain consistency.
> h2. Benefits
> # *Consistency*: Both Elasticsearch and OpenSearch use the same Docker-based
> approach
> # *Simplification*: Removes workaround code for template issue
> # *Reliability*: Docker containers are more isolated and predictable
> # *Maintainability*: Easier to update Elasticsearch versions (just change
> image tag)
> # *Resource Management*: Better container lifecycle management
> # *CI/CD*: Docker-based approach is more standard in modern CI/CD pipelines
> h2. Potential Issues & Considerations
> # *Docker Requirement*: Tests will require Docker to be available (same as
> OpenSearch tests)
> # *Port Conflicts*: Ensure port 9400 is available (same as current setup)
> # *Elasticsearch License*: Ensure Docker image license is compatible
> (official images may have different licensing)
> # *Startup Time*: Docker container startup may have different timing than
> direct execution
> # *Network Configuration*: Ensure container networking allows connection from
> host
> # *Heap Size Increase*: Changing from 4GB to 8GB may require more system
> resources, but aligns with OpenSearch and ES 9 best practices
> # *Dashboard Deployment*: OpenSearch does NOT deploy dashboard in integration
> tests (only in docker-compose files), so Elasticsearch should also NOT deploy
> Kibana to maintain consistency
> h2. Integration Test Impact Analysis
> h3. Files Requiring Updates
> * *{{BaseIT.java}}*:
> ** {{fixDefaultTemplateIfNeeded()}} method (lines 1431-1543) - *Remove*
> (workaround no longer needed)
> ** {{checkSearchEngine()}} method (line 216) - *Remove* call to
> {{fixDefaultTemplateIfNeeded()}}
> ** {{getSearchPort()}} method (lines 1294-1304) - *Verify* works correctly
> (uses {{System.getProperty("elasticsearch.port", "9400")}})
> ** ES configuration in {{config()}} method (lines 575-578) - *Verify* works
> with Docker setup
> * *{{Migrate16xToCurrentVersionIT.java}}*:
> ** Hardcoded {{ES_BASE_URL = "http://localhost:9400"}} (line 56) - *Replace*
> with dynamic port using {{getSearchPort()}}
> ** All references to {{ES_BASE_URL}} throughout the file - *Update* to use
> dynamic port resolution
> ** This test directly connects to ES for migration testing and snapshot
> operations, so port must be correct
> h3. Tests That Should Work Without Changes
> Most integration tests use the {{PersistenceService}} abstraction layer and
> should work without modification:
> * All tests extending {{BaseIT}} that use {{persistenceService}}
> * Tests that rely on Unomi services rather than direct ES connections
> * Tests that use {{getSearchPort()}} method (already dynamic)
> h3. Potential Issues
> * *Port Hardcoding*: {{Migrate16xToCurrentVersionIT}} has hardcoded port that
> must be fixed
> * *Startup Timing*: Docker container startup may have different timing than
> direct execution - HTTP wait condition should handle this
> * *Network Access*: Ensure Docker container networking allows connection from
> host (default bridge network should work)
> h2. Related Files
> * {{itests/pom.xml}} - Main configuration file
> * {{itests/src/test/java/org/apache/unomi/itests/BaseIT.java}} - Test base
> class with workaround
> *
> {{itests/src/test/java/org/apache/unomi/itests/migration/Migrate16xToCurrentVersionIT.java}}
> - Migration test with hardcoded port
> * {{build.sh}} - Build script (needs Docker check)
> * {{pom.xml}} - Root POM with plugin version properties
> h2. Dependencies
> * {{docker-maven-plugin}} version 0.46.0 (already in root pom.xml)
> * Docker daemon must be running
> * Elasticsearch Docker image must be pullable
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
