sergehuber opened a new pull request, #715: URL: https://github.com/apache/unomi/pull/715
Dear Apache Unomi developers and users, After 8 months of development, and while I'm at it to start 2025 on a high note, I am proud to share with you and submit for review my contribution on the support of OpenSearch as an alternative to ElasticSearch for Apache Unomi. A lot of work has gone into this contribution, and I can fully understand that some of you will have questions. I propose to organize a meeting to go over the changes and also to answer any questions you might have. LIVE DEMONSTRATION ANNOUNCEMENT ----------------------------- I'm pleased to invite you to a live presentation and demonstration of the OpenSearch integration and new development tools. Date: January 13th, 2025 Time: 17:00 CET Format: Online presentation with live demonstration Link: https://calendar.app.google/7ZffzAgwPwhamAAU6 Agenda: 1. Overview of OpenSearch integration 2. Demonstrations: - Configuring and starting Apache Unomi with OpenSearch - New development tools in action - Integration testing capabilities 3. Q&A session OPENSEARCH CONTRIBUTION RATIONALE --------- - Enable true vendor independence by providing a fully open-source search engine alternative - Address growing community concerns about ElasticSearch licensing changes and potential future restrictions - Reduce total cost of ownership for Apache Unomi deployments, especially in large-scale environments - Future-proof Apache Unomi's architecture through a modular persistence layer design - Enhance developer experience with flexible backend choices - Leverage OpenSearch's strong commitment to the Apache License 2.0 - Enable seamless scaling without licensing constraints - Benefit from OpenSearch's active development and innovation in the search engine space - Strengthen Apache Unomi's position as a truly free and open-source CDP platform - Support the latest and greatest version of OpenSearch - Don't break existing ElasticSearch support for existing users - Use existing integration tests to validate contribution for both backend implementations WHERE TO FIND IT —————————— https://github.com/apache/unomi/tree/opensearch-persistence WHAT’S INCLUDED ————————— - 100% feature parity between ElasticSearch and OpenSearch - 100% integration tests passing on both ElasticSearch and OpenSearch - Updates to in-project documentation (website updates will be done once these changes are reviewed) WHAT IS NOT INCLUDED ———————————— - Migration from ElasticSearch to OpenSearch: this contribution is aimed at new projects that are starting to deploy Apache Unomi. That doesn’t mean to say that migration is not possible or even difficult (apart from data size), it just wasn’t in the initial scope of this contribution. DETAILED CHANGES SUMMARY ----------------------- I'd like to summarize the significant improvements and changes implemented in the opensearch-persistence branch. This work represents a major enhancement to Apache Unomi's persistence layer and development tooling. Key Features and Improvements: 1. OpenSearch Integration - Full support for OpenSearch 2.18.0 as an alternative to ElasticSearch - Complete implementation of persistence layer for OpenSearch - Support for authenticated and encrypted OpenSearch configurations - Implementation of roll-over policy for OpenSearch - Backend-independent implementation of GeoDistance and DateMath utilities - Integration with Health Check servlet 2. Testing and CI/CD Improvements - Integration tests now fully compatible with both OpenSearch and ElasticSearch - Docker-based OpenSearch testing environment - Prepared GitHub Workflows matrix (pending testing) - New integration tests slowest tests report for performance monitoring - Maven Cache integration for faster builds - Support for Karaf Debugging in Docker environments 3. Architecture and Configuration - Feature-based persistence implementation selection - Configurable startup process via org.apache.unomi.start.cfg - Clean separation between OpenSearch and ElasticSearch deployments - Split Unomi features into smaller, more manageable chunks - Enhanced configuration options for both persistence implementations 4. Development Tools and Scripts - New build script with comprehensive options for: * Building and running the application * Running integration tests with either ElasticSearch or OpenSearch * Maven Cache management integration (including purging/bypassing) - New Karaf Tools helper script for integration tests: * Log file viewing and tailing * Container management (start/stop) * Debug mode support * Grep functionality for logs 5. Command Line Interface Improvements - Enhanced unomi:start command with persistence selection - New shell commands for resource management including removal of items (session, profiles, actions, …) - Improved error handling and user feedback - Added confirmation prompts for destructive operations 6. Code Quality and Maintenance - Improved logging consistency - Enhanced error handling - Better resource cleanup - Updated servlet configurations - Version bump to 2.7.0-SNAPSHOT Test Performance Insights: The slowest test analysis reveals areas for potential optimization, with some tests taking up to 100 seconds to complete. This information will be valuable for future performance improvements. Also a intermediate progress report is now printed indicating how many tests have succeeded or failed, as well as a time estimation until they are completed and a progress bar. Next Steps: 1. Code review and merge of the branch 2. Complete testing of the GitHub Workflows matrix 3. Documentation updates to reflect new features and configurations 4. Release new version 5. Update website with latest OpenSearch support information The changes represent a significant step forward in making Apache Unomi more flexible and maintainable, while providing developers with better tools for development and debugging. ---- **Please** following this checklist to help us incorporate your contribution quickly and easily: - [X] Make sure there is a [JIRA issue](https://issues.apache.org/jira/browse/UNOMI) filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes. - [X] Format the pull request title like `[UNOMI-XXX] - Title of the pull request` - [X] Provide integration tests for your changes, especially if you are changing the behavior of existing code or adding significant new parts of code. - [X] Write a pull request description that is detailed enough to understand what the pull request does, how, and why. Copy the description to the related JIRA issue - [X] Run `mvn clean install -P integration-tests` to make sure basic checks pass. A more thorough check will be performed on your pull request automatically. Trivial changes like typos do not require a JIRA issue (javadoc, project build changes, small doc changes, comments...). If this is your first contribution, you have to read the [Contribution Guidelines](https://unomi.apache.org/contribute.html) If your pull request is about ~20 lines of code you don't need to sign an [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf) if you are unsure please ask on the developers list. To make clear that you license your contribution under the [Apache License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0) you have to acknowledge this by using the following check-box. - [X] I hereby declare this contribution to be licenced under the [Apache License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@unomi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
