Dear Apache Unomi developers and users,
After 8 months of development, and while I'm at it to start 2025 on a high note, I am proud to share with you and submit for review my contribution on the support of OpenSearch as an alternative to ElasticSearch for Apache Unomi. A lot of work has gone into this contribution, and I can fully understand that some of you will have questions. I propose to organize a meeting to go over the changes and also to answer any questions you might have. Btw as I am currently doing all this work on my own time and self-funded, if anybody/any company want to support my work by sponsoring me or via donations, that would be very appreciated. Just reach out to me via email directly at shuber at apache dot org LIVE DEMONSTRATION ANNOUNCEMENT ----------------------------- I'm pleased to invite you to a live presentation and demonstration of the OpenSearch integration and new development tools. Date: January 13th, 2025 Time: 17:00 CET Format: Online presentation with live demonstration Link: https://calendar.app.google/7ZffzAgwPwhamAAU6 Agenda: 1. Overview of OpenSearch integration 2. Demonstrations: - Configuring and starting Apache Unomi with OpenSearch - New development tools in action - Integration testing capabilities 3. Q&A session OPENSEARCH CONTRIBUTION RATIONALE --------- - Enable true vendor independence by providing a fully open-source search engine alternative - Address growing community concerns about ElasticSearch licensing changes and potential future restrictions - Reduce total cost of ownership for Apache Unomi deployments, especially in large-scale environments - Future-proof Apache Unomi's architecture through a modular persistence layer design - Enhance developer experience with flexible backend choices - Leverage OpenSearch's strong commitment to the Apache License 2.0 - Enable seamless scaling without licensing constraints - Benefit from OpenSearch's active development and innovation in the search engine space - Strengthen Apache Unomi's position as a truly free and open-source CDP platform - Support the latest and greatest version of OpenSearch - Don't break existing ElasticSearch support for existing users - Use existing integration tests to validate contribution for both backend implementations WHERE TO FIND IT —————————— https://github.com/apache/unomi/pull/715 WHAT’S INCLUDED ————————— - 100% feature parity between ElasticSearch and OpenSearch - 100% integration tests passing on both ElasticSearch and OpenSearch - Updates to in-project documentation (website updates will be done once these changes are reviewed) WHAT IS NOT INCLUDED ———————————— - Migration from ElasticSearch to OpenSearch: this contribution is aimed at new projects that are starting to deploy Apache Unomi. That doesn’t mean to say that migration is not possible or even difficult (apart from data size), it just wasn’t in the initial scope of this contribution. DETAILED CHANGES SUMMARY ----------------------- I'd like to summarize the significant improvements and changes implemented in the opensearch-persistence branch. This work represents a major enhancement to Apache Unomi's persistence layer and development tooling. Key Features and Improvements: 1. OpenSearch Integration - Full support for OpenSearch 2.18.0 as an alternative to ElasticSearch - Complete implementation of persistence layer for OpenSearch - Support for authenticated and encrypted OpenSearch configurations - Implementation of roll-over policy for OpenSearch - Backend-independent implementation of GeoDistance and DateMath utilities - Integration with Health Check servlet 2. Testing and CI/CD Improvements - Integration tests now fully compatible with both OpenSearch and ElasticSearch - Docker-based OpenSearch testing environment - Prepared GitHub Workflows matrix (pending testing) - New integration tests slowest tests report for performance monitoring - Maven Cache integration for faster builds - Support for Karaf Debugging in Docker environments 3. Architecture and Configuration - Feature-based persistence implementation selection - Configurable startup process via org.apache.unomi.start.cfg - Clean separation between OpenSearch and ElasticSearch deployments - Split Unomi features into smaller, more manageable chunks - Enhanced configuration options for both persistence implementations 4. Development Tools and Scripts - New build script with comprehensive options for: * Building and running the application * Running integration tests with either ElasticSearch or OpenSearch * Maven Cache management integration (including purging/bypassing) - New Karaf Tools helper script for integration tests: * Log file viewing and tailing * Container management (start/stop) * Debug mode support * Grep functionality for logs 5. Command Line Interface Improvements - Enhanced unomi:start command with persistence selection - New shell commands for resource management including removal of items (session, profiles, actions, …) - Improved error handling and user feedback - Added confirmation prompts for destructive operations 6. Code Quality and Maintenance - Improved logging consistency - Enhanced error handling - Better resource cleanup - Updated servlet configurations - Version bump to 2.7.0-SNAPSHOT Test Performance Insights: The slowest test analysis reveals areas for potential optimization, with some tests taking up to 100 seconds to complete. This information will be valuable for future performance improvements. Also a intermediate progress report is now printed indicating how many tests have succeeded or failed, as well as a time estimation until they are completed and a progress bar. Next Steps: 1. Code review and merge of the branch 2. Complete testing of the GitHub Workflows matrix 3. Documentation updates to reflect new features and configurations 4. Release new version 5. Update website with latest OpenSearch support information The changes represent a significant step forward in making Apache Unomi more flexible and maintainable, while providing developers with better tools for development and debugging. Best regards, Serge Huber. Founder of Inoyu & initial contributor of Apache Unomi
