Dear Apache Unomi Developers, I'm pleased to announce a comprehensive overview of the significant changes and new features in the unomi-3-dev branch. This branch represents a major architectural evolution of Apache Unomi with 145 commits introducing groundbreaking multi-tenancy support, enhanced clustering capabilities, and numerous architectural improvements. The branch is now considered stable for testing and all the unit tests and integration tests run successfully. As you will see this email is quite comprehensive and although I got some help from AI it should be mostly correct :) Let me know if you have any questions or if I missed anything.
========================================================================================= Branch Statistics Total Commits: 145 commits ahead of master Files Modified: 643 files across the entire codebase Lines Added: 86,838 insertions, 8,239 deletions Major Features: 15+ major feature categories implemented ========================================================================================= Multi-tenancy (UNOMI-139, UNOMI-880) - 24 commits Core Architecture ---------------------------------------- Complete Tenant Isolation: All data (profiles, events, segments, rules, schemas) is now tenant-specific Dual API Key System: Public keys for event collection, private keys for administrative operations Resource Quotas: Per-tenant limits on profiles, events, and requests Key Components Tenant class with API key management, resource quotas, and security settings TenantService, TenantSecurityService, TenantAuditService, TenantQuotaService Enhanced authentication filters supporting tenant-based access control Authentication Model Public Endpoints: Require public API key via X-Unomi-Api-Key header Private Endpoints: Support tenant authentication (tenantId:privateKey) or system admin fallback System Administration: Maintains backward compatibility with karaf/karaf credentials ========================================================================================= Cluster-aware Task Scheduling (UNOMI-878) - 11 commits Architecture Distributed Task Execution: Tasks executed across cluster nodes with automatic recovery Persistent Tasks: Tasks stored in persistence service for cluster-wide visibility Task Dependencies: Support for complex task dependency chains Key Components SchedulerService interface with cluster-aware operations TaskExecutor, TaskLockManager, TaskRecoveryManager, TaskMetricsManager Distributed locking, state management, and performance monitoring ========================================================================================= Unified Condition Validation Service (UNOMI-883) - 4 commits Framework Type-Safe Validation: Validates conditions against their type definitions Detailed Error Reporting: Rich error context with parameter names and validation types Extensible Validators: Plugin-based validation system for different value types Components ConditionValidationService interface with ValidationError class ValueTypeValidator implementations for all data types Support for nested validation and condition dependencies ========================================================================================= Unified CRUD Shell Commands (UNOMI-879) - 1 commit Interface Unified Command Structure: Consistent CRUD operations across all entity types Interactive Operations: Confirmation prompts and user-friendly interfaces Comprehensive Entity Support: Commands for profiles, events, rules, segments, tenants, etc. Components BaseCommand abstract class and BaseCrudCommand for standardized operations CrudCommand interface for entity-specific operations Specialized commands for each entity type with command completion support ========================================================================================= New Unit Testing Framework (UNOMI-881) - 10 commits Infrastructure Multi-tenant Testing: Support for testing tenant isolation and operations Integration Test Framework: Comprehensive integration testing capabilities In-Memory Persistence: Complete in-memory persistence implementation for fast, isolated testing In-Memory Persistence Service The InMemoryPersistenceServiceImpl provides a complete implementation of the persistence service interface: Full Persistence API Support: All CRUD operations, queries, aggregations, and script execution Tenant Isolation: Proper tenant context and data isolation for multi-tenant testing Advanced Features: Optimistic concurrency control, scroll queries, aggregation framework Script Execution: Support for update scripts including past event occurrences, profile ID updates, and scoring plans Test Categories Tenant Tests: Multi-tenant functionality validation Security Tests: Authentication and authorization testing Migration Tests: Data migration script validation Integration Tests: End-to-end functionality testing ========================================================================================= Enhanced Documentation and Diagrams (UNOMI-882) - 2 commits Documentation Multi-tenancy Guide: Complete guide to multi-tenant operations V2/V3 Compatibility: Detailed migration and compatibility documentation Architecture Diagrams: Visual representation of system architecture API Documentation: Enhanced API documentation with examples ========================================================================================= Migration Scripts for V3 (UNOMI-884) - 3 commits Framework Tenant Document Migration: Migration of existing data to tenant-specific structure Index Management: Automatic index creation and configuration Data Transformation: Transformation of existing data to new format Scripts migrate-3.0.0-00-tenantDocumentIds.groovy - Core tenant migration migrate-3.0.0-10-tenantInitialization.groovy - Tenant initialization Enhanced migration utilities with audit trails and error handling ========================================================================================= OpenSearch Support (UNOMI-828) - 2 commits Integration OpenSearch Client: Full OpenSearch client integration Query Builders: OpenSearch-specific query builders Index Management: OpenSearch index creation and management Components OpenSearchPersistenceServiceImpl - Complete OpenSearch persistence ConditionOSQueryBuilder - OpenSearch query building OSCustomObjectMapper - OpenSearch-specific object mapping Health check providers for OpenSearch ========================================================================================= Explain / Tracing Support (UNOMI-873) - 11 commits Framework Request Tracing: End-to-end request tracing capabilities Condition Evaluation Tracing: Detailed condition evaluation tracking Action Execution Tracing: Action execution monitoring Components TracerService interface for tracing management RequestTracer for per-request tracing TraceNode for hierarchical trace structure Integration across action and condition evaluators ========================================================================================= V2 Compatibility Mode Implementation (UNOMI-904) - 1 commit Complete V2 Compatibility Support V2 Compatibility Mode: Full support for V2 clients to interact with Unomi V3 without requiring API keys Dual Authentication Support: Seamless switching between V2 and V3 authentication modes Default Tenant Configuration: Configurable default tenant ID for V2 compatibility mode Key Features Authentication Filter Enhancement: Updated to handle both V2 and V3 authentication modes Public Endpoint Access: V2 compatibility mode allows public endpoints without authentication Third-Party Provider Support: Enhanced event authorization for V2-style protected events Components V2ThirdPartyConfigService - Third-party provider configuration management IPValidationUtils - IP address validation utilities V2CompatibilityModeIT - Comprehensive integration tests for V2 compatibility Enhanced AuthenticationFilter with dual-mode support Migration Support Seamless Migration: V2 clients can continue operating without immediate changes Gradual Migration: Support for gradual migration from V2 to V3 authentication Documentation: Comprehensive migration guides and compatibility documentation ========================================================================================= Enhanced Cluster Management (UNOMI-877) - 5 commits Health Monitoring Cluster Health Check Provider: Comprehensive cluster health monitoring and status reporting Dynamic Node Management: Real-time cluster node detection and management Stale Node Detection: Automatic detection and cleanup of stale cluster nodes Service Architecture Improvements ServiceTracker Elimination: Removed ServiceTracker for PersistenceService, simplifying service management Direct Service References: Updated methods to directly reference persistence service for enhanced clarity Resource Management: Added task cancellation during shutdown for better resource management Components ClusterHealthCheckProvider - Comprehensive cluster health monitoring Enhanced ClusterServiceImpl with improved node management and simplified architecture Dynamic service binding and unbinding capabilities ========================================================================================= Enhanced Groovy Actions Service (UNOMI-897) - 1 commit Advanced Script Management Pre-compilation Support: Enhanced Groovy script pre-compilation for improved performance Metadata Management: Comprehensive script metadata tracking and management Performance Optimization: Optimized script execution and caching Components GroovyActionsServiceImpl - Enhanced with pre-compilation and metadata management GroovyActionDispatcher - Improved script dispatching and execution ScriptMetadata - Enhanced metadata tracking capabilities ========================================================================================= Enhanced JSON Schema Management (UNOMI-888) - 1 commit Advanced Schema Management Multi-tenant Schema Support: Tenant-specific schema management and isolation Schema Inheritance: Support for schema inheritance and extension Dynamic Schema Validation: Real-time schema validation and enforcement Components SchemaServiceImpl - Enhanced with multi-tenant support and inheritance JsonSchemaListener - Improved schema change detection and handling JsonSchemaWrapper - Enhanced schema wrapping and validation ========================================================================================= New Services Common Module (New Architecture) Shared Service Infrastructure Abstract Service Classes: Common base classes for service implementations Security Infrastructure: Shared security and authentication components Caching Framework: Multi-type caching service infrastructure Key Components AbstractContextAwareService: Base class for context-aware services KarafSecurityService: Enhanced Karaf security integration AbstractMultiTypeCachingService: Multi-type caching infrastructure ExecutionContextManagerImpl: Enhanced execution context management Architecture Benefits Code Reuse: Reduced code duplication across services Consistent Patterns: Standardized service implementation patterns Enhanced Security: Centralized security management ========================================================================================= Enhanced Persistence SPI (New Architecture) Improved Persistence Framework Condition Evaluator Dispatcher: Enhanced condition evaluation framework Date Math Support: Advanced date math parsing and evaluation Geo Distance Calculations: Comprehensive geographic distance calculations Components ConditionEvaluatorDispatcherImpl - Centralized condition evaluation DateMathParser - Advanced date math parsing GeoDistance - Geographic distance calculations ConfigurationUpdateHelper - Enhanced configuration management ========================================================================================= Enhanced Build and Testing Infrastructure (UNOMI-892, UNOMI-887, UNOMI-875) Advanced Build System Maven Build Optimization: Enhanced Maven build configuration for integration tests Migration Test Options: Comprehensive migration testing capabilities Integration Test Framework: Advanced integration testing infrastructure Components Enhanced Maven build configuration Migration testing framework Integration test infrastructure Build script automation ========================================================================================= GraphQL Enhancements (UNOMI-680) GraphQL Bug Fixes and Improvements Metadata Handling: Enhanced metadata handling in GraphQL operations Date Type Support: Improved support for OffsetDateTime and Date types Null Safety: Enhanced null safety and error handling Components Enhanced CreateOrUpdateSourceCommand - Enhanced metadata handling ConditionFactory - Improved date type processing UnomiToGraphQLConverter - Enhanced null safety GraphQLSchemaProvider - Improved error logging ========================================================================================= Getting Started Building the Branch git checkout unomi-3-dev ./build.sh Running with Multi-tenancy # Start the server ./bin/start # Create your first tenant curl -X POST http://localhost:8181/cxs/tenants \ -H "Content-Type: application/json" \ -H "Authorization: Basic <ADMIN_CREDENTIALS>" \ -d '{"itemId": "my-tenant", "name": "My Organization"}' Key Configuration Files etc/org.apache.unomi.tenant.cfg - Tenant configuration manual/src/main/asciidoc/multitenancy.adoc - Multi-tenancy guide manual/src/main/asciidoc/migrations/v2-v3-compatibility.adoc - Migration guide ========================================================================================= Next Steps Review the Documentation: Start with the multi-tenancy and migration guides Test the Migration: Use the provided migration scripts to test V2 to V3 migration Explore New Features: Try the enhanced scheduler and tracing capabilities Provide Feedback: Share your experience and any issues you encounter ========================================================================================= Contributing We welcome contributions and feedback! Please: Test the new features and report any issues Review the documentation and suggest improvements Contribute to the migration scripts and testing framework Share your experience with the multi-tenancy features ========================================================================================= Support For questions and support: Mailing List: [email protected] JIRA: https://issues.apache.org/jira/browse/UNOMI Documentation: https://unomi.apache.org/manual/ This represents a major architectural evolution of Apache Unomi, bringing enterprise-grade multi-tenancy, enhanced clustering, comprehensive observability, and a completely reimagined service architecture to the platform. The introduction of the new services-common module, enhanced persistence SPI, comprehensive testing infrastructure, and complete V2 compatibility mode demonstrates the project's commitment to maintainability, scalability, developer experience, and backward compatibility. ========================================================================================= Key Architectural Improvements New Module Architecture Services Common Module: Centralized shared service infrastructure Enhanced Persistence SPI: Improved persistence framework with advanced condition evaluation Tracing Module: Comprehensive request tracing and debugging capabilities Enhanced Testing Infrastructure In-Memory Persistence Service: Complete persistence implementation for fast unit testing Multi-tenant Testing Support: Comprehensive testing framework for tenant isolation Integration Test Framework: Advanced integration testing capabilities Performance and Scalability Cluster-aware Task Scheduling: Distributed task execution across cluster nodes Enhanced Caching: Multi-type caching service with optimized strategies OpenSearch Support: Complete OpenSearch integration for modern search capabilities Backward Compatibility V2 Compatibility Mode: Complete support for V2 clients without requiring API keys Dual Authentication Support: Seamless switching between V2 and V3 authentication modes Gradual Migration Path: Support for gradual migration from V2 to V3 Third-Party Provider Support: Enhanced event authorization for V2-style protected events I'm excited to see how the community adopts and extends these new capabilities! Best regards, Serge Huber Apache Unomi PMC Chair
