Hi all, Below is a summary of the informal Unomi developer sync held on April 10, 2025. The discussion focused on progress with Unomi 3, clustering, migration, testing infrastructure, and ideas to improve resilience and contribution workflows.
As a reminder, no decisions were made during this call. All proposals and contributions must be brought forward and discussed on this mailing list in accordance with ASF governance. ⸻ Apache Unomi Developer Sync – Informal Summary Date: April 10, 2025 ⸻ Tracker Release Clarification • A new contributor asked whether there were any blockers to releasing the Unomi Tracker. • No specific blockers were raised during the call. • Reminder: any release must be formally proposed and approved on this list. ⸻ Release Automation • The team discussed improving the release process. • While many steps can be automated, artifact signing must still be done manually by a committer, in compliance with ASF policy. • It was suggested to contact Apache Infra to clarify the boundaries for release automation and secure key usage. ⸻ Docker-Based Test Infrastructure (Generalization Idea) • An idea was shared to generalize test infrastructure using Docker, replacing specific solutions like Maven plugins that launch Elasticsearch. • This approach could support a broader range of testing types, such as API and integration tests, with greater flexibility and environmental consistency. • This remains an unimplemented idea for future development. ⸻ Documentation • A ticket should be created to automate the generation of Unomi’s site documentation. • A broken link to the 2.6.x documentation was reported and should be tracked as a separate issue. ⸻ Clustering Overhaul • The clustering system has been rewritten to use Elasticsearch, removing Hazelcast and Karaf Cellar. • Nodes ping Elasticsearch and store timestamps to indicate availability. • The API remains compatible but now includes version metadata for nodes. • These changes will be contributed by Serge Huber once the Unomi 3 integration tests are stable. ⸻ Unomi 3 Development & Integration Tests • Work is underway to stabilize Unomi 3, with Serge currently the main contributor. • Progress continues despite long test/debug cycles. • Updates include: • A revamped progress reporting system for import/export operations, including start, end, and progress listeners. • Initial multi-tenancy support, based on tenant-specific key isolation. • Cleanup of previously private/proprietary code, making the fork ready for contribution back to the community. • It was suggested that once tests stabilize, contributions should be split into smaller PRs to simplify community review (e.g., clustering, data model, multi-tenancy). ⸻ Breaking Changes • Some breaking changes are expected, particularly in authentication and schema definitions. • The team is taking care to minimize disruption and ensure clear upgrade paths. ⸻ Migration Challenges • Migration tooling is evolving to support the new model but still presents challenges: • Reindexing time for large datasets can exceed 30 hours. • Elasticsearch mapping/type conflicts may occur. • Some tasks are incorrectly reported as successful even when errors are present. • Discussed solutions: • A meta-migration script to cover multiple version upgrades. • Event capture/replay strategies to avoid data loss. • Temporary parallel system operation with proxy-based routing. ⸻ Event Loss & Recovery • A bug was identified where failures in loading scopes or rules could result in a permanent crash of the event-processing thread, silently dropping events. • Unomi 3 introduces a resilient scheduler that ensures the system continues processing even if certain components fail. • The group discussed implementing a queue or retry mechanism, with ideas like SMTP-style exponential backoff. • This may be part of core functionality or an optional/commercial extension. ⸻ Other Noteworthy Topics • Multi-tenancy strategy: smaller tenants may share indexes; larger tenants could be split out to separate infrastructure as needed. • Tenant migration tooling was discussed conceptually — this remains an idea for the future, not yet implemented. • The most notable known difference between Elasticsearch and OpenSearch support is the index rollover behavior, alongside the fact that the Elasticsearch version in use is outdated, while OpenSearch support targets a modern version. • Brief mention of cleaning up deprecated features, including a potential removal of OGNL scripting. ⸻ ✅ Next Steps (For Discussion on This List) • Open issues for: • Broken 2.6.x documentation link • Site documentation automation • Continue stabilizing Unomi 3 and integration testing • Plan to split contributions into smaller PRs once ready • Explore formalizing event recovery strategies • Bring any proposals or feature ideas to this list for discussion ⸻ Thanks to everyone who participated. Feel free to respond here with any additions, clarifications, or proposals. Best regards,
