[
https://issues.apache.org/jira/browse/UNOMI-962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Serge Huber updated UNOMI-962:
------------------------------
Description:
h2. Summary
While validating Unomi 3.1 for release, several REST API problems block
migration testing, health checks, and the OpenAPI test suite. Operators and
integrators get server errors (HTTP 500) or wrong status codes when they should
get clear 404 or 400 responses. V2 compatibility mode also blocks Karaf admin
access on some private endpoints.
h2. What is going wrong today
* *V2 compatibility mode:* Karaf admin users ({{karaf:karaf}}) cannot use some
private REST endpoints. {{GET /cxs/privacy/info}} fails with a permission error
(missing {{AGGREGATE}} for the default tenant). Docker health checks and
version detection on the 3.1-es-v2compat stack depend on this endpoint.
* *Missing resources:* Requests for unknown value types, goals, goal reports,
or events can return HTTP 500 (null pointer) instead of HTTP 404.
* *Profile export:* CSV export and router oneshot export can crash when segment
metadata is missing or export configuration is incomplete (missing segment or
mapping).
* *OpenSearch queries:* Property conditions with null or missing values behave
differently from Elasticsearch and can cause unexpected failures instead of a
clear validation error.
h2. Impact
* Blocks *unomi-3.1.0* release validation under UNOMI-875
* Fails OpenAPI smoke tests on 3.1-es, 3.1-os, and 3.1-es-v2compat stacks
* Breaks V2 migration paths that rely on Karaf admin and {{/cxs/privacy/info}}
* Operators see confusing 500 errors instead of actionable 404/400 responses
h2. Who is affected
* Teams migrating from Unomi 2.x (V2 compatibility mode)
* CI and integrators running REST API / OpenAPI tests
* Operators using Docker health checks and admin tools
* Anyone calling definitions, goals, events, or profile export endpoints with
invalid or missing ids
h2. What good would look like
* With V2 compat enabled, {{GET /cxs/privacy/info}} with Karaf admin
credentials returns HTTP 200 with {{serverVersion}}
* Missing value types, goals, goal reports, and events return HTTP 404
* Profile CSV export and router export fail fast with clear errors when
configuration is invalid; export does not crash on missing segment metadata
* OpenSearch property conditions reject null/missing values the same way
Elasticsearch does
* Unit tests cover the REST, service, router, and OpenSearch fixes
* 3.1-es-v2compat Docker stack health check passes without permission errors in
logs
h2. Acceptance criteria
* V2 compat: Karaf-authenticated private endpoints work for admin operations
that need persistence permissions (including {{/cxs/privacy/info}})
* REST: {{GET}} for missing value type, goal, goal report, and event returns 404
* Services: goal report and profile CSV export handle missing data without NPE
* Router: export requires segment and mapping; returns clear error when missing
* OpenSearch: null/missing property condition values throw
{{IllegalArgumentException}} (parity with ES)
* Integration test added for {{GET /cxs/privacy/info}} under V2 compatibility
mode
* Unit tests added or updated for each fix area
* OpenAPI REST test stacks (3.1-es, 3.1-os, 3.1-es-v2compat) show fewer
server-side failures after rebuild
----
h2. Technical notes
*Parent:* [UNOMI-875|https://issues.apache.org/jira/browse/UNOMI-875]
*Related:* [UNOMI-904|https://issues.apache.org/jira/browse/UNOMI-904] (V2 API
Compatibility mode)
*PR:* [apache/unomi#822|https://github.com/apache/unomi/pull/822]
*Test stacks:* {{server/unomi-rest-api-tests/docker/}} (3.1-es, 3.1-os,
3.1-es-v2compat)
*Scope (server):*
* {{AuthenticationFilter}} — V2 compat Karaf admin on private endpoints
* REST endpoints — definitions (value types), goals (reports), events (get by
id)
* {{GoalsServiceImpl}}, {{ProfileServiceImpl}}, {{ProfileExportServiceImpl}}
* {{PropertyConditionOSQueryBuilder}} — null value parity with ES
* Blueprint — {{UserListServiceImpl}} {{contextManager}} injection
* ES/OS persistence — downgrade duplicate-mapping log noise from WARN to DEBUG
*Reproduction (V2 compat):*
{code}
cd server/unomi-rest-api-tests/docker
./build-unomi.sh
./run-tests.sh 3.1-es-v2compat
curl -fsS -u karaf:karaf http://localhost:8181/cxs/privacy/info
{code}
was:
h3. Problem
When {{v2.compatibilitymode.enabled=true}}, private REST endpoints
authenticated with Karaf basic auth ({{karaf:karaf}}) fail with HTTP 500 and:
{code}
SecurityException - Access denied: Missing permission for operation AGGREGATE
for tenant <defaultTenantId> and roles []
{code}
Observed on {{GET /cxs/privacy/info}} while running the {{3.1-es-v2compat}}
Docker test stack
({{UNOMI_REST_AUTHENTICATION_V2COMPATIBILITYMODEENABLED=true}},
{{UNOMI_REST_AUTHENTICATION_V2COMPATIBILITYDEFAULTTENANTID=test_tenant}}).
This blocks V2-style admin clients and health checks that call
{{/cxs/privacy/info}} during migration to 3.1.0.
h3. Root cause
# {{PrivacyServiceImpl.getServerInfo()}} calls
{{persistenceService.aggregateWithOptimizedQuery(...)}}, which requires the
*AGGREGATE* permission on the active tenant execution context.
# In normal V3 mode, Karaf-only admin requests (no {{X-Unomi-Tenant-Id}}) use
{{ExecutionContext.systemContext()}}, which bypasses tenant permission checks
({{AuthenticationFilter}} ~line 240).
# In V2 compatibility mode, {{handleV2CompatibilityMode()}} merges the JAAS
subject with a {{TenantPrincipal}} for {{v2CompatibilityDefaultTenantId}} and
calls {{executionContextManager.createContext(defaultTenantId)}} (~lines
315–320 in {{AuthenticationFilter.java}}).
# That tenant-scoped context is built with *no Unomi roles* ({{roles []}} in
the error). Karaf JAAS principals are not mapped to {{ROLE_UNOMI_TENANT_USER}}
/ {{ROLE_UNOMI_TENANT_ADMIN}} the way tenant API-key auth is via
{{KarafSecurityService.createSubject()}}.
# Result: authenticated Karaf admin cannot perform persistence operations that
require AGGREGATE (and likely other permissions on aggregate-heavy code paths).
h3. Expected behaviour (V2 compat contract)
Per v2-compatibility-mode.adoc and UNOMI-904 acceptance criteria:
* *Private endpoints* should work with *system administrator* Karaf
credentials, same as Unomi 2.x.
* {{GET /cxs/privacy/info}} must return 200 with {{serverVersion}} (used by
Docker health checks, version detection, and operators).
h3. Suggested fix
In {{AuthenticationFilter.handleV2CompatibilityMode()}}, after successful JAAS
auth on private endpoints, either:
* Option A (preferred for parity with V3 Karaf admin): use
{{ExecutionContext.systemContext()}} for data operations (like non-compat V3
path when no tenant header), *or*
* Option B: augment the merged subject with Unomi admin roles
({{ROLE_UNOMI_ADMIN}} / {{ROLE_UNOMI_TENANT_ADMIN}} +
{{ROLE_UNOMI_TENANT_USER}}) before {{createContext()}}, so
{{getPermissionsForRoles()}} grants AGGREGATE.
h3. Reproduction
{code:bash}
cd server/unomi-rest-api-tests/docker
./build-unomi.sh
./run-tests.sh 3.1-es-v2compat
# Or manually:
curl -fsS -u karaf:karaf http://localhost:8181/cxs/privacy/info
{code}
Server log:
{code}
Internal server error on GET /cxs/privacy/info - Root cause: SecurityException -
Access denied: Missing permission for operation AGGREGATE for tenant
test_tenant and roles []
{code}
h3. Test gaps
* {{V2CompatibilityModeIT}} tests {{GET /cxs/profiles/{id}}} with Karaf in V2
mode but does *not* cover {{GET /cxs/privacy/info}}.
* {{unomi-rest-api-tests}} {{3.1-es-v2compat}} stack relies on
{{/cxs/privacy/info}} for health check and version detection.
h3. Acceptance criteria
* With V2 compat enabled, {{GET /cxs/privacy/info}} with {{karaf:karaf}}
returns HTTP 200 and includes {{serverVersion}}.
* Karaf-authenticated private endpoints retain full admin capability for
persistence operations requiring AGGREGATE/QUERY (not only simple GET-by-id).
* Add integration test in {{V2CompatibilityModeIT}} (or REST API test) for
{{GET /cxs/privacy/info}} under V2 compatibility mode.
* {{3.1-es-v2compat}} Docker stack health check passes without AGGREGATE errors
in logs.
h3. References
* Parent epic: UNOMI-875
* Original feature: UNOMI-904 (V2 API Compatibility mode)
* Code:
rest/src/main/java/org/apache/unomi/rest/authentication/AuthenticationFilter.java
(handleV2CompatibilityMode)
* Code: extensions/privacy-extension/services/.../PrivacyServiceImpl.java
(getServerInfo → aggregate)
* Test stack:
server/unomi-rest-api-tests/docker/docker-compose.unomi-3.1-es-v2compat.yml
> 3.1 REST API validation failures: V2 compat admin auth, missing resources,
> and export errors
> --------------------------------------------------------------------------------------------
>
> Key: UNOMI-962
> URL: https://issues.apache.org/jira/browse/UNOMI-962
> Project: Apache Unomi
> Issue Type: Sub-task
> Components: unomi(-core)
> Affects Versions: unomi-3.1.0
> Reporter: Serge Huber
> Assignee: Serge Huber
> Priority: Major
> Fix For: unomi-3.1.0
>
>
> h2. Summary
> While validating Unomi 3.1 for release, several REST API problems block
> migration testing, health checks, and the OpenAPI test suite. Operators and
> integrators get server errors (HTTP 500) or wrong status codes when they
> should get clear 404 or 400 responses. V2 compatibility mode also blocks
> Karaf admin access on some private endpoints.
> h2. What is going wrong today
> * *V2 compatibility mode:* Karaf admin users ({{karaf:karaf}}) cannot use
> some private REST endpoints. {{GET /cxs/privacy/info}} fails with a
> permission error (missing {{AGGREGATE}} for the default tenant). Docker
> health checks and version detection on the 3.1-es-v2compat stack depend on
> this endpoint.
> * *Missing resources:* Requests for unknown value types, goals, goal reports,
> or events can return HTTP 500 (null pointer) instead of HTTP 404.
> * *Profile export:* CSV export and router oneshot export can crash when
> segment metadata is missing or export configuration is incomplete (missing
> segment or mapping).
> * *OpenSearch queries:* Property conditions with null or missing values
> behave differently from Elasticsearch and can cause unexpected failures
> instead of a clear validation error.
> h2. Impact
> * Blocks *unomi-3.1.0* release validation under UNOMI-875
> * Fails OpenAPI smoke tests on 3.1-es, 3.1-os, and 3.1-es-v2compat stacks
> * Breaks V2 migration paths that rely on Karaf admin and {{/cxs/privacy/info}}
> * Operators see confusing 500 errors instead of actionable 404/400 responses
> h2. Who is affected
> * Teams migrating from Unomi 2.x (V2 compatibility mode)
> * CI and integrators running REST API / OpenAPI tests
> * Operators using Docker health checks and admin tools
> * Anyone calling definitions, goals, events, or profile export endpoints with
> invalid or missing ids
> h2. What good would look like
> * With V2 compat enabled, {{GET /cxs/privacy/info}} with Karaf admin
> credentials returns HTTP 200 with {{serverVersion}}
> * Missing value types, goals, goal reports, and events return HTTP 404
> * Profile CSV export and router export fail fast with clear errors when
> configuration is invalid; export does not crash on missing segment metadata
> * OpenSearch property conditions reject null/missing values the same way
> Elasticsearch does
> * Unit tests cover the REST, service, router, and OpenSearch fixes
> * 3.1-es-v2compat Docker stack health check passes without permission errors
> in logs
> h2. Acceptance criteria
> * V2 compat: Karaf-authenticated private endpoints work for admin operations
> that need persistence permissions (including {{/cxs/privacy/info}})
> * REST: {{GET}} for missing value type, goal, goal report, and event returns
> 404
> * Services: goal report and profile CSV export handle missing data without NPE
> * Router: export requires segment and mapping; returns clear error when
> missing
> * OpenSearch: null/missing property condition values throw
> {{IllegalArgumentException}} (parity with ES)
> * Integration test added for {{GET /cxs/privacy/info}} under V2 compatibility
> mode
> * Unit tests added or updated for each fix area
> * OpenAPI REST test stacks (3.1-es, 3.1-os, 3.1-es-v2compat) show fewer
> server-side failures after rebuild
> ----
> h2. Technical notes
> *Parent:* [UNOMI-875|https://issues.apache.org/jira/browse/UNOMI-875]
> *Related:* [UNOMI-904|https://issues.apache.org/jira/browse/UNOMI-904] (V2
> API Compatibility mode)
> *PR:* [apache/unomi#822|https://github.com/apache/unomi/pull/822]
> *Test stacks:* {{server/unomi-rest-api-tests/docker/}} (3.1-es, 3.1-os,
> 3.1-es-v2compat)
> *Scope (server):*
> * {{AuthenticationFilter}} — V2 compat Karaf admin on private endpoints
> * REST endpoints — definitions (value types), goals (reports), events (get by
> id)
> * {{GoalsServiceImpl}}, {{ProfileServiceImpl}}, {{ProfileExportServiceImpl}}
> * {{PropertyConditionOSQueryBuilder}} — null value parity with ES
> * Blueprint — {{UserListServiceImpl}} {{contextManager}} injection
> * ES/OS persistence — downgrade duplicate-mapping log noise from WARN to DEBUG
> *Reproduction (V2 compat):*
> {code}
> cd server/unomi-rest-api-tests/docker
> ./build-unomi.sh
> ./run-tests.sh 3.1-es-v2compat
> curl -fsS -u karaf:karaf http://localhost:8181/cxs/privacy/info
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
