[
https://issues.apache.org/jira/browse/UNOMI-918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Serge Huber updated UNOMI-918:
------------------------------
Description:
h2. Summary
Plan and execute migration from Unomi's custom {{custom.system.properties}}
file to Karaf's native environment variable and system property override
mechanism. This will simplify configuration management, align with Karaf best
practices, and eliminate the need for a custom configuration loading mechanism.
h2. Current State
h3. How custom.system.properties Works
Apache Unomi currently uses a centralized configuration file
{{$KARAF_HOME/etc/custom.system.properties}} that:
# Contains ~203 system properties with environment variable references using
the pattern: {{{}org.apache.unomi.{*}=$\{env:UNOMI_{*}:-defaultValue{}}}}
# Uses Karaf's {{${optionals}=unomi.custom.system.properties}} mechanism to
load user overrides from {{$KARAF_HOME/etc/unomi.custom.system.properties}}
# Provides variable substitution in {{.cfg}} files using
{{{}$\{org.apache.unomi.*{}}}} syntax
# Is documented as the primary way to configure Unomi via environment variables
h3. Current Environment Variable Pattern
All environment variables follow the pattern: UNOMI_CATEGORY_PROPERTY_NAME
*Examples:*
* {{UNOMI_HTTP_PORT}} → {{org.osgi.service.http.port}}
* {{UNOMI_ELASTICSEARCH_ADDRESSES}} →
{{org.apache.unomi.elasticsearch.addresses}}
* {{UNOMI_CLUSTER_NODEID}} → {{org.apache.unomi.cluster.nodeId}}
h3. User Customization Methods
Users can override configuration in two ways:
# *{{unomi.custom.system.properties}} file* — Create
{{$KARAF_HOME/etc/unomi.custom.system.properties}} with property overrides
# *Environment variables* — Set {{UNOMI_*}} environment variables
h3. Documentation References
The following documentation sections reference {{{}custom.system.properties{}}}:
* {{manual/src/main/asciidoc/configuration.adoc}} — Main configuration
documentation
* {{manual/src/main/asciidoc/clustering.adoc}} — Cluster configuration
* {{manual/src/main/asciidoc/profile-import-export.adoc}} — Router configuration
* {{manual/src/main/asciidoc/connectors/salesforce-connector.adoc}} — Connector
configuration
* {{CODING_GUIDELINES.md}} — Developer guidelines
h2. Target State: Karaf Native System
h3. How Karaf's Native Override System Works
Apache Karaf provides a built-in mechanism to override configuration properties
using environment variables or system properties:
*Environment Variable Pattern:* {{PID_PROPERTY}} (uppercase, dots become
underscores)
*System Property Pattern:* {{pid.property}} (lowercase, dots remain)
*Example:*
* Configuration PID: {{org.apache.karaf.shell}}
* Property name: {{sshPort}}
* Environment variable: {{ORG_APACHE_KARAF_SHELL_SSHPORT}}
* System property: {{-Dorg.apache.karaf.shell.sshPort}}
h3. Benefits of Migration
# *Simpler architecture* — No custom configuration loading mechanism needed
# *Standard Karaf approach* — Aligns with Karaf best practices
# *Better integration* — Works seamlessly with Karaf's
{{KarafConfigurationPlugin}}
# *Reduced maintenance* — Less custom code to maintain
# *Consistent with Karaf ecosystem* — Other Karaf-based projects use this
approach
h3. Challenges
# *Breaking change* — All environment variable names will change (requires
major version release)
# *Migration effort* — Users must update their environment variables,
Docker/Kubernetes deployments, and CI/CD pipelines
# *Documentation updates* — Extensive documentation changes required across
multiple files
h2. Migration Strategy
h3. Phase 1: Preparation (Non-Breaking)
# *Create comprehensive environment variable mapping table* — Document all 203+
variables
# *Create migration script/tool* — Help users convert their environment
variables
# *Add deprecation warnings* — Log warnings when old {{UNOMI_*}} variables are
detected (if possible)
h3. Phase 2: Implementation (Breaking Change - Major Version)
# *Remove {{custom.system.properties}} file and
{{${optionals}=unomi.custom.system.properties}} mechanism*
# *Update {{.cfg}} files* — Remove {{{}$\{org.apache.unomi.*{}}}} variable
substitutions where appropriate
# *Update code* — Remove any custom loading logic for
{{custom.system.properties}}
# *Update {{BaseIT.java}}* — Replace {{editConfigurationFilePut}} calls with
Karaf native system properties
h3. Phase 3: Cleanup
# *Update examples* — All examples in documentation and code
# *Remove migration script* — After sufficient time has passed
h2. Environment Variable Mapping
h3. Current vs. New Naming Convention
*Current Pattern:* UNOMI_CATEGORY_PROPERTY
*New Pattern (Karaf Native):* {{PID_PROPERTY}} where:
* PID = Configuration Persistent ID (e.g., {{{}org.apache.unomi.cluster{}}})
* Property = Property name (e.g., {{{}nodeId{}}})
* Dots ({{{}.{}}}) become underscores ({{{}_{}}})
* All uppercase
h3. Example Mappings
||Current Environment Variable||New Environment Variable||Configuration
PID||Property Name||
|{{UNOMI_HTTP_PORT}}|{{ORG_OSGI_SERVICE_HTTP_PORT}}|{{org.osgi.service.http}}|{{port}}|
|{{UNOMI_ELASTICSEARCH_ADDRESSES}}|{{ORG_APACHE_UNOMI_ELASTICSEARCH_ADDRESSES}}|{{org.apache.unomi.elasticsearch}}|{{addresses}}|
|{{UNOMI_CLUSTER_NODEID}}|{{ORG_APACHE_UNOMI_CLUSTER_NODEID}}|{{org.apache.unomi.cluster}}|{{nodeId}}|
|{{UNOMI_CLUSTER_NODESTATISTICS_UPDATEFREQUENCY}}|{{ORG_APACHE_UNOMI_CLUSTER_NODESTATISTICSUPDATEFREQUENCY}}|{{org.apache.unomi.cluster}}|{{nodeStatisticsUpdateFrequency}}|
|{{UNOMI_ROUTER_KAFKA_HOST}}|{{ORG_APACHE_UNOMI_ROUTER_KAFKA_HOST}}|{{org.apache.unomi.router}}|{{kafka.host}}|
|{{UNOMI_MIGRATION_RECOVER_FROM_HISTORY}}|{{ORG_APACHE_UNOMI_MIGRATION_RECOVERFROMHISTORY}}|{{org.apache.unomi.migration}}|{{recoverFromHistory}}|
h3. Special Cases
Some properties may need special handling:
# *Properties with dots in the name* — Dots become underscores in environment
variable names
# *Properties that span multiple PIDs* — May need to be split or handled
specially
# *Properties used in multiple contexts* — Need to ensure correct PID mapping
h3. Complete Environment Variable List
*Total Count:* ~203 environment variables need to be mapped
*Categories:*
* Security settings (~10 variables)
* HTTP/SSL settings (~8 variables)
* Cluster settings (~4 variables)
* ElasticSearch settings (~30 variables)
* OpenSearch settings (~30 variables)
* Service settings (~20 variables)
* Third-party server settings (~5 variables)
* Profile cookie settings (~5 variables)
* SSH server settings (~8 variables)
* Log settings (~8 variables)
* GeoNames settings (~3 variables)
* Groovy action settings (~1 variable)
* MaxMind IP settings (~10 variables)
* Mail settings (~5 variables)
* Base plugin settings (~1 variable)
* Router settings (~13 variables)
* Salesforce connector settings (~10 variables)
* Weather update settings (~3 variables)
* GraphQL settings (~1 variable)
* Migration settings (~1 variable)
* HealthCheck settings (~4 variables)
* Goals and campaigns settings (~2 variables)
* REST API authorization settings (~2 variables)
h2. User Impact Assessment
h3. Impacted User Groups
# *Docker/Kubernetes users* — Must update all environment variables in
containers
# *System administrators* — Must update system environment variables
# *CI/CD pipeline maintainers* — Must update automation scripts
# *Custom distribution builders* — Must update build configurations
# *Users with {{unomi.custom.system.properties}} files* — Must migrate to
Karaf native approach
h3. Migration Steps for Users
h4. Step 1: Identify Current Environment Variables
Users need to identify all {{UNOMI_*}} environment variables they are currently
using:
{code:bash}
# Find all UNOMI environment variables
env | grep UNOMI
{code}
h4. Step 2: Map to New Variable Names
Use the migration mapping table to convert each variable:
*Example:*
{code:bash}
# OLD
export UNOMI_HTTP_PORT=8080
export UNOMI_ELASTICSEARCH_ADDRESSES=elasticsearch:9200
# NEW
export ORG_OSGI_SERVICE_HTTP_PORT=8080
export ORG_APACHE_UNOMI_ELASTICSEARCH_ADDRESSES=elasticsearch:9200
{code}
h4. Step 3: Update Configuration Files
# *Docker Compose* — Update {{docker-compose.yml}} files
# *Kubernetes* — Update ConfigMaps and Secrets
# *Systemd services* — Update {{.service}} files
# *Shell scripts* — Update {{setenv}} scripts
# *CI/CD pipelines* — Update pipeline configurations
h4. Step 4: Remove {{unomi.custom.system.properties}} Files
If users have created {{$KARAF_HOME/etc/unomi.custom.system.properties}} files,
they should:
# Convert property overrides to environment variables or system properties
# Remove the file (or it will be ignored in new version)
h3. Breaking Change Considerations
# *Major version required* — This is a breaking change and should be part of a
major version release (e.g., 4.0.0)
# *Deprecation period* — Consider deprecating in 3.x with warnings, remove in
4.x
# *Backward compatibility* — Not possible to maintain backward compatibility
with environment variable names
h2. Implementation Plan
h3. Pre-Implementation Tasks
# *Create complete environment variable mapping table* — All 203+ variables
# *Create migration script/tool* — Help users convert their configurations
# *Identify all code references* — Find all places that reference
{{{}custom.system.properties{}}}, including {{BaseIT.java}}
h3. Implementation Tasks
# *Remove {{custom.system.properties}} file and
{{${optionals}=unomi.custom.system.properties}} mechanism*
# *Update {{.cfg}} files* — Remove {{{}$\{org.apache.unomi.*{}}}} variable
substitutions where appropriate
# *Update code* — Remove any custom loading logic for
{{custom.system.properties}}
# *Update {{BaseIT.java}}* — Replace {{editConfigurationFilePut}} calls for
{{custom.system.properties}} with {{systemProperty()}} or environment variable
configuration (13 properties)
h3. Post-Implementation Tasks
# *Release migration tool* — Make available to users
# *Update all documentation and examples* — Replace references with Karaf
native approach
# *Monitor user feedback and provide support* — Track issues during migration
h2. Documentation Updates Required
h3. Files to Update
# *{{manual/src/main/asciidoc/configuration.adoc}}* — Complete rewrite of
configuration section
# *{{manual/src/main/asciidoc/clustering.adoc}}* — Update cluster
configuration examples
# *{{manual/src/main/asciidoc/profile-import-export.adoc}}* — Update router
configuration
# *{{manual/src/main/asciidoc/connectors/salesforce-connector.adoc}}* — Update
connector configuration
# *{{CODING_GUIDELINES.md}}* — Update environment variable guidelines
# *{{README.md}}* — Update quick start examples
# *Docker documentation* — Update Docker examples
# *Kubernetes documentation* — Update Kubernetes examples
h3. New Documentation Needed
# *Migration guide* — Step-by-step instructions with complete environment
variable mapping table
# *Karaf configuration guide* — How to use Karaf's native system
# *FAQ* — Common migration questions
h2. BaseIT Integration Test Changes
h3. Current Usage
The {{BaseIT.java}} integration test base class currently uses
{{editConfigurationFilePut}} to set properties in
{{etc/custom.system.properties}} for test configuration. There are 13
properties being set:
# {{org.apache.unomi.graphql.feature.activated}}
# {{org.apache.unomi.elasticsearch.cluster.name}}
# {{org.apache.unomi.elasticsearch.addresses}}
# {{org.apache.unomi.elasticsearch.taskWaitingPollingInterval}}
# {{org.apache.unomi.elasticsearch.rollover.maxDocs}}
# {{org.apache.unomi.opensearch.cluster.name}}
# {{org.apache.unomi.opensearch.addresses}}
# {{org.apache.unomi.opensearch.username}}
# {{org.apache.unomi.opensearch.password}}
# {{org.apache.unomi.opensearch.sslEnable}}
# {{org.apache.unomi.opensearch.sslTrustAllCertificates}}
# {{org.apache.unomi.opensearch.minimalClusterState}}
# {{org.apache.unomi.migration.tenant.id}}
h3. Required Changes
When {{custom.system.properties}} is removed, these properties must be set
using Karaf's native system property mechanism instead:
*Replace:*
{code:java}
editConfigurationFilePut("etc/custom.system.properties",
"org.apache.unomi.elasticsearch.addresses", "localhost:" + getSearchPort())
{code}
*With:*
{code:java}
systemProperty("org.apache.unomi.elasticsearch.addresses").value("localhost:" +
getSearchPort())
{code}
*Or using environment variables in PaxExam configuration:*
{code:java}
environment("ORG_APACHE_UNOMI_ELASTICSEARCH_ADDRESSES", "localhost:" +
getSearchPort())
{code}
h3. Impact
# *All 13 {{editConfigurationFilePut}} calls for {{custom.system.properties}}
must be converted*
# *Test configuration will use Karaf's native system property mechanism*
# *Tests will verify that Karaf's native override system works correctly*
h2. Risk Assessment
h3. High Risk Areas
# *Production deployments* — Risk of configuration errors during migration
# *Docker/Kubernetes users* — Many environment variables to update
# *Custom distributions* — May have complex configurations
# *Documentation gaps* — Users may not find migration information
h3. Mitigation Strategies
# *Comprehensive migration guide and tool* — Clear, step-by-step instructions
with automated conversion script
# *Deprecation period* — Warn users in advance (if possible in 3.x)
# *Extensive testing* — Test all scenarios before release
# *Support channels* — Provide help during migration period
h2. Success Criteria
# *All 203+ environment variables successfully migrated, mapped, and tested*
# *Documentation complete* — All references updated with migration guide and
examples
# *Migration tool available* — Users can easily convert configurations
# *Code simplified* — {{custom.system.properties}} mechanism removed
# *No critical issues* — Migration proceeds smoothly for majority of users
h2. Related Issues
* Configuration Admin override problem : UNOMI-917
h2. References
* [Karaf Configuration
Documentation|https://karaf.apache.org/manual/latest/#_environment_variables_system_properties]
* [Current custom.system.properties
file|https://github.com/apache/unomi/blob/master/package/src/main/resources/etc/custom.system.properties]
* [Unomi Configuration
Documentation|https://unomi.apache.org/manual/latest/configuration.html]
was:
h2. Summary
Plan and execute migration from Unomi's custom {{custom.system.properties}}
file to Karaf's native environment variable and system property override
mechanism. This will simplify configuration management, align with Karaf best
practices, and eliminate the need for a custom configuration loading mechanism.
h2. Current State
h3. How custom.system.properties Works
Apache Unomi currently uses a centralized configuration file
{{$KARAF_HOME/etc/custom.system.properties}} that:
# Contains ~203 system properties with environment variable references using
the pattern: {{{}org.apache.unomi.{*}=$\{env:UNOMI_{*}:-defaultValue{}}}}
# Uses Karaf's {{${optionals}=unomi.custom.system.properties}} mechanism to
load user overrides from {{$KARAF_HOME/etc/unomi.custom.system.properties}}
# Provides variable substitution in {{.cfg}} files using
{{{}$\{org.apache.unomi.*{}}}} syntax
# Is documented as the primary way to configure Unomi via environment variables
h3. Current Environment Variable Pattern
All environment variables follow the pattern:
\{{UNOMI_{CATEGORY}_\{PROPERTY_NAME}}}
*Examples:*
* {{UNOMI_HTTP_PORT}} → {{org.osgi.service.http.port}}
* {{UNOMI_ELASTICSEARCH_ADDRESSES}} →
{{org.apache.unomi.elasticsearch.addresses}}
* {{UNOMI_CLUSTER_NODEID}} → {{org.apache.unomi.cluster.nodeId}}
h3. User Customization Methods
Users can override configuration in two ways:
# *{{unomi.custom.system.properties}} file* — Create
{{$KARAF_HOME/etc/unomi.custom.system.properties}} with property overrides
# *Environment variables* — Set {{UNOMI_*}} environment variables
h3. Documentation References
The following documentation sections reference {{{}custom.system.properties{}}}:
* {{manual/src/main/asciidoc/configuration.adoc}} — Main configuration
documentation
* {{manual/src/main/asciidoc/clustering.adoc}} — Cluster configuration
* {{manual/src/main/asciidoc/profile-import-export.adoc}} — Router configuration
* {{manual/src/main/asciidoc/connectors/salesforce-connector.adoc}} — Connector
configuration
* {{CODING_GUIDELINES.md}} — Developer guidelines
h2. Target State: Karaf Native System
h3. How Karaf's Native Override System Works
Apache Karaf provides a built-in mechanism to override configuration properties
using environment variables or system properties:
*Environment Variable Pattern:* {{PID_PROPERTY}} (uppercase, dots become
underscores)
*System Property Pattern:* {{pid.property}} (lowercase, dots remain)
*Example:*
* Configuration PID: {{org.apache.karaf.shell}}
* Property name: {{sshPort}}
* Environment variable: {{ORG_APACHE_KARAF_SHELL_SSHPORT}}
* System property: {{-Dorg.apache.karaf.shell.sshPort}}
h3. Benefits of Migration
# *Simpler architecture* — No custom configuration loading mechanism needed
# *Standard Karaf approach* — Aligns with Karaf best practices
# *Better integration* — Works seamlessly with Karaf's
{{KarafConfigurationPlugin}}
# *Reduced maintenance* — Less custom code to maintain
# *Consistent with Karaf ecosystem* — Other Karaf-based projects use this
approach
h3. Challenges
# *Breaking change* — All environment variable names will change (requires
major version release)
# *Migration effort* — Users must update their environment variables,
Docker/Kubernetes deployments, and CI/CD pipelines
# *Documentation updates* — Extensive documentation changes required across
multiple files
h2. Migration Strategy
h3. Phase 1: Preparation (Non-Breaking)
# *Create comprehensive environment variable mapping table* — Document all 203+
variables
# *Create migration script/tool* — Help users convert their environment
variables
# *Add deprecation warnings* — Log warnings when old {{UNOMI_*}} variables are
detected (if possible)
h3. Phase 2: Implementation (Breaking Change - Major Version)
# *Remove {{custom.system.properties}} file and
{{${optionals}=unomi.custom.system.properties}} mechanism*
# *Update {{.cfg}} files* — Remove {{{}$\{org.apache.unomi.*{}}}} variable
substitutions where appropriate
# *Update code* — Remove any custom loading logic for
{{custom.system.properties}}
# *Update {{BaseIT.java}}* — Replace {{editConfigurationFilePut}} calls with
Karaf native system properties
h3. Phase 3: Cleanup
# *Update examples* — All examples in documentation and code
# *Remove migration script* — After sufficient time has passed
h2. Environment Variable Mapping
h3. Current vs. New Naming Convention
*Current Pattern:* \{{UNOMI_{CATEGORY}
_
{PROPERTY}
}}
*New Pattern (Karaf Native):* {{PID_PROPERTY}} where:
* PID = Configuration Persistent ID (e.g., {{{}org.apache.unomi.cluster{}}})
* Property = Property name (e.g., {{{}nodeId{}}})
* Dots ({{{}.{}}}) become underscores ({{{}_{}}})
* All uppercase
h3. Example Mappings
||Current Environment Variable||New Environment Variable||Configuration
PID||Property Name||
|{{UNOMI_HTTP_PORT}}|{{ORG_OSGI_SERVICE_HTTP_PORT}}|{{org.osgi.service.http}}|{{port}}|
|{{UNOMI_ELASTICSEARCH_ADDRESSES}}|{{ORG_APACHE_UNOMI_ELASTICSEARCH_ADDRESSES}}|{{org.apache.unomi.elasticsearch}}|{{addresses}}|
|{{UNOMI_CLUSTER_NODEID}}|{{ORG_APACHE_UNOMI_CLUSTER_NODEID}}|{{org.apache.unomi.cluster}}|{{nodeId}}|
|{{UNOMI_CLUSTER_NODESTATISTICS_UPDATEFREQUENCY}}|{{ORG_APACHE_UNOMI_CLUSTER_NODESTATISTICSUPDATEFREQUENCY}}|{{org.apache.unomi.cluster}}|{{nodeStatisticsUpdateFrequency}}|
|{{UNOMI_ROUTER_KAFKA_HOST}}|{{ORG_APACHE_UNOMI_ROUTER_KAFKA_HOST}}|{{org.apache.unomi.router}}|{{kafka.host}}|
|{{UNOMI_MIGRATION_RECOVER_FROM_HISTORY}}|{{ORG_APACHE_UNOMI_MIGRATION_RECOVERFROMHISTORY}}|{{org.apache.unomi.migration}}|{{recoverFromHistory}}|
h3. Special Cases
Some properties may need special handling:
# *Properties with dots in the name* — Dots become underscores in environment
variable names
# *Properties that span multiple PIDs* — May need to be split or handled
specially
# *Properties used in multiple contexts* — Need to ensure correct PID mapping
h3. Complete Environment Variable List
*Total Count:* ~203 environment variables need to be mapped
*Categories:*
* Security settings (~10 variables)
* HTTP/SSL settings (~8 variables)
* Cluster settings (~4 variables)
* ElasticSearch settings (~30 variables)
* OpenSearch settings (~30 variables)
* Service settings (~20 variables)
* Third-party server settings (~5 variables)
* Profile cookie settings (~5 variables)
* SSH server settings (~8 variables)
* Log settings (~8 variables)
* GeoNames settings (~3 variables)
* Groovy action settings (~1 variable)
* MaxMind IP settings (~10 variables)
* Mail settings (~5 variables)
* Base plugin settings (~1 variable)
* Router settings (~13 variables)
* Salesforce connector settings (~10 variables)
* Weather update settings (~3 variables)
* GraphQL settings (~1 variable)
* Migration settings (~1 variable)
* HealthCheck settings (~4 variables)
* Goals and campaigns settings (~2 variables)
* REST API authorization settings (~2 variables)
h2. User Impact Assessment
h3. Impacted User Groups
# *Docker/Kubernetes users* — Must update all environment variables in
containers
# *System administrators* — Must update system environment variables
# *CI/CD pipeline maintainers* — Must update automation scripts
# *Custom distribution builders* — Must update build configurations
# *Users with {{unomi.custom.system.properties}} files* — Must migrate to
Karaf native approach
h3. Migration Steps for Users
h4. Step 1: Identify Current Environment Variables
Users need to identify all {{UNOMI_*}} environment variables they are currently
using:
{code:bash}
# Find all UNOMI environment variables
env | grep UNOMI
{code}
h4. Step 2: Map to New Variable Names
Use the migration mapping table to convert each variable:
*Example:*
{code:bash}
# OLD
export UNOMI_HTTP_PORT=8080
export UNOMI_ELASTICSEARCH_ADDRESSES=elasticsearch:9200
# NEW
export ORG_OSGI_SERVICE_HTTP_PORT=8080
export ORG_APACHE_UNOMI_ELASTICSEARCH_ADDRESSES=elasticsearch:9200
{code}
h4. Step 3: Update Configuration Files
# *Docker Compose* — Update {{docker-compose.yml}} files
# *Kubernetes* — Update ConfigMaps and Secrets
# *Systemd services* — Update {{.service}} files
# *Shell scripts* — Update {{setenv}} scripts
# *CI/CD pipelines* — Update pipeline configurations
h4. Step 4: Remove {{unomi.custom.system.properties}} Files
If users have created {{$KARAF_HOME/etc/unomi.custom.system.properties}} files,
they should:
# Convert property overrides to environment variables or system properties
# Remove the file (or it will be ignored in new version)
h3. Breaking Change Considerations
# *Major version required* — This is a breaking change and should be part of a
major version release (e.g., 4.0.0)
# *Deprecation period* — Consider deprecating in 3.x with warnings, remove in
4.x
# *Backward compatibility* — Not possible to maintain backward compatibility
with environment variable names
h2. Implementation Plan
h3. Pre-Implementation Tasks
# *Create complete environment variable mapping table* — All 203+ variables
# *Create migration script/tool* — Help users convert their configurations
# *Identify all code references* — Find all places that reference
{{{}custom.system.properties{}}}, including {{BaseIT.java}}
h3. Implementation Tasks
# *Remove {{custom.system.properties}} file and
{{${optionals}=unomi.custom.system.properties}} mechanism*
# *Update {{.cfg}} files* — Remove {{{}$\{org.apache.unomi.*{}}}} variable
substitutions where appropriate
# *Update code* — Remove any custom loading logic for
{{custom.system.properties}}
# *Update {{BaseIT.java}}* — Replace {{editConfigurationFilePut}} calls for
{{custom.system.properties}} with {{systemProperty()}} or environment variable
configuration (13 properties)
h3. Post-Implementation Tasks
# *Release migration tool* — Make available to users
# *Update all documentation and examples* — Replace references with Karaf
native approach
# *Monitor user feedback and provide support* — Track issues during migration
h2. Documentation Updates Required
h3. Files to Update
# *{{manual/src/main/asciidoc/configuration.adoc}}* — Complete rewrite of
configuration section
# *{{manual/src/main/asciidoc/clustering.adoc}}* — Update cluster
configuration examples
# *{{manual/src/main/asciidoc/profile-import-export.adoc}}* — Update router
configuration
# *{{manual/src/main/asciidoc/connectors/salesforce-connector.adoc}}* — Update
connector configuration
# *{{CODING_GUIDELINES.md}}* — Update environment variable guidelines
# *{{README.md}}* — Update quick start examples
# *Docker documentation* — Update Docker examples
# *Kubernetes documentation* — Update Kubernetes examples
h3. New Documentation Needed
# *Migration guide* — Step-by-step instructions with complete environment
variable mapping table
# *Karaf configuration guide* — How to use Karaf's native system
# *FAQ* — Common migration questions
h2. BaseIT Integration Test Changes
h3. Current Usage
The {{BaseIT.java}} integration test base class currently uses
{{editConfigurationFilePut}} to set properties in
{{etc/custom.system.properties}} for test configuration. There are 13
properties being set:
# {{org.apache.unomi.graphql.feature.activated}}
# {{org.apache.unomi.elasticsearch.cluster.name}}
# {{org.apache.unomi.elasticsearch.addresses}}
# {{org.apache.unomi.elasticsearch.taskWaitingPollingInterval}}
# {{org.apache.unomi.elasticsearch.rollover.maxDocs}}
# {{org.apache.unomi.opensearch.cluster.name}}
# {{org.apache.unomi.opensearch.addresses}}
# {{org.apache.unomi.opensearch.username}}
# {{org.apache.unomi.opensearch.password}}
# {{org.apache.unomi.opensearch.sslEnable}}
# {{org.apache.unomi.opensearch.sslTrustAllCertificates}}
# {{org.apache.unomi.opensearch.minimalClusterState}}
# {{org.apache.unomi.migration.tenant.id}}
h3. Required Changes
When {{custom.system.properties}} is removed, these properties must be set
using Karaf's native system property mechanism instead:
*Replace:*
{code:java}
editConfigurationFilePut("etc/custom.system.properties",
"org.apache.unomi.elasticsearch.addresses", "localhost:" + getSearchPort())
{code}
*With:*
{code:java}
systemProperty("org.apache.unomi.elasticsearch.addresses").value("localhost:" +
getSearchPort())
{code}
*Or using environment variables in PaxExam configuration:*
{code:java}
environment("ORG_APACHE_UNOMI_ELASTICSEARCH_ADDRESSES", "localhost:" +
getSearchPort())
{code}
h3. Impact
# *All 13 {{editConfigurationFilePut}} calls for {{custom.system.properties}}
must be converted*
# *Test configuration will use Karaf's native system property mechanism*
# *Tests will verify that Karaf's native override system works correctly*
h2. Risk Assessment
h3. High Risk Areas
# *Production deployments* — Risk of configuration errors during migration
# *Docker/Kubernetes users* — Many environment variables to update
# *Custom distributions* — May have complex configurations
# *Documentation gaps* — Users may not find migration information
h3. Mitigation Strategies
# *Comprehensive migration guide and tool* — Clear, step-by-step instructions
with automated conversion script
# *Deprecation period* — Warn users in advance (if possible in 3.x)
# *Extensive testing* — Test all scenarios before release
# *Support channels* — Provide help during migration period
h2. Success Criteria
# *All 203+ environment variables successfully migrated, mapped, and tested*
# *Documentation complete* — All references updated with migration guide and
examples
# *Migration tool available* — Users can easily convert configurations
# *Code simplified* — {{custom.system.properties}} mechanism removed
# *No critical issues* — Migration proceeds smoothly for majority of users
h2. Related Issues
* Configuration Admin override problem : UNOMI-917
h2. References
* [Karaf Configuration
Documentation|https://karaf.apache.org/manual/latest/#_environment_variables_system_properties]
* [Current custom.system.properties
file|https://github.com/apache/unomi/blob/master/package/src/main/resources/etc/custom.system.properties]
* [Unomi Configuration
Documentation|https://unomi.apache.org/manual/latest/configuration.html]
> Migrate from custom.system.properties to Karaf Native Environment Variable
> and System Property Override System
> --------------------------------------------------------------------------------------------------------------
>
> Key: UNOMI-918
> URL: https://issues.apache.org/jira/browse/UNOMI-918
> Project: Apache Unomi
> Issue Type: Bug
> Components: unomi(-core)
> Affects Versions: unomi-3.0.0, unomi-2.7.0, unomi-3.1.0
> Reporter: Serge Huber
> Priority: Major
> Fix For: unomi-4.0.0
>
>
> h2. Summary
> Plan and execute migration from Unomi's custom {{custom.system.properties}}
> file to Karaf's native environment variable and system property override
> mechanism. This will simplify configuration management, align with Karaf best
> practices, and eliminate the need for a custom configuration loading
> mechanism.
> h2. Current State
> h3. How custom.system.properties Works
> Apache Unomi currently uses a centralized configuration file
> {{$KARAF_HOME/etc/custom.system.properties}} that:
> # Contains ~203 system properties with environment variable references using
> the pattern: {{{}org.apache.unomi.{*}=$\{env:UNOMI_{*}:-defaultValue{}}}}
> # Uses Karaf's {{${optionals}=unomi.custom.system.properties}} mechanism to
> load user overrides from {{$KARAF_HOME/etc/unomi.custom.system.properties}}
> # Provides variable substitution in {{.cfg}} files using
> {{{}$\{org.apache.unomi.*{}}}} syntax
> # Is documented as the primary way to configure Unomi via environment
> variables
> h3. Current Environment Variable Pattern
> All environment variables follow the pattern: UNOMI_CATEGORY_PROPERTY_NAME
> *Examples:*
> * {{UNOMI_HTTP_PORT}} → {{org.osgi.service.http.port}}
> * {{UNOMI_ELASTICSEARCH_ADDRESSES}} →
> {{org.apache.unomi.elasticsearch.addresses}}
> * {{UNOMI_CLUSTER_NODEID}} → {{org.apache.unomi.cluster.nodeId}}
> h3. User Customization Methods
> Users can override configuration in two ways:
> # *{{unomi.custom.system.properties}} file* — Create
> {{$KARAF_HOME/etc/unomi.custom.system.properties}} with property overrides
> # *Environment variables* — Set {{UNOMI_*}} environment variables
> h3. Documentation References
> The following documentation sections reference
> {{{}custom.system.properties{}}}:
> * {{manual/src/main/asciidoc/configuration.adoc}} — Main configuration
> documentation
> * {{manual/src/main/asciidoc/clustering.adoc}} — Cluster configuration
> * {{manual/src/main/asciidoc/profile-import-export.adoc}} — Router
> configuration
> * {{manual/src/main/asciidoc/connectors/salesforce-connector.adoc}} —
> Connector configuration
> * {{CODING_GUIDELINES.md}} — Developer guidelines
> h2. Target State: Karaf Native System
> h3. How Karaf's Native Override System Works
> Apache Karaf provides a built-in mechanism to override configuration
> properties using environment variables or system properties:
> *Environment Variable Pattern:* {{PID_PROPERTY}} (uppercase, dots become
> underscores)
> *System Property Pattern:* {{pid.property}} (lowercase, dots remain)
> *Example:*
> * Configuration PID: {{org.apache.karaf.shell}}
> * Property name: {{sshPort}}
> * Environment variable: {{ORG_APACHE_KARAF_SHELL_SSHPORT}}
> * System property: {{-Dorg.apache.karaf.shell.sshPort}}
> h3. Benefits of Migration
> # *Simpler architecture* — No custom configuration loading mechanism needed
> # *Standard Karaf approach* — Aligns with Karaf best practices
> # *Better integration* — Works seamlessly with Karaf's
> {{KarafConfigurationPlugin}}
> # *Reduced maintenance* — Less custom code to maintain
> # *Consistent with Karaf ecosystem* — Other Karaf-based projects use this
> approach
> h3. Challenges
> # *Breaking change* — All environment variable names will change (requires
> major version release)
> # *Migration effort* — Users must update their environment variables,
> Docker/Kubernetes deployments, and CI/CD pipelines
> # *Documentation updates* — Extensive documentation changes required across
> multiple files
> h2. Migration Strategy
> h3. Phase 1: Preparation (Non-Breaking)
> # *Create comprehensive environment variable mapping table* — Document all
> 203+ variables
> # *Create migration script/tool* — Help users convert their environment
> variables
> # *Add deprecation warnings* — Log warnings when old {{UNOMI_*}} variables
> are detected (if possible)
> h3. Phase 2: Implementation (Breaking Change - Major Version)
> # *Remove {{custom.system.properties}} file and
> {{${optionals}=unomi.custom.system.properties}} mechanism*
> # *Update {{.cfg}} files* — Remove {{{}$\{org.apache.unomi.*{}}}} variable
> substitutions where appropriate
> # *Update code* — Remove any custom loading logic for
> {{custom.system.properties}}
> # *Update {{BaseIT.java}}* — Replace {{editConfigurationFilePut}} calls with
> Karaf native system properties
> h3. Phase 3: Cleanup
> # *Update examples* — All examples in documentation and code
> # *Remove migration script* — After sufficient time has passed
> h2. Environment Variable Mapping
> h3. Current vs. New Naming Convention
> *Current Pattern:* UNOMI_CATEGORY_PROPERTY
> *New Pattern (Karaf Native):* {{PID_PROPERTY}} where:
> * PID = Configuration Persistent ID (e.g., {{{}org.apache.unomi.cluster{}}})
> * Property = Property name (e.g., {{{}nodeId{}}})
> * Dots ({{{}.{}}}) become underscores ({{{}_{}}})
> * All uppercase
> h3. Example Mappings
> ||Current Environment Variable||New Environment Variable||Configuration
> PID||Property Name||
> |{{UNOMI_HTTP_PORT}}|{{ORG_OSGI_SERVICE_HTTP_PORT}}|{{org.osgi.service.http}}|{{port}}|
> |{{UNOMI_ELASTICSEARCH_ADDRESSES}}|{{ORG_APACHE_UNOMI_ELASTICSEARCH_ADDRESSES}}|{{org.apache.unomi.elasticsearch}}|{{addresses}}|
> |{{UNOMI_CLUSTER_NODEID}}|{{ORG_APACHE_UNOMI_CLUSTER_NODEID}}|{{org.apache.unomi.cluster}}|{{nodeId}}|
> |{{UNOMI_CLUSTER_NODESTATISTICS_UPDATEFREQUENCY}}|{{ORG_APACHE_UNOMI_CLUSTER_NODESTATISTICSUPDATEFREQUENCY}}|{{org.apache.unomi.cluster}}|{{nodeStatisticsUpdateFrequency}}|
> |{{UNOMI_ROUTER_KAFKA_HOST}}|{{ORG_APACHE_UNOMI_ROUTER_KAFKA_HOST}}|{{org.apache.unomi.router}}|{{kafka.host}}|
> |{{UNOMI_MIGRATION_RECOVER_FROM_HISTORY}}|{{ORG_APACHE_UNOMI_MIGRATION_RECOVERFROMHISTORY}}|{{org.apache.unomi.migration}}|{{recoverFromHistory}}|
> h3. Special Cases
> Some properties may need special handling:
> # *Properties with dots in the name* — Dots become underscores in
> environment variable names
> # *Properties that span multiple PIDs* — May need to be split or handled
> specially
> # *Properties used in multiple contexts* — Need to ensure correct PID mapping
> h3. Complete Environment Variable List
> *Total Count:* ~203 environment variables need to be mapped
> *Categories:*
> * Security settings (~10 variables)
> * HTTP/SSL settings (~8 variables)
> * Cluster settings (~4 variables)
> * ElasticSearch settings (~30 variables)
> * OpenSearch settings (~30 variables)
> * Service settings (~20 variables)
> * Third-party server settings (~5 variables)
> * Profile cookie settings (~5 variables)
> * SSH server settings (~8 variables)
> * Log settings (~8 variables)
> * GeoNames settings (~3 variables)
> * Groovy action settings (~1 variable)
> * MaxMind IP settings (~10 variables)
> * Mail settings (~5 variables)
> * Base plugin settings (~1 variable)
> * Router settings (~13 variables)
> * Salesforce connector settings (~10 variables)
> * Weather update settings (~3 variables)
> * GraphQL settings (~1 variable)
> * Migration settings (~1 variable)
> * HealthCheck settings (~4 variables)
> * Goals and campaigns settings (~2 variables)
> * REST API authorization settings (~2 variables)
> h2. User Impact Assessment
> h3. Impacted User Groups
> # *Docker/Kubernetes users* — Must update all environment variables in
> containers
> # *System administrators* — Must update system environment variables
> # *CI/CD pipeline maintainers* — Must update automation scripts
> # *Custom distribution builders* — Must update build configurations
> # *Users with {{unomi.custom.system.properties}} files* — Must migrate to
> Karaf native approach
> h3. Migration Steps for Users
> h4. Step 1: Identify Current Environment Variables
> Users need to identify all {{UNOMI_*}} environment variables they are
> currently using:
> {code:bash}
> # Find all UNOMI environment variables
> env | grep UNOMI
> {code}
> h4. Step 2: Map to New Variable Names
> Use the migration mapping table to convert each variable:
> *Example:*
> {code:bash}
> # OLD
> export UNOMI_HTTP_PORT=8080
> export UNOMI_ELASTICSEARCH_ADDRESSES=elasticsearch:9200
> # NEW
> export ORG_OSGI_SERVICE_HTTP_PORT=8080
> export ORG_APACHE_UNOMI_ELASTICSEARCH_ADDRESSES=elasticsearch:9200
> {code}
> h4. Step 3: Update Configuration Files
> # *Docker Compose* — Update {{docker-compose.yml}} files
> # *Kubernetes* — Update ConfigMaps and Secrets
> # *Systemd services* — Update {{.service}} files
> # *Shell scripts* — Update {{setenv}} scripts
> # *CI/CD pipelines* — Update pipeline configurations
> h4. Step 4: Remove {{unomi.custom.system.properties}} Files
> If users have created {{$KARAF_HOME/etc/unomi.custom.system.properties}}
> files, they should:
> # Convert property overrides to environment variables or system properties
> # Remove the file (or it will be ignored in new version)
> h3. Breaking Change Considerations
> # *Major version required* — This is a breaking change and should be part of
> a major version release (e.g., 4.0.0)
> # *Deprecation period* — Consider deprecating in 3.x with warnings, remove
> in 4.x
> # *Backward compatibility* — Not possible to maintain backward compatibility
> with environment variable names
> h2. Implementation Plan
> h3. Pre-Implementation Tasks
> # *Create complete environment variable mapping table* — All 203+ variables
> # *Create migration script/tool* — Help users convert their configurations
> # *Identify all code references* — Find all places that reference
> {{{}custom.system.properties{}}}, including {{BaseIT.java}}
> h3. Implementation Tasks
> # *Remove {{custom.system.properties}} file and
> {{${optionals}=unomi.custom.system.properties}} mechanism*
> # *Update {{.cfg}} files* — Remove {{{}$\{org.apache.unomi.*{}}}} variable
> substitutions where appropriate
> # *Update code* — Remove any custom loading logic for
> {{custom.system.properties}}
> # *Update {{BaseIT.java}}* — Replace {{editConfigurationFilePut}} calls for
> {{custom.system.properties}} with {{systemProperty()}} or environment
> variable configuration (13 properties)
> h3. Post-Implementation Tasks
> # *Release migration tool* — Make available to users
> # *Update all documentation and examples* — Replace references with Karaf
> native approach
> # *Monitor user feedback and provide support* — Track issues during migration
> h2. Documentation Updates Required
> h3. Files to Update
> # *{{manual/src/main/asciidoc/configuration.adoc}}* — Complete rewrite of
> configuration section
> # *{{manual/src/main/asciidoc/clustering.adoc}}* — Update cluster
> configuration examples
> # *{{manual/src/main/asciidoc/profile-import-export.adoc}}* — Update router
> configuration
> # *{{manual/src/main/asciidoc/connectors/salesforce-connector.adoc}}* —
> Update connector configuration
> # *{{CODING_GUIDELINES.md}}* — Update environment variable guidelines
> # *{{README.md}}* — Update quick start examples
> # *Docker documentation* — Update Docker examples
> # *Kubernetes documentation* — Update Kubernetes examples
> h3. New Documentation Needed
> # *Migration guide* — Step-by-step instructions with complete environment
> variable mapping table
> # *Karaf configuration guide* — How to use Karaf's native system
> # *FAQ* — Common migration questions
> h2. BaseIT Integration Test Changes
> h3. Current Usage
> The {{BaseIT.java}} integration test base class currently uses
> {{editConfigurationFilePut}} to set properties in
> {{etc/custom.system.properties}} for test configuration. There are 13
> properties being set:
> # {{org.apache.unomi.graphql.feature.activated}}
> # {{org.apache.unomi.elasticsearch.cluster.name}}
> # {{org.apache.unomi.elasticsearch.addresses}}
> # {{org.apache.unomi.elasticsearch.taskWaitingPollingInterval}}
> # {{org.apache.unomi.elasticsearch.rollover.maxDocs}}
> # {{org.apache.unomi.opensearch.cluster.name}}
> # {{org.apache.unomi.opensearch.addresses}}
> # {{org.apache.unomi.opensearch.username}}
> # {{org.apache.unomi.opensearch.password}}
> # {{org.apache.unomi.opensearch.sslEnable}}
> # {{org.apache.unomi.opensearch.sslTrustAllCertificates}}
> # {{org.apache.unomi.opensearch.minimalClusterState}}
> # {{org.apache.unomi.migration.tenant.id}}
> h3. Required Changes
> When {{custom.system.properties}} is removed, these properties must be set
> using Karaf's native system property mechanism instead:
> *Replace:*
> {code:java}
> editConfigurationFilePut("etc/custom.system.properties",
> "org.apache.unomi.elasticsearch.addresses", "localhost:" + getSearchPort())
> {code}
> *With:*
> {code:java}
> systemProperty("org.apache.unomi.elasticsearch.addresses").value("localhost:"
> + getSearchPort())
> {code}
> *Or using environment variables in PaxExam configuration:*
> {code:java}
> environment("ORG_APACHE_UNOMI_ELASTICSEARCH_ADDRESSES", "localhost:" +
> getSearchPort())
> {code}
> h3. Impact
> # *All 13 {{editConfigurationFilePut}} calls for
> {{custom.system.properties}} must be converted*
> # *Test configuration will use Karaf's native system property mechanism*
> # *Tests will verify that Karaf's native override system works correctly*
> h2. Risk Assessment
> h3. High Risk Areas
> # *Production deployments* — Risk of configuration errors during migration
> # *Docker/Kubernetes users* — Many environment variables to update
> # *Custom distributions* — May have complex configurations
> # *Documentation gaps* — Users may not find migration information
> h3. Mitigation Strategies
> # *Comprehensive migration guide and tool* — Clear, step-by-step
> instructions with automated conversion script
> # *Deprecation period* — Warn users in advance (if possible in 3.x)
> # *Extensive testing* — Test all scenarios before release
> # *Support channels* — Provide help during migration period
> h2. Success Criteria
> # *All 203+ environment variables successfully migrated, mapped, and tested*
> # *Documentation complete* — All references updated with migration guide and
> examples
> # *Migration tool available* — Users can easily convert configurations
> # *Code simplified* — {{custom.system.properties}} mechanism removed
> # *No critical issues* — Migration proceeds smoothly for majority of users
> h2. Related Issues
> * Configuration Admin override problem : UNOMI-917
> h2. References
> * [Karaf Configuration
> Documentation|https://karaf.apache.org/manual/latest/#_environment_variables_system_properties]
> * [Current custom.system.properties
> file|https://github.com/apache/unomi/blob/master/package/src/main/resources/etc/custom.system.properties]
> * [Unomi Configuration
> Documentation|https://unomi.apache.org/manual/latest/configuration.html]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
