sergehuber commented on code in PR #472: URL: https://github.com/apache/unomi/pull/472#discussion_r954971800
########## manual/src/main/asciidoc/migrations/migrate-1.6-to-2.0.adoc: ########## @@ -0,0 +1,189 @@ +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// + +=== Migration Overview + +Apache Unomi 2.0 is a major release, and as such it does introduce breaking changes. This portion of the document detail the various steps we recommend following to successfully migrate your environment from Apache Unomi 1.6 to Apache Unomi 2.0. + +There are two main steps in preparing your migration to Apache Unomi 2.0: +- Updating applications consuming Unomi +- Migrating your existing data + +=== Updating applications consuming Unomi + +Since Apache Unomi is an engine, you've probably built multiple applications consuming its APIs, you might also have built extensions directly running in Unomi. + +As you begin updating applications consuming Apache Unomi, it is generally a good practice to <<Enable debug mode>>. +Doing so will display any errors when processing events (such as JSON Schema validations), and will provide useful indications towards solving issues. + +==== Data Model changes + +There has been changes to Unomi Data model, please make sure to review those in the << what_s_new>> section of the user manual. + +==== Create JSON schemas + +Once you updated your applications to align with Unomi 2 data model, the next step will be to create the necessary JSON Schemas. + +Any event (and more generally, any object) received through Unomi public endpoints do require a valid JSON schema. +Apache Unomi ships, out of the box, with all of the necessary JSON Schemas for its own operation but you will need to create schemas for any custom event. + +When creating your new schemas, reviewing debug messages in the logs (using: `log:set DEBUG org.apache.unomi.schema.impl.SchemaServiceImpl` in Karaf console), +will point to errors in your schemas. Review Comment: ```suggestion will point to errors in your schemas or will help you diagnose why the events are not being accepted. ``` ########## manual/src/main/asciidoc/migrations/migrate-1.6-to-2.0.adoc: ########## @@ -0,0 +1,189 @@ +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// + +=== Migration Overview + +Apache Unomi 2.0 is a major release, and as such it does introduce breaking changes. This portion of the document detail the various steps we recommend following to successfully migrate your environment from Apache Unomi 1.6 to Apache Unomi 2.0. + +There are two main steps in preparing your migration to Apache Unomi 2.0: +- Updating applications consuming Unomi +- Migrating your existing data + +=== Updating applications consuming Unomi + +Since Apache Unomi is an engine, you've probably built multiple applications consuming its APIs, you might also have built extensions directly running in Unomi. + +As you begin updating applications consuming Apache Unomi, it is generally a good practice to <<Enable debug mode>>. +Doing so will display any errors when processing events (such as JSON Schema validations), and will provide useful indications towards solving issues. + +==== Data Model changes + +There has been changes to Unomi Data model, please make sure to review those in the << what_s_new>> section of the user manual. + +==== Create JSON schemas + +Once you updated your applications to align with Unomi 2 data model, the next step will be to create the necessary JSON Schemas. + +Any event (and more generally, any object) received through Unomi public endpoints do require a valid JSON schema. +Apache Unomi ships, out of the box, with all of the necessary JSON Schemas for its own operation but you will need to create schemas for any custom event. Review Comment: ```suggestion Apache Unomi ships, out of the box, with all of the necessary JSON Schemas for its own operation but you will need to create schemas for any custom event you may be using. ``` ########## manual/src/main/asciidoc/migrations/migrate-1.6-to-2.0.adoc: ########## @@ -0,0 +1,189 @@ +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// + +=== Migration Overview + +Apache Unomi 2.0 is a major release, and as such it does introduce breaking changes. This portion of the document detail the various steps we recommend following to successfully migrate your environment from Apache Unomi 1.6 to Apache Unomi 2.0. + +There are two main steps in preparing your migration to Apache Unomi 2.0: +- Updating applications consuming Unomi +- Migrating your existing data + +=== Updating applications consuming Unomi + +Since Apache Unomi is an engine, you've probably built multiple applications consuming its APIs, you might also have built extensions directly running in Unomi. + +As you begin updating applications consuming Apache Unomi, it is generally a good practice to <<Enable debug mode>>. +Doing so will display any errors when processing events (such as JSON Schema validations), and will provide useful indications towards solving issues. + +==== Data Model changes + +There has been changes to Unomi Data model, please make sure to review those in the << what_s_new>> section of the user manual. + +==== Create JSON schemas + +Once you updated your applications to align with Unomi 2 data model, the next step will be to create the necessary JSON Schemas. + +Any event (and more generally, any object) received through Unomi public endpoints do require a valid JSON schema. +Apache Unomi ships, out of the box, with all of the necessary JSON Schemas for its own operation but you will need to create schemas for any custom event. + +When creating your new schemas, reviewing debug messages in the logs (using: `log:set DEBUG org.apache.unomi.schema.impl.SchemaServiceImpl` in Karaf console), +will point to errors in your schemas. + +Note that it is currently not possible to modify or surcharge an existing JSON schema via the REST API. Review Comment: ```suggestion Note that it is currently not possible to modify or surcharge an existing system-deployed JSON schema via the REST API. It is however possible to deploy new schemas and manage them through the REST API on the `/cxs/jsonSchema` endpoint. ``` ########## manual/src/main/asciidoc/migrations/migrate-1.6-to-2.0.adoc: ########## @@ -0,0 +1,189 @@ +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// + +=== Migration Overview + +Apache Unomi 2.0 is a major release, and as such it does introduce breaking changes. This portion of the document detail the various steps we recommend following to successfully migrate your environment from Apache Unomi 1.6 to Apache Unomi 2.0. + +There are two main steps in preparing your migration to Apache Unomi 2.0: +- Updating applications consuming Unomi +- Migrating your existing data + +=== Updating applications consuming Unomi + +Since Apache Unomi is an engine, you've probably built multiple applications consuming its APIs, you might also have built extensions directly running in Unomi. + +As you begin updating applications consuming Apache Unomi, it is generally a good practice to <<Enable debug mode>>. +Doing so will display any errors when processing events (such as JSON Schema validations), and will provide useful indications towards solving issues. + +==== Data Model changes + +There has been changes to Unomi Data model, please make sure to review those in the << what_s_new>> section of the user manual. + +==== Create JSON schemas + +Once you updated your applications to align with Unomi 2 data model, the next step will be to create the necessary JSON Schemas. + +Any event (and more generally, any object) received through Unomi public endpoints do require a valid JSON schema. +Apache Unomi ships, out of the box, with all of the necessary JSON Schemas for its own operation but you will need to create schemas for any custom event. + +When creating your new schemas, reviewing debug messages in the logs (using: `log:set DEBUG org.apache.unomi.schema.impl.SchemaServiceImpl` in Karaf console), +will point to errors in your schemas. + +Note that it is currently not possible to modify or surcharge an existing JSON schema via the REST API. +If you are using custom properties on an event type for which an Apache Unomi schema already exists, Review Comment: ```suggestion If you are currently using custom properties on an Apache Unomi-provided event type, ``` ########## manual/src/main/asciidoc/migrations/migrate-1.6-to-2.0.adoc: ########## @@ -0,0 +1,189 @@ +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// + +=== Migration Overview + +Apache Unomi 2.0 is a major release, and as such it does introduce breaking changes. This portion of the document detail the various steps we recommend following to successfully migrate your environment from Apache Unomi 1.6 to Apache Unomi 2.0. + +There are two main steps in preparing your migration to Apache Unomi 2.0: +- Updating applications consuming Unomi +- Migrating your existing data + +=== Updating applications consuming Unomi + +Since Apache Unomi is an engine, you've probably built multiple applications consuming its APIs, you might also have built extensions directly running in Unomi. + +As you begin updating applications consuming Apache Unomi, it is generally a good practice to <<Enable debug mode>>. +Doing so will display any errors when processing events (such as JSON Schema validations), and will provide useful indications towards solving issues. + +==== Data Model changes + +There has been changes to Unomi Data model, please make sure to review those in the << what_s_new>> section of the user manual. + +==== Create JSON schemas + +Once you updated your applications to align with Unomi 2 data model, the next step will be to create the necessary JSON Schemas. + +Any event (and more generally, any object) received through Unomi public endpoints do require a valid JSON schema. +Apache Unomi ships, out of the box, with all of the necessary JSON Schemas for its own operation but you will need to create schemas for any custom event. + +When creating your new schemas, reviewing debug messages in the logs (using: `log:set DEBUG org.apache.unomi.schema.impl.SchemaServiceImpl` in Karaf console), +will point to errors in your schemas. + +Note that it is currently not possible to modify or surcharge an existing JSON schema via the REST API. +If you are using custom properties on an event type for which an Apache Unomi schema already exists, +you will need to either change its eventType and create the corresponding Schema or to create a Unomi extension. + +You can use as a source of inspiration for creating new schemas, Apache Unomi 2.0 schema located at: + https://github.com/apache/unomi/tree/master/extensions/json-schema/services/src/main/resources/META-INF/cxs/schemas[extensions/json-schema/services/src/main/resources/META-INF/cxs/schemas]. + +Finally, and altuough it is technically feasible, we recommend against creating permissive JSON Schemas allowing any event payload. Review Comment: ```suggestion Finally, and although it is technically feasible, we recommend against creating permissive JSON Schemas allowing any event payload. This requires making sure that you don't allow open properties by using JSON schema keywords such as https://json-schema.org/understanding-json-schema/reference/object.html#unevaluated-properties[unevaluated properties] ``` ########## manual/src/main/asciidoc/migrations/migrate-1.6-to-2.0.adoc: ########## @@ -0,0 +1,189 @@ +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// + +=== Migration Overview + +Apache Unomi 2.0 is a major release, and as such it does introduce breaking changes. This portion of the document detail the various steps we recommend following to successfully migrate your environment from Apache Unomi 1.6 to Apache Unomi 2.0. + +There are two main steps in preparing your migration to Apache Unomi 2.0: +- Updating applications consuming Unomi +- Migrating your existing data + +=== Updating applications consuming Unomi + +Since Apache Unomi is an engine, you've probably built multiple applications consuming its APIs, you might also have built extensions directly running in Unomi. + +As you begin updating applications consuming Apache Unomi, it is generally a good practice to <<Enable debug mode>>. +Doing so will display any errors when processing events (such as JSON Schema validations), and will provide useful indications towards solving issues. + +==== Data Model changes + +There has been changes to Unomi Data model, please make sure to review those in the << what_s_new>> section of the user manual. + +==== Create JSON schemas + +Once you updated your applications to align with Unomi 2 data model, the next step will be to create the necessary JSON Schemas. + +Any event (and more generally, any object) received through Unomi public endpoints do require a valid JSON schema. +Apache Unomi ships, out of the box, with all of the necessary JSON Schemas for its own operation but you will need to create schemas for any custom event. + +When creating your new schemas, reviewing debug messages in the logs (using: `log:set DEBUG org.apache.unomi.schema.impl.SchemaServiceImpl` in Karaf console), +will point to errors in your schemas. + +Note that it is currently not possible to modify or surcharge an existing JSON schema via the REST API. +If you are using custom properties on an event type for which an Apache Unomi schema already exists, +you will need to either change its eventType and create the corresponding Schema or to create a Unomi extension. + +You can use as a source of inspiration for creating new schemas, Apache Unomi 2.0 schema located at: + https://github.com/apache/unomi/tree/master/extensions/json-schema/services/src/main/resources/META-INF/cxs/schemas[extensions/json-schema/services/src/main/resources/META-INF/cxs/schemas]. + +Finally, and altuough it is technically feasible, we recommend against creating permissive JSON Schemas allowing any event payload. + +=== Migrating your existing data + +==== Elasticsearch version and capacity + +While still using Unomi 1.6, the first step will be to upgrade your Elasticsearch to 7.17.5. +Documentation is available on https://www.elastic.co/guide/en/elasticsearch/reference/7.17/setup-upgrade.html[Elasticsearch's website]. + +Your Elasticsearch cluster must have enough capacity to handle the migration. +At a minimum, the required capacity must be greater than the size of the dataset in production + the size of the largest index. + +==== Migrate custom data + +Apache Unomi 2.0 knows how to migrate its own data from the new model to the old one, but it does not know how to migrate custom events you might be using in your environment. + +It relies on a set of groovy scripts to perform its data migration, +located in https://github.com/apache/unomi/tree/master/tools/shell-commands/src/main/resources/META-INF/cxs/migration[tools/shell-commands/src/main/resources/META-INF/cxs/migration], +these scripts are sorted alphabetically and executed sequentially when migration is started. You can use these scripts as a source of inspiration for creating your own. + +In most cases, migration steps consist of an Elasticsearch painless script that will handle the data changes. + +At runtime, and when starting the migration, Unomi 2.0 will take its own scripts, any additional scripts located in `data/migration/scripts`, will sort the resulting list alphabetically and execute each migration script sequentially. + +==== Perform the migration + +===== Checklist + +Before starting the mirgation, please ensure that: + + - You do have a backup of your data + - You did practice the migration in a staging environment, NEVER migrate a production environment without prior validation + - You verified your applications were operational with Apache Unomi 2.0 (JSON schemas created, client applications updated, ...) + - You are running Elasticsearch 7.17.5 (or a later 7.x version) + - Your Elasticsearch cluster has enough capacity to handle the migration + - You are currently running Apache Unomi 1.6 (or a later 1.x version) + - You will be using the same Apache Unomi instance for the entire migration progress. Do not start the migration on one node, and resume an interrupted migration on another node. + +===== Migration process overview + +The migration is performed by means of a dedicated Apache Unomi 2.0 node started in a particular migration mode. + +In a nutshell, the migration process will consist in the following steps: + +- Shutdown your Apache Unomi 1.6 cluster +- Start an Apache Unomi 2.0 migration node +- Wait for data migration to complete +- Start you Apache Unomi 2.0 cluster +- (optional) Import additional JSON Schemas + +Each migration step maintains its execution state, meaning if a step fails you can fix the issue, and resume the migration from the failed step. + +===== Configuration + +The following environment variables are used for the migration: + +|=== +|Environment Variable|Unomi Setting|Default + +|UNOMI_ELASTICSEARCH_ADDRESSES +|org.apache.unomi.elasticsearch.addresses +|localhost:9200 + +|UNOMI_ELASTICSEARCH_SSL_ENABLE +|org.apache.unomi.elasticsearch.sslEnable +|false + +|UNOMI_ELASTICSEARCH_USERNAME +|org.apache.unomi.elasticsearch.username +| + +|UNOMI_ELASTICSEARCH_PASSWORD +|org.apache.unomi.elasticsearch.password +| + +|UNOMI_ELASTICSEARCH_SSL_TRUST_ALL_CERTIFICATES +|org.apache.unomi.elasticsearch.sslTrustAllCertificates +|false + +|UNOMI_ELASTICSEARCH_INDEXPREFIX +|org.apache.unomi.elasticsearch.index.prefix +|context + +|UNOMI_MIGRATION_RECOVER_FROM_HISTORY +|org.apache.unomi.migration.recoverFromHistory +|true + +|=== + +If there is a need for advanced configuratiion, the configuration file used by Apache Unomi 2.0 is located in: `etc/org.apache.unomi.migration.cfg` + +===== Migrate manually + +You can migrate manually using the Karaf console. + +After having started Apache Unomi 2.0 with the `./karaf` command, you will be presented with the Karaf shell. + +From there you have two options: + + - The necessary configuration variables (see above) are set, the migration can be started using: `unomi:migrate 1.6.0` + - Or, you want to provide the configuration settings via the terminal, in that case you can start the migration in interactive mode using: `unomi:migrate 1.6.0` + +The value of the migrate command (1.6.0 in the example above), corresponds to the version you're migrating from. Review Comment: ```suggestion The parameter of the migrate command (1.6.0 in the example above) corresponds to the version you're migrating from. ``` ########## manual/src/main/asciidoc/migrations/migrate-1.6-to-2.0.adoc: ########## @@ -0,0 +1,189 @@ +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// + +=== Migration Overview + +Apache Unomi 2.0 is a major release, and as such it does introduce breaking changes. This portion of the document detail the various steps we recommend following to successfully migrate your environment from Apache Unomi 1.6 to Apache Unomi 2.0. + +There are two main steps in preparing your migration to Apache Unomi 2.0: +- Updating applications consuming Unomi +- Migrating your existing data + +=== Updating applications consuming Unomi + +Since Apache Unomi is an engine, you've probably built multiple applications consuming its APIs, you might also have built extensions directly running in Unomi. + +As you begin updating applications consuming Apache Unomi, it is generally a good practice to <<Enable debug mode>>. +Doing so will display any errors when processing events (such as JSON Schema validations), and will provide useful indications towards solving issues. + +==== Data Model changes + +There has been changes to Unomi Data model, please make sure to review those in the << what_s_new>> section of the user manual. + +==== Create JSON schemas + +Once you updated your applications to align with Unomi 2 data model, the next step will be to create the necessary JSON Schemas. + +Any event (and more generally, any object) received through Unomi public endpoints do require a valid JSON schema. +Apache Unomi ships, out of the box, with all of the necessary JSON Schemas for its own operation but you will need to create schemas for any custom event. + +When creating your new schemas, reviewing debug messages in the logs (using: `log:set DEBUG org.apache.unomi.schema.impl.SchemaServiceImpl` in Karaf console), +will point to errors in your schemas. + +Note that it is currently not possible to modify or surcharge an existing JSON schema via the REST API. +If you are using custom properties on an event type for which an Apache Unomi schema already exists, +you will need to either change its eventType and create the corresponding Schema or to create a Unomi extension. + +You can use as a source of inspiration for creating new schemas, Apache Unomi 2.0 schema located at: + https://github.com/apache/unomi/tree/master/extensions/json-schema/services/src/main/resources/META-INF/cxs/schemas[extensions/json-schema/services/src/main/resources/META-INF/cxs/schemas]. + +Finally, and altuough it is technically feasible, we recommend against creating permissive JSON Schemas allowing any event payload. + +=== Migrating your existing data + +==== Elasticsearch version and capacity + +While still using Unomi 1.6, the first step will be to upgrade your Elasticsearch to 7.17.5. +Documentation is available on https://www.elastic.co/guide/en/elasticsearch/reference/7.17/setup-upgrade.html[Elasticsearch's website]. + +Your Elasticsearch cluster must have enough capacity to handle the migration. +At a minimum, the required capacity must be greater than the size of the dataset in production + the size of the largest index. + +==== Migrate custom data + +Apache Unomi 2.0 knows how to migrate its own data from the new model to the old one, but it does not know how to migrate custom events you might be using in your environment. + +It relies on a set of groovy scripts to perform its data migration, +located in https://github.com/apache/unomi/tree/master/tools/shell-commands/src/main/resources/META-INF/cxs/migration[tools/shell-commands/src/main/resources/META-INF/cxs/migration], +these scripts are sorted alphabetically and executed sequentially when migration is started. You can use these scripts as a source of inspiration for creating your own. + +In most cases, migration steps consist of an Elasticsearch painless script that will handle the data changes. + +At runtime, and when starting the migration, Unomi 2.0 will take its own scripts, any additional scripts located in `data/migration/scripts`, will sort the resulting list alphabetically and execute each migration script sequentially. + +==== Perform the migration + +===== Checklist + +Before starting the mirgation, please ensure that: + + - You do have a backup of your data + - You did practice the migration in a staging environment, NEVER migrate a production environment without prior validation + - You verified your applications were operational with Apache Unomi 2.0 (JSON schemas created, client applications updated, ...) + - You are running Elasticsearch 7.17.5 (or a later 7.x version) + - Your Elasticsearch cluster has enough capacity to handle the migration + - You are currently running Apache Unomi 1.6 (or a later 1.x version) + - You will be using the same Apache Unomi instance for the entire migration progress. Do not start the migration on one node, and resume an interrupted migration on another node. + +===== Migration process overview + +The migration is performed by means of a dedicated Apache Unomi 2.0 node started in a particular migration mode. + +In a nutshell, the migration process will consist in the following steps: + +- Shutdown your Apache Unomi 1.6 cluster +- Start an Apache Unomi 2.0 migration node +- Wait for data migration to complete +- Start you Apache Unomi 2.0 cluster +- (optional) Import additional JSON Schemas + +Each migration step maintains its execution state, meaning if a step fails you can fix the issue, and resume the migration from the failed step. + +===== Configuration + +The following environment variables are used for the migration: + +|=== +|Environment Variable|Unomi Setting|Default + +|UNOMI_ELASTICSEARCH_ADDRESSES +|org.apache.unomi.elasticsearch.addresses +|localhost:9200 + +|UNOMI_ELASTICSEARCH_SSL_ENABLE +|org.apache.unomi.elasticsearch.sslEnable +|false + +|UNOMI_ELASTICSEARCH_USERNAME +|org.apache.unomi.elasticsearch.username +| + +|UNOMI_ELASTICSEARCH_PASSWORD +|org.apache.unomi.elasticsearch.password +| + +|UNOMI_ELASTICSEARCH_SSL_TRUST_ALL_CERTIFICATES +|org.apache.unomi.elasticsearch.sslTrustAllCertificates +|false + +|UNOMI_ELASTICSEARCH_INDEXPREFIX +|org.apache.unomi.elasticsearch.index.prefix +|context + +|UNOMI_MIGRATION_RECOVER_FROM_HISTORY +|org.apache.unomi.migration.recoverFromHistory +|true + +|=== + +If there is a need for advanced configuratiion, the configuration file used by Apache Unomi 2.0 is located in: `etc/org.apache.unomi.migration.cfg` + +===== Migrate manually + +You can migrate manually using the Karaf console. + +After having started Apache Unomi 2.0 with the `./karaf` command, you will be presented with the Karaf shell. + +From there you have two options: + + - The necessary configuration variables (see above) are set, the migration can be started using: `unomi:migrate 1.6.0` Review Comment: ```suggestion - The necessary configuration variables (see above) have already been set, you can start the migration using the command: `unomi:migrate 1.6.0` ``` ########## manual/src/main/asciidoc/recipes.adoc: ########## @@ -146,40 +187,150 @@ sending them to Apache Unomi whenever you like) of type 'contactInfoSubmitted' a course change any of the property names to find your needs. For example you might want to prefix the profile properties with the source of the event, such as 'mobileApp:firstName'. -You could then simply send the `contactInfoSubmitted` event using a request similar to this one: +Now that our rule is defined, the next step is to create a scope and a JSON Schema corresponding to the event to be submitted. + +We will start by creating a scope called "example" scope: +[source] +---- +curl --location --request POST 'http://localhost:8181/cxs/scopes' \ +-u 'karaf:karaf' \ +--header 'Content-Type: application/json' \ +--data-raw '{ +"itemId": "example", +"itemType": "scope" +}' +---- + +The next step consist in creating a JSON Schema to validate our event. + +[source] +---- +curl --location --request POST 'http://localhost:8181/cxs/jsonSchema' \ +-u 'karaf:karaf' \ +--header 'Content-Type: application/json' \ +--data-raw '{ + "$id": "https://unomi.apache.org/schemas/json/events/contactInfoSubmitted/1-0-0";, + "$schema": "https://json-schema.org/draft/2019-09/schema";, + "self": { + "vendor": "org.apache.unomi", + "name": "contactInfoSubmitted", + "format": "jsonschema", + "target": "events", + "version": "1-0-0" + }, + "title": "contactInfoSubmittedEvent", + "type": "object", + "allOf": [{ "$ref": "https://unomi.apache.org/schemas/json/event/1-0-0"; }], + "properties": { + "source" : { + "$ref" : "https://unomi.apache.org/schemas/json/item/1-0-0"; + }, + "target" : { + "$ref" : "https://unomi.apache.org/schemas/json/item/1-0-0"; + }, + "properties": { + "type": "object", + "properties": { + "firstName": { + "type": ["null", "string"] + }, + "lastName": { + "type": ["null", "string"] + }, + "email": { + "type": ["null", "string"] + } + } + } + }, + "unevaluatedProperties": false +}' +---- + +You can notice the following in the above schema: + +* We are creating a schema of type "events" ("self.target" equals "events") +* The name of this schema is "contactInfoSubmitted", this MUST match the value of the "eventType" field in the event itself (below) +* To simplify our schema declaration, we're referring to an already existing schema (https://unomi.apache.org/schemas/json/item/1-0-0) to validate the "source" and "target" properties. Apache Unomi ships with a set of predefined JSON Schemas, detailed here: https://github.com/apache/unomi/tree/master/extensions/json-schema/services/src/main/resources/META-INF/cxs/schemas. +* `"unevaluatedProperties": false` indicates that the event should be rejected if it contains any additional metadata. + +Finally, send the `contactInfoSubmitted` event using a request similar to this one: [source] ---- curl -X POST http://localhost:8181/cxs/eventcollector \ -H "Content-Type: application/json" \ --d @- <<'EOF' -{ +--data-raw '{ "sessionId" : "1234", "events":[ { "eventType":"contactInfoSubmitted", "scope": "example", "source":{ "itemType": "site", - "scope":"example", + "scope": "example", "itemId": "mysite" }, "target":{ - "itemType":"form", - "scope":"example", - "itemId":"contactForm" + "itemType": "form", + "scope": "example", + "itemId": "contactForm" }, "properties" : { - "firstName" : "John", - "lastName" : "Doe", - "email" : "[email protected]" + "firstName": "John", + "lastName": "Doe", + "email": "[email protected]" } } ] -} -EOF +}' ---- +The event we just submitted can be retrieved using the following request: + +[source] +---- +curl -X POST http://localhost:8181/cxs/events/search \ +--user karaf:karaf \ +-H "Content-Type: application/json" \ +--data-raw '{ + "offset" : 0, + "limit" : 20, + "condition" : { + "type": "eventPropertyCondition", + "parameterValues" : { + "propertyName" : "properties.firstName", + "comparisonOperator" : "equals", + "propertyValue" : "John" + } + } +}' +---- + +===== Debug common errors Review Comment: ```suggestion ===== Troubleshooting common errors ``` ########## manual/src/main/asciidoc/whats-new.adoc: ########## @@ -0,0 +1,110 @@ +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +=== What's new in Apache Unomi 2.0 + +Apache Unomi 2 is a new release focused on improving core functionalities and robustness of the product. +The introduction of tighter data validation with JSON Schemas required some changes in the product data model, which in turn presented an opportunity for noticeable improvements of the overall performances. + +This new release also introduces a first (beta) version of Unomi GraphQL API. + +==== Scopes are required + +Scopes are required in Unomi 2. When submitting an event and specifying a scope, Review Comment: ```suggestion Scopes declarations are now required in Unomi 2. When submitting an event and specifying a scope, ``` ########## manual/src/main/asciidoc/whats-new.adoc: ########## @@ -0,0 +1,110 @@ +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +=== What's new in Apache Unomi 2.0 + +Apache Unomi 2 is a new release focused on improving core functionalities and robustness of the product. +The introduction of tighter data validation with JSON Schemas required some changes in the product data model, which in turn presented an opportunity for noticeable improvements of the overall performances. + +This new release also introduces a first (beta) version of Unomi GraphQL API. Review Comment: ```suggestion This new release also introduces a first (beta) version of the Unomi GraphQL API. ``` ########## manual/src/main/asciidoc/whats-new.adoc: ########## @@ -0,0 +1,110 @@ +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +=== What's new in Apache Unomi 2.0 + +Apache Unomi 2 is a new release focused on improving core functionalities and robustness of the product. +The introduction of tighter data validation with JSON Schemas required some changes in the product data model, which in turn presented an opportunity for noticeable improvements of the overall performances. Review Comment: ```suggestion The introduction of tighter data validation with JSON Schemas required some changes in the product data model, which presented an opportunity for noticeable improvements in the overall performance. ``` ########## manual/src/main/asciidoc/recipes.adoc: ########## @@ -10,14 +10,61 @@ // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. -// +// === Recipes ==== Introduction In this section of the documentation we provide quick recipes focused on helping you achieve a specific result with Apache Unomi. +==== Enable debug mode Review Comment: ```suggestion ==== Enabling debug mode ``` ########## manual/src/main/asciidoc/migrations/migrate-1.6-to-2.0.adoc: ########## @@ -0,0 +1,189 @@ +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// + +=== Migration Overview + +Apache Unomi 2.0 is a major release, and as such it does introduce breaking changes. This portion of the document detail the various steps we recommend following to successfully migrate your environment from Apache Unomi 1.6 to Apache Unomi 2.0. + +There are two main steps in preparing your migration to Apache Unomi 2.0: +- Updating applications consuming Unomi +- Migrating your existing data + +=== Updating applications consuming Unomi + +Since Apache Unomi is an engine, you've probably built multiple applications consuming its APIs, you might also have built extensions directly running in Unomi. + +As you begin updating applications consuming Apache Unomi, it is generally a good practice to <<Enable debug mode>>. +Doing so will display any errors when processing events (such as JSON Schema validations), and will provide useful indications towards solving issues. + +==== Data Model changes + +There has been changes to Unomi Data model, please make sure to review those in the << what_s_new>> section of the user manual. + +==== Create JSON schemas + +Once you updated your applications to align with Unomi 2 data model, the next step will be to create the necessary JSON Schemas. + +Any event (and more generally, any object) received through Unomi public endpoints do require a valid JSON schema. +Apache Unomi ships, out of the box, with all of the necessary JSON Schemas for its own operation but you will need to create schemas for any custom event. + +When creating your new schemas, reviewing debug messages in the logs (using: `log:set DEBUG org.apache.unomi.schema.impl.SchemaServiceImpl` in Karaf console), +will point to errors in your schemas. + +Note that it is currently not possible to modify or surcharge an existing JSON schema via the REST API. +If you are using custom properties on an event type for which an Apache Unomi schema already exists, +you will need to either change its eventType and create the corresponding Schema or to create a Unomi extension. Review Comment: ```suggestion you will need to either change to use a new custom eventType and create the corresponding schema or to create a Unomi schema extension. ``` ########## manual/src/main/asciidoc/migrations/migrate-1.6-to-2.0.adoc: ########## @@ -0,0 +1,189 @@ +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// + +=== Migration Overview + +Apache Unomi 2.0 is a major release, and as such it does introduce breaking changes. This portion of the document detail the various steps we recommend following to successfully migrate your environment from Apache Unomi 1.6 to Apache Unomi 2.0. + +There are two main steps in preparing your migration to Apache Unomi 2.0: +- Updating applications consuming Unomi +- Migrating your existing data + +=== Updating applications consuming Unomi + +Since Apache Unomi is an engine, you've probably built multiple applications consuming its APIs, you might also have built extensions directly running in Unomi. + +As you begin updating applications consuming Apache Unomi, it is generally a good practice to <<Enable debug mode>>. +Doing so will display any errors when processing events (such as JSON Schema validations), and will provide useful indications towards solving issues. + +==== Data Model changes + +There has been changes to Unomi Data model, please make sure to review those in the << what_s_new>> section of the user manual. + +==== Create JSON schemas + +Once you updated your applications to align with Unomi 2 data model, the next step will be to create the necessary JSON Schemas. + +Any event (and more generally, any object) received through Unomi public endpoints do require a valid JSON schema. +Apache Unomi ships, out of the box, with all of the necessary JSON Schemas for its own operation but you will need to create schemas for any custom event. + +When creating your new schemas, reviewing debug messages in the logs (using: `log:set DEBUG org.apache.unomi.schema.impl.SchemaServiceImpl` in Karaf console), +will point to errors in your schemas. + +Note that it is currently not possible to modify or surcharge an existing JSON schema via the REST API. +If you are using custom properties on an event type for which an Apache Unomi schema already exists, +you will need to either change its eventType and create the corresponding Schema or to create a Unomi extension. + +You can use as a source of inspiration for creating new schemas, Apache Unomi 2.0 schema located at: + https://github.com/apache/unomi/tree/master/extensions/json-schema/services/src/main/resources/META-INF/cxs/schemas[extensions/json-schema/services/src/main/resources/META-INF/cxs/schemas]. + +Finally, and altuough it is technically feasible, we recommend against creating permissive JSON Schemas allowing any event payload. + +=== Migrating your existing data + +==== Elasticsearch version and capacity + +While still using Unomi 1.6, the first step will be to upgrade your Elasticsearch to 7.17.5. +Documentation is available on https://www.elastic.co/guide/en/elasticsearch/reference/7.17/setup-upgrade.html[Elasticsearch's website]. + +Your Elasticsearch cluster must have enough capacity to handle the migration. +At a minimum, the required capacity must be greater than the size of the dataset in production + the size of the largest index. + +==== Migrate custom data + +Apache Unomi 2.0 knows how to migrate its own data from the new model to the old one, but it does not know how to migrate custom events you might be using in your environment. + +It relies on a set of groovy scripts to perform its data migration, +located in https://github.com/apache/unomi/tree/master/tools/shell-commands/src/main/resources/META-INF/cxs/migration[tools/shell-commands/src/main/resources/META-INF/cxs/migration], +these scripts are sorted alphabetically and executed sequentially when migration is started. You can use these scripts as a source of inspiration for creating your own. + +In most cases, migration steps consist of an Elasticsearch painless script that will handle the data changes. + +At runtime, and when starting the migration, Unomi 2.0 will take its own scripts, any additional scripts located in `data/migration/scripts`, will sort the resulting list alphabetically and execute each migration script sequentially. + +==== Perform the migration + +===== Checklist + +Before starting the mirgation, please ensure that: Review Comment: ```suggestion Before starting the migration, please ensure that: ``` ########## manual/src/main/asciidoc/migrations/migrate-1.6-to-2.0.adoc: ########## @@ -0,0 +1,189 @@ +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// + +=== Migration Overview + +Apache Unomi 2.0 is a major release, and as such it does introduce breaking changes. This portion of the document detail the various steps we recommend following to successfully migrate your environment from Apache Unomi 1.6 to Apache Unomi 2.0. + +There are two main steps in preparing your migration to Apache Unomi 2.0: +- Updating applications consuming Unomi +- Migrating your existing data + +=== Updating applications consuming Unomi + +Since Apache Unomi is an engine, you've probably built multiple applications consuming its APIs, you might also have built extensions directly running in Unomi. + +As you begin updating applications consuming Apache Unomi, it is generally a good practice to <<Enable debug mode>>. +Doing so will display any errors when processing events (such as JSON Schema validations), and will provide useful indications towards solving issues. + +==== Data Model changes + +There has been changes to Unomi Data model, please make sure to review those in the << what_s_new>> section of the user manual. + +==== Create JSON schemas + +Once you updated your applications to align with Unomi 2 data model, the next step will be to create the necessary JSON Schemas. + +Any event (and more generally, any object) received through Unomi public endpoints do require a valid JSON schema. +Apache Unomi ships, out of the box, with all of the necessary JSON Schemas for its own operation but you will need to create schemas for any custom event. + +When creating your new schemas, reviewing debug messages in the logs (using: `log:set DEBUG org.apache.unomi.schema.impl.SchemaServiceImpl` in Karaf console), +will point to errors in your schemas. + +Note that it is currently not possible to modify or surcharge an existing JSON schema via the REST API. +If you are using custom properties on an event type for which an Apache Unomi schema already exists, +you will need to either change its eventType and create the corresponding Schema or to create a Unomi extension. + +You can use as a source of inspiration for creating new schemas, Apache Unomi 2.0 schema located at: Review Comment: ```suggestion You can use, as a source of inspiration for creating new schemas, Apache Unomi 2.0 schema located at: ``` ########## manual/src/main/asciidoc/recipes.adoc: ########## @@ -146,40 +187,150 @@ sending them to Apache Unomi whenever you like) of type 'contactInfoSubmitted' a course change any of the property names to find your needs. For example you might want to prefix the profile properties with the source of the event, such as 'mobileApp:firstName'. -You could then simply send the `contactInfoSubmitted` event using a request similar to this one: +Now that our rule is defined, the next step is to create a scope and a JSON Schema corresponding to the event to be submitted. Review Comment: ```suggestion Now that our rule is defined, the next step is to create a scope and a JSON Schema corresponding to the event to be submitted. ``` ########## manual/src/main/asciidoc/migrations/migrate-1.6-to-2.0.adoc: ########## @@ -0,0 +1,189 @@ +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// + +=== Migration Overview + +Apache Unomi 2.0 is a major release, and as such it does introduce breaking changes. This portion of the document detail the various steps we recommend following to successfully migrate your environment from Apache Unomi 1.6 to Apache Unomi 2.0. + +There are two main steps in preparing your migration to Apache Unomi 2.0: +- Updating applications consuming Unomi +- Migrating your existing data + +=== Updating applications consuming Unomi + +Since Apache Unomi is an engine, you've probably built multiple applications consuming its APIs, you might also have built extensions directly running in Unomi. + +As you begin updating applications consuming Apache Unomi, it is generally a good practice to <<Enable debug mode>>. +Doing so will display any errors when processing events (such as JSON Schema validations), and will provide useful indications towards solving issues. + +==== Data Model changes + +There has been changes to Unomi Data model, please make sure to review those in the << what_s_new>> section of the user manual. + +==== Create JSON schemas + +Once you updated your applications to align with Unomi 2 data model, the next step will be to create the necessary JSON Schemas. + +Any event (and more generally, any object) received through Unomi public endpoints do require a valid JSON schema. +Apache Unomi ships, out of the box, with all of the necessary JSON Schemas for its own operation but you will need to create schemas for any custom event. + +When creating your new schemas, reviewing debug messages in the logs (using: `log:set DEBUG org.apache.unomi.schema.impl.SchemaServiceImpl` in Karaf console), +will point to errors in your schemas. + +Note that it is currently not possible to modify or surcharge an existing JSON schema via the REST API. +If you are using custom properties on an event type for which an Apache Unomi schema already exists, +you will need to either change its eventType and create the corresponding Schema or to create a Unomi extension. + +You can use as a source of inspiration for creating new schemas, Apache Unomi 2.0 schema located at: + https://github.com/apache/unomi/tree/master/extensions/json-schema/services/src/main/resources/META-INF/cxs/schemas[extensions/json-schema/services/src/main/resources/META-INF/cxs/schemas]. + +Finally, and altuough it is technically feasible, we recommend against creating permissive JSON Schemas allowing any event payload. + +=== Migrating your existing data + +==== Elasticsearch version and capacity + +While still using Unomi 1.6, the first step will be to upgrade your Elasticsearch to 7.17.5. +Documentation is available on https://www.elastic.co/guide/en/elasticsearch/reference/7.17/setup-upgrade.html[Elasticsearch's website]. + +Your Elasticsearch cluster must have enough capacity to handle the migration. +At a minimum, the required capacity must be greater than the size of the dataset in production + the size of the largest index. + +==== Migrate custom data + +Apache Unomi 2.0 knows how to migrate its own data from the new model to the old one, but it does not know how to migrate custom events you might be using in your environment. + +It relies on a set of groovy scripts to perform its data migration, +located in https://github.com/apache/unomi/tree/master/tools/shell-commands/src/main/resources/META-INF/cxs/migration[tools/shell-commands/src/main/resources/META-INF/cxs/migration], +these scripts are sorted alphabetically and executed sequentially when migration is started. You can use these scripts as a source of inspiration for creating your own. + +In most cases, migration steps consist of an Elasticsearch painless script that will handle the data changes. + +At runtime, and when starting the migration, Unomi 2.0 will take its own scripts, any additional scripts located in `data/migration/scripts`, will sort the resulting list alphabetically and execute each migration script sequentially. + +==== Perform the migration + +===== Checklist + +Before starting the mirgation, please ensure that: + + - You do have a backup of your data + - You did practice the migration in a staging environment, NEVER migrate a production environment without prior validation + - You verified your applications were operational with Apache Unomi 2.0 (JSON schemas created, client applications updated, ...) + - You are running Elasticsearch 7.17.5 (or a later 7.x version) + - Your Elasticsearch cluster has enough capacity to handle the migration + - You are currently running Apache Unomi 1.6 (or a later 1.x version) + - You will be using the same Apache Unomi instance for the entire migration progress. Do not start the migration on one node, and resume an interrupted migration on another node. + +===== Migration process overview + +The migration is performed by means of a dedicated Apache Unomi 2.0 node started in a particular migration mode. + +In a nutshell, the migration process will consist in the following steps: + +- Shutdown your Apache Unomi 1.6 cluster +- Start an Apache Unomi 2.0 migration node +- Wait for data migration to complete +- Start you Apache Unomi 2.0 cluster +- (optional) Import additional JSON Schemas + +Each migration step maintains its execution state, meaning if a step fails you can fix the issue, and resume the migration from the failed step. + +===== Configuration + +The following environment variables are used for the migration: + +|=== +|Environment Variable|Unomi Setting|Default + +|UNOMI_ELASTICSEARCH_ADDRESSES +|org.apache.unomi.elasticsearch.addresses +|localhost:9200 + +|UNOMI_ELASTICSEARCH_SSL_ENABLE +|org.apache.unomi.elasticsearch.sslEnable +|false + +|UNOMI_ELASTICSEARCH_USERNAME +|org.apache.unomi.elasticsearch.username +| + +|UNOMI_ELASTICSEARCH_PASSWORD +|org.apache.unomi.elasticsearch.password +| + +|UNOMI_ELASTICSEARCH_SSL_TRUST_ALL_CERTIFICATES +|org.apache.unomi.elasticsearch.sslTrustAllCertificates +|false + +|UNOMI_ELASTICSEARCH_INDEXPREFIX +|org.apache.unomi.elasticsearch.index.prefix +|context + +|UNOMI_MIGRATION_RECOVER_FROM_HISTORY +|org.apache.unomi.migration.recoverFromHistory +|true + +|=== + +If there is a need for advanced configuratiion, the configuration file used by Apache Unomi 2.0 is located in: `etc/org.apache.unomi.migration.cfg` + +===== Migrate manually + +You can migrate manually using the Karaf console. + +After having started Apache Unomi 2.0 with the `./karaf` command, you will be presented with the Karaf shell. + +From there you have two options: + + - The necessary configuration variables (see above) are set, the migration can be started using: `unomi:migrate 1.6.0` + - Or, you want to provide the configuration settings via the terminal, in that case you can start the migration in interactive mode using: `unomi:migrate 1.6.0` Review Comment: ```suggestion - Or, you want to provide the configuration settings interactively via the terminal, in that case you can start the migration in interactive mode using: `unomi:migrate 1.6.0` ``` ########## manual/src/main/asciidoc/migrations/migrate-1.6-to-2.0.adoc: ########## @@ -0,0 +1,189 @@ +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// + +=== Migration Overview + +Apache Unomi 2.0 is a major release, and as such it does introduce breaking changes. This portion of the document detail the various steps we recommend following to successfully migrate your environment from Apache Unomi 1.6 to Apache Unomi 2.0. + +There are two main steps in preparing your migration to Apache Unomi 2.0: +- Updating applications consuming Unomi +- Migrating your existing data + +=== Updating applications consuming Unomi + +Since Apache Unomi is an engine, you've probably built multiple applications consuming its APIs, you might also have built extensions directly running in Unomi. + +As you begin updating applications consuming Apache Unomi, it is generally a good practice to <<Enable debug mode>>. +Doing so will display any errors when processing events (such as JSON Schema validations), and will provide useful indications towards solving issues. + +==== Data Model changes + +There has been changes to Unomi Data model, please make sure to review those in the << what_s_new>> section of the user manual. + +==== Create JSON schemas + +Once you updated your applications to align with Unomi 2 data model, the next step will be to create the necessary JSON Schemas. + +Any event (and more generally, any object) received through Unomi public endpoints do require a valid JSON schema. +Apache Unomi ships, out of the box, with all of the necessary JSON Schemas for its own operation but you will need to create schemas for any custom event. + +When creating your new schemas, reviewing debug messages in the logs (using: `log:set DEBUG org.apache.unomi.schema.impl.SchemaServiceImpl` in Karaf console), +will point to errors in your schemas. + +Note that it is currently not possible to modify or surcharge an existing JSON schema via the REST API. +If you are using custom properties on an event type for which an Apache Unomi schema already exists, +you will need to either change its eventType and create the corresponding Schema or to create a Unomi extension. + +You can use as a source of inspiration for creating new schemas, Apache Unomi 2.0 schema located at: + https://github.com/apache/unomi/tree/master/extensions/json-schema/services/src/main/resources/META-INF/cxs/schemas[extensions/json-schema/services/src/main/resources/META-INF/cxs/schemas]. + +Finally, and altuough it is technically feasible, we recommend against creating permissive JSON Schemas allowing any event payload. + +=== Migrating your existing data + +==== Elasticsearch version and capacity + +While still using Unomi 1.6, the first step will be to upgrade your Elasticsearch to 7.17.5. +Documentation is available on https://www.elastic.co/guide/en/elasticsearch/reference/7.17/setup-upgrade.html[Elasticsearch's website]. + +Your Elasticsearch cluster must have enough capacity to handle the migration. +At a minimum, the required capacity must be greater than the size of the dataset in production + the size of the largest index. + +==== Migrate custom data + +Apache Unomi 2.0 knows how to migrate its own data from the new model to the old one, but it does not know how to migrate custom events you might be using in your environment. + +It relies on a set of groovy scripts to perform its data migration, +located in https://github.com/apache/unomi/tree/master/tools/shell-commands/src/main/resources/META-INF/cxs/migration[tools/shell-commands/src/main/resources/META-INF/cxs/migration], +these scripts are sorted alphabetically and executed sequentially when migration is started. You can use these scripts as a source of inspiration for creating your own. + +In most cases, migration steps consist of an Elasticsearch painless script that will handle the data changes. + +At runtime, and when starting the migration, Unomi 2.0 will take its own scripts, any additional scripts located in `data/migration/scripts`, will sort the resulting list alphabetically and execute each migration script sequentially. + +==== Perform the migration + +===== Checklist + +Before starting the mirgation, please ensure that: + + - You do have a backup of your data + - You did practice the migration in a staging environment, NEVER migrate a production environment without prior validation + - You verified your applications were operational with Apache Unomi 2.0 (JSON schemas created, client applications updated, ...) + - You are running Elasticsearch 7.17.5 (or a later 7.x version) + - Your Elasticsearch cluster has enough capacity to handle the migration + - You are currently running Apache Unomi 1.6 (or a later 1.x version) + - You will be using the same Apache Unomi instance for the entire migration progress. Do not start the migration on one node, and resume an interrupted migration on another node. + +===== Migration process overview + +The migration is performed by means of a dedicated Apache Unomi 2.0 node started in a particular migration mode. + +In a nutshell, the migration process will consist in the following steps: + +- Shutdown your Apache Unomi 1.6 cluster +- Start an Apache Unomi 2.0 migration node +- Wait for data migration to complete +- Start you Apache Unomi 2.0 cluster +- (optional) Import additional JSON Schemas + +Each migration step maintains its execution state, meaning if a step fails you can fix the issue, and resume the migration from the failed step. + +===== Configuration + +The following environment variables are used for the migration: + +|=== +|Environment Variable|Unomi Setting|Default + +|UNOMI_ELASTICSEARCH_ADDRESSES +|org.apache.unomi.elasticsearch.addresses +|localhost:9200 + +|UNOMI_ELASTICSEARCH_SSL_ENABLE +|org.apache.unomi.elasticsearch.sslEnable +|false + +|UNOMI_ELASTICSEARCH_USERNAME +|org.apache.unomi.elasticsearch.username +| + +|UNOMI_ELASTICSEARCH_PASSWORD +|org.apache.unomi.elasticsearch.password +| + +|UNOMI_ELASTICSEARCH_SSL_TRUST_ALL_CERTIFICATES +|org.apache.unomi.elasticsearch.sslTrustAllCertificates +|false + +|UNOMI_ELASTICSEARCH_INDEXPREFIX +|org.apache.unomi.elasticsearch.index.prefix +|context + +|UNOMI_MIGRATION_RECOVER_FROM_HISTORY +|org.apache.unomi.migration.recoverFromHistory +|true + +|=== + +If there is a need for advanced configuratiion, the configuration file used by Apache Unomi 2.0 is located in: `etc/org.apache.unomi.migration.cfg` + +===== Migrate manually + +You can migrate manually using the Karaf console. + +After having started Apache Unomi 2.0 with the `./karaf` command, you will be presented with the Karaf shell. + +From there you have two options: + + - The necessary configuration variables (see above) are set, the migration can be started using: `unomi:migrate 1.6.0` + - Or, you want to provide the configuration settings via the terminal, in that case you can start the migration in interactive mode using: `unomi:migrate 1.6.0` + +The value of the migrate command (1.6.0 in the example above), corresponds to the version you're migrating from. + +At the end of the migration, you can start Unomi 2.0 as usual using: `unomi:start`. + +===== Migrate with Docker + +The migration can also be performed using Docker images, the migration itself can be started by passing a specific value to the `KARAF_OPTS` environment variable. + +In the context of this migration guide, we will asssume that: + - Custom migration scripts are located in `/home/unomi/migration/scripts/` + - Painless scripts, or more generally any migration assets are located in `/home/unomi/migration/assets/`, these scripts will be mounted under `/tmp/assets/` inside the Docker container. + +[source] +---- +docker run \ + -e UNOMI_ELASTICSEARCH_ADDRESSES=localhost:9200 \ + -e KARAF_OPTS="-Dunomi.autoMigrate=1.6.0" \ + --v /home/unomi/migration/scripts/:/opt/apache-unomi/data/migration/scripts \ + --v /home/unomi/migration/assets/:/tmp/assets/ \ + apache/unomi:2.0.0-SNAPSHOT +---- + +You might need to provide additional variables (see table above) depending of your environment. + +If the migration fail, you can simply restart this command. Review Comment: ```suggestion If the migration fails, you can simply restart this command. ``` ########## manual/src/main/asciidoc/whats-new.adoc: ########## @@ -0,0 +1,110 @@ +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +=== What's new in Apache Unomi 2.0 + +Apache Unomi 2 is a new release focused on improving core functionalities and robustness of the product. +The introduction of tighter data validation with JSON Schemas required some changes in the product data model, which in turn presented an opportunity for noticeable improvements of the overall performances. + +This new release also introduces a first (beta) version of Unomi GraphQL API. + +==== Scopes are required Review Comment: ```suggestion ==== Scopes declarations are now required ``` ########## manual/src/main/asciidoc/recipes.adoc: ########## @@ -146,40 +187,150 @@ sending them to Apache Unomi whenever you like) of type 'contactInfoSubmitted' a course change any of the property names to find your needs. For example you might want to prefix the profile properties with the source of the event, such as 'mobileApp:firstName'. -You could then simply send the `contactInfoSubmitted` event using a request similar to this one: +Now that our rule is defined, the next step is to create a scope and a JSON Schema corresponding to the event to be submitted. + +We will start by creating a scope called "example" scope: +[source] +---- +curl --location --request POST 'http://localhost:8181/cxs/scopes' \ +-u 'karaf:karaf' \ +--header 'Content-Type: application/json' \ +--data-raw '{ +"itemId": "example", +"itemType": "scope" +}' +---- + +The next step consist in creating a JSON Schema to validate our event. + +[source] +---- +curl --location --request POST 'http://localhost:8181/cxs/jsonSchema' \ +-u 'karaf:karaf' \ +--header 'Content-Type: application/json' \ +--data-raw '{ + "$id": "https://unomi.apache.org/schemas/json/events/contactInfoSubmitted/1-0-0";, + "$schema": "https://json-schema.org/draft/2019-09/schema";, + "self": { + "vendor": "org.apache.unomi", + "name": "contactInfoSubmitted", + "format": "jsonschema", + "target": "events", + "version": "1-0-0" + }, + "title": "contactInfoSubmittedEvent", + "type": "object", + "allOf": [{ "$ref": "https://unomi.apache.org/schemas/json/event/1-0-0"; }], + "properties": { + "source" : { + "$ref" : "https://unomi.apache.org/schemas/json/item/1-0-0"; + }, + "target" : { + "$ref" : "https://unomi.apache.org/schemas/json/item/1-0-0"; + }, + "properties": { + "type": "object", + "properties": { + "firstName": { + "type": ["null", "string"] + }, + "lastName": { + "type": ["null", "string"] + }, + "email": { + "type": ["null", "string"] + } + } + } + }, + "unevaluatedProperties": false +}' +---- + +You can notice the following in the above schema: + +* We are creating a schema of type "events" ("self.target" equals "events") +* The name of this schema is "contactInfoSubmitted", this MUST match the value of the "eventType" field in the event itself (below) +* To simplify our schema declaration, we're referring to an already existing schema (https://unomi.apache.org/schemas/json/item/1-0-0) to validate the "source" and "target" properties. Apache Unomi ships with a set of predefined JSON Schemas, detailed here: https://github.com/apache/unomi/tree/master/extensions/json-schema/services/src/main/resources/META-INF/cxs/schemas. +* `"unevaluatedProperties": false` indicates that the event should be rejected if it contains any additional metadata. + +Finally, send the `contactInfoSubmitted` event using a request similar to this one: [source] ---- curl -X POST http://localhost:8181/cxs/eventcollector \ -H "Content-Type: application/json" \ --d @- <<'EOF' -{ +--data-raw '{ "sessionId" : "1234", "events":[ { "eventType":"contactInfoSubmitted", "scope": "example", "source":{ "itemType": "site", - "scope":"example", + "scope": "example", "itemId": "mysite" }, "target":{ - "itemType":"form", - "scope":"example", - "itemId":"contactForm" + "itemType": "form", + "scope": "example", + "itemId": "contactForm" }, "properties" : { - "firstName" : "John", - "lastName" : "Doe", - "email" : "[email protected]" + "firstName": "John", + "lastName": "Doe", + "email": "[email protected]" } } ] -} -EOF +}' ---- +The event we just submitted can be retrieved using the following request: + +[source] +---- +curl -X POST http://localhost:8181/cxs/events/search \ +--user karaf:karaf \ +-H "Content-Type: application/json" \ +--data-raw '{ + "offset" : 0, + "limit" : 20, + "condition" : { + "type": "eventPropertyCondition", + "parameterValues" : { + "propertyName" : "properties.firstName", + "comparisonOperator" : "equals", + "propertyValue" : "John" + } + } +}' +---- + +===== Debug common errors + +There could be two types of common errors while customizing the above requests: +* The schema is invalid +* The event is invalid + +While first submitting the schema during its creation, Apache Unomi will validate it is syntaxically correct (JSON) +but will not perform any further validation. Since the schema will be process for the first time when events are submitted, Review Comment: ```suggestion but will not perform any further validation. Since the schema will be processed for the first time when events are submitted, ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
