Modified: unomi/website/manual/latest/index.html URL: http://svn.apache.org/viewvc/unomi/website/manual/latest/index.html?rev=1903789&r1=1903788&r2=1903789&view=diff ============================================================================== --- unomi/website/manual/latest/index.html (original) +++ unomi/website/manual/latest/index.html Wed Aug 31 11:00:23 2022 @@ -6,421 +6,462 @@ <meta name="viewport" content="width=device-width, initial-scale=1.0"> <meta name="generator" content="Asciidoctor 1.5.8"> <meta name="author" content="Apache Software Foundation"> -<title>Apache Unomi 1.x - Documentation</title> +<title>Apache Unomi 2.x - Documentation</title> <link rel="stylesheet" href="./apache.css"> </head> <body class="article toc2 toc-left"> <div id="header"> -<h1>Apache Unomi 1.x - Documentation</h1> +<h1>Apache Unomi 2.x - Documentation</h1> <div class="details"> <span id="author" class="author">Apache Software Foundation</span><br> </div> <div id="toc" class="toc2"> <div id="toctitle">Table of Contents</div> <ul class="sectlevel1"> -<li><a href="#_quick_start">1. Quick start</a> +<li><a href="#_whats_new">1. What’s new</a> <ul class="sectlevel2"> -<li><a href="#_five_minutes_quickstart">1.1. Five Minutes QuickStart</a></li> +<li><a href="#_whats_new_in_apache_unomi_2_0">1.1. What’s new in Apache Unomi 2.0</a> +<ul class="sectlevel3"> +<li><a href="#_scopes_declarations_are_now_required">1.1.1. Scopes declarations are now required</a></li> +<li><a href="#_json_schemas">1.1.2. JSON Schemas</a></li> +<li><a href="#_updated_data_model">1.1.3. Updated data model</a></li> +<li><a href="#_removal_of_the_web_tracker">1.1.4. Removal of the Web Tracker</a></li> +<li><a href="#_graphql_api_beta">GraphQL API (beta)</a></li> +<li><a href="#_migrate_from_unomi_1_x">1.1.5. Migrate from Unomi 1.x</a></li> +<li><a href="#_elasticsearch_compatibility">1.1.6. Elasticsearch compatibility</a></li> +</ul> +</li> +</ul> +</li> +<li><a href="#_quick_start">2. Quick start</a> +<ul class="sectlevel2"> +<li><a href="#_quick_start_with_docker">2.1. Quick start with Docker</a></li> +<li><a href="#_quick_start_manually">2.2. Quick Start manually</a></li> </ul> </li> -<li><a href="#_first_steps_with_apache_unomi">2. First steps with Apache Unomi</a> +<li><a href="#_first_steps_with_apache_unomi">3. First steps with Apache Unomi</a> <ul class="sectlevel2"> -<li><a href="#_getting_started_with_unomi">2.1. Getting started with Unomi</a> +<li><a href="#_getting_started_with_unomi">3.1. Getting started with Unomi</a> <ul class="sectlevel3"> -<li><a href="#_prerequisites">2.1.1. Prerequisites</a></li> -<li><a href="#_running_unomi">2.1.2. Running Unomi</a></li> +<li><a href="#_prerequisites">3.1.1. Prerequisites</a></li> +<li><a href="#_running_unomi">3.1.2. Running Unomi</a></li> </ul> </li> -<li><a href="#_recipes">2.2. Recipes</a> +<li><a href="#_recipes">3.2. Recipes</a> <ul class="sectlevel3"> -<li><a href="#_introduction">2.2.1. Introduction</a></li> -<li><a href="#_how_to_read_a_profile">2.2.2. How to read a profile</a></li> -<li><a href="#_how_to_update_a_profile_from_the_public_internet">2.2.3. How to update a profile from the public internet</a></li> -<li><a href="#_how_to_search_for_profile_events">2.2.4. How to search for profile events</a></li> -<li><a href="#_how_to_create_a_new_rule">2.2.5. How to create a new rule</a></li> -<li><a href="#_how_to_search_for_profiles">2.2.6. How to search for profiles</a></li> -<li><a href="#_getting_updating_consents">2.2.7. Getting / updating consents</a></li> -<li><a href="#_how_to_send_a_login_event_to_unomi">2.2.8. How to send a login event to Unomi</a></li> +<li><a href="#_introduction">3.2.1. Introduction</a></li> +<li><a href="#_enabling_debug_mode">3.2.2. Enabling debug mode</a></li> +<li><a href="#_how_to_read_a_profile">3.2.3. How to read a profile</a></li> +<li><a href="#_how_to_update_a_profile_from_the_public_internet">3.2.4. How to update a profile from the public internet</a></li> +<li><a href="#_how_to_search_for_profile_events">3.2.5. How to search for profile events</a></li> +<li><a href="#_how_to_create_a_new_rule">3.2.6. How to create a new rule</a></li> +<li><a href="#_how_to_search_for_profiles">3.2.7. How to search for profiles</a></li> +<li><a href="#_getting_updating_consents">3.2.8. Getting / updating consents</a></li> +<li><a href="#_how_to_send_a_login_event_to_unomi">3.2.9. How to send a login event to Unomi</a></li> </ul> </li> -<li><a href="#_request_examples">2.3. Request examples</a> +<li><a href="#_request_examples">3.3. Request examples</a> <ul class="sectlevel3"> -<li><a href="#_retrieving_your_first_context">2.3.1. Retrieving your first context</a></li> -<li><a href="#_retrieving_a_context_as_a_json_object">2.3.2. Retrieving a context as a JSON object.</a></li> -<li><a href="#_accessing_profile_properties_in_a_context">2.3.3. Accessing profile properties in a context</a></li> -<li><a href="#_sending_events_using_the_context_servlet">2.3.4. Sending events using the context servlet</a></li> -<li><a href="#_sending_events_using_the_eventcollector_servlet">2.3.5. Sending events using the eventcollector servlet</a></li> -<li><a href="#_where_to_go_from_here">2.3.6. Where to go from here</a></li> +<li><a href="#_retrieving_your_first_context">3.3.1. Retrieving your first context</a></li> +<li><a href="#_retrieving_a_context_as_a_json_object">3.3.2. Retrieving a context as a JSON object.</a></li> +<li><a href="#_accessing_profile_properties_in_a_context">3.3.3. Accessing profile properties in a context</a></li> +<li><a href="#_sending_events_using_the_context_servlet">3.3.4. Sending events using the context servlet</a></li> +<li><a href="#_sending_events_using_the_eventcollector_servlet">3.3.5. Sending events using the eventcollector servlet</a></li> +<li><a href="#_where_to_go_from_here">3.3.6. Where to go from here</a></li> </ul> </li> </ul> </li> -<li><a href="#_configuration">3. Configuration</a> +<li><a href="#_configuration">4. Configuration</a> <ul class="sectlevel2"> -<li><a href="#_centralized_configuration">3.1. Centralized configuration</a></li> -<li><a href="#_changing_the_default_configuration_using_environment_variables_i_e_docker_configuration">3.2. Changing the default configuration using environment variables (i.e. Docker configuration)</a></li> -<li><a href="#_changing_the_default_configuration_using_property_files">3.3. Changing the default configuration using property files</a></li> -<li><a href="#_secured_events_configuration">3.4. Secured events configuration</a></li> -<li><a href="#_installing_the_maxmind_geoiplite2_ip_lookup_database">3.5. Installing the MaxMind GeoIPLite2 IP lookup database</a></li> -<li><a href="#_installing_geonames_database">3.6. Installing Geonames database</a></li> -<li><a href="#_rest_api_security">3.7. REST API Security</a></li> -<li><a href="#_scripting_security">3.8. Scripting security</a> +<li><a href="#_centralized_configuration">4.1. Centralized configuration</a></li> +<li><a href="#_changing_the_default_configuration_using_environment_variables_i_e_docker_configuration">4.2. Changing the default configuration using environment variables (i.e. Docker configuration)</a></li> +<li><a href="#_changing_the_default_configuration_using_property_files">4.3. Changing the default configuration using property files</a></li> +<li><a href="#_secured_events_configuration">4.4. Secured events configuration</a></li> +<li><a href="#_installing_the_maxmind_geoiplite2_ip_lookup_database">4.5. Installing the MaxMind GeoIPLite2 IP lookup database</a></li> +<li><a href="#_installing_geonames_database">4.6. Installing Geonames database</a></li> +<li><a href="#_rest_api_security">4.7. REST API Security</a></li> +<li><a href="#_scripting_security">4.8. Scripting security</a> <ul class="sectlevel3"> -<li><a href="#_multi_layer_scripting_filtering_system">3.8.1. Multi-layer scripting filtering system</a></li> -<li><a href="#_scripts_and_expressions">3.8.2. Scripts and expressions</a></li> -<li><a href="#_scripting_expression_filtering_configuration_parameters">3.8.3. Scripting expression filtering configuration parameters</a></li> -<li><a href="#_groovy_actions">3.8.4. Groovy Actions</a></li> -<li><a href="#_scripting_roadmap">3.8.5. Scripting roadmap</a></li> +<li><a href="#_multi_layer_scripting_filtering_system">4.8.1. Multi-layer scripting filtering system</a></li> +<li><a href="#_scripts_and_expressions">4.8.2. Scripts and expressions</a></li> +<li><a href="#_scripting_expression_filtering_configuration_parameters">4.8.3. Scripting expression filtering configuration parameters</a></li> +<li><a href="#_groovy_actions">4.8.4. Groovy Actions</a></li> +<li><a href="#_scripting_roadmap">4.8.5. Scripting roadmap</a></li> </ul> </li> -<li><a href="#_automatic_profile_merging">3.9. Automatic profile merging</a></li> -<li><a href="#_securing_a_production_environment">3.10. Securing a production environment</a></li> -<li><a href="#_integrating_with_an_apache_http_web_server">3.11. Integrating with an Apache HTTP web server</a></li> -<li><a href="#_changing_the_default_tracking_location">3.12. Changing the default tracking location</a></li> -<li><a href="#_apache_karaf_ssh_console">3.13. Apache Karaf SSH Console</a></li> -<li><a href="#_elasticsearch_authentication_and_security">3.14. ElasticSearch authentication and security</a> +<li><a href="#_automatic_profile_merging">4.9. Automatic profile merging</a></li> +<li><a href="#_securing_a_production_environment">4.10. Securing a production environment</a></li> +<li><a href="#_integrating_with_an_apache_http_web_server">4.11. Integrating with an Apache HTTP web server</a></li> +<li><a href="#_changing_the_default_tracking_location">4.12. Changing the default tracking location</a></li> +<li><a href="#_apache_karaf_ssh_console">4.13. Apache Karaf SSH Console</a></li> +<li><a href="#_elasticsearch_authentication_and_security">4.14. ElasticSearch authentication and security</a> <ul class="sectlevel3"> -<li><a href="#_user_authentication">3.14.1. User authentication !</a></li> -<li><a href="#_ssl_communication">3.14.2. SSL communication</a></li> +<li><a href="#_user_authentication">4.14.1. User authentication !</a></li> +<li><a href="#_ssl_communication">4.14.2. SSL communication</a></li> +<li><a href="#_permissions">4.14.3. Permissions</a></li> </ul> </li> </ul> </li> -<li><a href="#_json_schemas">4. JSON schemas</a> +<li><a href="#_json_schemas_2">5. JSON schemas</a> <ul class="sectlevel2"> -<li><a href="#_introduction_2">4.1. Introduction</a> +<li><a href="#_introduction_2">5.1. Introduction</a> +<ul class="sectlevel3"> +<li><a href="#_what_is_a_json_schema">5.1.1. What is a JSON Schema</a></li> +<li><a href="#_key_concepts">5.1.2. Key concepts</a></li> +<li><a href="#_how_are_json_schema_used_in_unomi">5.1.3. How are JSON Schema used in Unomi</a></li> +</ul> +</li> +<li><a href="#_json_schema_api">5.2. JSON schema API</a> <ul class="sectlevel3"> -<li><a href="#_what_is_a_json_schema">4.1.1. What is a JSON Schema</a></li> -<li><a href="#_key_concepts">4.1.2. Key concepts</a></li> -<li><a href="#_how_are_json_schema_used_in_unomi">4.1.3. How are JSON Schema used in Unomi</a></li> +<li><a href="#_list_existing_schemas">5.2.1. List existing schemas</a></li> +<li><a href="#_read_a_schema">5.2.2. Read a schema</a></li> +<li><a href="#_create_update_a_json_schema_to_validate_an_event">5.2.3. Create / update a JSON schema to validate an event</a></li> +<li><a href="#_deleting_a_schema">5.2.4. Deleting a schema</a></li> +<li><a href="#_error_management">5.2.5. Error Management</a></li> +<li><a href="#_details_on_invalid_events">5.2.6. Details on invalid events</a></li> </ul> </li> -<li><a href="#_json_schema_api">4.2. JSON schema API</a> +<li><a href="#_extend_an_existing_schema">5.3. Extend an existing schema</a> <ul class="sectlevel3"> -<li><a href="#_list_existing_schemas">4.2.1. List existing schemas</a></li> -<li><a href="#_read_a_schema">4.2.2. Read a schema</a></li> -<li><a href="#_create_update_a_json_schema_to_validate_an_event">4.2.3. Create / update a JSON schema to validate an event</a></li> -<li><a href="#_deleting_a_schema">4.2.4. Deleting a schema</a></li> -<li><a href="#_error_management">4.2.5. Error Management</a></li> -<li><a href="#_details_on_invalid_events">4.2.6. Details on invalid events</a></li> +<li><a href="#_when_a_extension_is_needed">5.3.1. When a extension is needed?</a></li> +<li><a href="#_understanding_how_extensions_are_merged_in_unomi">5.3.2. Understanding how extensions are merged in unomi</a></li> +<li><a href="#_how_to_add_an_extension_through_the_api">5.3.3. How to add an extension through the API</a></li> </ul> </li> -<li><a href="#_extend_an_existing_schema">4.3. Extend an existing schema</a> +<li><a href="#_graphql_api">5.4. GraphQL API</a> <ul class="sectlevel3"> -<li><a href="#_when_a_extension_is_needed">4.3.1. When a extension is needed?</a></li> -<li><a href="#_understanding_how_extensions_are_merged_in_unomi">4.3.2. Understanding how extensions are merged in unomi</a></li> -<li><a href="#_how_to_add_an_extension_through_the_api">4.3.3. How to add an extension through the API</a></li> +<li><a href="#_enabling_the_api">5.4.1. Enabling/ the API</a></li> +<li><a href="#_endpoints">5.4.2. Endpoints</a></li> +<li><a href="#_graphql_schema">5.4.3. GraphQL Schema</a></li> </ul> </li> </ul> </li> -<li><a href="#_migrations">5. Migrations</a> +<li><a href="#_migrations">6. Migrations</a> <ul class="sectlevel2"> -<li><a href="#_from_version_1_4_to_1_5">5.1. From version 1.4 to 1.5</a> +<li><a href="#_from_version_1_6_to_2_0">6.1. From version 1.6 to 2.0</a></li> +<li><a href="#_migration_overview">6.2. Migration Overview</a></li> +<li><a href="#_updating_applications_consuming_unomi">6.3. Updating applications consuming Unomi</a> +<ul class="sectlevel3"> +<li><a href="#_data_model_changes">6.3.1. Data Model changes</a></li> +<li><a href="#_create_json_schemas">6.3.2. Create JSON schemas</a></li> +</ul> +</li> +<li><a href="#_migrating_your_existing_data">6.4. Migrating your existing data</a> <ul class="sectlevel3"> -<li><a href="#_data_model_and_elasticsearch_7">5.1.1. Data model and ElasticSearch 7</a></li> -<li><a href="#_api_changes">5.1.2. API changes</a></li> -<li><a href="#_migration_steps">5.1.3. Migration steps</a></li> +<li><a href="#_elasticsearch_version_and_capacity">6.4.1. Elasticsearch version and capacity</a></li> +<li><a href="#_migrate_custom_data">6.4.2. Migrate custom data</a></li> +<li><a href="#_perform_the_migration">6.4.3. Perform the migration</a></li> </ul> </li> -<li><a href="#_important_changes_in_public_servlets_since_version_1_5_5_and_2_0_0">5.2. Important changes in public servlets since version 1.5.5 and 2.0.0</a></li> +<li><a href="#_from_version_1_5_to_1_6">6.5. From version 1.5 to 1.6</a></li> +<li><a href="#_from_version_1_4_to_1_5">6.6. From version 1.4 to 1.5</a> +<ul class="sectlevel3"> +<li><a href="#_data_model_and_elasticsearch_7">6.6.1. Data model and ElasticSearch 7</a></li> +<li><a href="#_api_changes">6.6.2. API changes</a></li> +<li><a href="#_migration_steps">6.6.3. Migration steps</a></li> +</ul> +</li> +<li><a href="#_important_changes_in_public_servlets_since_version_1_5_5_and_2_0_0">6.7. Important changes in public servlets since version 1.5.5 and 2.0.0</a></li> </ul> </li> -<li><a href="#_queries_and_aggregations">6. Queries and aggregations</a> +<li><a href="#_queries_and_aggregations">7. Queries and aggregations</a> <ul class="sectlevel2"> -<li><a href="#_query_counts">6.1. Query counts</a></li> -<li><a href="#_metrics">6.2. Metrics</a></li> -<li><a href="#_aggregations">6.3. Aggregations</a> +<li><a href="#_query_counts">7.1. Query counts</a></li> +<li><a href="#_metrics">7.2. Metrics</a></li> +<li><a href="#_aggregations">7.3. Aggregations</a> <ul class="sectlevel3"> -<li><a href="#_aggregation_types">6.3.1. Aggregation types</a></li> +<li><a href="#_aggregation_types">7.3.1. Aggregation types</a></li> </ul> </li> </ul> </li> -<li><a href="#_profile_import_export">7. Profile import & export</a> +<li><a href="#_profile_import_export">8. Profile import & export</a> <ul class="sectlevel2"> -<li><a href="#_importing_profiles">7.1. Importing profiles</a> +<li><a href="#_importing_profiles">8.1. Importing profiles</a> <ul class="sectlevel3"> -<li><a href="#_import_api">7.1.1. Import API</a></li> +<li><a href="#_import_api">8.1.1. Import API</a></li> </ul> </li> -<li><a href="#_exporting_profiles">7.2. Exporting profiles</a> +<li><a href="#_exporting_profiles">8.2. Exporting profiles</a> <ul class="sectlevel3"> -<li><a href="#_export_api">7.2.1. Export API</a></li> +<li><a href="#_export_api">8.2.1. Export API</a></li> </ul> </li> -<li><a href="#_configuration_in_details">7.3. Configuration in details</a></li> +<li><a href="#_configuration_in_details">8.3. Configuration in details</a></li> </ul> </li> -<li><a href="#_consent_management">8. Consent management</a> +<li><a href="#_consent_management">9. Consent management</a> <ul class="sectlevel2"> -<li><a href="#_consent_api">8.1. Consent API</a> +<li><a href="#_consent_api">9.1. Consent API</a> <ul class="sectlevel3"> -<li><a href="#_profiles_with_consents">8.1.1. Profiles with consents</a></li> -<li><a href="#_consent_type_definitions">8.1.2. Consent type definitions</a></li> -<li><a href="#_creating_update_a_visitor_consent">8.1.3. Creating / update a visitor consent</a></li> -<li><a href="#_how_it_works_internally">8.1.4. How it works (internally)</a></li> +<li><a href="#_profiles_with_consents">9.1.1. Profiles with consents</a></li> +<li><a href="#_consent_type_definitions">9.1.2. Consent type definitions</a></li> +<li><a href="#_creating_update_a_visitor_consent">9.1.3. Creating / update a visitor consent</a></li> +<li><a href="#_how_it_works_internally">9.1.4. How it works (internally)</a></li> </ul> </li> </ul> </li> -<li><a href="#_privacy_management">9. Privacy management</a> +<li><a href="#_privacy_management">10. Privacy management</a> <ul class="sectlevel2"> -<li><a href="#_setting_up_access_to_the_privacy_endpoint">9.1. Setting up access to the privacy endpoint</a></li> -<li><a href="#_anonymizing_a_profile">9.2. Anonymizing a profile</a></li> -<li><a href="#_downloading_profile_data">9.3. Downloading profile data</a></li> -<li><a href="#_deleting_a_profile">9.4. Deleting a profile</a></li> -<li><a href="#_related">9.5. Related</a></li> +<li><a href="#_setting_up_access_to_the_privacy_endpoint">10.1. Setting up access to the privacy endpoint</a></li> +<li><a href="#_anonymizing_a_profile">10.2. Anonymizing a profile</a></li> +<li><a href="#_downloading_profile_data">10.3. Downloading profile data</a></li> +<li><a href="#_deleting_a_profile">10.4. Deleting a profile</a></li> +<li><a href="#_related">10.5. Related</a></li> </ul> </li> -<li><a href="#_cluster_setup">10. Cluster setup</a> +<li><a href="#_cluster_setup">11. Cluster setup</a> <ul class="sectlevel2"> -<li><a href="#_cluster_setup_2">10.1. Cluster setup</a></li> +<li><a href="#_cluster_setup_2">11.1. Cluster setup</a></li> </ul> </li> -<li><a href="#_reference">11. Reference</a> +<li><a href="#_reference">12. Reference</a> <ul class="sectlevel2"> -<li><a href="#_useful_apache_unomi_urls">11.1. Useful Apache Unomi URLs</a></li> -<li><a href="#_how_profile_tracking_works">11.2. How profile tracking works</a> +<li><a href="#_useful_apache_unomi_urls">12.1. Useful Apache Unomi URLs</a></li> +<li><a href="#_how_profile_tracking_works">12.2. How profile tracking works</a> <ul class="sectlevel3"> -<li><a href="#_steps">11.2.1. Steps</a></li> +<li><a href="#_steps">12.2.1. Steps</a></li> </ul> </li> -<li><a href="#_context_request_flow">11.3. Context Request Flow</a></li> -<li><a href="#_data_model_overview">11.4. Data Model Overview</a></li> -<li><a href="#_scope">11.5. Scope</a> +<li><a href="#_context_request_flow">12.3. Context Request Flow</a></li> +<li><a href="#_data_model_overview">12.4. Data Model Overview</a></li> +<li><a href="#_scope">12.5. Scope</a> <ul class="sectlevel3"> -<li><a href="#_example">11.5.1. Example</a></li> +<li><a href="#_example">12.5.1. Example</a></li> </ul> </li> -<li><a href="#_item">11.6. Item</a> +<li><a href="#_item">12.6. Item</a> <ul class="sectlevel3"> -<li><a href="#_structure_definition">11.6.1. Structure definition</a></li> +<li><a href="#_structure_definition">12.6.1. Structure definition</a></li> </ul> </li> -<li><a href="#_metadata">11.7. Metadata</a> +<li><a href="#_metadata">12.7. Metadata</a> <ul class="sectlevel3"> -<li><a href="#_structure_definition_2">11.7.1. Structure definition</a></li> -<li><a href="#_example_2">11.7.2. Example</a></li> +<li><a href="#_structure_definition_2">12.7.1. Structure definition</a></li> +<li><a href="#_example_2">12.7.2. Example</a></li> </ul> </li> -<li><a href="#_metadataitem">11.8. MetadataItem</a> +<li><a href="#_metadataitem">12.8. MetadataItem</a> <ul class="sectlevel3"> -<li><a href="#_structure_definition_3">11.8.1. Structure definition</a></li> -<li><a href="#_example_3">11.8.2. Example</a></li> +<li><a href="#_structure_definition_3">12.8.1. Structure definition</a></li> +<li><a href="#_example_3">12.8.2. Example</a></li> </ul> </li> -<li><a href="#_event">11.9. Event</a> +<li><a href="#_event">12.9. Event</a> <ul class="sectlevel3"> -<li><a href="#_fields">11.9.1. Fields</a></li> -<li><a href="#_event_types">11.9.2. Event types</a></li> +<li><a href="#_fields">12.9.1. Fields</a></li> +<li><a href="#_event_types">12.9.2. Event types</a></li> </ul> </li> -<li><a href="#_profile">11.10. Profile</a> +<li><a href="#_profile">12.10. Profile</a> <ul class="sectlevel3"> -<li><a href="#_structure_definition_4">11.10.1. Structure definition</a></li> -<li><a href="#_example_4">11.10.2. Example</a></li> +<li><a href="#_structure_definition_4">12.10.1. Structure definition</a></li> +<li><a href="#_example_4">12.10.2. Example</a></li> </ul> </li> -<li><a href="#_persona">11.11. Persona</a> +<li><a href="#_persona">12.11. Persona</a> <ul class="sectlevel3"> -<li><a href="#_structure_definition_5">11.11.1. Structure definition</a></li> -<li><a href="#_example_5">11.11.2. Example</a></li> +<li><a href="#_structure_definition_5">12.11.1. Structure definition</a></li> +<li><a href="#_example_5">12.11.2. Example</a></li> </ul> </li> -<li><a href="#_consent">11.12. Consent</a> +<li><a href="#_consent">12.12. Consent</a> <ul class="sectlevel3"> -<li><a href="#_structure_definition_6">11.12.1. Structure definition</a></li> -<li><a href="#_example_6">11.12.2. Example</a></li> +<li><a href="#_structure_definition_6">12.12.1. Structure definition</a></li> +<li><a href="#_example_6">12.12.2. Example</a></li> </ul> </li> -<li><a href="#_session">11.13. Session</a> +<li><a href="#_session">12.13. Session</a> <ul class="sectlevel3"> -<li><a href="#_structure_definition_7">11.13.1. Structure definition</a></li> -<li><a href="#_example_7">11.13.2. Example</a></li> +<li><a href="#_structure_definition_7">12.13.1. Structure definition</a></li> +<li><a href="#_example_7">12.13.2. Example</a></li> </ul> </li> -<li><a href="#_segment">11.14. Segment</a> +<li><a href="#_segment">12.14. Segment</a> <ul class="sectlevel3"> -<li><a href="#_structure_definition_8">11.14.1. Structure definition</a></li> -<li><a href="#_example_8">11.14.2. Example</a></li> +<li><a href="#_structure_definition_8">12.14.1. Structure definition</a></li> +<li><a href="#_example_8">12.14.2. Example</a></li> </ul> </li> -<li><a href="#_condition">11.15. Condition</a> +<li><a href="#_condition">12.15. Condition</a> <ul class="sectlevel3"> -<li><a href="#_structure_definition_9">11.15.1. Structure definition</a></li> -<li><a href="#_example_9">11.15.2. Example</a></li> +<li><a href="#_structure_definition_9">12.15.1. Structure definition</a></li> +<li><a href="#_example_9">12.15.2. Example</a></li> </ul> </li> -<li><a href="#_rule">11.16. Rule</a> +<li><a href="#_rule">12.16. Rule</a> <ul class="sectlevel3"> -<li><a href="#_structure_definition_10">11.16.1. Structure definition</a></li> -<li><a href="#_example_10">11.16.2. Example</a></li> +<li><a href="#_structure_definition_10">12.16.1. Structure definition</a></li> +<li><a href="#_example_10">12.16.2. Example</a></li> </ul> </li> -<li><a href="#_action">11.17. Action</a> +<li><a href="#_action">12.17. Action</a> <ul class="sectlevel3"> -<li><a href="#_structure_definition_11">11.17.1. Structure definition</a></li> -<li><a href="#_example_11">11.17.2. Example</a></li> +<li><a href="#_structure_definition_11">12.17.1. Structure definition</a></li> +<li><a href="#_example_11">12.17.2. Example</a></li> </ul> </li> -<li><a href="#_list">11.18. List</a> +<li><a href="#_list">12.18. List</a> <ul class="sectlevel3"> -<li><a href="#_structure_definition_12">11.18.1. Structure definition</a></li> -<li><a href="#_example_12">11.18.2. Example</a></li> +<li><a href="#_structure_definition_12">12.18.1. Structure definition</a></li> +<li><a href="#_example_12">12.18.2. Example</a></li> </ul> </li> -<li><a href="#_goal">11.19. Goal</a> +<li><a href="#_goal">12.19. Goal</a> <ul class="sectlevel3"> -<li><a href="#_structure_definition_13">11.19.1. Structure definition</a></li> -<li><a href="#_example_13">11.19.2. Example</a></li> +<li><a href="#_structure_definition_13">12.19.1. Structure definition</a></li> +<li><a href="#_example_13">12.19.2. Example</a></li> </ul> </li> -<li><a href="#_campaign">11.20. Campaign</a> +<li><a href="#_campaign">12.20. Campaign</a> <ul class="sectlevel3"> -<li><a href="#_structure_definition_14">11.20.1. Structure definition</a></li> -<li><a href="#_example_14">11.20.2. Example</a></li> +<li><a href="#_structure_definition_14">12.20.1. Structure definition</a></li> +<li><a href="#_example_14">12.20.2. Example</a></li> </ul> </li> -<li><a href="#_scoring_plan">11.21. Scoring plan</a> +<li><a href="#_scoring_plan">12.21. Scoring plan</a> <ul class="sectlevel3"> -<li><a href="#_structure_definition_15">11.21.1. Structure definition</a></li> -<li><a href="#_example_15">11.21.2. Example</a></li> +<li><a href="#_structure_definition_15">12.21.1. Structure definition</a></li> +<li><a href="#_example_15">12.21.2. Example</a></li> </ul> </li> -<li><a href="#_built_in_event_types">11.22. Built-in Event types</a> +<li><a href="#_built_in_event_types">12.22. Built-in Event types</a> <ul class="sectlevel3"> -<li><a href="#_login_event_type">11.22.1. Login event type</a></li> -<li><a href="#_view_event_type">11.22.2. View event type</a></li> -<li><a href="#_form_event_type">11.22.3. Form event type</a></li> -<li><a href="#_update_properties_event_type">11.22.4. Update properties event type</a></li> -<li><a href="#_identify_event_type">11.22.5. Identify event type</a></li> -<li><a href="#_session_created_event_type">11.22.6. Session created event type</a></li> -<li><a href="#_goal_event_type">11.22.7. Goal event type</a></li> -<li><a href="#_modify_consent_event_type">11.22.8. Modify consent event type</a></li> +<li><a href="#_login_event_type">12.22.1. Login event type</a></li> +<li><a href="#_view_event_type">12.22.2. View event type</a></li> +<li><a href="#_form_event_type">12.22.3. Form event type</a></li> +<li><a href="#_update_properties_event_type">12.22.4. Update properties event type</a></li> +<li><a href="#_identify_event_type">12.22.5. Identify event type</a></li> +<li><a href="#_session_created_event_type">12.22.6. Session created event type</a></li> +<li><a href="#_goal_event_type">12.22.7. Goal event type</a></li> +<li><a href="#_modify_consent_event_type">12.22.8. Modify consent event type</a></li> </ul> </li> -<li><a href="#_built_in_condition_types">11.23. Built-in condition types</a> +<li><a href="#_built_in_condition_types">12.23. Built-in condition types</a> <ul class="sectlevel3"> -<li><a href="#_existing_condition_type_descriptors">11.23.1. Existing condition type descriptors</a></li> +<li><a href="#_existing_condition_type_descriptors">12.23.1. Existing condition type descriptors</a></li> </ul> </li> -<li><a href="#_built_in_action_types">11.24. Built-in action types</a> +<li><a href="#_built_in_action_types">12.24. Built-in action types</a> <ul class="sectlevel3"> -<li><a href="#_existing_action_types_descriptors">11.24.1. Existing action types descriptors</a></li> +<li><a href="#_existing_action_types_descriptors">12.24.1. Existing action types descriptors</a></li> </ul> </li> -<li><a href="#_updating_events_using_the_context_servlet">11.25. Updating Events Using the Context Servlet</a> +<li><a href="#_updating_events_using_the_context_servlet">12.25. Updating Events Using the Context Servlet</a> <ul class="sectlevel3"> -<li><a href="#_solution">11.25.1. Solution</a></li> -<li><a href="#_defining_rules">11.25.2. Defining Rules</a></li> +<li><a href="#_solution">12.25.1. Solution</a></li> +<li><a href="#_defining_rules">12.25.2. Defining Rules</a></li> </ul> </li> </ul> </li> -<li><a href="#_integration_samples">12. Integration samples</a> +<li><a href="#_integration_samples">13. Integration samples</a> <ul class="sectlevel2"> -<li><a href="#_samples">12.1. Samples</a></li> -<li><a href="#_login_sample">12.2. Login sample</a> +<li><a href="#_samples">13.1. Samples</a></li> +<li><a href="#_login_sample">13.2. Login sample</a> <ul class="sectlevel3"> -<li><a href="#_warning">12.2.1. Warning !</a></li> -<li><a href="#_installing_the_samples">12.2.2. Installing the samples</a></li> +<li><a href="#_warning">13.2.1. Warning !</a></li> +<li><a href="#_installing_the_samples">13.2.2. Installing the samples</a></li> </ul> </li> -<li><a href="#_twitter_sample">12.3. Twitter sample</a> +<li><a href="#_twitter_sample">13.3. Twitter sample</a> <ul class="sectlevel3"> -<li><a href="#_overview">12.3.1. Overview</a></li> -<li><a href="#_interacting_with_the_context_server">12.3.2. Interacting with the context server</a></li> -<li><a href="#_retrieving_context_information_from_unomi_using_the_context_servlet">12.3.3. Retrieving context information from Unomi using the context servlet</a></li> +<li><a href="#_overview">13.3.1. Overview</a></li> +<li><a href="#_interacting_with_the_context_server">13.3.2. Interacting with the context server</a></li> +<li><a href="#_retrieving_context_information_from_unomi_using_the_context_servlet">13.3.3. Retrieving context information from Unomi using the context servlet</a></li> </ul> </li> -<li><a href="#_example_24">12.4. Example</a> +<li><a href="#_example_24">13.4. Example</a> <ul class="sectlevel3"> -<li><a href="#_html_page">12.4.1. HTML page</a></li> -<li><a href="#_javascript">12.4.2. Javascript</a></li> +<li><a href="#_html_page">13.4.1. HTML page</a></li> +<li><a href="#_javascript">13.4.2. Javascript</a></li> </ul> </li> -<li><a href="#_conclusion">12.5. Conclusion</a></li> -<li><a href="#_annex">12.6. Annex</a></li> -<li><a href="#_weather_update_sample">12.7. Weather update sample</a></li> +<li><a href="#_conclusion">13.5. Conclusion</a></li> +<li><a href="#_annex">13.6. Annex</a></li> +<li><a href="#_weather_update_sample">13.7. Weather update sample</a></li> </ul> </li> -<li><a href="#_connectors">13. Connectors</a> +<li><a href="#_connectors">14. Connectors</a> <ul class="sectlevel2"> -<li><a href="#_connectors_2">13.1. Connectors</a> +<li><a href="#_connectors_2">14.1. Connectors</a> <ul class="sectlevel3"> -<li><a href="#_call_for_contributors">13.1.1. Call for contributors</a></li> +<li><a href="#_call_for_contributors">14.1.1. Call for contributors</a></li> </ul> </li> -<li><a href="#_salesforce_connector">13.2. Salesforce Connector</a> +<li><a href="#_salesforce_connector">14.2. Salesforce Connector</a> <ul class="sectlevel3"> -<li><a href="#_getting_started">13.2.1. Getting started</a></li> -<li><a href="#_properties">13.2.2. Properties</a></li> -<li><a href="#_hot_deploying_updates_to_the_salesforce_connector_for_developers">13.2.3. Hot-deploying updates to the Salesforce connector (for developers)</a></li> -<li><a href="#_using_the_salesforce_workbench_for_testing_rest_api">13.2.4. Using the Salesforce Workbench for testing REST API</a></li> -<li><a href="#_setting_up_streaming_push_queries">13.2.5. Setting up Streaming Push queries</a></li> -<li><a href="#_executing_the_unit_tests">13.2.6. Executing the unit tests</a></li> +<li><a href="#_getting_started">14.2.1. Getting started</a></li> +<li><a href="#_properties">14.2.2. Properties</a></li> +<li><a href="#_hot_deploying_updates_to_the_salesforce_connector_for_developers">14.2.3. Hot-deploying updates to the Salesforce connector (for developers)</a></li> +<li><a href="#_using_the_salesforce_workbench_for_testing_rest_api">14.2.4. Using the Salesforce Workbench for testing REST API</a></li> +<li><a href="#_setting_up_streaming_push_queries">14.2.5. Setting up Streaming Push queries</a></li> +<li><a href="#_executing_the_unit_tests">14.2.6. Executing the unit tests</a></li> </ul> </li> -<li><a href="#_mailchimp_connector">13.3. MailChimp Connector</a> +<li><a href="#_mailchimp_connector">14.3. MailChimp Connector</a> <ul class="sectlevel3"> -<li><a href="#_getting_started_2">13.3.1. Getting started</a></li> +<li><a href="#_getting_started_2">14.3.1. Getting started</a></li> </ul> </li> </ul> </li> -<li><a href="#_developers">14. Developers</a> +<li><a href="#_developers">15. Developers</a> <ul class="sectlevel2"> -<li><a href="#_building">14.1. Building</a> +<li><a href="#_building">15.1. Building</a> <ul class="sectlevel3"> -<li><a href="#_initial_setup">14.1.1. Initial Setup</a></li> -<li><a href="#_building_2">14.1.2. Building</a></li> -<li><a href="#_installing_an_elasticsearch_server">14.1.3. Installing an ElasticSearch server</a></li> -<li><a href="#_deploying_the_generated_binary_package">14.1.4. Deploying the generated binary package</a></li> -<li><a href="#_deploying_into_an_existing_karaf_server">14.1.5. Deploying into an existing Karaf server</a></li> -<li><a href="#_jdk_selection_on_mac_os_x">14.1.6. JDK Selection on Mac OS X</a></li> -<li><a href="#_running_the_integration_tests">14.1.7. Running the integration tests</a></li> -<li><a href="#_testing_with_an_example_page">14.1.8. Testing with an example page</a></li> -</ul> -</li> -<li><a href="#_ssh_shell_commands">14.2. SSH Shell Commands</a> -<ul class="sectlevel3"> -<li><a href="#_using_the_shell">14.2.1. Using the shell</a></li> -<li><a href="#_lifecycle_commands">14.2.2. Lifecycle commands</a></li> -<li><a href="#_runtime_commands">14.2.3. Runtime commands</a></li> -</ul> -</li> -<li><a href="#_writing_plugins">14.3. Writing Plugins</a></li> -<li><a href="#_types_vs_instances">14.4. Types vs. instances</a></li> -<li><a href="#_plugin_structure">14.5. Plugin structure</a></li> -<li><a href="#_extension_points">14.6. Extension points</a> -<ul class="sectlevel3"> -<li><a href="#_actiontype">14.6.1. ActionType</a></li> -<li><a href="#_conditiontype">14.6.2. ConditionType</a></li> -<li><a href="#_persona_2">14.6.3. Persona</a></li> -<li><a href="#_propertymergestrategytype">14.6.4. PropertyMergeStrategyType</a></li> -<li><a href="#_propertytype">14.6.5. PropertyType</a></li> -<li><a href="#_rule_2">14.6.6. Rule</a></li> -<li><a href="#_scoring">14.6.7. Scoring</a></li> -<li><a href="#_segments">14.6.8. Segments</a></li> -<li><a href="#_tag">14.6.9. Tag</a></li> -<li><a href="#_valuetype">14.6.10. ValueType</a></li> -</ul> -</li> -<li><a href="#_custom_plugins">14.7. Custom plugins</a> -<ul class="sectlevel3"> -<li><a href="#_creating_a_plugin">14.7.1. Creating a plugin</a></li> -<li><a href="#_deployment_and_custom_definition">14.7.2. Deployment and custom definition</a></li> -<li><a href="#_predefined_segments">14.7.3. Predefined segments</a></li> -<li><a href="#_predefined_rules">14.7.4. Predefined rules</a></li> -<li><a href="#_predefined_properties">14.7.5. Predefined properties</a></li> -<li><a href="#_predefined_child_conditions">14.7.6. Predefined child conditions</a></li> -<li><a href="#_predefined_personas">14.7.7. Predefined personas</a></li> -<li><a href="#_custom_action_types">14.7.8. Custom action types</a></li> -<li><a href="#_custom_condition_types">14.7.9. Custom condition types</a></li> +<li><a href="#_initial_setup">15.1.1. Initial Setup</a></li> +<li><a href="#_building_2">15.1.2. Building</a></li> +<li><a href="#_installing_an_elasticsearch_server">15.1.3. Installing an ElasticSearch server</a></li> +<li><a href="#_deploying_the_generated_binary_package">15.1.4. Deploying the generated binary package</a></li> +<li><a href="#_deploying_into_an_existing_karaf_server">15.1.5. Deploying into an existing Karaf server</a></li> +<li><a href="#_jdk_selection_on_mac_os_x">15.1.6. JDK Selection on Mac OS X</a></li> +<li><a href="#_running_the_integration_tests">15.1.7. Running the integration tests</a></li> +<li><a href="#_testing_with_an_example_page">15.1.8. Testing with an example page</a></li> +</ul> +</li> +<li><a href="#_ssh_shell_commands">15.2. SSH Shell Commands</a> +<ul class="sectlevel3"> +<li><a href="#_using_the_shell">15.2.1. Using the shell</a></li> +<li><a href="#_lifecycle_commands">15.2.2. Lifecycle commands</a></li> +<li><a href="#_runtime_commands">15.2.3. Runtime commands</a></li> +</ul> +</li> +<li><a href="#_writing_plugins">15.3. Writing Plugins</a></li> +<li><a href="#_types_vs_instances">15.4. Types vs. instances</a></li> +<li><a href="#_plugin_structure">15.5. Plugin structure</a></li> +<li><a href="#_extension_points">15.6. Extension points</a> +<ul class="sectlevel3"> +<li><a href="#_actiontype">15.6.1. ActionType</a></li> +<li><a href="#_conditiontype">15.6.2. ConditionType</a></li> +<li><a href="#_persona_2">15.6.3. Persona</a></li> +<li><a href="#_propertymergestrategytype">15.6.4. PropertyMergeStrategyType</a></li> +<li><a href="#_propertytype">15.6.5. PropertyType</a></li> +<li><a href="#_rule_2">15.6.6. Rule</a></li> +<li><a href="#_scoring">15.6.7. Scoring</a></li> +<li><a href="#_segments">15.6.8. Segments</a></li> +<li><a href="#_tag">15.6.9. Tag</a></li> +<li><a href="#_valuetype">15.6.10. ValueType</a></li> +</ul> +</li> +<li><a href="#_custom_plugins">15.7. Custom plugins</a> +<ul class="sectlevel3"> +<li><a href="#_creating_a_plugin">15.7.1. Creating a plugin</a></li> +<li><a href="#_deployment_and_custom_definition">15.7.2. Deployment and custom definition</a></li> +<li><a href="#_predefined_segments">15.7.3. Predefined segments</a></li> +<li><a href="#_predefined_rules">15.7.4. Predefined rules</a></li> +<li><a href="#_predefined_properties">15.7.5. Predefined properties</a></li> +<li><a href="#_predefined_child_conditions">15.7.6. Predefined child conditions</a></li> +<li><a href="#_predefined_personas">15.7.7. Predefined personas</a></li> +<li><a href="#_custom_action_types">15.7.8. Custom action types</a></li> +<li><a href="#_custom_condition_types">15.7.9. Custom condition types</a></li> </ul> </li> -<li><a href="#_migration_patches">14.8. Migration patches</a></li> +<li><a href="#_migration_patches">15.8. Migration patches</a></li> </ul> </li> </ul> @@ -437,16 +478,188 @@ </div> </div> <div class="sect1"> -<h2 id="_quick_start">1. Quick start</h2> +<h2 id="_whats_new">1. What’s new</h2> <div class="sectionbody"> <div class="sect2"> -<h3 id="_five_minutes_quickstart">1.1. Five Minutes QuickStart</h3> +<h3 id="_whats_new_in_apache_unomi_2_0">1.1. What’s new in Apache Unomi 2.0</h3> +<div class="paragraph"> +<p>Apache Unomi 2 is a new release focused on improving core functionalities and robustness of the product. +The introduction of tighter data validation with JSON Schemas required some changes in the product data model, which presented an opportunity for noticeable improvements in the overall performance.</p> +</div> +<div class="paragraph"> +<p>This new release also introduces a first (beta) version of the Unomi GraphQL API.</p> +</div> +<div class="sect3"> +<h4 id="_scopes_declarations_are_now_required">1.1.1. Scopes declarations are now required</h4> +<div class="paragraph"> +<p>Scopes declarations are now required in Unomi 2. When submitting an event and specifying a scope, +that scope must already be declared on the platform.</p> +</div> +<div class="paragraph"> +<p>Scopes can be easily created via the corresponding REST API (<code>cxs/scopes</code>)</p> +</div> +<div class="paragraph"> +<p>For example, an "apache" scope can be created using the following API call.</p> +</div> +<div class="listingblock"> +<div class="content"> +<pre class="highlight"><code>curl --location --request POST 'http://localhost:8181/cxs/scopes' \ +-u 'karaf:karaf' \ +--header 'Content-Type: application/json' \ +--data-raw '{ +"itemId": "apache", +"itemType": "scope" +}'</code></pre> +</div> +</div> +</div> +<div class="sect3"> +<h4 id="_json_schemas">1.1.2. JSON Schemas</h4> +<div class="paragraph"> +<p>Apache Unomi 2 introduces support for <a href="https://json-schema.org/specification.html";>JSON Schema</a> for all of its publicly exposed endpoints. +Data received by Apache Unomi 2 will first be validated against a known schema to make sure it complies with an expected payload. +If the received payload does not match a known schema, it will be rejected by Apache Unomi 2.</p> +</div> +<div class="paragraph"> +<p>Apache Unomi 2 also introduces a set of administrative endpoints allowing new schemas and/or schemas extensions to be registered.</p> +</div> +<div class="paragraph"> +<p>More details about JSON Schemas implementation are available in the <a href="#_json_schemas_2">corresponding section</a> of the documentation.</p> +</div> +</div> +<div class="sect3"> +<h4 id="_updated_data_model">1.1.3. Updated data model</h4> +<div class="paragraph"> +<p>The introduction of JSON schema required us to modify Apache Unomi data model, One of the key differences is the removal of open maps.</p> +</div> +<div class="paragraph"> +<p>For example, the following event model in Apache Unomi 1.x:</p> +</div> +<div class="listingblock"> +<div class="content"> +<pre class="highlight"><code>{ + "TODO": "ADD JSON" +}</code></pre> +</div> +</div> +<div class="paragraph"> +<p>Is replaced by the following in Apache Unomi 2.x:</p> +</div> +<div class="listingblock"> +<div class="content"> +<pre class="highlight"><code>{ + "TODO": "ADD JSON" +}</code></pre> +</div> +</div> +<div class="paragraph"> +<p>Most objects were refactored as part of this new release.</p> +</div> +<div class="paragraph"> +<p>If using the default Apache 1.x data model, our Unomi 2 migration process will handle the data model changes for you. +If you are using custom events/objects, please refer to the detailed migration guide for more details.</p> +</div> +</div> +<div class="sect3"> +<h4 id="_removal_of_the_web_tracker">1.1.4. Removal of the Web Tracker</h4> +<div class="paragraph"> +<p>Apache Unomi 2.0 Web Tracker, previously located in <code>extensions/web-tracker/</code> has been removed. +We considered it as outdated and instead recommend implementing your own tracker logic based on your project +use case.</p> +</div> +</div> +<div class="sect3"> +<h4 id="_graphql_api_beta">GraphQL API (beta)</h4> +<div class="paragraph"> +<p>Apache Unomi 2.0 sees the introduction of a new (beta) GraphQL API. +Available behind a feature flag (the API disabled by default), the GraphQL API is available for you to play with.</p> +</div> +<div class="paragraph"> +<p>More details about how to enable/disable the GraphQL API are available in the <a href="#_graphql_api">corresponding section</a> of the documentation.</p> +</div> +<div class="paragraph"> +<p>We welcome tickets/PRs to improve its robustness and progressively make it ready for prime time.</p> +</div> +</div> +<div class="sect3"> +<h4 id="_migrate_from_unomi_1_x">1.1.5. Migrate from Unomi 1.x</h4> +<div class="paragraph"> +<p>To facilitate migration we prepared a set of scripts that will automatically handle the migration of your data from Apache Unomi 1.5+ to Apache Unomi 2.0.</p> +</div> +<div class="paragraph"> +<p>It is worth keeping in mind that for Apache Unomi 2.0 we do not support âhotâ migration, +the migration process will require a shutdown of your cluster to guarantee that no new events will be collected while data migration is in progress.</p> +</div> +<div class="paragraph"> +<p>Special caution must be taken if you declared custom events as our migration scripts can only handle objects we know of. +More details about migration (incl. of custom events) is available in the corresponding section <a href="#_migrations">corresponding section</a> of the documentation.</p> +</div> +</div> +<div class="sect3"> +<h4 id="_elasticsearch_compatibility">1.1.6. Elasticsearch compatibility</h4> +<div class="paragraph"> +<p>We currently recommend using Elasticsearch 7.17.5 with Apache Unomi 2.0, +this ensure you are on a recent version that is not impacted by the log4j vulnerabilities (fixed in Elasticsearch 7.16.3).</p> +</div> +<div class="paragraph"> +<p>This version increase is releated to Apache Unomi 2.0 makeing use of a new Elasticsearch field type +called <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.17/flattened.html";>Flattened</a>, +and although it was available in prior versions of Elasticsearch, we do not recommend using those +due to the above-mentioned log4j vulnerabilities.</p> +</div> +</div> +</div> +</div> +</div> +<div class="sect1"> +<h2 id="_quick_start">2. Quick start</h2> +<div class="sectionbody"> +<div class="sect2"> +<h3 id="_quick_start_with_docker">2.1. Quick start with Docker</h3> +<div class="paragraph"> +<p>Begin by creating a <code>docker-compose.yml</code> file with the following content:</p> +</div> +<div class="listingblock"> +<div class="content"> +<pre class="highlight"><code>version: '3.8' +services: + elasticsearch: + image: docker.elastic.co/elasticsearch/elasticsearch:7.17.5 + environment: + - discovery.type=single-node + ports: + - 9200:9200 + unomi: + # Unomi version can be updated based on your needs + image: apache/unomi:2.0.0 + environment: + - UNOMI_ELASTICSEARCH_ADDRESSES=elasticsearch:9200 + - UNOMI_THIRDPARTY_PROVIDER1_IPADDRESSES=0.0.0.0/0,::1,127.0.0.1 + ports: + - 8181:8181 + - 9443:9443 + - 8102:8102 + links: + - elasticsearch + depends_on: + - elasticsearch</code></pre> +</div> +</div> +<div class="paragraph"> +<p>From the same folder, start the environment using <code>docker-compose up</code> and wait for the startup to complete.</p> +</div> +<div class="paragraph"> +<p>Try accessing <a href="https://localhost:9443/cxs/cluster"; class="bare">https://localhost:9443/cxs/cluster</a> with username/password: karaf/karaf . You might get a certificate warning in your browser, just accept it despite the warning it is safe.</p> +</div> +</div> +<div class="sect2"> +<h3 id="_quick_start_manually">2.2. Quick Start manually</h3> <div class="paragraph"> <p>1) Install JDK 8 (<a href="https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html"; class="bare">https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html</a>) and make sure you set the JAVA_HOME variable <a href="https://docs.oracle.com/cd/E19182-01/820-7851/inst_cli_jdk_javahome_t/"; class="bare">https://docs.oracle.com/cd/E19182-01/820-7851/inst_cli_jdk_javahome_t/</a> (see our <a href="#_jdk_compatibility">Getting Started</a> guide for more information on JDK compatibility)</p> </div> <div class="paragraph"> -<p>2) Download ElasticSearch here : <a href="https://www.elastic.co/downloads/past-releases/elasticsearch-7-4-2"; class="bare">https://www.elastic.co/downloads/past-releases/elasticsearch-7-4-2</a> (please <strong>make sure</strong> you use the proper version : 7.4.2)</p> +<p>2) Download ElasticSearch here : <a href="https://www.elastic.co/downloads/past-releases/elasticsearch-7-17-5"; class="bare">https://www.elastic.co/downloads/past-releases/elasticsearch-7-17-5</a> (please <strong>make sure</strong> you use the proper version : 7.17.5)</p> </div> <div class="paragraph"> <p>3) Uncompress it and change the <code>config/elasticsearch.yml</code> to include the following config : <code>cluster.name: contextElasticSearch</code></p> @@ -490,16 +703,16 @@ make sure you are using the proper versi </div> </div> <div class="sect1"> -<h2 id="_first_steps_with_apache_unomi">2. First steps with Apache Unomi</h2> +<h2 id="_first_steps_with_apache_unomi">3. First steps with Apache Unomi</h2> <div class="sectionbody"> <div class="sect2"> -<h3 id="_getting_started_with_unomi">2.1. Getting started with Unomi</h3> +<h3 id="_getting_started_with_unomi">3.1. Getting started with Unomi</h3> <div class="paragraph"> <p>We will first get you up and running with an example. We will then lift the corner of the cover somewhat and explain in greater details what just happened.</p> </div> <div class="sect3"> -<h4 id="_prerequisites">2.1.1. Prerequisites</h4> +<h4 id="_prerequisites">3.1.1. Prerequisites</h4> <div class="paragraph"> <p>This document assumes working knowledge of <a href="https://git-scm.com/";>git</a> to be able to retrieve the code for Unomi and the example. Additionally, you will require a working Java 8 or above install. Refer to <a href="http://www.oracle.com/technetwork/java/javase/";>http://www.oracle.com/technetwork/java/javase/</a> for details on how to download and install Java SE 8 or greater.</p> @@ -519,22 +732,20 @@ them at your own risks.</p> </div> </div> <div class="sect4"> -<h5 id="_elasticsearch_compatibility">ElasticSearch compatibility</h5> +<h5 id="_elasticsearch_compatibility_2">ElasticSearch compatibility</h5> <div class="paragraph"> -<p>Starting with version 1.5.0 Apache Unomi adds compatibility with ElasticSearch 7.4 . It is highly recommended to use the -ElasticSearch version provided by the documentation when possible. However minor versions (7.4.x) should also work, and -one version higher (7.5) will usually work. Going higher than that is risky given the way that ElasticSearch is developed -and breaking changes are introduced quite often. If in doubt, don’t hesitate to check with the Apache Unomi community +<p>Starting with version 2.0.0 Apache Unomi adds compatibility with ElasticSearch 7.17.5 . It is highly recommended to use the +ElasticSearch version specified in the documentation whenever possible. If in doubt, don’t hesitate to check with the Apache Unomi community to get the latest information about ElasticSearch version compatibility.</p> </div> </div> </div> <div class="sect3"> -<h4 id="_running_unomi">2.1.2. Running Unomi</h4> +<h4 id="_running_unomi">3.1.2. Running Unomi</h4> <div class="sect4"> <h5 id="_start_unomi">Start Unomi</h5> <div class="paragraph"> -<p>Start Unomi according to the <a href="#_five_minutes_quickstart">five minutes quick start</a> or by compiling using the +<p>Start Unomi according to the <a href="#Five Minutes QuickStart">quick start with docker</a> or by compiling using the <a href="#_building">building instructions</a>. Once you have Karaf running, you should wait until you see the following messages on the Karaf console:</p> </div> @@ -566,16 +777,69 @@ requests you can do once your server is </div> </div> <div class="sect2"> -<h3 id="_recipes">2.2. Recipes</h3> +<h3 id="_recipes">3.2. Recipes</h3> <div class="sect3"> -<h4 id="_introduction">2.2.1. Introduction</h4> +<h4 id="_introduction">3.2.1. Introduction</h4> <div class="paragraph"> <p>In this section of the documentation we provide quick recipes focused on helping you achieve a specific result with Apache Unomi.</p> </div> </div> <div class="sect3"> -<h4 id="_how_to_read_a_profile">2.2.2. How to read a profile</h4> +<h4 id="_enabling_debug_mode">3.2.2. Enabling debug mode</h4> +<div class="paragraph"> +<p>Although the examples provided in this documentation are correct (they will work "as-is"), +you might be tempted to modify them to fit your use case, which might result in errors.</p> +</div> +<div class="paragraph"> +<p>The best approach during development is to enable Apache Unomi debug mode, which will provide +you with more detailed logs about events processing.</p> +</div> +<div class="paragraph"> +<p>The debug mode can be activated via the karaf SSH console (default credentials are karaf/karaf):</p> +</div> +<div class="listingblock"> +<div class="content"> +<pre class="highlight"><code>ubuntu@ip-10-0-3-252:~/$ ssh -p 8102 karaf@localhost +Password authentication +Password: + __ __ ____ + / //_/____ __________ _/ __/ + / ,< / __ `/ ___/ __ `/ /_ + / /| |/ /_/ / / / /_/ / __/ + /_/ |_|\__,_/_/ \__,_/_/ + + Apache Karaf (4.2.15) + +Hit '<tab>' for a list of available commands +and '[cmd] --help' for help on a specific command. +Hit 'system:shutdown' to shutdown Karaf. +Hit '<ctrl-d>' or type 'logout' to disconnect shell from current session. + +karaf@root()> log:set DEBUG org.apache.unomi.schema.impl.SchemaServiceImpl</code></pre> +</div> +</div> +<div class="paragraph"> +<p>You can then either watch the logs via your preferred logging mechanism (docker logs, log file, …​) or +simply tail the logs to the terminal you used to enable debug mode.</p> +</div> +<div class="listingblock"> +<div class="content"> +<pre class="highlight"><code>karaf@root()> log:tail +08:55:28.128 DEBUG [qtp1422628821-128] Schema validation found 2 errors while validating against schema: https://unomi.apache.org/schemas/json/events/view/1-0-0 +08:55:28.138 DEBUG [qtp1422628821-128] Validation error: There are unevaluated properties at following paths $.source.properties +08:55:28.140 DEBUG [qtp1422628821-128] Validation error: There are unevaluated properties at following paths $.source.itemId, $.source.itemType, $.source.scope, $.source.properties +08:55:28.142 ERROR [qtp1422628821-128] An event was rejected - switch to DEBUG log level for more information</code></pre> +</div> +</div> +<div class="paragraph"> +<p>The example above shows schema validation failure at the <code>$.source.properties</code> path. +Note that the validation will output one log line for the exact failing path and a log line for its parent, +therefore to find the source of a schema validation issue it’s best to start from the top.</p> +</div> +</div> +<div class="sect3"> +<h4 id="_how_to_read_a_profile">3.2.3. How to read a profile</h4> <div class="paragraph"> <p>The simplest way to retrieve profile data for the current profile is to simply send a request to the /cxs/context.json endpoint. However you will need to send a body along with that request. Here’s an example:</p> @@ -587,8 +851,7 @@ endpoint. However you will need to send <div class="content"> <pre class="highlight"><code>curl -X POST http://localhost:8181/cxs/context.json?sessionId=1234 \ -H "Content-Type: application/json" \ --d @- <<'EOF' -{ +--data-raw '{ "source": { "itemId":"homepage", "itemType":"page", @@ -598,15 +861,14 @@ endpoint. However you will need to send "requiredSessionProperties":["*"], "requireSegments":true, "requireScores":true -} -EOF</code></pre> +}'</code></pre> </div> </div> <div class="paragraph"> <p>The <code>requiredProfileProperties</code> and <code>requiredSessionProperties</code> are properties that take an array of property names that should be retrieved. In this case we use the wildcard character '*' to say we want to retrieve all the available -properties. The structure of the JSON object that you should send is a JSON-serialized version of the <a href="http://unomi.apache.org/unomi-api/apidocs/org/apache/unomi/api/ContextRequest.html";>ContextRequest</a> -Java class.</p> +properties. The structure of the JSON object that you should send is a JSON-serialized version of the +<a href="http://unomi.apache.org/unomi-api/apidocs/org/apache/unomi/api/ContextRequest.html";>ContextRequest</a> Java class.</p> </div> <div class="paragraph"> <p>Note that it is also possible to access a profile’s data through the /cxs/profiles/ endpoint but that really should be @@ -615,7 +877,7 @@ and security.</p> </div> </div> <div class="sect3"> -<h4 id="_how_to_update_a_profile_from_the_public_internet">2.2.3. How to update a profile from the public internet</h4> +<h4 id="_how_to_update_a_profile_from_the_public_internet">3.2.4. How to update a profile from the public internet</h4> <div class="paragraph"> <p>Before we get into how to update a profile directly from a request coming from the public internet, we’ll quickly talk first about how NOT to do it, because we often see users using the following anti-patterns.</p> @@ -655,10 +917,8 @@ ones that you might not want to be overr <ul> <li> <p>(Preferred) Use you own custom event(s) to send data you want to be inserted in a profile, and use rules to map the -event data to the profile. This is simpler than it sounds, as usually all it requires is setting up a simple rule and -you’re ready to update profiles using events. This is also the safest way to update a profile because if you design your -events to be as specific as possible to your needs, only the data that you specified will be copied to the profile, -making sure that even in the case an attacker tries to send more data using your custom event it will simply be ignored.</p> +event data to the profile. This is simpler than it sounds, as usually all it requires is setting up a simple rule, +defining the corresponding JSON schema and you’re ready to update profiles using events.</p> </li> <li> <p>Use the protected built-in "updateProperties" event. This event is designed to be used for administrative purposes @@ -677,8 +937,7 @@ tracing.</p> <pre class="highlight"><code>curl -X POST http://localhost:8181/cxs/rules \ --user karaf:karaf \ -H "Content-Type: application/json" \ --d @- <<'EOF' -{ +--data-raw '{ "metadata": { "id": "setContactInfo", "name": "Copy the received contact info to the current profile", @@ -717,8 +976,7 @@ tracing.</p> } } ] -} -EOF</code></pre> +}'</code></pre> </div> </div> <div class="paragraph"> @@ -729,14 +987,96 @@ course change any of the property names with the source of the event, such as 'mobileApp:firstName'.</p> </div> <div class="paragraph"> -<p>You could then simply send the <code>contactInfoSubmitted</code> event using a request similar to this one:</p> +<p>Now that our rule is defined, the next step is to create a scope and a JSON Schema corresponding to the event to be submitted.</p> +</div> +<div class="paragraph"> +<p>We will start by creating a scope called "example" scope:</p> +</div> +<div class="listingblock"> +<div class="content"> +<pre class="highlight"><code>curl --location --request POST 'http://localhost:8181/cxs/scopes' \ +-u 'karaf:karaf' \ +--header 'Content-Type: application/json' \ +--data-raw '{ +"itemId": "example", +"itemType": "scope" +}'</code></pre> +</div> +</div> +<div class="paragraph"> +<p>The next step consist in creating a JSON Schema to validate our event.</p> +</div> +<div class="listingblock"> +<div class="content"> +<pre class="highlight"><code>curl --location --request POST 'http://localhost:8181/cxs/jsonSchema' \ +-u 'karaf:karaf' \ +--header 'Content-Type: application/json' \ +--data-raw '{ + "$id": "https://unomi.apache.org/schemas/json/events/contactInfoSubmitted/1-0-0";, + "$schema": "https://json-schema.org/draft/2019-09/schema";, + "self": { + "vendor": "org.apache.unomi", + "name": "contactInfoSubmitted", + "format": "jsonschema", + "target": "events", + "version": "1-0-0" + }, + "title": "contactInfoSubmittedEvent", + "type": "object", + "allOf": [{ "$ref": "https://unomi.apache.org/schemas/json/event/1-0-0"; }], + "properties": { + "source" : { + "$ref" : "https://unomi.apache.org/schemas/json/item/1-0-0"; + }, + "target" : { + "$ref" : "https://unomi.apache.org/schemas/json/item/1-0-0"; + }, + "properties": { + "type": "object", + "properties": { + "firstName": { + "type": ["null", "string"] + }, + "lastName": { + "type": ["null", "string"] + }, + "email": { + "type": ["null", "string"] + } + } + } + }, + "unevaluatedProperties": false +}'</code></pre> +</div> +</div> +<div class="paragraph"> +<p>You can notice the following in the above schema:</p> +</div> +<div class="ulist"> +<ul> +<li> +<p>We are creating a schema of type "events" ("self.target" equals "events")</p> +</li> +<li> +<p>The name of this schema is "contactInfoSubmitted", this MUST match the value of the "eventType" field in the event itself (below)</p> +</li> +<li> +<p>To simplify our schema declaration, we’re referring to an already existing schema (<a href="https://unomi.apache.org/schemas/json/item/1-0-0"; class="bare">https://unomi.apache.org/schemas/json/item/1-0-0</a>) to validate the "source" and "target" properties. Apache Unomi ships with a set of predefined JSON Schemas, detailed here: <a href="https://github.com/apache/unomi/tree/master/extensions/json-schema/services/src/main/resources/META-INF/cxs/schemas"; class="bare">https://github.com/apache/unomi/tree/master/extensions/json-schema/services/src/main/resources/META-INF/cxs/schemas</a>.</p> +</li> +<li> +<p><code>"unevaluatedProperties": false</code> indicates that the event should be rejected if it contains any additional metadata.</p> +</li> +</ul> +</div> +<div class="paragraph"> +<p>Finally, send the <code>contactInfoSubmitted</code> event using a request similar to this one:</p> </div> <div class="listingblock"> <div class="content"> <pre class="highlight"><code>curl -X POST http://localhost:8181/cxs/eventcollector \ -H "Content-Type: application/json" \ --d @- <<'EOF' -{ +--data-raw '{ "sessionId" : "1234", "events":[ { @@ -744,29 +1084,82 @@ with the source of the event, such as 'm "scope": "example", "source":{ "itemType": "site", - "scope":"example", + "scope": "example", "itemId": "mysite" }, "target":{ - "itemType":"form", - "scope":"example", - "itemId":"contactForm" + "itemType": "form", + "scope": "example", + "itemId": "contactForm" }, "properties" : { - "firstName" : "John", - "lastName" : "Doe", - "email" : "[email protected]" + "firstName": "John", + "lastName": "Doe", + "email": "[email protected]" } } ] -} -EOF</code></pre> +}'</code></pre> +</div> +</div> +<div class="paragraph"> +<p>The event we just submitted can be retrieved using the following request:</p> +</div> +<div class="listingblock"> +<div class="content"> +<pre class="highlight"><code>curl -X POST http://localhost:8181/cxs/events/search \ +--user karaf:karaf \ +-H "Content-Type: application/json" \ +--data-raw '{ + "offset" : 0, + "limit" : 20, + "condition" : { + "type": "eventPropertyCondition", + "parameterValues" : { + "propertyName" : "properties.firstName", + "comparisonOperator" : "equals", + "propertyValue" : "John" + } + } +}'</code></pre> +</div> +</div> +</div> +<div class="sect4"> +<h5 id="_troubleshooting_common_errors">Troubleshooting common errors</h5> +<div class="paragraph"> +<p>There could be two types of common errors while customizing the above requests: +* The schema is invalid +* The event is invalid</p> +</div> +<div class="paragraph"> +<p>While first submitting the schema during its creation, Apache Unomi will validate it is syntaxically correct (JSON) +but will not perform any further validation. Since the schema will be processed for the first time when events are submitted, +errors might be noticeable at that time.</p> +</div> +<div class="paragraph"> +<p>Those errors are usually self-explanatory, such as this one pointing to an incorrect lcoation for the "firstName" keyword:</p> +</div> +<div class="listingblock"> +<div class="content"> +<pre class="highlight"><code>09:35:56.573 WARN [qtp1421852915-83] Unknown keyword firstName - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword</code></pre> +</div> +</div> +<div class="paragraph"> +<p>If an event is invalid, the logs will contain details about the part of the event that did not validate against the schema. +In the example below, an extra property "abcd" was added to the event:</p> +</div> +<div class="listingblock"> +<div class="content"> +<pre class="highlight"><code>12:27:04.269 DEBUG [qtp1421852915-481] Schema validation found 1 errors while validating against schema: https://unomi.apache.org/schemas/json/events/contactInfoSubmitted/1-0-0 +12:27:04.272 DEBUG [qtp1421852915-481] Validation error: There are unevaluated properties at following paths $.properties.abcd +12:27:04.273 ERROR [qtp1421852915-481] An event was rejected - switch to DEBUG log level for more information</code></pre> </div> </div> </div> </div> <div class="sect3"> -<h4 id="_how_to_search_for_profile_events">2.2.4. How to search for profile events</h4> +<h4 id="_how_to_search_for_profile_events">3.2.5. How to search for profile events</h4> <div class="paragraph"> <p>Sometimes you want to retrieve events for a known profile. You will need to provide a query in the body of the request that looks something like this (and <a href="https://unomi.apache.org/rest-api-doc/#1768188821";>documentation is available in the REST API</a>) :</p> @@ -776,8 +1169,8 @@ that looks something like this (and <a h <pre class="highlight"><code>curl -X POST http://localhost:8181/cxs/events/search \ --user karaf:karaf \ -H "Content-Type: application/json" \ --d @- <<'EOF' -{ "offset" : 0, +--data-raw '{ + "offset" : 0, "limit" : 20, "condition" : { "type": "eventPropertyCondition", @@ -787,8 +1180,7 @@ that looks something like this (and <a h "propertyValue" : "PROFILE_ID" } } -} -EOF</code></pre> +}'</code></pre> </div> </div> <div class="paragraph"> @@ -796,7 +1188,7 @@ EOF</code></pre> </div> </div> <div class="sect3"> -<h4 id="_how_to_create_a_new_rule">2.2.5. How to create a new rule</h4> +<h4 id="_how_to_create_a_new_rule">3.2.6. How to create a new rule</h4> <div class="paragraph"> <p>There are basically two ways to create a new rule :</p> </div> @@ -820,8 +1212,7 @@ on the Apache Unomi server.</p> <pre class="highlight"><code>curl -X POST http://localhost:8181/cxs/rules \ --user karaf:karaf \ -H "Content-Type: application/json" \ --d @- <<'EOF' -{ +--data-raw '{ "metadata": { "id": "exampleEventCopy", "name": "Example Copy Event to Profile", @@ -840,8 +1231,7 @@ on the Apache Unomi server.</p> "type": "allEventToProfilePropertiesAction" } ] -} -EOF</code></pre> +}'</code></pre> </div> </div> <div class="paragraph"> @@ -850,7 +1240,7 @@ contained in the event to the current pr </div> </div> <div class="sect3"> -<h4 id="_how_to_search_for_profiles">2.2.6. How to search for profiles</h4> +<h4 id="_how_to_search_for_profiles">3.2.7. How to search for profiles</h4> <div class="paragraph"> <p>In order to search for profiles you will have to use the /cxs/profiles/search endpoint that requires a Query JSON structure. Here’s an example of a profile search with a Query object:</p> @@ -860,8 +1250,7 @@ structure. Here’s an example of a <pre class="highlight"><code>curl -X POST http://localhost:8181/cxs/profiles/search \ --user karaf:karaf \ -H "Content-Type: application/json" \ --d @- <<'EOF' -{ +--data-raw '{ "text" : "unomi", "offset" : 0, "limit" : 10, @@ -888,8 +1277,7 @@ structure. Here’s an example of a ] } } -} -EOF</code></pre> +}'</code></pre> </div> </div> <div class="paragraph"> @@ -904,13 +1292,13 @@ on the server and potentially this could </div> </div> <div class="sect3"> -<h4 id="_getting_updating_consents">2.2.7. Getting / updating consents</h4> +<h4 id="_getting_updating_consents">3.2.8. Getting / updating consents</h4> <div class="paragraph"> <p>You can find information on how to retrieve or create/update consents in the <a href="#_consent_api">Consent API</a> section.</p> </div> </div> <div class="sect3"> -<h4 id="_how_to_send_a_login_event_to_unomi">2.2.8. How to send a login event to Unomi</h4> +<h4 id="_how_to_send_a_login_event_to_unomi">3.2.9. How to send a login event to Unomi</h4> <div class="paragraph"> <p>Tracking logins must be done carefully with Unomi. A login event is considered a "privileged" event and therefore for not be initiated from the public internet. Ideally user authentication should always be validated by a trusted third- @@ -976,9 +1364,9 @@ security issue since it could be a way t </div> </div> <div class="sect2"> -<h3 id="_request_examples">2.3. Request examples</h3> +<h3 id="_request_examples">3.3. Request examples</h3> <div class="sect3"> -<h4 id="_retrieving_your_first_context">2.3.1. Retrieving your first context</h4> +<h4 id="_retrieving_your_first_context">3.3.1. Retrieving your first context</h4> <div class="paragraph"> <p>You can retrieve a context using curl like this :</p> </div> @@ -994,7 +1382,7 @@ events using the cxs.collectEvents() fun </div> </div> <div class="sect3"> -<h4 id="_retrieving_a_context_as_a_json_object">2.3.2. Retrieving a context as a JSON object.</h4> +<h4 id="_retrieving_a_context_as_a_json_object">3.3.2. Retrieving a context as a JSON object.</h4> <div class="paragraph"> <p>If you prefer to retrieve a pure JSON object, you can simply use a request formed like this:</p> </div> @@ -1005,7 +1393,7 @@ events using the cxs.collectEvents() fun </div> </div> <div class="sect3"> -<h4 id="_accessing_profile_properties_in_a_context">2.3.3. Accessing profile properties in a context</h4> +<h4 id="_accessing_profile_properties_in_a_context">3.3.3. Accessing profile properties in a context</h4> <div class="paragraph"> <p>By default, in order to optimize the amount of data sent over the network, Apache Unomi will not send the content of the profile or session properties. If you need this data, you must send a JSON object to configure the resulting output @@ -1042,7 +1430,7 @@ Java class.</p> </div> </div> <div class="sect3"> -<h4 id="_sending_events_using_the_context_servlet">2.3.4. Sending events using the context servlet</h4> +<h4 id="_sending_events_using_the_context_servlet">3.3.4. Sending events using the context servlet</h4> <div class="paragraph"> <p>At the same time as you are retrieving the context, you can also directly send events in the ContextRequest object as illustrated in the following example:</p> @@ -1073,7 +1461,7 @@ illustrated in the following example:</p "itemId":"homepage", "properties":{ "pageInfo":{ - "referringURL":"" + "referringURL":"https://apache.org/"; } } } @@ -1091,7 +1479,7 @@ to send events in a network optimal way </div> </div> <div class="sect3"> -<h4 id="_sending_events_using_the_eventcollector_servlet">2.3.5. Sending events using the eventcollector servlet</h4> +<h4 id="_sending_events_using_the_eventcollector_servlet">3.3.5. Sending events using the eventcollector servlet</h4> <div class="paragraph"> <p>If you only need to send events without retrieving a context, you should use the eventcollector servlet that is optimized respond quickly and minimize network traffic. Here is an example of using this servlet:</p> @@ -1118,7 +1506,7 @@ respond quickly and minimize network tra "itemId":"homepage", "properties":{ "pageInfo":{ - "referringURL":"" + "referringURL":"https://apache.org/"; } } } @@ -1134,7 +1522,7 @@ to send additional events.</p> </div> </div> <div class="sect3"> -<h4 id="_where_to_go_from_here">2.3.6. Where to go from here</h4> +<h4 id="_where_to_go_from_here">3.3.6. Where to go from here</h4> <div class="ulist"> <ul> <li> @@ -1150,10 +1538,10 @@ to send additional events.</p> </div> </div> <div class="sect1"> -<h2 id="_configuration">3. Configuration</h2> +<h2 id="_configuration">4. Configuration</h2> <div class="sectionbody"> <div class="sect2"> -<h3 id="_centralized_configuration">3.1. Centralized configuration</h3> +<h3 id="_centralized_configuration">4.1. Centralized configuration</h3> <div class="paragraph"> <p>Apache Unomi uses a centralized configuration file that contains both system properties and configuration properties. These settings are then fed to the OSGi and other configuration files using placeholder that look something like this:</p> @@ -1179,7 +1567,7 @@ the values in the <code>$MY_KARAF_HOME/e </div> </div> <div class="sect2"> -<h3 id="_changing_the_default_configuration_using_environment_variables_i_e_docker_configuration">3.2. Changing the default configuration using environment variables (i.e. Docker configuration)</h3> +<h3 id="_changing_the_default_configuration_using_environment_variables_i_e_docker_configuration">4.2. Changing the default configuration using environment variables (i.e. Docker configuration)</h3> <div class="paragraph"> <p>You might want to use environment variables to change the default system configuration, especially if you intend to run Apache Unomi inside a Docker container. You can find the list of all the environment variable names in the following file:</p> @@ -1197,7 +1585,7 @@ variables you want to use.</p> </div> </div> <div class="sect2"> -<h3 id="_changing_the_default_configuration_using_property_files">3.3. Changing the default configuration using property files</h3> +<h3 id="_changing_the_default_configuration_using_property_files">4.3. Changing the default configuration using property files</h3> <div class="paragraph"> <p>If you want to change the default configuration using property files instead of environment variables, you can perform any modification you want in the <code>$MY_KARAF_HOME/etc/unomi.custom.system.properties</code> file.</p> @@ -1244,7 +1632,7 @@ org.apache.unomi.elasticsearch.addresses </div> </div> <div class="sect2"> -<h3 id="_secured_events_configuration">3.4. Secured events configuration</h3> +<h3 id="_secured_events_configuration">4.4. Secured events configuration</h3> <div class="paragraph"> <p>Apache Unomi secures some events by default. It comes out of the box with a default configuration that you can adjust by using the centralized configuration file override in <code>$MY_KARAF_HOME/etc/unomi.custom.system.properties</code></p> @@ -1336,7 +1724,7 @@ thirdparty.provider1.allowedEvents=login </div> </div> <div class="sect2"> -<h3 id="_installing_the_maxmind_geoiplite2_ip_lookup_database">3.5. Installing the MaxMind GeoIPLite2 IP lookup database</h3> +<h3 id="_installing_the_maxmind_geoiplite2_ip_lookup_database">4.5. Installing the MaxMind GeoIPLite2 IP lookup database</h3> <div class="paragraph"> <p>Apache Unomi requires an IP database in order to resolve IP addresses to user location. The GeoLite2 database can be downloaded from MaxMind here : @@ -1347,7 +1735,7 @@ The GeoLite2 database can be downloaded </div> </div> <div class="sect2"> -<h3 id="_installing_geonames_database">3.6. Installing Geonames database</h3> +<h3 id="_installing_geonames_database">4.6. Installing Geonames database</h3> <div class="paragraph"> <p>Apache Unomi includes a geocoding service based on the geonames database ( <a href="http://www.geonames.org/";>http://www.geonames.org/</a> ). It can be used to create conditions on countries or cities.</p> @@ -1365,7 +1753,7 @@ At the end, you should have about 4 mill </div> </div> <div class="sect2"> -<h3 id="_rest_api_security">3.7. REST API Security</h3> +<h3 id="_rest_api_security">4.7. REST API Security</h3> <div class="paragraph"> <p>The Apache Unomi Context Server REST API is protected using JAAS authentication and using Basic or Digest HTTP auth. By default, the login/password for the REST API full administrative access is "karaf/karaf".</p> @@ -1398,9 +1786,9 @@ org.ops4j.pax.web.ssl.keypassword=${env: </div> </div> <div class="sect2"> -<h3 id="_scripting_security">3.8. Scripting security</h3> +<h3 id="_scripting_security">4.8. Scripting security</h3> <div class="sect3"> -<h4 id="_multi_layer_scripting_filtering_system">3.8.1. Multi-layer scripting filtering system</h4> +<h4 id="_multi_layer_scripting_filtering_system">4.8.1. Multi-layer scripting filtering system</h4> <div class="paragraph"> <p>The scripting security system is multi-layered.</p> </div> @@ -1435,7 +1823,7 @@ otherwise the rest of the filtering syst </div> </div> <div class="sect3"> -<h4 id="_scripts_and_expressions">3.8.2. Scripts and expressions</h4> +<h4 id="_scripts_and_expressions">4.8.2. Scripts and expressions</h4> <div class="paragraph"> <p>Apache Unomi allows using different types of expressions in the following subsystems:</p> </div> @@ -1581,7 +1969,7 @@ list, avoiding the possibility to provid </div> </div> <div class="sect3"> -<h4 id="_scripting_expression_filtering_configuration_parameters">3.8.3. Scripting expression filtering configuration parameters</h4> +<h4 id="_scripting_expression_filtering_configuration_parameters">4.8.3. Scripting expression filtering configuration parameters</h4> <div class="paragraph"> <p>Alongside with the allow-listing technology, there are new configuration parameters to control the security of the scripting engines:</p> </div> @@ -1622,7 +2010,7 @@ org.apache.unomi.security.personalizatio </div> </div> <div class="sect3"> -<h4 id="_groovy_actions">3.8.4. Groovy Actions</h4> +<h4 id="_groovy_actions">4.8.4. Groovy Actions</h4> <div class="paragraph"> <p>Groovy actions offer the ability to define a set of actions and action types (aka action descriptors) purely from Groovy scripts defined at runtime.</p> </div> @@ -1755,7 +2143,7 @@ org.apache.unomi.security.personalizatio <div class="listingblock"> <div class="content"> <pre class="highlight"><code class="language-groovy" data-lang="groovy">@Action(id = "helloWorldGroovyAction", - actionExecutor = "groovy:helloWorldAction") + actionExecutor = "groovy:helloWorldAction", parameters = [@Parameter(id = "location", type = "string", multivalued = false)]) def execute() { logger.info("Hello {}", action.getParameterValues().get("location")) @@ -1801,7 +2189,7 @@ def execute() { "parameterValues": { "location": "world!" }, - "type": "helloWorldAction" + "type": "helloWorldGroovyAction" } ] }'</code></pre> @@ -1834,7 +2222,7 @@ def execute() { </div> </div> <div class="sect3"> -<h4 id="_scripting_roadmap">3.8.5. Scripting roadmap</h4> +<h4 id="_scripting_roadmap">4.8.5. Scripting roadmap</h4> <div class="paragraph"> <p>Scripting will probably undergo major changes in future versions of Apache Unomi, with the likely retirement of MVEL in favor of Groovy Actions detailed above.</p> </div> @@ -1845,7 +2233,7 @@ versions will of course maintain compati </div> </div> <div class="sect2"> -<h3 id="_automatic_profile_merging">3.9. Automatic profile merging</h3> +<h3 id="_automatic_profile_merging">4.9. Automatic profile merging</h3> <div class="paragraph"> <p>Apache Unomi is capable of merging profiles based on a common property value. In order to use this, you must add the MergeProfileOnPropertyAction to a rule (such as a login rule for example), and configure it with the name @@ -1864,7 +2252,7 @@ Upon sending one of the events, all matc </div> </div> <div class="sect2"> -<h3 id="_securing_a_production_environment">3.10. Securing a production environment</h3> +<h3 id="_securing_a_production_environment">4.10. Securing a production environment</h3> <div class="paragraph"> <p>Before going live with a project, you should <em>absolutely</em> read the following section that will help you setup a proper secure environment for running your context server.</p> @@ -1974,7 +2362,7 @@ a proxy, and use it to filter any commun </div> </div> <div class="sect2"> -<h3 id="_integrating_with_an_apache_http_web_server">3.11. Integrating with an Apache HTTP web server</h3> +<h3 id="_integrating_with_an_apache_http_web_server">4.11. Integrating with an Apache HTTP web server</h3> <div class="paragraph"> <p>If you want to setup an Apache HTTP web server in from of Apache Unomi, here is an example configuration using mod_proxy.</p> @@ -2072,7 +2460,7 @@ ProxyPassReverse / http://localhost:8181 </div> </div> <div class="sect2"> -<h3 id="_changing_the_default_tracking_location">3.12. Changing the default tracking location</h3> +<h3 id="_changing_the_default_tracking_location">4.12. Changing the default tracking location</h3> <div class="paragraph"> <p>When performing localhost requests to Apache Unomi, a default location will be used to insert values into the session to make the location-based personalization still work. You can modify the default location settings using the @@ -2100,7 +2488,7 @@ org.apache.unomi.ip.default.longitude=${ </div> </div> <div class="sect2"> -<h3 id="_apache_karaf_ssh_console">3.13. Apache Karaf SSH Console</h3> +<h3 id="_apache_karaf_ssh_console">4.13. Apache Karaf SSH Console</h3> <div class="paragraph"> <p>The Apache Karaf SSH console is available inside Apache Unomi, but the port has been changed from the default value of 8101 to 8102 to avoid conflicts with other Karaf-based products. So to connect to the SSH console you should use:</p> @@ -2116,20 +2504,14 @@ shell commands in the "Shell commands" s </div> </div> <div class="sect2"> -<h3 id="_elasticsearch_authentication_and_security">3.14. ElasticSearch authentication and security</h3> -<div class="paragraph"> -<p>With ElasticSearch 7, it’s possible to secure the access to your data. (<a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.5/secure-cluster.html";>https://www.elastic.co/guide/en/elasticsearch/reference/7.5/secure-cluster.html</a>)</p> -</div> +<h3 id="_elasticsearch_authentication_and_security">4.14. ElasticSearch authentication and security</h3> <div class="paragraph"> -<p>Depending on your ElasticSearch license you may need to install Kibana and enable xpack security: <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.5/configuring-security.html";>https://www.elastic.co/guide/en/elasticsearch/reference/7.5/configuring-security.html</a></p> +<p>With ElasticSearch 7, it’s possible to secure the access to your data. (see <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.17/configuring-stack-security.html";>https://www.elastic.co/guide/en/elasticsearch/reference/7.17/configuring-stack-security.html</a> and <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.17/secure-cluster.html";>https://www.elastic.co/guide/en/elasticsearch/reference/7.17/secure-cluster.html</a>)</p> </div> <div class="sect3"> -<h4 id="_user_authentication">3.14.1. User authentication !</h4> -<div class="paragraph"> -<p>If your ElasticSearch have been configured to be only accessible by authenticated users (<a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.5/setting-up-authentication.html";>https://www.elastic.co/guide/en/elasticsearch/reference/7.5/setting-up-authentication.html</a>)</p> -</div> +<h4 id="_user_authentication">4.14.1. User authentication !</h4> <div class="paragraph"> -<p>Just edit <code>etc/org.apache.unomi.persistence.elasticsearch.cfg</code> to add the following settings:</p> +<p>If your ElasticSearch have been configured to be only accessible by authenticated users, edit <code>etc/org.apache.unomi.persistence.elasticsearch.cfg</code> to add the following settings:</p> </div> <div class="listingblock"> <div class="content"> @@ -2139,26 +2521,13 @@ password=PASSWORD</code></pre> </div> </div> <div class="sect3"> -<h4 id="_ssl_communication">3.14.2. SSL communication</h4> +<h4 id="_ssl_communication">4.14.2. SSL communication</h4> <div class="paragraph"> <p>By default Unomi will communicate with ElasticSearch using <code>http</code> but you can configure your ElasticSearch server(s) to allow encrypted request using <code>https</code>.</p> </div> <div class="paragraph"> -<p>You can follow this documentation to enable SSL on your ElasticSearch server(s):</p> -</div> -<div class="ulist"> -<ul> -<li> -<p><a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.5/ssl-tls.html";>Full documentation</a></p> -</li> -<li> -<p><a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.5/configuring-tls.html#node-certificates";>Configure certificates</a></p> -</li> -<li> -<p><a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.5/configuring-tls.html#tls-http";>Encrypt HTTP communications</a></p> -</li> -</ul> +<p>You can follow this documentation to enable SSL on your ElasticSearch server(s): <a href="https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-basic-setup-https.html";>https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-basic-setup-https.html</a></p> </div> <div class="paragraph"> <p>If your ElasticSearch is correctly configure to encrypt communications on <code>https</code>:</p> @@ -2181,16 +2550,41 @@ of the ElasticSearch server(s). But if y </div> </div> </div> +<div class="sect3"> +<h4 id="_permissions">4.14.3. Permissions</h4> +<div class="paragraph"> +<p>Apache Unomi requires a particular set of Elasticsearch permissions for its operation.</p> +</div> +<div class="paragraph"> +<p>If you are using Elasticsearch in a production environment, you will most likely need to fine tune permissions given to the user used by Unomi.</p> +</div> +<div class="paragraph"> +<p>The following permissions are required by Unomi:</p> +</div> +<div class="ulist"> +<ul> +<li> +<p>required cluster privileges: <code>manage</code> OR `al`l</p> +</li> +<li> +<p>required index privileges on unomi indices: <code>write</code>) AND (<code>manage</code>) AND (<code>read</code> OR <code>all</code></p> +</li> +</ul> +</div> +</div> </div> </div> </div> <div class="sect1"> -<h2 id="_json_schemas">4. JSON schemas</h2> +<h2 id="_json_schemas_2">5. JSON schemas</h2> <div class="sectionbody"> <div class="sect2"> -<h3 id="_introduction_2">4.1. Introduction</h3> +<h3 id="_introduction_2">5.1. Introduction</h3> +<div class="paragraph"> +<p>Introduced with Apache Unomi 2.0, JSON-Schema are used to validate data submitted through all of the public (unprotected) API endpoints.</p> +</div> <div class="sect3"> -<h4 id="_what_is_a_json_schema">4.1.1. What is a JSON Schema</h4> +<h4 id="_what_is_a_json_schema">5.1.1. What is a JSON Schema</h4> <div class="paragraph"> <p><a href="https://json-schema.org/specification.html";>JSON Schema</a> is a powerful standard for validating the structure of JSON data. Described as a JSON object, a JSON schema file contains format, types, patterns, and more. @@ -2240,7 +2634,7 @@ More details about JSON schema are avail </div> </div> <div class="sect3"> -<h4 id="_key_concepts">4.1.2. Key concepts</h4> +<h4 id="_key_concepts">5.1.2. Key concepts</h4> <div class="paragraph"> <p>This section details concepts that are important to understand in order to use JSON schema validation with Apache Unomi.</p> </div> @@ -2435,7 +2829,7 @@ When setting the <strong>unevaluatedProp </div> </div> <div class="sect3"> -<h4 id="_how_are_json_schema_used_in_unomi">4.1.3. How are JSON Schema used in Unomi</h4> +<h4 id="_how_are_json_schema_used_in_unomi">5.1.3. How are JSON Schema used in Unomi</h4> <div class="paragraph"> <p>JSON Schema is used in Unomi to validate the data coming from the two public endpoints <strong>/contextRequest</strong> and <strong>/eventCollector</strong>. Both endpoints have a custom deserializer which will begin by validating the payload of the request, then will filter invalid events present in this payload. @@ -2494,12 +2888,12 @@ Schemas persisted in Elasticsearch do no </div> </div> <div class="sect2"> -<h3 id="_json_schema_api">4.2. JSON schema API</h3> +<h3 id="_json_schema_api">5.2. JSON schema API</h3> <div class="paragraph"> <p>The JSON schema endpoints are private, so the user has to be authenticated to manage the JSON schema in Unomi.</p> </div> <div class="sect3"> -<h4 id="_list_existing_schemas">4.2.1. List existing schemas</h4> +<h4 id="_list_existing_schemas">5.2.1. List existing schemas</h4> <div class="paragraph"> <p>The REST endpoint GET <code>{{url}}/cxs/jsonSchema</code> allows to get all ids of available schemas and subschemas.</p> </div> @@ -2540,7 +2934,7 @@ Schemas persisted in Elasticsearch do no </div> </div> <div class="sect3"> -<h4 id="_read_a_schema">4.2.2. Read a schema</h4> +<h4 id="_read_a_schema">5.2.2. Read a schema</h4> <div class="paragraph"> <p>Itâs possible to get a schema by its id by calling the endpoint <code>POST {{url}}/cxs/jsonSchema/query</code> with the id of the schema in the payload of the query.</p> </div>
[... 1829 lines stripped ...]
