[
https://issues.apache.org/jira/browse/UNOMI-724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Francois Gerthoffert updated UNOMI-724:
---------------------------------------
Description:
A POC have been done to validate the usage of Rollover ES API that combine a
lifecycle policy and aliases to replace the current monthly indices rotation.
The goal behind that is to apply current ElasticSearch recommandation on
Index/Shards sizing to reduce costs related to monthly indices not being sized
correctly due to unpredictable amount of events/sessions per months.
POC PR: [https://github.com/apache/unomi/pull/567]
h2. Prerequisite before starting this ticket:
Read carefully ES documentation regarding:
* Rollover API:
[https://www.elastic.co/guide/en/elasticsearch/reference/7.17/ilm-rollover.html]
h2. What remain to be done in current ticket ?
* Cleanup the POC
** cleanup used old functions from previous implem
** deprecate old param not used like: *dateHint*
** global code review and cleanup of the new implem (if necessary contact me
directly to discuss improvement point)
* make the rollover configurable using configuration file ({*}max_age,
max_size, max_primary_shard_size, max_docs.{*} See documentations regarding the
rollover capabilities:
[https://www.elastic.co/guide/en/elasticsearch/reference/7.17/ilm-rollover.html]
)
* Disable/quarantine tests associated with the purge, since these tests will
be broken as a result of this story
Out of current ticket scope:
- check performance for session loaded using query:
https://issues.apache.org/jira/browse/UNOMI-725
- fixing the purge system: https://issues.apache.org/jira/browse/UNOMI-726
- fixing the merge system: https://issues.apache.org/jira/browse/UNOMI-727
- handle migration: https://issues.apache.org/jira/browse/UNOMI-728
h2. Automated test
Query Elasticsearch to verify lifecycle policy once the configuration has been
applied in Unomi (this way we do not test Elasticsearch rollover mechanism but
ensure the configuration is properly applied).
h2. Manual Test Case:
- Pick one of the configuration (such as max_docs), configure Unomi with a
very low value
- Create anonymous session
- Verify rotation is happening when sessions are created and get above the
configured threshold.
was:
A POC have been done to validate the usage of Rollover ES API that combine a
lifecycle policy and aliases to replace the current monthly indices rotation.
The goal behind that is to apply current ElasticSearch recommandation on
Index/Shards sizing to reduce costs related to monthly indices not being sized
correctly due to unpredictable amount of events/sessions per months.
POC PR: [https://github.com/apache/unomi/pull/567]
h2. Prerequisite before starting this ticket:
Read carefully ES documentation regarding:
* Rollover API:
[https://www.elastic.co/guide/en/elasticsearch/reference/7.17/ilm-rollover.html]
h2. What remain to be done in current ticket ?
* Cleanup the POC
** cleanup used old functions from previous implem
** deprecate old param not used like: *dateHint*
** global code review and cleanup of the new implem (if necessary contact me
directly to discuss improvement point)
* make the rollover configurable using configuration file ({*}max_age,
max_size, max_primary_shard_size, max_docs.{*} See documentations regarding the
rollover capabilities:
[https://www.elastic.co/guide/en/elasticsearch/reference/7.17/ilm-rollover.html]
)
Out of current ticket scope:
- check performance for session loaded using query:
https://issues.apache.org/jira/browse/UNOMI-725
- fixing the purge system: https://issues.apache.org/jira/browse/UNOMI-726
- fixing the merge system: https://issues.apache.org/jira/browse/UNOMI-727
- handle migration: https://issues.apache.org/jira/browse/UNOMI-728
h2. Automated test
Query Elasticsearch to verify lifecycle policy once the configuration has been
applied in Unomi (this way we do not test Elasticsearch rollover mechanism but
ensure the configuration is properly applied).
h2. Manual Test Case:
- Pick one of the configuration (such as max_docs), configure Unomi with a
very low value
- Create anonymous session
- Verify rotation is happening when sessions are created and get above the
configured threshold.
> Session/Event index rollover: Implement ElasticSearch rollover recommandation
> and system to manage current monthly indices
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: UNOMI-724
> URL: https://issues.apache.org/jira/browse/UNOMI-724
> Project: Apache Unomi
> Issue Type: Improvement
> Affects Versions: unomi-2.1.0
> Reporter: Kevan Jahanshahi
> Priority: Major
> Fix For: unomi-2.2.0
>
>
> A POC have been done to validate the usage of Rollover ES API that combine a
> lifecycle policy and aliases to replace the current monthly indices rotation.
> The goal behind that is to apply current ElasticSearch recommandation on
> Index/Shards sizing to reduce costs related to monthly indices not being
> sized correctly due to unpredictable amount of events/sessions per months.
> POC PR: [https://github.com/apache/unomi/pull/567]
> h2. Prerequisite before starting this ticket:
> Read carefully ES documentation regarding:
> * Rollover API:
> [https://www.elastic.co/guide/en/elasticsearch/reference/7.17/ilm-rollover.html]
>
> h2. What remain to be done in current ticket ?
> * Cleanup the POC
> ** cleanup used old functions from previous implem
> ** deprecate old param not used like: *dateHint*
> ** global code review and cleanup of the new implem (if necessary contact me
> directly to discuss improvement point)
> * make the rollover configurable using configuration file ({*}max_age,
> max_size, max_primary_shard_size, max_docs.{*} See documentations regarding
> the rollover capabilities:
> [https://www.elastic.co/guide/en/elasticsearch/reference/7.17/ilm-rollover.html]
> )
> * Disable/quarantine tests associated with the purge, since these tests will
> be broken as a result of this story
> Out of current ticket scope:
> - check performance for session loaded using query:
> https://issues.apache.org/jira/browse/UNOMI-725
> - fixing the purge system: https://issues.apache.org/jira/browse/UNOMI-726
> - fixing the merge system: https://issues.apache.org/jira/browse/UNOMI-727
> - handle migration: https://issues.apache.org/jira/browse/UNOMI-728
> h2. Automated test
> Query Elasticsearch to verify lifecycle policy once the configuration has
> been applied in Unomi (this way we do not test Elasticsearch rollover
> mechanism but ensure the configuration is properly applied).
> h2. Manual Test Case:
> - Pick one of the configuration (such as max_docs), configure Unomi with a
> very low value
> - Create anonymous session
> - Verify rotation is happening when sessions are created and get above the
> configured threshold.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
