[
https://issues.apache.org/jira/browse/UNOMI-670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17607076#comment-17607076
]
Kevan Jahanshahi commented on UNOMI-670:
----------------------------------------
Workaround advised: in the meantime the purge system fixes, if you encounter
any issue you can:
h2. Disable the purge on Unomi servers:
Set this conf on the Unomi servers:
{code:java}
org.apache.unomi.profile.purge.existTime = -1
org.apache.unomi.profile.purge.inactiveTime = -1
org.apache.unomi.monthly.index.purge.existTime = -1 {code}
Or use the env variable:
{code:java}
export UNOMI_PROFILE_PURGE_INACTIVETIME=-1
export UNOMI_PROFILE_PURGE_EXISTTIME=-1
export UNOMI_MONTHLY_INDEX_PURGE_EXISTTIME=-1{code}
*A restart of the server is required for the conf to be applied (sorry about
that ...)*
{color:#de350b}*Be careful, it's important to disable all this 3 properties at
once, there is a bug when you only disable the profiles purge and want to keep
the monthly index purge, that will erase all existing profiles every day.*
{color}
h2. *Drawback:*
Purge system will be disable completely meaining, no profile, no events, no
sessions will be purge no matter how old they are.
But this should be ok, as we will provide a fix in a next version of Unomi and
the purge could be re-enabled at that moment, and will take care of cleaning
potential documents that need to be.
Also it's still possible to purge manually the documents from ES if it's
require.
h2. Manual purge (optional, but in case you need it, here is the queries)
For monthly indices (event/session): just delete the indices that are older
than 12 month:
[https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-delete-index.html]
For profiles: do a delete by query on ES:
[https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete-by-query.html]
and the query would be:
{code:java}
{
"bool" : {
"must" : [
{
"match_all" : {
"boost" : 1.0
}
}
],
"filter" : [
{
"range" : {
"properties.lastVisit" : {
"from" : null,
"to" : "now-180d",
"include_lower" : true,
"include_upper" : true,
"boost" : 1.0
}
}
}
],
"adjust_pure_negative" : true,
"boost" : 1.0
}
} {code}
> Unomi purge system is not working
> ---------------------------------
>
> Key: UNOMI-670
> URL: https://issues.apache.org/jira/browse/UNOMI-670
> Project: Apache Unomi
> Issue Type: Bug
> Components: unomi(-core)
> Affects Versions: unomi-2.0.0, unomi-1.6.1
> Reporter: Kevan Jahanshahi
> Priority: Major
>
> h2. Bug 1: disabling the profile purge will end up with all profiles to be
> purge everyday
> Using this conf:
> {code:java}
> org.apache.unomi.profile.purge.existTime = -1
> org.apache.unomi.profile.purge.inactiveTime = -1
> org.apache.unomi.monthly.index.purge.existTime= 12 {code}
> To only enable events/session purge will have a bad side effect: all profiles
> of the system will be removed everyday.
> h2. bug 2: OOM issue for profile purge.
> The persistence service is implementing a method:
> {code:java}
> /**
> * Deletes items with the specified Item subclass matching the specified
> {@link Condition}.
> *
> * @param <T> the type of the Item subclass we want to delete
> * @param query a {@link Condition} identifying which elements we want to
> delete
> * @param clazz the {@link Item} subclass of the items we want to delete
> * @return {@code true} if the deletion was successful, {@code false}
> otherwise
> */
> <T extends Item> boolean removeByQuery(Condition query, Class<T> clazz);
> {code}
> The issue with the implem is that it's doing the following:
> * query profiles to get the IDs of documents
> * building a batch request with a delete request per profile ID
> * sending the batch.
> So in case of a lot profiles to be purge at once, the batch can grow a lot,
> and consume all the memory available.
> We should not do like that, instead we should:
> * just use the ES query:
> [https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete-by-query.html]
> directly with the query to identify the profiles
> h2. Bug 3: What is the cache system implemented in the persistence service.
> Seem's coming from: https://issues.apache.org/jira/browse/UNOMI-166
> It's used by the persistence service, and it explain why the removeByQuery
> implem is bad.
> The reason why the *removeByQuery* is querying all the documents IDs is to
> invalidate the cache.
> But it's strange, the cache system is not used at all:
> {code:java}
> <cm:property name="itemClassesToCache" value="" /> {code}
> So I would say, we should remove this cache system, because:
> * it's not tested
> * nobody know what would be the effect if enabled
> * it's not possible to activate it with ENV conf, or global unomi conf.
> * it's not documented
> * it's impacting implems like: removeByQuery, even if the cache system is
> not used
> * it's mostly dead code inside the core persistence of Unomi, witch is
> dangerous to keep.
> h2. Bug 4: Purge job seem's never unregistered.
> We do register the purge job when the profile service is started, but in case
> the profile service is stopped (stop the bundle for example). Then it's not
> unregistered.
> h2. Bug 5: what should we do in case of unomi cluster, should all of them do
> the purge ?
> In case of unomi cluster, all of them may try to trigger the purge, I dont
> think it's an issue, but it look like something could be done to avoid doing
> unecessary operations twice or more.
> h2. Final: We should also add integrations tests to cover this feature.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
