[
https://issues.apache.org/jira/browse/UNOMI-764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kevan Jahanshahi updated UNOMI-764:
-----------------------------------
Description:
We experienced issues with profile merge action with errors like:
{code:java}
java.lang.Exception: Error loading itemType=org.apache.unomi.api.Profile
itemId=a9abefb6-8e21-4a0b-9a38-92c8e2fcf512
at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$6.execute(ElasticSearchPersistenceServiceImpl.java:772)
~[?:?]
at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$6.execute(ElasticSearchPersistenceServiceImpl.java:740)
~[?:?]
at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$InClassLoaderExecute.executeInClassLoader(ElasticSearchPersistenceServiceImpl.java:2233)
~[?:?]
at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$InClassLoaderExecute.catchingExecuteInClassLoader(ElasticSearchPersistenceServiceImpl.java:2244)
~[?:?]
at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl.load(ElasticSearchPersistenceServiceImpl.java:780)
~[?:?]
at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl.load(ElasticSearchPersistenceServiceImpl.java:735)
~[?:?]
at Proxy6d4b2959_6bef_4c34_bb5a_88734cd23613.load(Unknown Source) ~[?:?]
at
org.apache.unomi.plugins.baseplugin.actions.MergeProfilesOnPropertyAction$1.execute(MergeProfilesOnPropertyAction.java:226)
~[?:?]
at
org.apache.unomi.services.impl.events.EventServiceImpl.send(EventServiceImpl.java:175)
~[?:?]
at
org.apache.unomi.services.impl.events.EventServiceImpl.send(EventServiceImpl.java:144)
~[?:?]
at
org.apache.unomi.rest.service.impl.RestServiceUtilsImpl.handleEvents(RestServiceUtilsImpl.java:112)
~[?:?]
at
org.apache.unomi.rest.endpoints.ContextJsonEndpoint.handleRequest(ContextJsonEndpoint.java:364)
~[?:?]
at
org.apache.unomi.rest.endpoints.ContextJsonEndpoint.contextJSONAsPost(ContextJsonEndpoint.java:311)
~[?:?]{code}
{code:java}
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch
exception [type=circuit_breaking_exception, reason=[parent] Data too large,
data for [indices:data/read/get[s]] would be [8308883688/7.7gb], which is
larger than the limit of [8160437862/7.5gb], real usage: [8308883456/7.7gb],
new bytes reserved: [232/232b], usages [request=0/0b, fielddata=36816/35.9kb,
in_flight_requests=232/232b, model_inference=0/0b, eql_sequence=0/0b,
accounting=106608640/101.6mb]] at
org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177)
~[?:?] at
org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1727)
~[?:?] at
org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1704)
~[?:?] at
org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1467)
~[?:?] at
org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1424)
~[?:?] at
org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1394)
~[?:?] at
org.elasticsearch.client.RestHighLevelClient.get(RestHighLevelClient.java:714)
~[?:?] at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$6.execute(ElasticSearchPersistenceServiceImpl.java:757)
~[?:?] ... 99 more Suppressed: org.elasticsearch.client.ResponseException:
method [GET], host [https://cdp-data.societe.mma.fr:9220], URI
[/context-profile/_doc/a9abefb6-8e21-4a0b-9a38-92c8e2fcf512], status line
[HTTP/1.1 429 Too Many Requests]
{"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[parent]
Data too large, data for [indices:data/read/get[s]] would be
[8308883688/7.7gb], which is larger than the limit of [8160437862/7.5gb], real
usage: [8308883456/7.7gb], new bytes reserved: [232/232b], usages
[request=0/0b, fielddata=36816/35.9kb, in_flight_requests=232/232b,
model_inference=0/0b, eql_sequence=0/0b,
accounting=106608640/101.6mb]","bytes_wanted":8308883688,"bytes_limit":8160437862,"durability":"PERMANENT"}],"type":"circuit_breaking_exception","reason":"[parent]
Data too large, data for [indices:data/read/get[s]] would be
[8308883688/7.7gb], which is larger than the limit of [8160437862/7.5gb], real
usage: [8308883456/7.7gb], new bytes reserved: [232/232b], usages
[request=0/0b, fielddata=36816/35.9kb, in_flight_requests=232/232b,
model_inference=0/0b, eql_sequence=0/0b,
accounting=106608640/101.6mb]","bytes_wanted":8308883688,"bytes_limit":8160437862,"durability":"PERMANENT"},"status":429}
at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:253)
~[?:?] at
org.elasticsearch.client.RestClient.performRequest(RestClient.java:231) ~[?:?]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:205)
~[?:?] at
org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1454)
~[?:?] at
org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1424)
~[?:?] at
org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1394)
~[?:?] at
org.elasticsearch.client.RestHighLevelClient.get(RestHighLevelClient.java:714)
~[?:?] at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$6.execute(ElasticSearchPersistenceServiceImpl.java:757)
~[?:?] at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$6.execute(ElasticSearchPersistenceServiceImpl.java:740)
~[?:?] at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$InClassLoaderExecute.executeInClassLoader(ElasticSearchPersistenceServiceImpl.java:2233)
~[?:?] at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$InClassLoaderExecute.catchingExecuteInClassLoader(ElasticSearchPersistenceServiceImpl.java:2244)
~[?:?] at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl.load(ElasticSearchPersistenceServiceImpl.java:780)
~[?:?] at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl.load(ElasticSearchPersistenceServiceImpl.java:735)
~[?:?] at Proxy6d4b2959_6bef_4c34_bb5a_88734cd23613.load(Unknown Source)
~[?:?] at
org.apache.unomi.plugins.baseplugin.actions.MergeProfilesOnPropertyAction$1.execute(MergeProfilesOnPropertyAction.java:226)
~[?:?] at
org.apache.unomi.services.impl.events.EventServiceImpl.send(EventServiceImpl.java:175)
~[?:?] at
org.apache.unomi.services.impl.events.EventServiceImpl.send(EventServiceImpl.java:144)
~[?:?] at
org.apache.unomi.rest.service.impl.RestServiceUtilsImpl.handleEvents(RestServiceUtilsImpl.java:112)
~[?:?] at
org.apache.unomi.rest.endpoints.ContextJsonEndpoint.handleRequest(ContextJsonEndpoint.java:364)
~[?:?] at
org.apache.unomi.rest.endpoints.ContextJsonEndpoint.contextJSONAsPost(ContextJsonEndpoint.java:311)
~[?:?]
{code}
Or:
{code:java}
Error while executing in class loader
java.lang.Exception: Error loading itemType=org.apache.unomi.api.Profile query={
"bool" : {
"must" : [
{
"match_all" : {
"boost" : 1.0
}
}
],
"filter" : [
{
"bool" : {
"must" : [
{
"term" : {
"systemProperties.coveaIdentifier" : {
"value" : "d4e55ba9b1004d49e0548ac200e561ac-mma",
"boost" : 1.0
}
}
},
{
"bool" : {
"must_not" : [
{
"exists" : {
"field" : "mergedWith",
"boost" : 1.0
}
}
],
"adjust_pure_negative" : true,
"boost" : 1.0
}
}
],
"adjust_pure_negative" : true,
"boost" : 1.0
}
}
],
"adjust_pure_negative" : true,
"boost" : 1.0
}
} sortBy=properties.firstVisit
at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$24.execute(ElasticSearchPersistenceServiceImpl.java:1757)
~[?:?]
at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$24.execute(ElasticSearchPersistenceServiceImpl.java:1650)
~[?:?]
at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$InClassLoaderExecute.executeInClassLoader(ElasticSearchPersistenceServiceImpl.java:2233)
~[?:?]
at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$InClassLoaderExecute.catchingExecuteInClassLoader(ElasticSearchPersistenceServiceImpl.java:2244)
~[?:?]
at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl.query(ElasticSearchPersistenceServiceImpl.java:1767)
~[?:?]
at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl.query(ElasticSearchPersistenceServiceImpl.java:1570)
~[?:?]
at Proxy6d4b2959_6bef_4c34_bb5a_88734cd23613.query(Unknown Source)
~[?:?]
at
org.apache.unomi.plugins.baseplugin.actions.MergeProfilesOnPropertyAction.execute(MergeProfilesOnPropertyAction.java:94)
~[?:?]
at
org.apache.unomi.services.actions.impl.ActionExecutorDispatcherImpl$8.execute(ActionExecutorDispatcherImpl.java:227)
~[?:?]
at
org.apache.unomi.services.actions.impl.ActionExecutorDispatcherImpl$8.execute(ActionExecutorDispatcherImpl.java:224)
~[?:?]
at
org.apache.unomi.metrics.MetricAdapter.runWithTimer(MetricAdapter.java:38)
~[?:?]
at
org.apache.unomi.services.actions.impl.ActionExecutorDispatcherImpl.execute(ActionExecutorDispatcherImpl.java:229)
~[?:?]
at
org.apache.unomi.services.impl.rules.RulesServiceImpl.onEvent(RulesServiceImpl.java:327)
~[?:?]
at
org.apache.unomi.services.impl.events.EventServiceImpl.send(EventServiceImpl.java:170)
~[?:?]
at
org.apache.unomi.services.impl.events.EventServiceImpl.send(EventServiceImpl.java:144)
~[?:?]
at
org.apache.unomi.rest.service.impl.RestServiceUtilsImpl.handleEvents(RestServiceUtilsImpl.java:112)
~[?:?]
at
org.apache.unomi.rest.endpoints.ContextJsonEndpoint.handleRequest(ContextJsonEndpoint.java:364)
~[?:?]
at
org.apache.unomi.rest.endpoints.ContextJsonEndpoint.contextJSONAsPost(ContextJsonEndpoint.java:311)
~[?:?] {code}
The merge is doing scroll queries from code:
{code:java}
List<Profile> profiles = persistenceService.query(c, "properties.firstVisit",
Profile.class, 0, maxProfilesInOneMerge).getList(); {code}
Where *maxProfilesInOneMerge* is by default set to {*}-1{*}, in persistence
service when doing a query using size {*}-1{*}:
{code:java}
if (size == Integer.MIN_VALUE) {
searchSourceBuilder.size(defaultQueryLimit);
} else if (size != -1) {
searchSourceBuilder.size(size);
} else {
// size == -1, use scroll query to retrieve all the results
searchRequest.scroll(keepAlive);
} {code}
With a *keepAlive* of 1hour by default. So that can be a lot, in case of lot
merge requests happen in a short period of time, the scroll will accumulate in
RAM.
The workaround is to set the *maxProfilesInOneMerge* to something else than
*-1* to avoid scroll queries.
For that we can use this configuration:
* Unomi config: *org.apache.unomi.plugins.base.maxProfilesInOneMerge*
* Env config: *UNOMI_MAX_PROFILES_IN_ONE_MERGE*
A value of *10* seem's good as we never expect a lot of profiles to be merged
together.
*But for the current ticket we should fix the default value and we should never
do scroll queries at all at that place. Creating a new scroll on every request
is a bad idea.*
was:
We experienced issues with profile merge action with errors like:
{code:java}
java.lang.Exception: Error loading itemType=org.apache.unomi.api.Profile
itemId=a9abefb6-8e21-4a0b-9a38-92c8e2fcf512
at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$6.execute(ElasticSearchPersistenceServiceImpl.java:772)
~[?:?]
at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$6.execute(ElasticSearchPersistenceServiceImpl.java:740)
~[?:?]
at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$InClassLoaderExecute.executeInClassLoader(ElasticSearchPersistenceServiceImpl.java:2233)
~[?:?]
at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$InClassLoaderExecute.catchingExecuteInClassLoader(ElasticSearchPersistenceServiceImpl.java:2244)
~[?:?]
at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl.load(ElasticSearchPersistenceServiceImpl.java:780)
~[?:?]
at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl.load(ElasticSearchPersistenceServiceImpl.java:735)
~[?:?]
at Proxy6d4b2959_6bef_4c34_bb5a_88734cd23613.load(Unknown Source) ~[?:?]
at
org.apache.unomi.plugins.baseplugin.actions.MergeProfilesOnPropertyAction$1.execute(MergeProfilesOnPropertyAction.java:226)
~[?:?]
at
org.apache.unomi.services.impl.events.EventServiceImpl.send(EventServiceImpl.java:175)
~[?:?]
at
org.apache.unomi.services.impl.events.EventServiceImpl.send(EventServiceImpl.java:144)
~[?:?]
at
org.apache.unomi.rest.service.impl.RestServiceUtilsImpl.handleEvents(RestServiceUtilsImpl.java:112)
~[?:?]
at
org.apache.unomi.rest.endpoints.ContextJsonEndpoint.handleRequest(ContextJsonEndpoint.java:364)
~[?:?]
at
org.apache.unomi.rest.endpoints.ContextJsonEndpoint.contextJSONAsPost(ContextJsonEndpoint.java:311)
~[?:?]{code}
{code:java}
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch
exception [type=circuit_breaking_exception, reason=[parent] Data too large,
data for [indices:data/read/get[s]] would be [8308883688/7.7gb], which is
larger than the limit of [8160437862/7.5gb], real usage: [8308883456/7.7gb],
new bytes reserved: [232/232b], usages [request=0/0b, fielddata=36816/35.9kb,
in_flight_requests=232/232b, model_inference=0/0b, eql_sequence=0/0b,
accounting=106608640/101.6mb]] at
org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177)
~[?:?] at
org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1727)
~[?:?] at
org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1704)
~[?:?] at
org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1467)
~[?:?] at
org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1424)
~[?:?] at
org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1394)
~[?:?] at
org.elasticsearch.client.RestHighLevelClient.get(RestHighLevelClient.java:714)
~[?:?] at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$6.execute(ElasticSearchPersistenceServiceImpl.java:757)
~[?:?] ... 99 more Suppressed: org.elasticsearch.client.ResponseException:
method [GET], host [https://cdp-data.societe.mma.fr:9220], URI
[/context-profile/_doc/a9abefb6-8e21-4a0b-9a38-92c8e2fcf512], status line
[HTTP/1.1 429 Too Many Requests]
{"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[parent]
Data too large, data for [indices:data/read/get[s]] would be
[8308883688/7.7gb], which is larger than the limit of [8160437862/7.5gb], real
usage: [8308883456/7.7gb], new bytes reserved: [232/232b], usages
[request=0/0b, fielddata=36816/35.9kb, in_flight_requests=232/232b,
model_inference=0/0b, eql_sequence=0/0b,
accounting=106608640/101.6mb]","bytes_wanted":8308883688,"bytes_limit":8160437862,"durability":"PERMANENT"}],"type":"circuit_breaking_exception","reason":"[parent]
Data too large, data for [indices:data/read/get[s]] would be
[8308883688/7.7gb], which is larger than the limit of [8160437862/7.5gb], real
usage: [8308883456/7.7gb], new bytes reserved: [232/232b], usages
[request=0/0b, fielddata=36816/35.9kb, in_flight_requests=232/232b,
model_inference=0/0b, eql_sequence=0/0b,
accounting=106608640/101.6mb]","bytes_wanted":8308883688,"bytes_limit":8160437862,"durability":"PERMANENT"},"status":429}
at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:253)
~[?:?] at
org.elasticsearch.client.RestClient.performRequest(RestClient.java:231) ~[?:?]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:205)
~[?:?] at
org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1454)
~[?:?] at
org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1424)
~[?:?] at
org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1394)
~[?:?] at
org.elasticsearch.client.RestHighLevelClient.get(RestHighLevelClient.java:714)
~[?:?] at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$6.execute(ElasticSearchPersistenceServiceImpl.java:757)
~[?:?] at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$6.execute(ElasticSearchPersistenceServiceImpl.java:740)
~[?:?] at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$InClassLoaderExecute.executeInClassLoader(ElasticSearchPersistenceServiceImpl.java:2233)
~[?:?] at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$InClassLoaderExecute.catchingExecuteInClassLoader(ElasticSearchPersistenceServiceImpl.java:2244)
~[?:?] at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl.load(ElasticSearchPersistenceServiceImpl.java:780)
~[?:?] at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl.load(ElasticSearchPersistenceServiceImpl.java:735)
~[?:?] at Proxy6d4b2959_6bef_4c34_bb5a_88734cd23613.load(Unknown Source)
~[?:?] at
org.apache.unomi.plugins.baseplugin.actions.MergeProfilesOnPropertyAction$1.execute(MergeProfilesOnPropertyAction.java:226)
~[?:?] at
org.apache.unomi.services.impl.events.EventServiceImpl.send(EventServiceImpl.java:175)
~[?:?] at
org.apache.unomi.services.impl.events.EventServiceImpl.send(EventServiceImpl.java:144)
~[?:?] at
org.apache.unomi.rest.service.impl.RestServiceUtilsImpl.handleEvents(RestServiceUtilsImpl.java:112)
~[?:?] at
org.apache.unomi.rest.endpoints.ContextJsonEndpoint.handleRequest(ContextJsonEndpoint.java:364)
~[?:?] at
org.apache.unomi.rest.endpoints.ContextJsonEndpoint.contextJSONAsPost(ContextJsonEndpoint.java:311)
~[?:?]
{code}
Or:
{code:java}
Error while executing in class loader
java.lang.Exception: Error loading itemType=org.apache.unomi.api.Profile query={
"bool" : {
"must" : [
{
"match_all" : {
"boost" : 1.0
}
}
],
"filter" : [
{
"bool" : {
"must" : [
{
"term" : {
"systemProperties.coveaIdentifier" : {
"value" : "d4e55ba9b1004d49e0548ac200e561ac-mma",
"boost" : 1.0
}
}
},
{
"bool" : {
"must_not" : [
{
"exists" : {
"field" : "mergedWith",
"boost" : 1.0
}
}
],
"adjust_pure_negative" : true,
"boost" : 1.0
}
}
],
"adjust_pure_negative" : true,
"boost" : 1.0
}
}
],
"adjust_pure_negative" : true,
"boost" : 1.0
}
} sortBy=properties.firstVisit
at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$24.execute(ElasticSearchPersistenceServiceImpl.java:1757)
~[?:?]
at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$24.execute(ElasticSearchPersistenceServiceImpl.java:1650)
~[?:?]
at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$InClassLoaderExecute.executeInClassLoader(ElasticSearchPersistenceServiceImpl.java:2233)
~[?:?]
at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$InClassLoaderExecute.catchingExecuteInClassLoader(ElasticSearchPersistenceServiceImpl.java:2244)
~[?:?]
at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl.query(ElasticSearchPersistenceServiceImpl.java:1767)
~[?:?]
at
org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl.query(ElasticSearchPersistenceServiceImpl.java:1570)
~[?:?]
at Proxy6d4b2959_6bef_4c34_bb5a_88734cd23613.query(Unknown Source)
~[?:?]
at
org.apache.unomi.plugins.baseplugin.actions.MergeProfilesOnPropertyAction.execute(MergeProfilesOnPropertyAction.java:94)
~[?:?]
at
org.apache.unomi.services.actions.impl.ActionExecutorDispatcherImpl$8.execute(ActionExecutorDispatcherImpl.java:227)
~[?:?]
at
org.apache.unomi.services.actions.impl.ActionExecutorDispatcherImpl$8.execute(ActionExecutorDispatcherImpl.java:224)
~[?:?]
at
org.apache.unomi.metrics.MetricAdapter.runWithTimer(MetricAdapter.java:38)
~[?:?]
at
org.apache.unomi.services.actions.impl.ActionExecutorDispatcherImpl.execute(ActionExecutorDispatcherImpl.java:229)
~[?:?]
at
org.apache.unomi.services.impl.rules.RulesServiceImpl.onEvent(RulesServiceImpl.java:327)
~[?:?]
at
org.apache.unomi.services.impl.events.EventServiceImpl.send(EventServiceImpl.java:170)
~[?:?]
at
org.apache.unomi.services.impl.events.EventServiceImpl.send(EventServiceImpl.java:144)
~[?:?]
at
org.apache.unomi.rest.service.impl.RestServiceUtilsImpl.handleEvents(RestServiceUtilsImpl.java:112)
~[?:?]
at
org.apache.unomi.rest.endpoints.ContextJsonEndpoint.handleRequest(ContextJsonEndpoint.java:364)
~[?:?]
at
org.apache.unomi.rest.endpoints.ContextJsonEndpoint.contextJSONAsPost(ContextJsonEndpoint.java:311)
~[?:?] {code}
The merge is doing scroll queries from code:
* nal List<Profile> profiles = persistenceService.query(c,
"properties.firstVisit", Profile.class, 0, maxProfilesInOneMerge).getList();
Where *maxProfilesInOneMerge* is by default set to {*}-1{*}, in persistence
service when doing a query using size {*}-1{*}:
{code:java}
if (size == Integer.MIN_VALUE) {
searchSourceBuilder.size(defaultQueryLimit);
} else if (size != -1) {
searchSourceBuilder.size(size);
} else {
// size == -1, use scroll query to retrieve all the results
searchRequest.scroll(keepAlive);
} {code}
With a keepAlive of 1hour by default. So that can be a lot, in case of lot
merge requests happen in a short period of time, the scroll will accumulate in
RAM.
The workaround is to set the *maxProfilesInOneMerge* to something else than
*-1* to avoid scroll queries.
For that we can use this configuration:
* Unomi config: *org.apache.unomi.plugins.base.maxProfilesInOneMerge*
* Env config: *UNOMI_MAX_PROFILES_IN_ONE_MERGE*
A value of *10* seem's good as we never expect a lot of profiles to be merged
together.
*But for the current ticket we should fix the default value and we should never
do scroll queries at all at that place. Creating a new scroll on every request
is a bad idea.*
> Profile merge system is doing a lot of scroll queries
> -----------------------------------------------------
>
> Key: UNOMI-764
> URL: https://issues.apache.org/jira/browse/UNOMI-764
> Project: Apache Unomi
> Issue Type: Bug
> Affects Versions: unomi-2.2.0, unomi-1.8.0
> Reporter: Kevan Jahanshahi
> Priority: Major
> Fix For: unomi-2.3.0, unomi-1.9.0
>
>
> We experienced issues with profile merge action with errors like:
> {code:java}
> java.lang.Exception: Error loading itemType=org.apache.unomi.api.Profile
> itemId=a9abefb6-8e21-4a0b-9a38-92c8e2fcf512
> at
> org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$6.execute(ElasticSearchPersistenceServiceImpl.java:772)
> ~[?:?]
> at
> org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$6.execute(ElasticSearchPersistenceServiceImpl.java:740)
> ~[?:?]
> at
> org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$InClassLoaderExecute.executeInClassLoader(ElasticSearchPersistenceServiceImpl.java:2233)
> ~[?:?]
> at
> org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$InClassLoaderExecute.catchingExecuteInClassLoader(ElasticSearchPersistenceServiceImpl.java:2244)
> ~[?:?]
> at
> org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl.load(ElasticSearchPersistenceServiceImpl.java:780)
> ~[?:?]
> at
> org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl.load(ElasticSearchPersistenceServiceImpl.java:735)
> ~[?:?]
> at Proxy6d4b2959_6bef_4c34_bb5a_88734cd23613.load(Unknown Source) ~[?:?]
> at
> org.apache.unomi.plugins.baseplugin.actions.MergeProfilesOnPropertyAction$1.execute(MergeProfilesOnPropertyAction.java:226)
> ~[?:?]
> at
> org.apache.unomi.services.impl.events.EventServiceImpl.send(EventServiceImpl.java:175)
> ~[?:?]
> at
> org.apache.unomi.services.impl.events.EventServiceImpl.send(EventServiceImpl.java:144)
> ~[?:?]
> at
> org.apache.unomi.rest.service.impl.RestServiceUtilsImpl.handleEvents(RestServiceUtilsImpl.java:112)
> ~[?:?]
> at
> org.apache.unomi.rest.endpoints.ContextJsonEndpoint.handleRequest(ContextJsonEndpoint.java:364)
> ~[?:?]
> at
> org.apache.unomi.rest.endpoints.ContextJsonEndpoint.contextJSONAsPost(ContextJsonEndpoint.java:311)
> ~[?:?]{code}
> {code:java}
> Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch
> exception [type=circuit_breaking_exception, reason=[parent] Data too large,
> data for [indices:data/read/get[s]] would be [8308883688/7.7gb], which is
> larger than the limit of [8160437862/7.5gb], real usage: [8308883456/7.7gb],
> new bytes reserved: [232/232b], usages [request=0/0b, fielddata=36816/35.9kb,
> in_flight_requests=232/232b, model_inference=0/0b, eql_sequence=0/0b,
> accounting=106608640/101.6mb]] at
> org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177)
> ~[?:?] at
> org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1727)
> ~[?:?] at
> org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1704)
> ~[?:?] at
> org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1467)
> ~[?:?] at
> org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1424)
> ~[?:?] at
> org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1394)
> ~[?:?] at
> org.elasticsearch.client.RestHighLevelClient.get(RestHighLevelClient.java:714)
> ~[?:?] at
> org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$6.execute(ElasticSearchPersistenceServiceImpl.java:757)
> ~[?:?] ... 99 more Suppressed: org.elasticsearch.client.ResponseException:
> method [GET], host [https://cdp-data.societe.mma.fr:9220], URI
> [/context-profile/_doc/a9abefb6-8e21-4a0b-9a38-92c8e2fcf512], status line
> [HTTP/1.1 429 Too Many Requests]
> {"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[parent]
> Data too large, data for [indices:data/read/get[s]] would be
> [8308883688/7.7gb], which is larger than the limit of [8160437862/7.5gb],
> real usage: [8308883456/7.7gb], new bytes reserved: [232/232b], usages
> [request=0/0b, fielddata=36816/35.9kb, in_flight_requests=232/232b,
> model_inference=0/0b, eql_sequence=0/0b,
> accounting=106608640/101.6mb]","bytes_wanted":8308883688,"bytes_limit":8160437862,"durability":"PERMANENT"}],"type":"circuit_breaking_exception","reason":"[parent]
> Data too large, data for [indices:data/read/get[s]] would be
> [8308883688/7.7gb], which is larger than the limit of [8160437862/7.5gb],
> real usage: [8308883456/7.7gb], new bytes reserved: [232/232b], usages
> [request=0/0b, fielddata=36816/35.9kb, in_flight_requests=232/232b,
> model_inference=0/0b, eql_sequence=0/0b,
> accounting=106608640/101.6mb]","bytes_wanted":8308883688,"bytes_limit":8160437862,"durability":"PERMANENT"},"status":429}
> at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:253)
> ~[?:?] at
> org.elasticsearch.client.RestClient.performRequest(RestClient.java:231)
> ~[?:?] at
> org.elasticsearch.client.RestClient.performRequest(RestClient.java:205)
> ~[?:?] at
> org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1454)
> ~[?:?] at
> org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1424)
> ~[?:?] at
> org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1394)
> ~[?:?] at
> org.elasticsearch.client.RestHighLevelClient.get(RestHighLevelClient.java:714)
> ~[?:?] at
> org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$6.execute(ElasticSearchPersistenceServiceImpl.java:757)
> ~[?:?] at
> org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$6.execute(ElasticSearchPersistenceServiceImpl.java:740)
> ~[?:?] at
> org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$InClassLoaderExecute.executeInClassLoader(ElasticSearchPersistenceServiceImpl.java:2233)
> ~[?:?] at
> org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$InClassLoaderExecute.catchingExecuteInClassLoader(ElasticSearchPersistenceServiceImpl.java:2244)
> ~[?:?] at
> org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl.load(ElasticSearchPersistenceServiceImpl.java:780)
> ~[?:?] at
> org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl.load(ElasticSearchPersistenceServiceImpl.java:735)
> ~[?:?] at Proxy6d4b2959_6bef_4c34_bb5a_88734cd23613.load(Unknown Source)
> ~[?:?] at
> org.apache.unomi.plugins.baseplugin.actions.MergeProfilesOnPropertyAction$1.execute(MergeProfilesOnPropertyAction.java:226)
> ~[?:?] at
> org.apache.unomi.services.impl.events.EventServiceImpl.send(EventServiceImpl.java:175)
> ~[?:?] at
> org.apache.unomi.services.impl.events.EventServiceImpl.send(EventServiceImpl.java:144)
> ~[?:?] at
> org.apache.unomi.rest.service.impl.RestServiceUtilsImpl.handleEvents(RestServiceUtilsImpl.java:112)
> ~[?:?] at
> org.apache.unomi.rest.endpoints.ContextJsonEndpoint.handleRequest(ContextJsonEndpoint.java:364)
> ~[?:?] at
> org.apache.unomi.rest.endpoints.ContextJsonEndpoint.contextJSONAsPost(ContextJsonEndpoint.java:311)
> ~[?:?]
> {code}
> Or:
> {code:java}
> Error while executing in class loader
> java.lang.Exception: Error loading itemType=org.apache.unomi.api.Profile
> query={
> "bool" : {
> "must" : [
> {
> "match_all" : {
> "boost" : 1.0
> }
> }
> ],
> "filter" : [
> {
> "bool" : {
> "must" : [
> {
> "term" : {
> "systemProperties.coveaIdentifier" : {
> "value" : "d4e55ba9b1004d49e0548ac200e561ac-mma",
> "boost" : 1.0
> }
> }
> },
> {
> "bool" : {
> "must_not" : [
> {
> "exists" : {
> "field" : "mergedWith",
> "boost" : 1.0
> }
> }
> ],
> "adjust_pure_negative" : true,
> "boost" : 1.0
> }
> }
> ],
> "adjust_pure_negative" : true,
> "boost" : 1.0
> }
> }
> ],
> "adjust_pure_negative" : true,
> "boost" : 1.0
> }
> } sortBy=properties.firstVisit
> at
> org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$24.execute(ElasticSearchPersistenceServiceImpl.java:1757)
> ~[?:?]
> at
> org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$24.execute(ElasticSearchPersistenceServiceImpl.java:1650)
> ~[?:?]
> at
> org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$InClassLoaderExecute.executeInClassLoader(ElasticSearchPersistenceServiceImpl.java:2233)
> ~[?:?]
> at
> org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$InClassLoaderExecute.catchingExecuteInClassLoader(ElasticSearchPersistenceServiceImpl.java:2244)
> ~[?:?]
> at
> org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl.query(ElasticSearchPersistenceServiceImpl.java:1767)
> ~[?:?]
> at
> org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl.query(ElasticSearchPersistenceServiceImpl.java:1570)
> ~[?:?]
> at Proxy6d4b2959_6bef_4c34_bb5a_88734cd23613.query(Unknown Source)
> ~[?:?]
> at
> org.apache.unomi.plugins.baseplugin.actions.MergeProfilesOnPropertyAction.execute(MergeProfilesOnPropertyAction.java:94)
> ~[?:?]
> at
> org.apache.unomi.services.actions.impl.ActionExecutorDispatcherImpl$8.execute(ActionExecutorDispatcherImpl.java:227)
> ~[?:?]
> at
> org.apache.unomi.services.actions.impl.ActionExecutorDispatcherImpl$8.execute(ActionExecutorDispatcherImpl.java:224)
> ~[?:?]
> at
> org.apache.unomi.metrics.MetricAdapter.runWithTimer(MetricAdapter.java:38)
> ~[?:?]
> at
> org.apache.unomi.services.actions.impl.ActionExecutorDispatcherImpl.execute(ActionExecutorDispatcherImpl.java:229)
> ~[?:?]
> at
> org.apache.unomi.services.impl.rules.RulesServiceImpl.onEvent(RulesServiceImpl.java:327)
> ~[?:?]
> at
> org.apache.unomi.services.impl.events.EventServiceImpl.send(EventServiceImpl.java:170)
> ~[?:?]
> at
> org.apache.unomi.services.impl.events.EventServiceImpl.send(EventServiceImpl.java:144)
> ~[?:?]
> at
> org.apache.unomi.rest.service.impl.RestServiceUtilsImpl.handleEvents(RestServiceUtilsImpl.java:112)
> ~[?:?]
> at
> org.apache.unomi.rest.endpoints.ContextJsonEndpoint.handleRequest(ContextJsonEndpoint.java:364)
> ~[?:?]
> at
> org.apache.unomi.rest.endpoints.ContextJsonEndpoint.contextJSONAsPost(ContextJsonEndpoint.java:311)
> ~[?:?] {code}
> The merge is doing scroll queries from code:
> {code:java}
> List<Profile> profiles = persistenceService.query(c, "properties.firstVisit",
> Profile.class, 0, maxProfilesInOneMerge).getList(); {code}
> Where *maxProfilesInOneMerge* is by default set to {*}-1{*}, in persistence
> service when doing a query using size {*}-1{*}:
> {code:java}
> if (size == Integer.MIN_VALUE) {
> searchSourceBuilder.size(defaultQueryLimit);
> } else if (size != -1) {
> searchSourceBuilder.size(size);
> } else {
> // size == -1, use scroll query to retrieve all the results
> searchRequest.scroll(keepAlive);
> } {code}
> With a *keepAlive* of 1hour by default. So that can be a lot, in case of lot
> merge requests happen in a short period of time, the scroll will accumulate
> in RAM.
>
> The workaround is to set the *maxProfilesInOneMerge* to something else than
> *-1* to avoid scroll queries.
> For that we can use this configuration:
> * Unomi config: *org.apache.unomi.plugins.base.maxProfilesInOneMerge*
> * Env config: *UNOMI_MAX_PROFILES_IN_ONE_MERGE*
> A value of *10* seem's good as we never expect a lot of profiles to be merged
> together.
> *But for the current ticket we should fix the default value and we should
> never do scroll queries at all at that place. Creating a new scroll on every
> request is a bad idea.*
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
