[
https://issues.apache.org/jira/browse/UNOMI-172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400142#comment-16400142
]
Serge Huber commented on UNOMI-172:
-----------------------------------
Hello Don,
As I understand you are building segments using pastEvent conditions ? Are you
limiting these past event conditions in time ? Because you should avoid from
retrieving past events without any limits ?
Can you maybe tell us more about what you are doing with the segments and what
your segment looks like ?
Another way around these types of problems is to use rules to update profiles
and then avoid querying on past events but simply have a segment condition on a
property that is updated by a rule.
Regards,
Serge...
> Better way to create large profile segments
> -------------------------------------------
>
> Key: UNOMI-172
> URL: https://issues.apache.org/jira/browse/UNOMI-172
> Project: Apache Unomi
> Issue Type: Improvement
> Reporter: Don Hinshaw
> Priority: Minor
>
> Right now the aggregateQueryBucketSize is the limit for how large a segment
> can be when it's created. New events coming in are added, but in order to
> create a large segment of profiles that already exist, it requires increasing
> the aggregateQueryBucketSize.
>
> I increased the bucketSize to 100,000 and it took ~10min to create the
> segment. It was a pastEventCondition on a 3 node cluster with good
> resources. Our needs go well beyond 100,000.
>
> I understand that queries of that size in ES should be paginated. Are you
> aware of a better way to achieve large segments of existing data or is this a
> limitation at the moment?
>
> I even looked into using batchProfileUpdate to add segments directly to the
> profile but the performance was the same or worse since the condition is the
> bottleneck.
>
> Any insight into this issue would be greatly appreciated.
>
> Thanks,
> Donnie
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
