Don Hinshaw created UNOMI-172:
---------------------------------
Summary: Better way to create large profile segments
Key: UNOMI-172
URL: https://issues.apache.org/jira/browse/UNOMI-172
Project: Apache Unomi
Issue Type: Improvement
Reporter: Don Hinshaw
Right now the aggregateQueryBucketSize is the limit for how large a segment can
be when it's created. New events coming in are added, but in order to create a
large segment of profiles that already exist, it requires increasing the
aggregateQueryBucketSize.
I increased the bucketSize to 100,000 and it took ~10min to create the segment.
It was a pastEventCondition on a 3 node cluster with good resources. Our
needs go well beyond 100,000.
I understand that queries of that size in ES should be paginated. Are you
aware of a better way to achieve large segments of existing data or is this a
limitation at the moment?
I even looked into using batchProfileUpdate to add segments directly to the
profile but the performance was the same or worse since the condition is the
bottleneck.
Any insight into this issue would be greatly appreciated.
Thanks,
Donnie
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
