[
https://issues.apache.org/jira/browse/LUCENE-10025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangchao.es updated LUCENE-10025:
----------------------------------
Attachment: image-2021-07-14-16-52-34-740.png
flamegraph.html
Description:
In lucene-8246, numDeletesToMerge is added in SoftDeletesRetentionMergePolicy.
if soft deleted doc is very more, and they are also in retention lease,the
numDeletesToMerge funcation have performance issue
for instance，a update indexing is writing to elasticsearch, then we move one a
shard to an other node,If the moving continues for a long time, the size of old
shard will become very big,because soft-deleted operations need to held by
retention lease. The more soft-deleted documents, the slower the indexing. if
the shard size is about 20GB, we can get the below flamegraph
!image-2021-07-14-16-52-34-740.png!
was:In lucene-8246, numDeletesToMerge is added in
SoftDeletesRetentionMergePolicy.
> SoftDeletesRetentionMergePolicy#numDeletesToMerge caused indexing backlogged
> ----------------------------------------------------------------------------
>
> Key: LUCENE-10025
> URL: https://issues.apache.org/jira/browse/LUCENE-10025
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/index
> Affects Versions: 8.4
> Reporter: zhangchao.es
> Priority: Major
> Labels: indexing, soft-delete
> Attachments: flamegraph.html, image-2021-07-14-16-52-34-740.png
>
>
> In lucene-8246, numDeletesToMerge is added in SoftDeletesRetentionMergePolicy.
> if soft deleted doc is very more, and they are also in retention lease,the
> numDeletesToMerge funcation have performance issue
> for instance，a update indexing is writing to elasticsearch, then we move one
> a shard to an other node,If the moving continues for a long time, the size of
> old shard will become very big,because soft-deleted operations need to held
> by retention lease. The more soft-deleted documents, the slower the indexing.
> if the shard size is about 20GB, we can get the below flamegraph
>
> !image-2021-07-14-16-52-34-740.png!
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
