[
https://issues.apache.org/jira/browse/NIFI-756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14703460#comment-14703460
]
ASF subversion and git services commented on NIFI-756:
------------------------------------------------------
Commit 16dc5d5fd9a1538880312fea9438e0f00e4b7538 in nifi's branch
refs/heads/master from [~markap14]
[ https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=16dc5d5 ]
NIFI-756: Do not remove documents from a Lucene Index; instead, wait until the
entire index is 'expired' and delete the index's directory
> Persistent Provenance Repository can avoid deleting events from lucene
> ----------------------------------------------------------------------
>
> Key: NIFI-756
> URL: https://issues.apache.org/jira/browse/NIFI-756
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Reporter: Mark Payne
> Assignee: Mark Payne
> Fix For: 0.3.0
>
> Attachments:
> 0001-NIFI-756-Do-not-remove-documents-from-a-Lucene-Index.patch
>
>
> Currently, when events expire in the repository, they are deleted from the
> indices. This is very expensive. Since the index is sharded (by default at
> 500 MB), we can instead just ensure that searches always have a start date
> no earlier than the first provenance event. This way, we won't retrieve any
> expired records, but they can remain in the index. When all events in the
> index have expired (we know, based on the earliest event of the next index),
> we can simply close all readers/writers for the expired index and delete the
> entire index. This is far cheaper than continually updating the Lucene
> indices and would make a huge difference in performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
