[
https://issues.apache.org/jira/browse/NIFI-756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14703428#comment-14703428
]
Aldrin Piri commented on NIFI-756:
----------------------------------
Build and contrib all look good. Logic to optimize the deletion process looks
good. Thanks for capturing that in the test case; extremely helpful. +1
> Persistent Provenance Repository can avoid deleting events from lucene
> ----------------------------------------------------------------------
>
> Key: NIFI-756
> URL: https://issues.apache.org/jira/browse/NIFI-756
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Reporter: Mark Payne
> Assignee: Mark Payne
> Fix For: 0.3.0
>
> Attachments:
> 0001-NIFI-756-Do-not-remove-documents-from-a-Lucene-Index.patch
>
>
> Currently, when events expire in the repository, they are deleted from the
> indices. This is very expensive. Since the index is sharded (by default at
> 500 MB), we can instead just ensure that searches always have a start date
> no earlier than the first provenance event. This way, we won't retrieve any
> expired records, but they can remain in the index. When all events in the
> index have expired (we know, based on the earliest event of the next index),
> we can simply close all readers/writers for the expired index and delete the
> entire index. This is far cheaper than continually updating the Lucene
> indices and would make a huge difference in performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
