[ 
https://issues.apache.org/jira/browse/NIFI-16416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael W Moser updated NIFI-16416:
-----------------------------------
    Status: Patch Available  (was: In Progress)

> Optimize WriteAheadProvenanceRepository storage maintenance and bug fixes
> -------------------------------------------------------------------------
>
>                 Key: NIFI-16416
>                 URL: https://issues.apache.org/jira/browse/NIFI-16416
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Core Framework
>            Reporter: Michael W Moser
>            Assignee: Michael W Moser
>            Priority: Minor
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> h3. Background
> WriteAheadProvenanceRepository has a background maintenance task that checks 
> if the max.storage.time and max.storage.size is exceeded and cleanup needs to 
> happen.  The maintenance task performs many provenance repo directory 
> listings, including a new listing after each file is deleted during 
> max.storage.size checks.  This process repeats until enough files are deleted 
> to bring the provenance repository within storage limits.  (Task 1) This 
> could be optimized.
> h3. Bug Reports
> Users have also reported various other issues with provenance repo 
> maintenance, including:
>  * (Task 2) an orphaned events file with a large event ID "3437277653.prov" 
> could never get cleaned up because files with lower event IDs "572531.prov" 
> are cleaned up first
>  * (Task 3) junk files in the prov_repo directory count toward the max 
> storage size limit, which can cause the maintenance task to get stuck in an 
> infinite loop if 0 files can be deleted
>  * (Task 4) .toc files can be removed if the corresponding .prov/.prov.gz 
> file could not be removed
> h3. Goals
>  * (Task 1) We can improve the provenance repo maintenance by performing only 
> one directory listing, then reusing that list throughout the maintenance task.
>  * (Task 2) Sort event files by lastModified instead of event ID in filename
>  * (Task 3) Avoid infinite loop when prov_repo size exceeds max.storage.size, 
> but event files cannot be deleted
>  * (Task 4) Properly handle removing .toc files only if its .prov file is 
> removed
> h3. Stretch Goal
>  * Remove events files when disk partition usage exceeds 98% (NiFi does this 
> in FileSystemRepository when content_repo archiving is on, removing archived 
> data). This could help avoid filling the disk partition where prov_repo 
> stores data.
> h3. Other Observations
>  * When using multiple provenance repo partitions, the cleanup task deletes 
> one file from each prov_repo partition per pass.  This can delete files that 
> are not the oldest events overall.  This is a minor issue, though, because 
> deleted events are still very close in time to older events in other 
> partitions.
> This was discussed in Slack in May 2026 and August 2026.  Feedback on 
> proposed goals appreciated.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to