mosermw opened a new pull request, #11790: URL: https://github.com/apache/nifi/pull/11790
# Summary [NIFI-16416](https://issues.apache.org/jira/browse/NIFI-16416) Optimize WriteAheadProvenanceRepository storage maintenance task by doing only 1 full directory listing, plus bug fixes This PR modifies the WriteAheadProvenanceRepository background maintenance task that purges old provenance events based on configured max age or max disk usage. The changes include: * Performs one directory listing of provenance repo partitions and reuses that listing between age-based and capacity-based provenance event removal * this is an optimization which helps when the provenance repo partitions contain many files * Changes capacity-based removal from ordering by event ID to file modification time * this helps cleanup orphaned provenance event files with large ID number (reported on Slack) * Stops capacity-based cleanup when no files can be deleted, resolving an infinite loop risk * resolves a possible infinte loop (possible if a large unrelated file is placed in the provenance repo) * Preserves TOC files when corresponding event file cannot be deleted * resolves an issue when listing a ".prov" file but it is compressed to ".prov.gz" before deletion Did not make any changes for Stretch Goal or Other Observations as described in the Jira ticket. ## Alternatives Considered Considered changing the EventStorePartition interface to be told exactly which files to delete, and moving the decision on which files to delete into PartitionedEventStore. But this seemed like a significant change that may not yield better results. # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-00000` - Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-00000` - Pull request contains [commits signed](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits) with a registered key indicating `Verified` status ### Pull Request Formatting - Pull Request based on current revision of the `main` branch - Pull Request refers to a feature branch with one commit containing changes # Verification * initial code review performed by [gpt-6.1-sol] * Ran NiFi with a busy flow generating many provenance events, and measured time spent in PartitionedEventStore performMaintenance(). Ran before and after these changes and observed improved timing. performMaintenance() time increases as more files are deleted ### Build - [x] Build completed using `./mvnw clean install -P contrib-check` - [x] JDK 21 - [ ] JDK 25 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
