markap14 opened a new pull request, #11936: URL: https://github.com/apache/nifi/pull/11936
# Summary [NIFI-16599](https://issues.apache.org/jira/browse/NIFI-16599) When a connection holds more FlowFiles than its swap threshold, FlowFiles could come out of the queue in the wrong order after NiFi restarts. This change fixes three causes. Restart with no swap files. On restart, the FlowFiles that were held in memory are loaded from the FlowFile Repository in no particular order. If there are more of them than the active queue holds, whichever ones are loaded first fill the active queue, and the rest go to the swap queue. NIFI-16580 fixed this when swap files exist, by keeping in the active queue only the FlowFiles that come before the oldest swap file. This change replaces that method with `SwappablePriorityQueue.restoreFlowFilesInPriorityOrder`, which handles both cases. It sorts the restored FlowFiles using the queue's prioritizers. With swap files, it keeps the same rule as NIFI-16580. Without swap files, it keeps the same number of FlowFiles in the active queue, chosen from the highest priority ones. Selective drop. A selective drop rewrites a swap file with the remaining FlowFiles under a new name. Swap files are recovered on restart in the order of the timestamp in their names, so the rewritten file moved behind swap files holding newer FlowFiles. `FlowFileSwapManager` now has a default method `swapOut(flowFiles, queue, partitionName, replacedSwapLocation)` that falls back to the existing `swapOut`. `FileSystemSwapManager` overrides it so that the new swap file takes the timestamp and counter of the file it replaces, and `SwappablePriorityQueue` uses it when rewriting a swap file for a selective drop. Swap files written in the same millisecond. Swap files with the same timestamp were ordered by the random UUID in their names. Swap file names now include a counter, `<timestamp>-<queue id>-<counter>-<random uuid>`, and swap files are recovered in order of timestamp, then counter. Names written by earlier versions have no counter and still sort by timestamp. Earlier versions can still read the new names, because the timestamp and queue identifier are in the same positions. ### Tests - `TestSwappablePriorityQueue.testRecoverSwappedFlowFilesPreservesOrder` now also restores more FlowFiles than the active queue holds with no swap file, and verifies that every FlowFile is polled in order. - `TestSwappablePriorityQueue.testSelectiveDropPreservesSwapFileOrdering` now uses a real `FileSystemSwapManager` and verifies that, after a selective drop, a new swap manager recovers the rewritten swap file ahead of the swap file that holds newer FlowFiles. - `TestFileSystemSwapManager.testRecoverSwapLocationsInOrderWritten` writes 100 swap files and verifies that a new swap manager recovers them in the order they were written. Each test fails without its change and passes with it. # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-00000` - [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-00000` - [x] Pull request contains [commits signed](https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits) with a registered key indicating `Verified` status ### Pull Request Formatting - [x] Pull Request based on current revision of the `main` branch - [x] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `./mvnw clean install -P contrib-check` - [ ] JDK 21 - [ ] JDK 25 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
