Mark Payne created NIFI-16599:
---------------------------------
Summary: Preserve FlowFile order across restart for queues that
are swapping
Key: NIFI-16599
URL: https://issues.apache.org/jira/browse/NIFI-16599
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Reporter: Mark Payne
Assignee: Mark Payne
When a connection holds more FlowFiles than its swap threshold, FlowFiles can
come out of the queue in the wrong order after NiFi restarts. There are three
causes.
If more FlowFiles are restored from the FlowFile Repository than the active
queue can hold, but there are no swap files, whichever FlowFiles are restored
first fill the active queue. The restore order is arbitrary, so newer FlowFiles
can be processed ahead of older ones. NIFI-16580 fixed this only for the case
where swap files exist.
A selective drop rewrites a swap file under a new name that uses the current
time. On restart, swap files are sorted by the time in their names, so the
rewritten file is placed behind swap files that hold newer FlowFiles.
Swap files written within the same millisecond are ordered by a random UUID in
their names, so their order after a restart is random.
SwappablePriorityQueue should restore FlowFiles in priority order when there
are no swap files. FileSystemSwapManager should name swap files so that a
rewritten file keeps its place and files written in the same millisecond sort
in the order they were written.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)