Mark Payne created NIFI-16599:
---------------------------------

             Summary: Preserve FlowFile order across restart for queues that 
are swapping
                 Key: NIFI-16599
                 URL: https://issues.apache.org/jira/browse/NIFI-16599
             Project: Apache NiFi
          Issue Type: Bug
          Components: Core Framework
            Reporter: Mark Payne
            Assignee: Mark Payne


When a connection holds more FlowFiles than its swap threshold, FlowFiles can 
come out of the queue in the wrong order after NiFi restarts. There are three 
causes.

If more FlowFiles are restored from the FlowFile Repository than the active 
queue can hold, but there are no swap files, whichever FlowFiles are restored 
first fill the active queue. The restore order is arbitrary, so newer FlowFiles 
can be processed ahead of older ones. NIFI-16580 fixed this only for the case 
where swap files exist.

A selective drop rewrites a swap file under a new name that uses the current 
time. On restart, swap files are sorted by the time in their names, so the 
rewritten file is placed behind swap files that hold newer FlowFiles.

Swap files written within the same millisecond are ordered by a random UUID in 
their names, so their order after a restart is random.

SwappablePriorityQueue should restore FlowFiles in priority order when there 
are no swap files. FileSystemSwapManager should name swap files so that a 
rewritten file keeps its place and files written in the same millisecond sort 
in the order they were written.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to