[
https://issues.apache.org/jira/browse/NIFI-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17014831#comment-17014831
]
ASF subversion and git services commented on NIFI-7011:
-------------------------------------------------------
Commit 66d5ab80eb22f535da0898ae0d6a4a5da2dd7bd9 in nifi's branch
refs/heads/master from Mark Payne
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=66d5ab8 ]
NIFI-7011: This closes #3983. SwappablePriorityQueue contains two internal data
structures: activeQueue, swapQueue. activeQueue is intended to be pulled from
for processing. swapQueue is intended to hold FlowFiles that are waiting to be
swapped out. SinWe want to ensure that we first swap in any data that has
already been swapped out before processing the swap queue, in order to ensure
that we process the data in the correct order. This fix ddresses an issue where
data was being swapped out by writing the lowest priority data to a swap file,
then adding the highest priority data to activeQueue and the 'middle' priority
data back to swapQueue. As a result, when polling from the queue we got highest
priority data, followed by lowest priority data, followed by middle priority
data. This is addressed by avoiding putting anything back on swapQueue when we
swap out. Instead, write data to the swap file, then push everything else to
activeQueue. This way, any new data that comes in will still go to the
swapQueue, as it should, but all data that didn't get written to the Swap file
will be processed before the low priority data in the swap file.
NIFI-7011: Addressed corner case where data could be inserted out of order
still if added while swapping was taking place
NIFI-7011: Fixed ordering issue with swap queue that can occur if data is
migrated from swap queue to active queue instead of being swapped out
> FlowFile ordering can become incorrect when swapping data
> ---------------------------------------------------------
>
> Key: NIFI-7011
> URL: https://issues.apache.org/jira/browse/NIFI-7011
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Reporter: Mark Payne
> Assignee: Mark Payne
> Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> NiFi provides weak ordering guarantees when using prioritizers in conjunction
> with swapped data. When data is swapped out, it is always the lowest priority
> data (according to the selected prioritizers) that is swapped out first. When
> data is swapped, it is always swapped back in, in the same order.
> However, I've come across a problem where data is swapped out. Then, the data
> that is not swapped out gets processed, then data waiting to be swapped out
> (lowest priority data) is processed, then the swapped data. It should always
> be that the lowest priority data, waiting to be swapped out, should be
> processed after all data that is already swapped out gets swapped back in.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
