[ https://issues.apache.org/jira/browse/NIFI-3332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15871526#comment-15871526 ]
Koji Kawamura commented on NIFI-3332: ------------------------------------- [~mosermw] I tried the patch on NIFI-3213 PR1335 branch. I got the same result, Expected :5 Actual :4 AssertionError. NIFI-3213 ensures that the file whose timestamp is the latest within a listing activity will be listed if the timestamp is old enough, specifically older than current - LISTING_LAG_NANOS(100ms). This JIRA issue is not addressed by NIFI-3213. I've looked at the old implementation before we switched to use managed state. Found following comment: {quote} we need to keep track of all files that we pulled in that had a modification time equal to lastListingTime so that we can avoid pulling those files in again. We can't just ignore any files that have a mod time equal to that timestamp because more files may come in with the same timestamp later in the same millisecond. {quote} https://github.com/ijokarumawak/nifi/blob/b0322d9ffe8d117aae4faf7dd3e2881a28940f96/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractListProcessor.java#L279 We need to bring those code back to the current implementation to store serialized identifier list of files those were listed last time. I will try to fix it. > Bug in ListXXX causes matching timestamps to be ignored on later runs > --------------------------------------------------------------------- > > Key: NIFI-3332 > URL: https://issues.apache.org/jira/browse/NIFI-3332 > Project: Apache NiFi > Issue Type: Bug > Components: Core Framework > Affects Versions: 0.7.1, 1.1.1 > Reporter: Joe Skora > Priority: Critical > Attachments: Test-showing-ListFile-timestamp-bug.log, > Test-showing-ListFile-timestamp-bug.patch > > > The new state implementation for the ListXXX processors based on > AbstractListProcessor creates a race conditions when processor runs occur > while a batch of files is being written with the same timestamp. > The changes to state management dropped tracking of the files processed for a > given timestamp. Without the record of files processed, the remainder of the > batch is ignored on the next processor run since their timestamp is not > greater than the one timestamp stored in processor state. With the file > tracking it was possible to process files that matched the timestamp exactly > and exclude the previously processed files. > A basic time goes as follows. > T0 - system creates or receives batch of files with Tx timestamp where Tx > is more than the current timestamp in processor state. > T1 - system writes 1st half of Tx batch to the ListFile source directory. > T2 - ListFile runs picking up 1st half of Tx batch and stores Tx timestamp > in processor state. > T3 - system writes 2nd half of Tx batch to ListFile source directory. > T4 - ListFile runs ignoring any files with T <= Tx, eliminating 2nd half Tx > timestamp batch. > I've attached a patch[1] for TestListFile.java that adds an instrumented unit > test demonstrates the problem and a log[2] of the output from one such run. > The test writes 3 files each in two batches with processor runs after each > batch. Batch 2 writes files with timestamps older than, equal to, and newer > than the timestamp stored when batch 1 was processed, but only the newer file > is picked up. The older file is correctly ignored but file with the matchin > timestamp file should have been processed. > [1] Test-showing-ListFile-timestamp-bug.patch > [2] Test-showing-ListFile-timestamp-bug.log -- This message was sent by Atlassian JIRA (v6.3.15#6346)