Mark Payne created NIFI-10142:
---------------------------------
Summary: ListSFTP assumes that time on SFTP server is the same as
the time on local server
Key: NIFI-10142
URL: https://issues.apache.org/jira/browse/NIFI-10142
Project: Apache NiFi
Issue Type: Bug
Components: Extensions
Reporter: Mark Payne
The ListSFTP processor was updated in NIFI-5977 to add "Minimum File Age" and
"Maximum File Age" properties. This was done in order to prevent ListSFTP from
constantly listing files as they are being updated, and done in a way that is
similar to ListFile.
However, with ListFile (at least when the file is local) we can be assured that
the timestamp was generated on the system whose time is the same as the
localhost's time.
For an SFTP server, we cannot assume that. So we should not be comparing local
timestamp to the Last Modified time reported by the SFTP server.
We should deprecate the Max File Age property, and we should update the
behavior such that if Min File Age is set to say 10 mins, we don't ingest a
file until we see its Last Modified Time set to time t, and then at least 10
mins later see the Last Modified Time still set to t. This is the only way we
can guarantee that the file has not been modified for at least 10 minutes.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
