Matt Rodriguez created NIFI-7216:
------------------------------------
Summary: FetchSFTP can't delete or move files upon completion
Key: NIFI-7216
URL: https://issues.apache.org/jira/browse/NIFI-7216
Project: Apache NiFi
Issue Type: Bug
Components: Core Framework
Affects Versions: 1.10.0
Reporter: Matt Rodriguez
Possibly similar to NIFI-7177 as they're not able to delete files with the
GetSFTP processor, but they're getting a different error than I am.
I'm using the FetchSFTP processor to get data from a third party SFTP server, I
don't have a lot of details on version or configuration on their end. I have
noticed that neither the "Move File" or "Delete File" Completion Strategy
options work.
When using "Delete File" as the Completion Strategy I get no alert/bulletin
from NiFi to show that anything went wrong, however, the original file has not
been deleted. I can manually delete the file using an SFTP client with the same
username/password that NiFi connects as.
When using "Move File" as the Completion Strategy I get this warning message
when NiFi tries to move the file after completion:
{code:java}
2020-03-03 13:59:54,375 WARN [Timer-Driven Process Thread-2]
o.a.nifi.processors.standard.FetchSFTP
FetchSFTP[id=0344354b-3a49-316e-a571-adcca7b3e70e] Successfully fetched the
content for
StandardFlowFileRecord[uuid=dea58017-8f3f-4991-91a3-1ff8ca167b6c,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1583243711464-8, container=default,
section=8], offset=4625, length=7672936],offset=0,name=TEST.csv,size=7672936]
from ftp1.XXXXX.com:22/Test/TEST.csv but failed to rename the remote file due
to java.io.FileNotFoundException: No such file or directory: {code}
As with deletion, I am able manually move (rename) the file using an SFTP
client with the same username/password that NiFi connects as.
However, I will add that I'm only able to delete or move the file PRIOR to NiFi
fetching the file. If I try to do it immediately after NiFi has fetched the
file, I'll get the exact same "no such file or directory" error from the SFTP
server when using my local client. If I wait some arbitrary amount of time
after NiFi has fetched the file, I am then able to delete or move the file.
I've also noticed the below sequence of events:
# NiFi lists the files on the SFTP server
# NiFi fetches the files on the SFTP server
# NiFi attempts to delete or move the files on the SFTP server and fails
# I immediately attempt to delete or move the files on the SFTP server and fail
# I stop the FetchSFTP processor in NiFi
# I immediately attempt to delete or move the files on the SFTP server and it
succeeds
This leads me to believe that there is some sort of locking behavior happening
where the fetch operation keeps the file locked for some arbitrary amount of
time or until the fetch processor is stopped, which is preventing any other
operation from taking place on the file.
Interestingly enough, I was able to get both the "Delete File" and "Move File"
Completion Strategies to work when I spun up a basic SFTP server on my own. It
sounds like there is something specific with how locks are handled with this
third part SFTP server I'm connecting to and how NiFi uses (or re-uses) SFTP
clients/connections.
As mentioned above, I unfortunately don't have a lot of information about this
SFTP server I'm connecting to, the vendor has been very tight-lipped about
their configuration for some reason.
If there's anything else you need me to provide, please let me know.
Thanks!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
