Josef Zahner created NIFI-8483:
----------------------------------
Summary: Restart NiFi - duplicate SFTP flowfiles
Key: NIFI-8483
URL: https://issues.apache.org/jira/browse/NIFI-8483
Project: Apache NiFi
Issue Type: Bug
Affects Versions: 1.13.2
Environment: Java 1.8.0_282, CentOS 7, 8-Node Cluster
Reporter: Josef Zahner
Attachments: SFTP_failure.png
Since the upgrade from NiFi 1.11.4 to 1.13.2 we faced an issue with the
FetchSFTP & PutSFTP processors. We have a 8-Node NiFi cluster. Pattern is
always ListSFTP (tracking timestamp) - FetchSFTP (and delete) and PutSFTP.
If we do a restart of NiFi and NiFi comes back, we sometimes see flowfiles for
FetchSFTP (not found) and PutSFTP (already present on disk) which have been
processed successfully and have been stored already. So in fact we see
flowfiles in a failure queue which have been save to disk with PutSFTP, which
should never happen. The files are always small (a few MBs) and the network
connectivity is insanely fast. The cluster shutdown is always before the grace
period runs out. The attached screeshot shows an example where the FetchSFTP
and the PutSFTP failure queue has files. Especially for the FetchSFTP this
shouldn't be possible and if I do a restart with the command below, I would
expect that within the grace period the processor has been stopped and it can't
be processed twice.
{code:java}
/opt/nifi/bin/nifi.sh restart
{code}
At the moment we have no clue where the issue comes from and why it happens, so
I can't provide an exact scenario to reproduce it. I only know that it
sometimes happens after a restart of our 8-node cluster.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
