We're seeing some odd behavior using site-to-site. The input port on a
3-node cluster will eventually stop receiving new data. In the log, I see
the following:
2021-07-08 13:13:14,010 ERROR [NiFi Web Server-43017]
o.a.nifi.web.api.ApplicationResource Exception detail:
org.apache.nifi.processor.exception.ProcessException:
java.lang.InterruptedException
at
org.apache.nifi.remote.StandardPublicPort.receiveFlowFiles(StandardPublicPort.java:588)
at
org.apache.nifi.web.api.DataTransferResource.receiveFlowFiles(DataTransferResource.java:277)
...
Then many more similar messages:
2021-07-08 13:13:14,015 ERROR [NiFi Web Server-47691]
0.a.nifi.web.api.ApplicationResource Exception detail:
org.apache.nifi.processor.exception.ProcessException:
org.apache.nifi.processor.exception.ProcessException: Interrupted while
waiting for site-to-site request to be serviced
at
org.apache.nifi.remote.StandardPublicPort.receiveFlowFiles(StandardPublicPort.java:588)
at
org.apache.nifi.web.api.DataTransferResource.receiveFlowFiles(DataTransferResource.java:277)
...
It's unclear what is causing the exception (possibly some network
instability), but the only way we have been able to get data flowing again
is to restart the NiFi node. Even more concerning is that when NIFi is
restarted, there are many thousand messages indicating:
2021-07-08 15:29:12,097 INFO [main] o.a.n.c.repository.FileSystemRepository
Found unknown file /cont_repo/content/336/1625700387433-161104 (1333153
bytes) in File System Repository; removing file
I suspect the failed site-to-site transfer completed writing data (content)
to disk, but was interrupted prior to creating a flowfile and
committing the Process Session. If this is true, this could cause the repo
to fill with data that will never get cleaned up until a NiFi restart.
I'm looking for someone with detailed knowledge of the internals of
site-to-site to comment on this issue - either the hard stop on receiving
additional data via site-to-site, or the orphaned content.
NiFi Version: 1.12.1
Thanks,
Mark
