Guille, Thanks for confirming the same behavior on ListSFTP, and thanks Mark for reproducing the issue.
Apparently the problem is in the NiFi SFTPTransfer class, which is not distinguishing between a symbolic link for a file and for a directory. In this case, SFTPTransfer is attempting to process the symlinked file as a directory, causing the error. I have assigned NIFI-6699 and will plan on submitting a pull request to resolve the problem soon. Thanks again for reporting this issue and helping track down the problem! Regards, David Handermann On Thu, Feb 3, 2022 at 10:58 AM Mark Payne <[email protected]> wrote: > Guille, > > Thanks for the extra details. > > I just tried again. In my case, all worked as expected when I had a > symlink to a directory. But when I had a symlink to a file, I got the same > error and stack trace as you. So looks like we are handling the case > properly for symlinked directories but not symlinked files. > > Thanks > -Mark > > > > On Feb 3, 2022, at 11:43 AM, Guillermo Muñoz < > [email protected]> wrote: > > Hi, David. > > Sorry for the misunderstanding, my fault. Firstly, we tried using ListSFTP > and FetchSFTP, and when it didn't work, we tried another option (GetSFTP), > and I pasted the wrong stack trace. So, i've done the following tests: > > - ListSFTP + FetchSFTP: Error in ListSFTP [1] > - GetSFTP: Error [2] > - Generate flowfile + FetchSFTP with the name of the symlink in the * > Remote File* property: OK, the file is downloaded. > > So, it seems the issue is in ListSFTP and GetSFTP, but FetchSFTP works > fine. > > Thanks. Regards > > -- > Guille > > [1] > 2022-02-03 17:36:45,466 ERROR [Timer-Driven Process Thread-8] > o.a.nifi.processors.standard.ListSFTP > ListSFTP[id=64443154-ac76-1736-9e49-f2ca388dfbdf] Unable to get listing > from *****.gz; skipping: java.io.FileNotFoundException: Could not > perform listing on *****.gz because could not find the file on the > remote server > java.io.FileNotFoundException: Could not perform listing on *****.gz > because could not find the file on the remote server > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:350) > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:365) > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:262) > at > org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:120) > at > org.apache.nifi.processors.standard.ListSFTP.performListing(ListSFTP.java:150) > at > org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:112) > at > org.apache.nifi.processor.util.list.AbstractListProcessor.listByTrackingTimestamps(AbstractListProcessor.java:750) > at > org.apache.nifi.processor.util.list.AbstractListProcessor.onTrigger(AbstractListProcessor.java:525) > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273) > at > org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214) > at > org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63) > at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > > [2] > 2022-02-03 17:39:14,714 ERROR [Timer-Driven Process Thread-27] > o.a.nifi.processors.standard.GetSFTP > GetSFTP[id=c0300c77-017e-1000-ffff-fffffa9c1f31] Unable to get listing from > *****.gz; skipping: java.io.FileNotFoundException: Could not perform > listing on *****.gz because could not find the file on the remote server > java.io.FileNotFoundException: Could not perform listing on *****.gz > because could not find the file on the remote server > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:350) > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:365) > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:262) > at > org.apache.nifi.processors.standard.GetFileTransfer.fetchListing(GetFileTransfer.java:299) > at > org.apache.nifi.processors.standard.GetFileTransfer.onTrigger(GetFileTransfer.java:126) > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273) > at > org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214) > at > org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63) > at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > El jue, 3 feb 2022 a las 17:09, David Handermann (< > [email protected]>) escribió: > >> Hi Guille, >> >> Thanks for raising this issue and providing a stack trace. You mentioned >> using ListSFTP and FetchSFTP, but the stack references GetFileTransfer, >> which corresponds to GetSFTP. >> >> Can you confirm the same error using FetchSFTP? If you can confirm the >> same issue with FetchSFTP, it would be very helpful to add those details on >> the newer Jira issue NIFI-6699. >> >> NiFi SFTP processors switched to a different SSH library after the >> resolution of NIFI-5560, so it is possible that some changes may be >> necessary. However, it would be helpful to confirm whether this is an >> issue with FetchSFTP, GetSFTP, or both processors. >> >> Regards, >> David Handermann >> >> On Thu, Feb 3, 2022 at 9:57 AM Guillermo Muñoz Salgado <[email protected]> >> wrote: >> >>> Hi all, >>> >>> We are developing a use case in which we have to get some files from a >>> server. We have implemented it by the listSFTP + FetchSFTP way in a 3 nodes >>> cluster running nifi 1.15.3. But we are having some issues when what >>> we want to get are symlinks instead of files. We have set true the property >>> *"Follow >>> symlink" *but we get the same results. Are we doing something wrong? Or >>> is it a bug or a known issue? We have found this issue [1] but it is old >>> and resolved and this other one [2], that is older and unresolved. We're >>> not sure if they are related to this behaviour or not. >>> >>> I paste our error log: >>> >>> 2022-02-03 16:27:41,002 ERROR [Timer-Driven Process Thread-18] >>> o.a.nifi.processors.standard.GetSFTP >>> GetSFTP[id=c0300c77-017e-1000-ffff-fff-ffa9c1f31] Unable to get listing >>> from testfile.gz; skipping: java.io.FileNotFoundException: Could not >>> perform listing on testfile.gz because could not find the file on the >>> remote server >>> java.io.FileNotFoundException: Could not perform listing on testfile.gz >>> because could not find the file on the remote server >>> at >>> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:350) >>> at >>> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:365) >>> at >>> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:262) >>> at >>> org.apache.nifi.processors.standard.GetFileTransfer.fetchListing(GetFileTransfer.java:299) >>> at >>> org.apache.nifi.processors.standard.GetFileTransfer.onTrigger(GetFileTransfer.java:126) >>> at >>> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) >>> at >>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273) >>> at >>> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214) >>> at >>> org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63) >>> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>> at java.lang.Thread.run(Thread.java:748) >>> >>> Thanks in advance >>> -- >>> Guille >>> >>> [1] https://issues.apache.org/jira/browse/NIFI-5560 >>> [2] https://issues.apache.org/jira/browse/NIFI-6699 >>> >> > > -- > Guillermo Muñoz Salgado > > >
