Re: ListSFTP doesn't follow symlinks
Thanks Mark and David. I will watch the Issue at Jira to be updated. Thanks again! Regards. El jue, 3 feb 2022 a las 18:28, David Handermann (< exceptionfact...@apache.org>) escribió: > Guille, > > Thanks for confirming the same behavior on ListSFTP, and thanks Mark for > reproducing the issue. > > Apparently the problem is in the NiFi SFTPTransfer class, which is not > distinguishing between a symbolic link for a file and for a directory. In > this case, SFTPTransfer is attempting to process the symlinked file as a > directory, causing the error. I have assigned NIFI-6699 and will plan on > submitting a pull request to resolve the problem soon. > > Thanks again for reporting this issue and helping track down the problem! > > Regards, > David Handermann > > On Thu, Feb 3, 2022 at 10:58 AM Mark Payne wrote: > >> Guille, >> >> Thanks for the extra details. >> >> I just tried again. In my case, all worked as expected when I had a >> symlink to a directory. But when I had a symlink to a file, I got the same >> error and stack trace as you. So looks like we are handling the case >> properly for symlinked directories but not symlinked files. >> >> Thanks >> -Mark >> >> >> >> On Feb 3, 2022, at 11:43 AM, Guillermo Muñoz < >> guillermo.munoz.salg...@gmail.com> wrote: >> >> Hi, David. >> >> Sorry for the misunderstanding, my fault. Firstly, we tried using >> ListSFTP and FetchSFTP, and when it didn't work, we tried another option >> (GetSFTP), and I pasted the wrong stack trace. So, i've done the following >> tests: >> >>- ListSFTP + FetchSFTP: Error in ListSFTP [1] >>- GetSFTP: Error [2] >>- Generate flowfile + FetchSFTP with the name of the symlink in the * >>Remote File* property: OK, the file is downloaded. >> >> So, it seems the issue is in ListSFTP and GetSFTP, but FetchSFTP works >> fine. >> >> Thanks. Regards >> >> -- >> Guille >> >> [1] >> 2022-02-03 17:36:45,466 ERROR [Timer-Driven Process Thread-8] >> o.a.nifi.processors.standard.ListSFTP >> ListSFTP[id=64443154-ac76-1736-9e49-f2ca388dfbdf] Unable to get listing >> from *.gz; skipping: java.io.FileNotFoundException: Could not >> perform listing on *.gz because could not find the file on the >> remote server >> java.io.FileNotFoundException: Could not perform listing on *.gz >> because could not find the file on the remote server >> at >> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:350) >> at >> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:365) >> at >> org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:262) >> at >> org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:120) >> at >> org.apache.nifi.processors.standard.ListSFTP.performListing(ListSFTP.java:150) >> at >> org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:112) >> at >> org.apache.nifi.processor.util.list.AbstractListProcessor.listByTrackingTimestamps(AbstractListProcessor.java:750) >> at >> org.apache.nifi.processor.util.list.AbstractListProcessor.onTrigger(AbstractListProcessor.java:525) >> at >> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) >> at >> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273) >> at >> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214) >> at >> org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63) >> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) >> at >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> at java.lang.Thread.run(Thread.java:748) >> >> >> [2] >> 2022-02-03 17:39:14,714 ERROR [Timer-Driven Process Thread-27] >> o.a.nifi.processors.standard.GetSFTP >> GetSFTP[id=c0300c77-017e-1000--fa9c1f31] Unable to get listing from >> *.gz; skipping: java.io.FileNotFoundException: Could not perform >> listing on *.gz because could not find the file on the remote server >> java.io.FileNotFoundException: Could not perform listing on *.gz >> because could not find the file on the remote server >> at >> org.apache.nifi.processors.standard.util.SFTPTransfer.getL
Re: ListSFTP doesn't follow symlinks
Guille, Thanks for confirming the same behavior on ListSFTP, and thanks Mark for reproducing the issue. Apparently the problem is in the NiFi SFTPTransfer class, which is not distinguishing between a symbolic link for a file and for a directory. In this case, SFTPTransfer is attempting to process the symlinked file as a directory, causing the error. I have assigned NIFI-6699 and will plan on submitting a pull request to resolve the problem soon. Thanks again for reporting this issue and helping track down the problem! Regards, David Handermann On Thu, Feb 3, 2022 at 10:58 AM Mark Payne wrote: > Guille, > > Thanks for the extra details. > > I just tried again. In my case, all worked as expected when I had a > symlink to a directory. But when I had a symlink to a file, I got the same > error and stack trace as you. So looks like we are handling the case > properly for symlinked directories but not symlinked files. > > Thanks > -Mark > > > > On Feb 3, 2022, at 11:43 AM, Guillermo Muñoz < > guillermo.munoz.salg...@gmail.com> wrote: > > Hi, David. > > Sorry for the misunderstanding, my fault. Firstly, we tried using ListSFTP > and FetchSFTP, and when it didn't work, we tried another option (GetSFTP), > and I pasted the wrong stack trace. So, i've done the following tests: > >- ListSFTP + FetchSFTP: Error in ListSFTP [1] >- GetSFTP: Error [2] >- Generate flowfile + FetchSFTP with the name of the symlink in the * >Remote File* property: OK, the file is downloaded. > > So, it seems the issue is in ListSFTP and GetSFTP, but FetchSFTP works > fine. > > Thanks. Regards > > -- > Guille > > [1] > 2022-02-03 17:36:45,466 ERROR [Timer-Driven Process Thread-8] > o.a.nifi.processors.standard.ListSFTP > ListSFTP[id=64443154-ac76-1736-9e49-f2ca388dfbdf] Unable to get listing > from *.gz; skipping: java.io.FileNotFoundException: Could not > perform listing on *.gz because could not find the file on the > remote server > java.io.FileNotFoundException: Could not perform listing on *.gz > because could not find the file on the remote server > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:350) > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:365) > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:262) > at > org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:120) > at > org.apache.nifi.processors.standard.ListSFTP.performListing(ListSFTP.java:150) > at > org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:112) > at > org.apache.nifi.processor.util.list.AbstractListProcessor.listByTrackingTimestamps(AbstractListProcessor.java:750) > at > org.apache.nifi.processor.util.list.AbstractListProcessor.onTrigger(AbstractListProcessor.java:525) > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273) > at > org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214) > at > org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63) > at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > > [2] > 2022-02-03 17:39:14,714 ERROR [Timer-Driven Process Thread-27] > o.a.nifi.processors.standard.GetSFTP > GetSFTP[id=c0300c77-017e-1000--fa9c1f31] Unable to get listing from > *.gz; skipping: java.io.FileNotFoundException: Could not perform > listing on *.gz because could not find the file on the remote server > java.io.FileNotFoundException: Could not perform listing on *.gz > because could not find the file on the remote server > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:350) > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:365) > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:262) > at > org.apache.nifi.processors.standard.GetFileTransfer.fetchListing(
Re: ListSFTP doesn't follow symlinks
Guille, Thanks for the extra details. I just tried again. In my case, all worked as expected when I had a symlink to a directory. But when I had a symlink to a file, I got the same error and stack trace as you. So looks like we are handling the case properly for symlinked directories but not symlinked files. Thanks -Mark On Feb 3, 2022, at 11:43 AM, Guillermo Muñoz mailto:guillermo.munoz.salg...@gmail.com>> wrote: Hi, David. Sorry for the misunderstanding, my fault. Firstly, we tried using ListSFTP and FetchSFTP, and when it didn't work, we tried another option (GetSFTP), and I pasted the wrong stack trace. So, i've done the following tests: * ListSFTP + FetchSFTP: Error in ListSFTP [1] * GetSFTP: Error [2] * Generate flowfile + FetchSFTP with the name of the symlink in the Remote File property: OK, the file is downloaded. So, it seems the issue is in ListSFTP and GetSFTP, but FetchSFTP works fine. Thanks. Regards -- Guille [1] 2022-02-03 17:36:45,466 ERROR [Timer-Driven Process Thread-8] o.a.nifi.processors.standard.ListSFTP ListSFTP[id=64443154-ac76-1736-9e49-f2ca388dfbdf] Unable to get listing from *.gz; skipping: java.io.FileNotFoundException: Could not perform listing on *.gz because could not find the file on the remote server java.io.FileNotFoundException: Could not perform listing on *.gz because could not find the file on the remote server at org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:350) at org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:365) at org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:262) at org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:120) at org.apache.nifi.processors.standard.ListSFTP.performListing(ListSFTP.java:150) at org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:112) at org.apache.nifi.processor.util.list.AbstractListProcessor.listByTrackingTimestamps(AbstractListProcessor.java:750) at org.apache.nifi.processor.util.list.AbstractListProcessor.onTrigger(AbstractListProcessor.java:525) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214) at org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) [2] 2022-02-03 17:39:14,714 ERROR [Timer-Driven Process Thread-27] o.a.nifi.processors.standard.GetSFTP GetSFTP[id=c0300c77-017e-1000--fa9c1f31] Unable to get listing from *.gz; skipping: java.io.FileNotFoundException: Could not perform listing on *.gz because could not find the file on the remote server java.io.FileNotFoundException: Could not perform listing on *.gz because could not find the file on the remote server at org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:350) at org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:365) at org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:262) at org.apache.nifi.processors.standard.GetFileTransfer.fetchListing(GetFileTransfer.java:299) at org.apache.nifi.processors.standard.GetFileTransfer.onTrigger(GetFileTransfer.java:126) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214) at org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurren
Re: ListSFTP doesn't follow symlinks
Hi, David. Sorry for the misunderstanding, my fault. Firstly, we tried using ListSFTP and FetchSFTP, and when it didn't work, we tried another option (GetSFTP), and I pasted the wrong stack trace. So, i've done the following tests: - ListSFTP + FetchSFTP: Error in ListSFTP [1] - GetSFTP: Error [2] - Generate flowfile + FetchSFTP with the name of the symlink in the *Remote File* property: OK, the file is downloaded. So, it seems the issue is in ListSFTP and GetSFTP, but FetchSFTP works fine. Thanks. Regards -- Guille [1] 2022-02-03 17:36:45,466 ERROR [Timer-Driven Process Thread-8] o.a.nifi.processors.standard.ListSFTP ListSFTP[id=64443154-ac76-1736-9e49-f2ca388dfbdf] Unable to get listing from *.gz; skipping: java.io.FileNotFoundException: Could not perform listing on *.gz because could not find the file on the remote server java.io.FileNotFoundException: Could not perform listing on *.gz because could not find the file on the remote server at org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:350) at org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:365) at org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:262) at org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:120) at org.apache.nifi.processors.standard.ListSFTP.performListing(ListSFTP.java:150) at org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:112) at org.apache.nifi.processor.util.list.AbstractListProcessor.listByTrackingTimestamps(AbstractListProcessor.java:750) at org.apache.nifi.processor.util.list.AbstractListProcessor.onTrigger(AbstractListProcessor.java:525) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214) at org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) [2] 2022-02-03 17:39:14,714 ERROR [Timer-Driven Process Thread-27] o.a.nifi.processors.standard.GetSFTP GetSFTP[id=c0300c77-017e-1000--fa9c1f31] Unable to get listing from *.gz; skipping: java.io.FileNotFoundException: Could not perform listing on *.gz because could not find the file on the remote server java.io.FileNotFoundException: Could not perform listing on *.gz because could not find the file on the remote server at org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:350) at org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:365) at org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:262) at org.apache.nifi.processors.standard.GetFileTransfer.fetchListing(GetFileTransfer.java:299) at org.apache.nifi.processors.standard.GetFileTransfer.onTrigger(GetFileTransfer.java:126) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214) at org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:62
Re: ListSFTP doesn't follow symlinks
Guille, I did a quick test on my MacBook and things worked as expected following symlinks. If I set the property to “false” it didn’t get the files. If I set it to “true” it did retrieve the files. Either way it didn’t error, though - just didn’t follow the symlink. Of course, that’s not to say that there’s not some issue, just that it’s not obviously always broken :) One thing that I notice in the log message there, though: "Could not perform listing on testfile.gz…” There are a couple of spaces there - it’s not looking for “testfile.gz” but rather “testfile.gz” - are there actually spaces in the filename? Or any other sort of character there, that is perhaps not being properly escaped? Thanks -Mark On Feb 3, 2022, at 10:57 AM, Guillermo Muñoz Salgado mailto:mun...@gmail.com>> wrote: Hi all, We are developing a use case in which we have to get some files from a server. We have implemented it by the listSFTP + FetchSFTP way in a 3 nodes cluster running nifi 1.15.3. But we are having some issues when what we want to get are symlinks instead of files. We have set true the property "Follow symlink" but we get the same results. Are we doing something wrong? Or is it a bug or a known issue? We have found this issue [1] but it is old and resolved and this other one [2], that is older and unresolved. We're not sure if they are related to this behaviour or not. I paste our error log: 2022-02-03 16:27:41,002 ERROR [Timer-Driven Process Thread-18] o.a.nifi.processors.standard.GetSFTP GetSFTP[id=c0300c77-017e-1000--fff-ffa9c1f31] Unable to get listing from testfile.gz; skipping: java.io.FileNotFoundException: Could not perform listing on testfile.gz because could not find the file on the remote server java.io.FileNotFoundException: Could not perform listing on testfile.gz because could not find the file on the remote server at org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:350) at org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:365) at org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:262) at org.apache.nifi.processors.standard.GetFileTransfer.fetchListing(GetFileTransfer.java:299) at org.apache.nifi.processors.standard.GetFileTransfer.onTrigger(GetFileTransfer.java:126) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214) at org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thanks in advance -- Guille [1] https://issues.apache.org/jira/browse/NIFI-5560 [2] https://issues.apache.org/jira/browse/NIFI-6699
Re: ListSFTP doesn't follow symlinks
Hi Guille, Thanks for raising this issue and providing a stack trace. You mentioned using ListSFTP and FetchSFTP, but the stack references GetFileTransfer, which corresponds to GetSFTP. Can you confirm the same error using FetchSFTP? If you can confirm the same issue with FetchSFTP, it would be very helpful to add those details on the newer Jira issue NIFI-6699. NiFi SFTP processors switched to a different SSH library after the resolution of NIFI-5560, so it is possible that some changes may be necessary. However, it would be helpful to confirm whether this is an issue with FetchSFTP, GetSFTP, or both processors. Regards, David Handermann On Thu, Feb 3, 2022 at 9:57 AM Guillermo Muñoz Salgado wrote: > Hi all, > > We are developing a use case in which we have to get some files from a > server. We have implemented it by the listSFTP + FetchSFTP way in a 3 nodes > cluster running nifi 1.15.3. But we are having some issues when what > we want to get are symlinks instead of files. We have set true the property > *"Follow > symlink" *but we get the same results. Are we doing something wrong? Or > is it a bug or a known issue? We have found this issue [1] but it is old > and resolved and this other one [2], that is older and unresolved. We're > not sure if they are related to this behaviour or not. > > I paste our error log: > > 2022-02-03 16:27:41,002 ERROR [Timer-Driven Process Thread-18] > o.a.nifi.processors.standard.GetSFTP > GetSFTP[id=c0300c77-017e-1000--fff-ffa9c1f31] Unable to get listing > from testfile.gz; skipping: java.io.FileNotFoundException: Could not > perform listing on testfile.gz because could not find the file on the > remote server > java.io.FileNotFoundException: Could not perform listing on testfile.gz > because could not find the file on the remote server > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:350) > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:365) > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:262) > at > org.apache.nifi.processors.standard.GetFileTransfer.fetchListing(GetFileTransfer.java:299) > at > org.apache.nifi.processors.standard.GetFileTransfer.onTrigger(GetFileTransfer.java:126) > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273) > at > org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214) > at > org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63) > at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > Thanks in advance > -- > Guille > > [1] https://issues.apache.org/jira/browse/NIFI-5560 > [2] https://issues.apache.org/jira/browse/NIFI-6699 >
ListSFTP doesn't follow symlinks
Hi all, We are developing a use case in which we have to get some files from a server. We have implemented it by the listSFTP + FetchSFTP way in a 3 nodes cluster running nifi 1.15.3. But we are having some issues when what we want to get are symlinks instead of files. We have set true the property *"Follow symlink" *but we get the same results. Are we doing something wrong? Or is it a bug or a known issue? We have found this issue [1] but it is old and resolved and this other one [2], that is older and unresolved. We're not sure if they are related to this behaviour or not. I paste our error log: 2022-02-03 16:27:41,002 ERROR [Timer-Driven Process Thread-18] o.a.nifi.processors.standard.GetSFTP GetSFTP[id=c0300c77-017e-1000--fff-ffa9c1f31] Unable to get listing from testfile.gz; skipping: java.io.FileNotFoundException: Could not perform listing on testfile.gz because could not find the file on the remote server java.io.FileNotFoundException: Could not perform listing on testfile.gz because could not find the file on the remote server at org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:350) at org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:365) at org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:262) at org.apache.nifi.processors.standard.GetFileTransfer.fetchListing(GetFileTransfer.java:299) at org.apache.nifi.processors.standard.GetFileTransfer.onTrigger(GetFileTransfer.java:126) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1273) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:214) at org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:63) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thanks in advance -- Guille [1] https://issues.apache.org/jira/browse/NIFI-5560 [2] https://issues.apache.org/jira/browse/NIFI-6699