[jira] [Commented] (NIFI-5764) Allow ListSftp connection parameter
[ https://issues.apache.org/jira/browse/NIFI-5764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683467#comment-16683467 ] Alfredo De Luca commented on NIFI-5764: --- Hi [~ijokarumawak] Just to let you know we fixed the issue. Wasn't either NIFI or Sftp server conf but the freeipa client due to selinux disabled. cheers > Allow ListSftp connection parameter > --- > > Key: NIFI-5764 > URL: https://issues.apache.org/jira/browse/NIFI-5764 > Project: Apache NiFi > Issue Type: Wish > Components: Extensions >Affects Versions: 1.6.0 >Reporter: dav >Priority: Critical > Labels: SFTP, customization, sftp > Attachments: dumpone > > > ListSftp and other Sftp processors should be able to add parameters > (like [-B buffer_size] [-b batchfile] [-c cipher] > [-D sftp_server_path] [-F ssh_config] [-i identity_file] [-l limit] > [-o ssh_option] [-P port] [-R num_requests] [-S program] > [-s subsystem | sftp_server] host > sftp [user@]host[:file ...] > sftp [user@]host[:dir[/]] > sftp -b batchfile [user@]host) > in order to edit the type of connection on Sftp Server. > For instance, I have this error on nifi: > 2018-10-29 11:06:09,462 ERROR [Timer-Driven Process Thread-5] > SimpleProcessLogger.java:254 > ListSFTP[id=766ac418-27ce-335a-5b13-52abe3495592] Failed to perform listing > on remote host due to java.io.IOException: Failed to obtain connection to > remote host due to com.jcraft.jsch.JSchException: Auth fail: {} > java.io.IOException: Failed to obtain connection to remote host due to > com.jcraft.jsch.JSchException: Auth fail > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getChannel(SFTPTransfer.java:468) > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:192) > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:156) > at > org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:105) > at > org.apache.nifi.processor.util.list.AbstractListProcessor.onTrigger(AbstractListProcessor.java:401) > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1147) > at > org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:175) > at > org.apache.nifi.controller.scheduling.QuartzSchedulingAgent$2.run(QuartzSchedulingAgent.java:140) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: com.jcraft.jsch.JSchException: Auth fail > at com.jcraft.jsch.Session.connect(Session.java:519) > at com.jcraft.jsch.Session.connect(Session.java:183) > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getChannel(SFTPTransfer.java:448) > ... 15 common frames omitted > This can be avoided by connect to Sftp server with this string: > *sftp -o “controlmaster auto” username@sftp_server* > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5764) Allow ListSftp connection parameter
[ https://issues.apache.org/jira/browse/NIFI-5764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16679458#comment-16679458 ] Alfredo De Luca commented on NIFI-5764: --- Thanks [~ijokarumawak] I am investigating on freeIPA/sss side as well. Cheers > Allow ListSftp connection parameter > --- > > Key: NIFI-5764 > URL: https://issues.apache.org/jira/browse/NIFI-5764 > Project: Apache NiFi > Issue Type: Wish > Components: Extensions >Affects Versions: 1.6.0 >Reporter: dav >Priority: Critical > Labels: SFTP, customization, sftp > Attachments: dumpone > > > ListSftp and other Sftp processors should be able to add parameters > (like [-B buffer_size] [-b batchfile] [-c cipher] > [-D sftp_server_path] [-F ssh_config] [-i identity_file] [-l limit] > [-o ssh_option] [-P port] [-R num_requests] [-S program] > [-s subsystem | sftp_server] host > sftp [user@]host[:file ...] > sftp [user@]host[:dir[/]] > sftp -b batchfile [user@]host) > in order to edit the type of connection on Sftp Server. > For instance, I have this error on nifi: > 2018-10-29 11:06:09,462 ERROR [Timer-Driven Process Thread-5] > SimpleProcessLogger.java:254 > ListSFTP[id=766ac418-27ce-335a-5b13-52abe3495592] Failed to perform listing > on remote host due to java.io.IOException: Failed to obtain connection to > remote host due to com.jcraft.jsch.JSchException: Auth fail: {} > java.io.IOException: Failed to obtain connection to remote host due to > com.jcraft.jsch.JSchException: Auth fail > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getChannel(SFTPTransfer.java:468) > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:192) > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:156) > at > org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:105) > at > org.apache.nifi.processor.util.list.AbstractListProcessor.onTrigger(AbstractListProcessor.java:401) > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1147) > at > org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:175) > at > org.apache.nifi.controller.scheduling.QuartzSchedulingAgent$2.run(QuartzSchedulingAgent.java:140) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: com.jcraft.jsch.JSchException: Auth fail > at com.jcraft.jsch.Session.connect(Session.java:519) > at com.jcraft.jsch.Session.connect(Session.java:183) > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getChannel(SFTPTransfer.java:448) > ... 15 common frames omitted > This can be avoided by connect to Sftp server with this string: > *sftp -o “controlmaster auto” username@sftp_server* > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5764) Allow ListSftp connection parameter
[ https://issues.apache.org/jira/browse/NIFI-5764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16677794#comment-16677794 ] Alfredo De Luca commented on NIFI-5764: --- Hi [~ijokarumawak]. > Do you see the ListSFTP processor instance works fine for sometime, but > encounter the auth failure issue some other times against the same SFTP > server without changing any ListSFTP configuration? {color:#FF}that's right. it goes ok and randomly we got the errors listed above. {color}{color:#33}{color} > Do you have access to the SFTP server to see any error happened? {color:#d04437}yes. I mange the SFTP server and we get the following error{color} {color:#d04437}Nov 7 08:30:09 sftp sshd[23487]: pam_sss(sshd:account): Access denied for user nifi_sftp: 4 (System error){color} {color:#d04437}We use freeipa for authentication and nifi is on 3 ndoes on Kubernetes PODs{color} {color:#d04437}{color:#33}> {color}{color}Did you confirm 'controlmaster' is the only difference between NiFi and the command you used? Did you perform ssh command without controlmaster option and if so, did it fail?? I mean: {color:#d04437}With a manually ssh not problem at all. It's when we have multiple connections that randomly we have that problem. {color} {code} {code} {color:#d04437} {color} {color:#d04437} {color} > Allow ListSftp connection parameter > --- > > Key: NIFI-5764 > URL: https://issues.apache.org/jira/browse/NIFI-5764 > Project: Apache NiFi > Issue Type: Wish > Components: Extensions >Affects Versions: 1.6.0 >Reporter: dav >Priority: Critical > Labels: SFTP, customization, sftp > Attachments: dumpone > > > ListSftp and other Sftp processors should be able to add parameters > (like [-B buffer_size] [-b batchfile] [-c cipher] > [-D sftp_server_path] [-F ssh_config] [-i identity_file] [-l limit] > [-o ssh_option] [-P port] [-R num_requests] [-S program] > [-s subsystem | sftp_server] host > sftp [user@]host[:file ...] > sftp [user@]host[:dir[/]] > sftp -b batchfile [user@]host) > in order to edit the type of connection on Sftp Server. > For instance, I have this error on nifi: > 2018-10-29 11:06:09,462 ERROR [Timer-Driven Process Thread-5] > SimpleProcessLogger.java:254 > ListSFTP[id=766ac418-27ce-335a-5b13-52abe3495592] Failed to perform listing > on remote host due to java.io.IOException: Failed to obtain connection to > remote host due to com.jcraft.jsch.JSchException: Auth fail: {} > java.io.IOException: Failed to obtain connection to remote host due to > com.jcraft.jsch.JSchException: Auth fail > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getChannel(SFTPTransfer.java:468) > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:192) > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:156) > at > org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:105) > at > org.apache.nifi.processor.util.list.AbstractListProcessor.onTrigger(AbstractListProcessor.java:401) > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1147) > at > org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:175) > at > org.apache.nifi.controller.scheduling.QuartzSchedulingAgent$2.run(QuartzSchedulingAgent.java:140) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: com.jcraft.jsch.JSchException: Auth fail > at com.jcraft.jsch.Session.connect(Session.java:519) > at com.jcraft.jsch.Session.connect(Session.java:183) > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getChannel(SFTPTransfer.java:448) > ... 15 common frames omitted > This can be avoided by connect to Sftp server with this string: > *sftp -o “controlmaster auto” username@sftp_server* > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (NIFI-5764) Allow ListSftp connection parameter
[ https://issues.apache.org/jira/browse/NIFI-5764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16675468#comment-16675468 ] Alfredo De Luca edited comment on NIFI-5764 at 11/5/18 5:45 PM: Hi [~ijokarumawak]. Thanks for that but I can't fix that issue that we have. So i did a few test (not with NiFi) and if I use a controlmaster on my ssh connection I don't get this error. Caused by: com.jcraft.jsch.JSchException: Auth fail Any idea/thoughts? Cheers was (Author: alfredo.deluca): Hi [~ijokarumawak]. Thanks for that but I can't fix that issue that we have. So i did a few test (not with NiFi) and if I use a controlmaster on my ssh connection I don't get this error. Caused by: com.jcraft.jsch.JSchException: Auth fail Any idea/thoughts? Cheerd > Allow ListSftp connection parameter > --- > > Key: NIFI-5764 > URL: https://issues.apache.org/jira/browse/NIFI-5764 > Project: Apache NiFi > Issue Type: Wish > Components: Extensions >Affects Versions: 1.6.0 >Reporter: dav >Priority: Critical > Labels: SFTP, customization, sftp > > ListSftp and other Sftp processors should be able to add parameters > (like [-B buffer_size] [-b batchfile] [-c cipher] > [-D sftp_server_path] [-F ssh_config] [-i identity_file] [-l limit] > [-o ssh_option] [-P port] [-R num_requests] [-S program] > [-s subsystem | sftp_server] host > sftp [user@]host[:file ...] > sftp [user@]host[:dir[/]] > sftp -b batchfile [user@]host) > in order to edit the type of connection on Sftp Server. > For instance, I have this error on nifi: > 2018-10-29 11:06:09,462 ERROR [Timer-Driven Process Thread-5] > SimpleProcessLogger.java:254 > ListSFTP[id=766ac418-27ce-335a-5b13-52abe3495592] Failed to perform listing > on remote host due to java.io.IOException: Failed to obtain connection to > remote host due to com.jcraft.jsch.JSchException: Auth fail: {} > java.io.IOException: Failed to obtain connection to remote host due to > com.jcraft.jsch.JSchException: Auth fail > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getChannel(SFTPTransfer.java:468) > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:192) > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:156) > at > org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:105) > at > org.apache.nifi.processor.util.list.AbstractListProcessor.onTrigger(AbstractListProcessor.java:401) > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1147) > at > org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:175) > at > org.apache.nifi.controller.scheduling.QuartzSchedulingAgent$2.run(QuartzSchedulingAgent.java:140) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: com.jcraft.jsch.JSchException: Auth fail > at com.jcraft.jsch.Session.connect(Session.java:519) > at com.jcraft.jsch.Session.connect(Session.java:183) > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getChannel(SFTPTransfer.java:448) > ... 15 common frames omitted > This can be avoided by connect to Sftp server with this string: > *sftp -o “controlmaster auto” username@sftp_server* > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (NIFI-5764) Allow ListSftp connection parameter
[ https://issues.apache.org/jira/browse/NIFI-5764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16675468#comment-16675468 ] Alfredo De Luca commented on NIFI-5764: --- Hi [~ijokarumawak]. Thanks for that but I can't fix that issue that we have. So i did a few test (not with NiFi) and if I use a controlmaster on my ssh connection I don't get this error. Caused by: com.jcraft.jsch.JSchException: Auth fail Any idea/thoughts? Cheerd > Allow ListSftp connection parameter > --- > > Key: NIFI-5764 > URL: https://issues.apache.org/jira/browse/NIFI-5764 > Project: Apache NiFi > Issue Type: Wish > Components: Extensions >Affects Versions: 1.6.0 >Reporter: dav >Priority: Critical > Labels: SFTP, customization, sftp > > ListSftp and other Sftp processors should be able to add parameters > (like [-B buffer_size] [-b batchfile] [-c cipher] > [-D sftp_server_path] [-F ssh_config] [-i identity_file] [-l limit] > [-o ssh_option] [-P port] [-R num_requests] [-S program] > [-s subsystem | sftp_server] host > sftp [user@]host[:file ...] > sftp [user@]host[:dir[/]] > sftp -b batchfile [user@]host) > in order to edit the type of connection on Sftp Server. > For instance, I have this error on nifi: > 2018-10-29 11:06:09,462 ERROR [Timer-Driven Process Thread-5] > SimpleProcessLogger.java:254 > ListSFTP[id=766ac418-27ce-335a-5b13-52abe3495592] Failed to perform listing > on remote host due to java.io.IOException: Failed to obtain connection to > remote host due to com.jcraft.jsch.JSchException: Auth fail: {} > java.io.IOException: Failed to obtain connection to remote host due to > com.jcraft.jsch.JSchException: Auth fail > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getChannel(SFTPTransfer.java:468) > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:192) > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getListing(SFTPTransfer.java:156) > at > org.apache.nifi.processors.standard.ListFileTransfer.performListing(ListFileTransfer.java:105) > at > org.apache.nifi.processor.util.list.AbstractListProcessor.onTrigger(AbstractListProcessor.java:401) > at > org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) > at > org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1147) > at > org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:175) > at > org.apache.nifi.controller.scheduling.QuartzSchedulingAgent$2.run(QuartzSchedulingAgent.java:140) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: com.jcraft.jsch.JSchException: Auth fail > at com.jcraft.jsch.Session.connect(Session.java:519) > at com.jcraft.jsch.Session.connect(Session.java:183) > at > org.apache.nifi.processors.standard.util.SFTPTransfer.getChannel(SFTPTransfer.java:448) > ... 15 common frames omitted > This can be avoided by connect to Sftp server with this string: > *sftp -o “controlmaster auto” username@sftp_server* > -- This message was sent by Atlassian JIRA (v7.6.3#76005)