Any thought Joe? thank so much On Tue, Oct 30, 2018 at 2:28 PM Alfredo De Luca <alfredo.del...@gmail.com> wrote:
> I have attached the nifi properties. Also these are more info ... > > - ListSftp is running on primary node of the cluster and with 1 concurrent > tasks > - nifi is in cluster with 3 nodes (pod in kubernetes) > --- > *Loading of nifi* > how many threads does nifi show are running? > *150 in average* > --- > How often does the problem happen? How often does the connection/flow > work? > > *Often, some processors does not list any file on sftp server because of > this problem. Other Processors could list file on server sftp but at > certain time listing fails.* > *In addition sometimes fails also the unique fetchSftp processor* > > SFTP Server > 1 CPU > 2Gb Ram > Load avg. load average: 0.20, 0.27, 0.31 > > Cheers > > > > On Mon, Oct 29, 2018 at 5:41 PM Alfredo De Luca <alfredo.del...@gmail.com> > wrote: > >> Thanks Joe. Appreciated. I will get back to you asap with the info. >> >> Cheers >> >> >> On Mon, Oct 29, 2018 at 5:26 PM Joe Witt <joe.w...@gmail.com> wrote: >> >>> Alfredo >>> >>> There are quite a few details that will be needed to help folks guess >>> at possible issues. >>> >>> Configuration of the machine running nifi >>> how many cores?, etc.. >>> >>> Configuration of nifi >>> you shared common properties from nifi.properties which is good. >>> how many threads does the flow controller have? >>> how many processors are on the graph? >>> How many threads does ListSFTP have? >>> Is this a NiFi cluster or single node? >>> >>> Loading of the system running nifi >>> cpu load avg >>> garbage collection behavior in the jVM? >>> >>> Loading of nifi >>> how many threads does nifi show are running? >>> >>> Similar info for the SFTP server side >>> System error 4 could mean a lot of things. Is that system taxed and >>> unable to establish connections at times? >>> >>> How often does the problem happen? How often does the connection/flow >>> work? >>> >>> Have you tried manually connecting the way nifi would by running >>> commands like 'sftp u...@host.sftp.server -vvv' repeatedly to see if >>> you can reproduce the issue? >>> >>> Does a file system get mounted for the sftp user and periodically is >>> this failing? >>> >>> Unfortunately it could be a lot of things so we'll probably need to >>> unpack a lot of info to get there. >>> >>> Thanks >>> On Mon, Oct 29, 2018 at 12:14 PM Alfredo De Luca >>> <alfredo.del...@gmail.com> wrote: >>> > >>> > hi all. any idea? I am trying everything..but not sure what cause this >>> error. >>> > >>> > Cheers >>> > >>> > On Mon, Oct 29, 2018 at 12:34 PM Alfredo De Luca < >>> alfredo.del...@gmail.com> wrote: >>> >> >>> >> nifi.swap.in.threads=1 >>> >> nifi.swap.out.threads=4 >>> >> nifi.cluster.node.protocol.threads=25 >>> >> nifi.cluster.node.protocol.max.threads=50 >>> >> >>> >> >>> >> >>> >> >>> >> On Mon, Oct 29, 2018 at 12:28 PM Alfredo De Luca < >>> alfredo.del...@gmail.com> wrote: >>> >>> >>> >>> anyway...this is the error >>> >>> 2018-10-29 11:21:06,768 ERROR [Timer-Driven Process Thread-13] >>> SimpleProcessLogger.java:254 >>> ListSFTP[id=766ac418-27ce-335a-97c9-8823f2cf5a96] Failed to perform listing >>> on remote host due to java.io.IOException: Failed to obtain connection to >>> remote host due to com.jcraft.jsch.JSchException: Auth fail: {} >>> >>> >>> >>> >>> >>> On Mon, Oct 29, 2018 at 12:27 PM Alfredo De Luca < >>> alfredo.del...@gmail.com> wrote: >>> >>>> >>> >>>> Hi Joe. >>> >>>> Sftp server allows 200 connections... not sure about the nifi. I ll >>> check >>> >>>> >>> >>>> >>> >>>> On Mon, Oct 29, 2018 at 12:21 PM Joe Witt <joe.w...@gmail.com> >>> wrote: >>> >>>>> >>> >>>>> how many connections does your sftp server allow at once and how >>> many threads is nifi given for communication? >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> On Mon, Oct 29, 2018, 7:10 AM Alfredo De Luca < >>> alfredo.del...@gmail.com> wrote: >>> >>>>>> >>> >>>>>> Hi all. We have some issue with our nifi (v. 1.6) where >>> occasionally we have an error saying ... >>> >>>>>> Auth fail. >>> >>>>>> >>> >>>>>> On the sftp server side we get >>> >>>>>> >>> >>>>>> Oct 29 11:48:06 sftp sshd[13845]: pam_sss(sshd:account): Access >>> denied for user <sftp user>: 4 (System error) >>> >>>>>> >>> >>>>>> Any idea? >>> >>>>>> >>> >>>>>> >>> >>>>>> -- >>> >>>>>> Alfredo >>> >>>>>> >>> >>>> >>> >>>> >>> >>>> -- >>> >>>> Alfredo >>> >>>> >>> >>> >>> >>> >>> >>> -- >>> >>> Alfredo >>> >>> >>> >> >>> >> >>> >> -- >>> >> Alfredo >>> >> >>> > >>> > >>> > -- >>> > Alfredo >>> > >>> >> >> >> -- >> *Alfredo* >> >> > > -- > *Alfredo* > > -- *Alfredo*