Hi Joe.. have you had any thought about this? I don't know what else to do. Cheers
On Tue, Oct 30, 2018 at 8:18 PM Alfredo De Luca <[email protected]> wrote: > Any thought Joe? thank so much > > On Tue, Oct 30, 2018 at 2:28 PM Alfredo De Luca <[email protected]> > wrote: > >> I have attached the nifi properties. Also these are more info ... >> >> - ListSftp is running on primary node of the cluster and with 1 >> concurrent tasks >> - nifi is in cluster with 3 nodes (pod in kubernetes) >> --- >> *Loading of nifi* >> how many threads does nifi show are running? >> *150 in average* >> --- >> How often does the problem happen? How often does the connection/flow >> work? >> >> *Often, some processors does not list any file on sftp server because of >> this problem. Other Processors could list file on server sftp but at >> certain time listing fails.* >> *In addition sometimes fails also the unique fetchSftp processor* >> >> SFTP Server >> 1 CPU >> 2Gb Ram >> Load avg. load average: 0.20, 0.27, 0.31 >> >> Cheers >> >> >> >> On Mon, Oct 29, 2018 at 5:41 PM Alfredo De Luca <[email protected]> >> wrote: >> >>> Thanks Joe. Appreciated. I will get back to you asap with the info. >>> >>> Cheers >>> >>> >>> On Mon, Oct 29, 2018 at 5:26 PM Joe Witt <[email protected]> wrote: >>> >>>> Alfredo >>>> >>>> There are quite a few details that will be needed to help folks guess >>>> at possible issues. >>>> >>>> Configuration of the machine running nifi >>>> how many cores?, etc.. >>>> >>>> Configuration of nifi >>>> you shared common properties from nifi.properties which is good. >>>> how many threads does the flow controller have? >>>> how many processors are on the graph? >>>> How many threads does ListSFTP have? >>>> Is this a NiFi cluster or single node? >>>> >>>> Loading of the system running nifi >>>> cpu load avg >>>> garbage collection behavior in the jVM? >>>> >>>> Loading of nifi >>>> how many threads does nifi show are running? >>>> >>>> Similar info for the SFTP server side >>>> System error 4 could mean a lot of things. Is that system taxed and >>>> unable to establish connections at times? >>>> >>>> How often does the problem happen? How often does the connection/flow >>>> work? >>>> >>>> Have you tried manually connecting the way nifi would by running >>>> commands like 'sftp [email protected] -vvv' repeatedly to see if >>>> you can reproduce the issue? >>>> >>>> Does a file system get mounted for the sftp user and periodically is >>>> this failing? >>>> >>>> Unfortunately it could be a lot of things so we'll probably need to >>>> unpack a lot of info to get there. >>>> >>>> Thanks >>>> On Mon, Oct 29, 2018 at 12:14 PM Alfredo De Luca >>>> <[email protected]> wrote: >>>> > >>>> > hi all. any idea? I am trying everything..but not sure what cause >>>> this error. >>>> > >>>> > Cheers >>>> > >>>> > On Mon, Oct 29, 2018 at 12:34 PM Alfredo De Luca < >>>> [email protected]> wrote: >>>> >> >>>> >> nifi.swap.in.threads=1 >>>> >> nifi.swap.out.threads=4 >>>> >> nifi.cluster.node.protocol.threads=25 >>>> >> nifi.cluster.node.protocol.max.threads=50 >>>> >> >>>> >> >>>> >> >>>> >> >>>> >> On Mon, Oct 29, 2018 at 12:28 PM Alfredo De Luca < >>>> [email protected]> wrote: >>>> >>> >>>> >>> anyway...this is the error >>>> >>> 2018-10-29 11:21:06,768 ERROR [Timer-Driven Process Thread-13] >>>> SimpleProcessLogger.java:254 >>>> ListSFTP[id=766ac418-27ce-335a-97c9-8823f2cf5a96] Failed to perform listing >>>> on remote host due to java.io.IOException: Failed to obtain connection to >>>> remote host due to com.jcraft.jsch.JSchException: Auth fail: {} >>>> >>> >>>> >>> >>>> >>> On Mon, Oct 29, 2018 at 12:27 PM Alfredo De Luca < >>>> [email protected]> wrote: >>>> >>>> >>>> >>>> Hi Joe. >>>> >>>> Sftp server allows 200 connections... not sure about the nifi. I >>>> ll check >>>> >>>> >>>> >>>> >>>> >>>> On Mon, Oct 29, 2018 at 12:21 PM Joe Witt <[email protected]> >>>> wrote: >>>> >>>>> >>>> >>>>> how many connections does your sftp server allow at once and how >>>> many threads is nifi given for communication? >>>> >>>>> >>>> >>>>> >>>> >>>>> >>>> >>>>> On Mon, Oct 29, 2018, 7:10 AM Alfredo De Luca < >>>> [email protected]> wrote: >>>> >>>>>> >>>> >>>>>> Hi all. We have some issue with our nifi (v. 1.6) where >>>> occasionally we have an error saying ... >>>> >>>>>> Auth fail. >>>> >>>>>> >>>> >>>>>> On the sftp server side we get >>>> >>>>>> >>>> >>>>>> Oct 29 11:48:06 sftp sshd[13845]: pam_sss(sshd:account): Access >>>> denied for user <sftp user>: 4 (System error) >>>> >>>>>> >>>> >>>>>> Any idea? >>>> >>>>>> >>>> >>>>>> >>>> >>>>>> -- >>>> >>>>>> Alfredo >>>> >>>>>> >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> >>>> Alfredo >>>> >>>> >>>> >>> >>>> >>> >>>> >>> -- >>>> >>> Alfredo >>>> >>> >>>> >> >>>> >> >>>> >> -- >>>> >> Alfredo >>>> >> >>>> > >>>> > >>>> > -- >>>> > Alfredo >>>> > >>>> >>> >>> >>> -- >>> *Alfredo* >>> >>> >> >> -- >> *Alfredo* >> >> > > -- > *Alfredo* > > -- *Alfredo*
