I have attached the nifi properties. Also these are more info ...
- ListSftp is running on primary node of the cluster and with 1 concurrent
tasks
- nifi is in cluster with 3 nodes (pod in kubernetes)
---
*Loading of nifi*
how many threads does nifi show are running?
*150 in average*
---
How often does the problem happen? How often does the connection/flow work?
*Often, some processors does not list any file on sftp server because of
this problem. Other Processors could list file on server sftp but at
certain time listing fails.*
*In addition sometimes fails also the unique fetchSftp processor*
SFTP Server
1 CPU
2Gb Ram
Load avg. load average: 0.20, 0.27, 0.31
Cheers
On Mon, Oct 29, 2018 at 5:41 PM Alfredo De Luca <[email protected]>
wrote:
> Thanks Joe. Appreciated. I will get back to you asap with the info.
>
> Cheers
>
>
> On Mon, Oct 29, 2018 at 5:26 PM Joe Witt <[email protected]> wrote:
>
>> Alfredo
>>
>> There are quite a few details that will be needed to help folks guess
>> at possible issues.
>>
>> Configuration of the machine running nifi
>> how many cores?, etc..
>>
>> Configuration of nifi
>> you shared common properties from nifi.properties which is good.
>> how many threads does the flow controller have?
>> how many processors are on the graph?
>> How many threads does ListSFTP have?
>> Is this a NiFi cluster or single node?
>>
>> Loading of the system running nifi
>> cpu load avg
>> garbage collection behavior in the jVM?
>>
>> Loading of nifi
>> how many threads does nifi show are running?
>>
>> Similar info for the SFTP server side
>> System error 4 could mean a lot of things. Is that system taxed and
>> unable to establish connections at times?
>>
>> How often does the problem happen? How often does the connection/flow
>> work?
>>
>> Have you tried manually connecting the way nifi would by running
>> commands like 'sftp [email protected] -vvv' repeatedly to see if
>> you can reproduce the issue?
>>
>> Does a file system get mounted for the sftp user and periodically is
>> this failing?
>>
>> Unfortunately it could be a lot of things so we'll probably need to
>> unpack a lot of info to get there.
>>
>> Thanks
>> On Mon, Oct 29, 2018 at 12:14 PM Alfredo De Luca
>> <[email protected]> wrote:
>> >
>> > hi all. any idea? I am trying everything..but not sure what cause this
>> error.
>> >
>> > Cheers
>> >
>> > On Mon, Oct 29, 2018 at 12:34 PM Alfredo De Luca <
>> [email protected]> wrote:
>> >>
>> >> nifi.swap.in.threads=1
>> >> nifi.swap.out.threads=4
>> >> nifi.cluster.node.protocol.threads=25
>> >> nifi.cluster.node.protocol.max.threads=50
>> >>
>> >>
>> >>
>> >>
>> >> On Mon, Oct 29, 2018 at 12:28 PM Alfredo De Luca <
>> [email protected]> wrote:
>> >>>
>> >>> anyway...this is the error
>> >>> 2018-10-29 11:21:06,768 ERROR [Timer-Driven Process Thread-13]
>> SimpleProcessLogger.java:254
>> ListSFTP[id=766ac418-27ce-335a-97c9-8823f2cf5a96] Failed to perform listing
>> on remote host due to java.io.IOException: Failed to obtain connection to
>> remote host due to com.jcraft.jsch.JSchException: Auth fail: {}
>> >>>
>> >>>
>> >>> On Mon, Oct 29, 2018 at 12:27 PM Alfredo De Luca <
>> [email protected]> wrote:
>> >>>>
>> >>>> Hi Joe.
>> >>>> Sftp server allows 200 connections... not sure about the nifi. I ll
>> check
>> >>>>
>> >>>>
>> >>>> On Mon, Oct 29, 2018 at 12:21 PM Joe Witt <[email protected]>
>> wrote:
>> >>>>>
>> >>>>> how many connections does your sftp server allow at once and how
>> many threads is nifi given for communication?
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> On Mon, Oct 29, 2018, 7:10 AM Alfredo De Luca <
>> [email protected]> wrote:
>> >>>>>>
>> >>>>>> Hi all. We have some issue with our nifi (v. 1.6) where
>> occasionally we have an error saying ...
>> >>>>>> Auth fail.
>> >>>>>>
>> >>>>>> On the sftp server side we get
>> >>>>>>
>> >>>>>> Oct 29 11:48:06 sftp sshd[13845]: pam_sss(sshd:account): Access
>> denied for user <sftp user>: 4 (System error)
>> >>>>>>
>> >>>>>> Any idea?
>> >>>>>>
>> >>>>>>
>> >>>>>> --
>> >>>>>> Alfredo
>> >>>>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Alfredo
>> >>>>
>> >>>
>> >>>
>> >>> --
>> >>> Alfredo
>> >>>
>> >>
>> >>
>> >> --
>> >> Alfredo
>> >>
>> >
>> >
>> > --
>> > Alfredo
>> >
>>
>
>
> --
> *Alfredo*
>
>
--
*Alfredo*
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Core Properties #
nifi.flow.configuration.file=./data/flow.xml.gz
nifi.flow.configuration.archive.enabled=true
nifi.flow.configuration.archive.dir=./data/archive/
nifi.flow.configuration.archive.max.time=30 days
nifi.flow.configuration.archive.max.storage=500 MB
nifi.flow.configuration.archive.max.count=
nifi.flowcontroller.autoResumeState=true
nifi.flowcontroller.graceful.shutdown.period=10 sec
nifi.flowservice.writedelay.interval=500 ms
nifi.administrative.yield.duration=30 sec
# If a component has no work to do (is "bored"), how long should we wait before checking again for work?
nifi.bored.yield.duration=10 millis
nifi.authorizer.configuration.file=./conf/authorizers.xml
nifi.login.identity.provider.configuration.file=./conf/login-identity-providers.xml
nifi.templates.directory=./data/templates
nifi.ui.banner.text=
nifi.ui.autorefresh.interval=30 sec
nifi.nar.library.directory=./lib
nifi.nar.working.directory=./work/nar/
nifi.documentation.working.directory=./work/docs/components
####################
# State Management #
####################
nifi.state.management.configuration.file=./conf/state-management.xml
# The ID of the local state provider
nifi.state.management.provider.local=local-provider
# The ID of the cluster-wide state provider. This will be ignored if NiFi is not clustered but must be populated if running in a cluster.
nifi.state.management.provider.cluster=zk-provider
# Specifies whether or not this instance of NiFi should run an embedded ZooKeeper server
nifi.state.management.embedded.zookeeper.start=false
# Properties file that provides the ZooKeeper properties to use if <nifi.state.management.embedded.zookeeper.start> is set to true
nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties
# H2 Settings
nifi.database.directory=./data/database_repository
nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE
# FlowFile Repository
nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
nifi.flowfile.repository.directory=./data/flowfile_repository
nifi.flowfile.repository.partitions=4
nifi.flowfile.repository.checkpoint.interval=2 mins
nifi.flowfile.repository.always.sync=true
nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
nifi.queue.swap.threshold=20000
nifi.swap.in.period=5 sec
nifi.swap.in.threads=1
nifi.swap.out.period=5 sec
nifi.swap.out.threads=4
# Content Repository
nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
nifi.content.claim.max.appendable.size=10 MB
nifi.content.claim.max.flow.files=100
nifi.content.repository.directory.default=./data/content_repository
nifi.content.repository.archive.max.retention.period=12 hours
nifi.content.repository.archive.max.usage.percentage=50%
nifi.content.repository.archive.enabled=true
nifi.content.repository.always.sync=false
nifi.content.viewer.url=/nifi-content-viewer/
# Provenance Repository Properties
nifi.provenance.repository.implementation=com.thinkbiganalytics.nifi.provenance.repo.KyloVolatileProvenanceEventRepository
# nifi.provenance.repository.implementation=org.apache.nifi.provenance.PersistentProvenanceRepository
nifi.provenance.repository.debug.frequency=1_000_000
nifi.provenance.repository.encryption.key.provider.implementation=
nifi.provenance.repository.encryption.key.provider.location=
nifi.provenance.repository.encryption.key.id=
nifi.provenance.repository.encryption.key=
# Persistent Provenance Repository Properties
nifi.provenance.repository.directory.default=./data/provenance_repository
nifi.provenance.repository.max.storage.time=24 hours
nifi.provenance.repository.max.storage.size=1 GB
nifi.provenance.repository.rollover.time=30 secs
nifi.provenance.repository.rollover.size=100 MB
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.index.threads=2
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.always.sync=false
nifi.provenance.repository.journal.count=16
nifi.provenance.repository.concurrent.merge.threads=1
# Comma-separated list of fields. Fields that are not indexed will not be searchable. Valid fields are:
# EventType, FlowFileUUID, Filename, TransitURI, ProcessorID, AlternateIdentifierURI, Relationship, Details
nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID, Relationship
# FlowFile Attributes that should be indexed and made searchable. Some examples to consider are filename, uuid, mime.type
nifi.provenance.repository.indexed.attributes=
# Large values for the shard size will result in more Java heap usage when searching the Provenance Repository
# but should provide better performance
nifi.provenance.repository.index.shard.size=500 MB
# Indicates the maximum length that a FlowFile attribute can be when retrieving a Provenance Event from
# the repository. If the length of any attribute exceeds this value, it will be truncated when the event is retrieved.
nifi.provenance.repository.max.attribute.length=65536
# Volatile Provenance Respository Properties
nifi.provenance.repository.buffer.size=100000
# Component Status Repository
nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository
nifi.components.status.repository.buffer.size=1440
nifi.components.status.snapshot.frequency=1 min
# Site to Site properties
nifi.remote.input.host=
nifi.remote.input.secure=false
nifi.remote.input.socket.port=9998
nifi.remote.input.http.enabled=true
nifi.remote.input.http.transaction.ttl=60 secs
nifi.remote.contents.cache.expiration=60 secs
# web properties #
nifi.web.war.directory=./lib
nifi.web.http.host=
nifi.web.http.port=8080
nifi.web.http.network.interface.default=eth0
nifi.web.https.host=
nifi.web.https.port=
nifi.web.https.network.interface.default=
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=200
# security properties #
nifi.sensitive.props.key=
nifi.sensitive.props.key.protected=
nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
nifi.sensitive.props.provider=BC
nifi.sensitive.props.additional.keys=
nifi.security.keystore=
nifi.security.keystoreType=
nifi.security.keystorePasswd=
nifi.security.keyPasswd=
nifi.security.truststore=
nifi.security.truststoreType=
nifi.security.truststorePasswd=
nifi.security.needClientAuth=
nifi.security.user.authorizer=file-provider
nifi.security.user.login.identity.provider=
nifi.security.ocsp.responder.url=
nifi.security.ocsp.responder.certificate=
# Identity Mapping Properties #
# These properties allow normalizing user identities such that identities coming from different identity providers
# (certificates, LDAP, Kerberos) can be treated the same internally in NiFi. The following example demonstrates normalizing
# DNs from certificates and principals from Kerberos into a common identity string:
#
# nifi.security.identity.mapping.pattern.dn=^CN=(.*?), OU=(.*?), O=(.*?), L=(.*?), ST=(.*?), C=(.*?)$
# nifi.security.identity.mapping.value.dn=$1@$2
# nifi.security.identity.mapping.pattern.kerb=^(.*?)/instance@(.*?)$
# nifi.security.identity.mapping.value.kerb=$1@$2
# cluster common properties (all nodes must have same values) #
nifi.cluster.protocol.heartbeat.interval=15 sec
nifi.cluster.protocol.is.secure=false
# cluster node properties (only configure for cluster nodes) #
nifi.cluster.is.node=true
nifi.cluster.node.address=
nifi.cluster.node.protocol.port=9999
nifi.cluster.node.protocol.threads=25
nifi.cluster.node.protocol.max.threads=50
nifi.cluster.node.event.history.size=25
nifi.cluster.node.connection.timeout=250 secs
nifi.cluster.node.read.timeout=250 secs
nifi.cluster.firewall.file=
nifi.cluster.flow.election.max.wait.time=5 mins
nifi.cluster.flow.election.max.candidates=2
nifi.cluster.node.max.concurrent.requests=1000
nifi.cluster.node.max.concurrent.requests=500
nifi.cluster.node.request.replication.claim.timeout=20 secs
# zookeeper properties, used for cluster management #
nifi.zookeeper.connect.string=master:2181,master-2:2181,master-3:2181
nifi.zookeeper.connect.timeout=10 secs
nifi.zookeeper.session.timeout=10 secs
nifi.zookeeper.root.node=/tba-nifi-cluster-2
# kerberos #
nifi.kerberos.krb5.file=/etc/krb5.conf
# kerberos service principal #
[email protected]
nifi.kerberos.service.keytab.location=/usr/nifi/daf.keytab
# kerberos spnego principal #
nifi.kerberos.spnego.principal=
nifi.kerberos.spnego.keytab.location=
nifi.kerberos.spnego.authentication.expiration=12 hours
# external properties files for variable registry
# supports a comma delimited list of file locations
nifi.variable.registry.properties=
nifi.web.http.network.interface.lo=lo
