[ 
https://issues.apache.org/jira/browse/NIFI-4905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16374391#comment-16374391
 ] 

Josef Zahner commented on NIFI-4905:
------------------------------------

Ok, so to find out which processor causes the high load I need to remove step 
by step the config - hmpf a lot of work :(.

Here what you requested:
{code:java}
[root@nifi2-08 ~]# grep -i "<class>" /tmp/flow.xml 
<class>org.apache.nifi.processors.standard.RouteOnAttribute</class>
<class>org.apache.nifi.processors.standard.PutSFTP</class>
<class>org.apache.nifi.processors.attributes.UpdateAttribute</class>
<class>org.apache.nifi.processors.standard.FetchSFTP</class>
<class>org.apache.nifi.processors.standard.CompressContent</class>
<class>org.apache.nifi.processors.standard.UpdateRecord</class>
<class>org.apache.nifi.processors.standard.ListSFTP</class>
<class>com.xyz.nifi.processors.kudu.PutKuduNull</class>
<class>org.apache.nifi.avro.AvroRecordSetWriter</class>
<class>org.apache.nifi.record.script.ScriptedAvscReader2</class>
<class>org.apache.nifi.processors.standard.UpdateCounter</class>
<class>com.xyz.nifi.processors.kudu.PutKuduNull</class>
<class>org.apache.nifi.processors.standard.UpdateCounter</class>
<class>org.apache.nifi.processors.parquet.PutParquet</class>
<class>org.apache.nifi.processors.standard.MergeRecord</class>
<class>org.apache.nifi.processors.attributes.UpdateAttribute</class>
<class>org.apache.nifi.processors.standard.PutFile</class>
<class>org.apache.nifi.processors.attributes.UpdateAttribute</class>
<class>org.apache.nifi.processors.standard.UpdateCounter</class>
<class>com.xyz.nifi.processors.kudu.PutKuduNull</class>
<class>com.xyz.nifi.processors.kudu.PutKuduNull</class>
<class>org.apache.nifi.avro.AvroRecordSetWriter</class>
<class>org.apache.nifi.avro.AvroReader</class>
<class>org.apache.nifi.processors.standard.PutSFTP</class>
<class>org.apache.nifi.processors.standard.ListSFTP</class>
<class>org.apache.nifi.processors.standard.FetchSFTP</class>
<class>com.xyz.nnp.processors.cdrdecode.CDR2AvroProcessor</class>
<class>org.apache.nifi.processors.standard.CompressContent</class>
<class>org.apache.nifi.processors.attributes.UpdateAttribute</class>
<class>org.apache.nifi.processors.standard.ListSFTP</class>
<class>org.apache.nifi.processors.standard.FetchSFTP</class>
<class>com.xyz.nifi.processors.kudu.PutKuduNull</class>
<class>org.apache.nifi.processors.standard.ConvertRecord</class>
<class>org.apache.nifi.processors.standard.CompressContent</class>
<class>org.apache.nifi.processors.standard.PutSFTP</class>
<class>org.apache.nifi.processors.attributes.UpdateAttribute</class>
<class>org.apache.nifi.avro.AvroRecordSetWriter</class>
<class>org.apache.nifi.record.script.ScriptedAvscReader2</class>
<class>org.apache.nifi.processors.standard.ConvertRecord</class>
<class>org.apache.nifi.processors.standard.ListSFTP</class>
<class>org.apache.nifi.processors.standard.CompressContent</class>
<class>org.apache.nifi.processors.standard.PutSFTP</class>
<class>org.apache.nifi.processors.attributes.UpdateAttribute</class>
<class>org.apache.nifi.processors.standard.FetchSFTP</class>
<class>org.apache.nifi.avro.AvroRecordSetWriter</class>
<class>com.xyz.lidr.processors.smeradius.SmeRadiusReader</class>
<class>com.xyz.nifi.processors.kudu.PutKuduNull</class>
<class>org.apache.nifi.processors.standard.PutSFTP</class>
<class>org.apache.nifi.processors.standard.UpdateCounter</class>
<class>org.apache.nifi.processors.standard.UpdateCounter</class>
<class>org.apache.nifi.processors.parquet.PutParquet</class>
<class>org.apache.nifi.processors.attributes.UpdateAttribute</class>
<class>org.apache.nifi.processors.standard.PutSFTP</class>
<class>org.apache.nifi.processors.standard.GetFile</class>
<class>org.apache.nifi.processors.standard.MergeRecord</class>
<class>org.apache.nifi.avro.AvroReader</class>
<class>org.apache.nifi.avro.AvroRecordSetWriter</class>
<class>org.apache.nifi.processors.attributes.UpdateAttribute</class>
<class>org.apache.nifi.processors.standard.ListSFTP</class>
<class>org.apache.nifi.processors.standard.PutSFTP</class>
<class>org.apache.nifi.processors.standard.CompressContent</class>
<class>org.apache.nifi.processors.standard.FetchSFTP</class>
<class>org.apache.nifi.processors.standard.ConvertRecord</class>
<class>com.xyz.lidr.processors.smeradius.SmeRadiusReader</class>
<class>org.apache.nifi.avro.AvroRecordSetWriter</class>
<class>org.apache.nifi.processors.standard.ListSFTP</class>
<class>com.xyz.nnp.processors.cdrdecode.CDR2AvroProcessor</class>
<class>org.apache.nifi.processors.attributes.UpdateAttribute</class>
<class>org.apache.nifi.processors.standard.FetchSFTP</class>
<class>org.apache.nifi.processors.standard.CompressContent</class>
<class>org.apache.nifi.processors.standard.PutSFTP</class>
<class>org.apache.nifi.processors.standard.GenerateFlowFile</class>
<class>org.apache.nifi.processors.attributes.UpdateAttribute</class>
<class>org.apache.nifi.processors.standard.FetchSFTP</class>
<class>com.xyz.nifi.processors.kudu.PutKuduNull</class>
<class>org.apache.nifi.processors.standard.ListSFTP</class>
<class>org.apache.nifi.processors.standard.CompressContent</class>
<class>org.apache.nifi.processors.standard.ConvertRecord</class>
<class>org.apache.nifi.processors.standard.PutSFTP</class>
<class>org.apache.nifi.avro.AvroReader</class>
<class>org.apache.nifi.avro.AvroRecordSetWriter</class>
<class>org.apache.nifi.record.script.ScriptedAvscReader2</class>
<class>com.xyz.nnp.processors.cdrdecode.CDR2AvroProcessor</class>
<class>org.apache.nifi.processors.standard.ListSFTP</class>
<class>org.apache.nifi.processors.attributes.UpdateAttribute</class>
<class>com.xyz.nifi.processors.kudu.PutKuduNull</class>
<class>org.apache.nifi.processors.standard.FetchSFTP</class>
<class>org.apache.nifi.processors.standard.PutSFTP</class>
<class>org.apache.nifi.processors.standard.ListSFTP</class>
<class>org.apache.nifi.processors.parquet.PutParquet</class>
<class>org.apache.nifi.processors.standard.UpdateRecordValues</class>
<class>org.apache.nifi.processors.standard.PutSFTP</class>
<class>org.apache.nifi.processors.kite.InferAvroSchema</class>
<class>org.apache.nifi.processors.attributes.UpdateAttribute</class>
<class>com.xyz.nifi.processors.kudu.PutKuduNull</class>
<class>com.xyz.nnp.processors.cdrdecode.CDR2AvroProcessor</class>
<class>org.apache.nifi.processors.standard.CompressContent</class>
<class>org.apache.nifi.processors.standard.FetchSFTP</class>
<class>org.apache.nifi.processors.kite.ConvertCSVToAvro</class>
<class>org.apache.nifi.processors.standard.ListSFTP</class>
<class>org.apache.nifi.processors.kiteext.FlatConvertAvroSchema</class>
<class>org.apache.nifi.processors.standard.UpdateCounter</class>
<class>org.apache.nifi.avro.AvroReader</class>
<class>org.apache.nifi.avro.AvroRecordSetWriter</class>
<class>org.apache.nifi.processors.standard.PutEmail</class>
<class>org.apache.nifi.processors.standard.ListSFTP</class>
<class>org.apache.nifi.processors.standard.ListSFTP</class>
<class>org.apache.nifi.processors.standard.CompressContent</class>
<class>org.apache.nifi.processors.standard.FetchSFTP</class>
<class>org.apache.nifi.processors.standard.ListSFTP</class>
<class>org.apache.nifi.processors.standard.PutSFTP</class>
<class>com.xyz.nnp.processors.cdrdecode.CDR2AvroProcessor</class>
<class>org.apache.nifi.processors.standard.CompressContent</class>
<class>org.apache.nifi.processors.standard.FetchSFTP</class>
<class>org.apache.nifi.processors.attributes.UpdateAttribute</class>
<class>com.xyz.nnp.processors.cdrdecode.CDR2AvroProcessor</class>
<class>org.apache.nifi.processors.standard.PutSFTP</class>
<class>com.xyz.nifi.processors.kudu.PutKuduNull</class>
<class>org.apache.nifi.processors.standard.FetchSFTP</class>
<class>org.apache.nifi.processors.standard.CompressContent</class>
<class>org.apache.nifi.processors.standard.ConvertRecord</class>
<class>org.apache.nifi.processors.standard.ListSFTP</class>
<class>org.apache.nifi.processors.attributes.UpdateAttribute</class>
<class>org.apache.nifi.record.script.ScriptedAvscReader2</class>
<class>org.apache.nifi.avro.AvroReader</class>
<class>org.apache.nifi.avro.AvroRecordSetWriter</class>
<class>org.apache.nifi.processors.standard.ListSFTP</class>
<class>org.apache.nifi.processors.standard.PutSFTP</class>
<class>org.apache.nifi.processors.standard.CompressContent</class>
<class>org.apache.nifi.processors.standard.ConvertRecord</class>
<class>org.apache.nifi.processors.attributes.UpdateAttribute</class>
<class>org.apache.nifi.processors.standard.FetchSFTP</class>
<class>org.apache.nifi.avro.AvroRecordSetWriter</class>
<class>com.xyz.lidr.processors.smeradius.SmeRadiusReader</class>
<class>org.apache.nifi.processors.standard.FetchSFTP</class>
<class>org.apache.nifi.processors.standard.PutSFTP</class>
<class>com.xyz.nnp.processors.cdrdecode.CDR2AvroProcessor</class>
<class>org.apache.nifi.processors.attributes.UpdateAttribute</class>
<class>org.apache.nifi.processors.standard.CompressContent</class>
<class>org.apache.nifi.processors.standard.ListSFTP</class>
<class>com.xyz.nnp.processors.cdrdecode.CDR2AvroProcessor</class>
<class>org.apache.nifi.processors.standard.CompressContent</class>
<class>org.apache.nifi.processors.standard.ListSFTP</class>
<class>org.apache.nifi.processors.standard.FetchSFTP</class>
<class>org.apache.nifi.processors.standard.PutSFTP</class>
<class>com.xyz.lidr.cdrdecode.FlatConvertAvroSchema</class>
<class>org.apache.nifi.processors.hive.PutHiveQL</class>
<class>org.apache.nifi.processors.standard.FetchSFTP</class>
<class>org.apache.nifi.processors.kafka.pubsub.PublishKafka_0_10</class>
<class>org.apache.nifi.processors.standard.ListSFTP</class>
<class>org.apache.nifi.processors.kafka.pubsub.PublishKafka_0_10</class>
<class>org.apache.nifi.processors.kafka.pubsub.ConsumeKafka_0_10</class>
<class>org.apache.nifi.processors.hadoop.PutHDFS</class>
<class>org.apache.nifi.processors.kafka.pubsub.PublishKafkaRecord_0_10</class>
<class>org.apache.nifi.processors.standard.ReplaceText</class>
<class>org.apache.nifi.avro.AvroReader</class>
<class>org.apache.nifi.avro.AvroRecordSetWriter</class>
<class>org.apache.nifi.processors.standard.PutEmail</class>
<class>org.apache.nifi.processors.standard.GetFile</class>
<class>org.apache.nifi.processors.standard.UpdateCounter</class>
<class>org.apache.nifi.processors.standard.UpdateCounter</class>
<class>org.apache.nifi.processors.standard.MonitorActivity</class>
<class>org.apache.nifi.processors.standard.HandleHttpRequest</class>
<class>org.apache.nifi.processors.standard.RouteOnAttribute</class>
<class>org.apache.nifi.processors.standard.HandleHttpResponse</class>
<class>org.apache.nifi.processors.standard.HandleHttpResponse</class>
<class>com.xyz.nifi.processors.kudu.GetKudu</class>
<class>com.xyz.nifi.processors.kudu.GetKudu</class>
<class>org.apache.nifi.csv.CSVRecordSetWriter</class>
<class>org.apache.nifi.record.script.ScriptedRecordSetWriter</class>
<class>com.xyz.nifi.processors.kudu.GenerateKuduQuery</class>
<class>org.apache.nifi.processors.standard.MergeRecord</class>
<class>com.xyz.nifi.processors.kudu.RunKuduQuery</class>
<class>org.apache.nifi.processors.standard.RouteOnAttribute</class>
<class>org.apache.nifi.csv.CSVRecordSetWriter</class>
<class>org.apache.nifi.avro.AvroReader</class>
<class>org.apache.nifi.avro.AvroRecordSetWriter</class>
<class>org.apache.nifi.json.JsonTreeReader</class>
<class>org.apache.nifi.json.JsonRecordSetWriter</class>
<class>org.apache.nifi.http.StandardHttpContextMap</class>
<class>org.apache.nifi.processors.standard.UpdateRecord</class>
<class>org.apache.nifi.processors.standard.GetFile</class>
<class>org.apache.nifi.processors.kudu.PutKudu</class>
<class>org.apache.nifi.avro.AvroRecordSetWriter</class>
<class>org.apache.nifi.record.script.ScriptedAvscReader</class>
<class>org.apache.nifi.avro.AvroReader</class>
<class>org.apache.nifi.processors.standard.FetchSFTP</class>
<class>org.apache.nifi.processors.standard.ReplaceText</class>
<class>org.apache.nifi.processors.standard.ListSFTP</class>
<class>org.apache.nifi.processors.attributes.UpdateAttribute</class>
<class>org.apache.nifi.processors.standard.FetchFile</class>
<class>org.apache.nifi.processors.standard.PutFile</class>
<class>org.apache.nifi.processors.standard.ReplaceText</class>
<class>org.apache.nifi.processors.attributes.UpdateAttribute</class>
<class>org.apache.nifi.processors.standard.ListFile</class>
<class>org.apache.nifi.processors.standard.MergeContent</class>
<class>org.apache.nifi.processors.standard.PutSFTP</class>
<class>org.apache.nifi.processors.standard.ListSFTP</class>
<class>org.apache.nifi.processors.attributes.UpdateAttribute</class>
<class>org.apache.nifi.processors.standard.FetchSFTP</class>
<class>org.apache.nifi.processors.standard.CompressContent</class>
<class>org.apache.nifi.processors.standard.ConvertRecord</class>
<class>com.xyz.lidr.processors.smeradius.SmeRadiusReader</class>
<class>org.apache.nifi.avro.AvroRecordSetWriter</class>
<class>org.apache.nifi.schemaregistry.services.AvroSchemaRegistry</class>
<class>org.apache.nifi.reporting.prometheus.PrometheusReportingTask</class>
{code}

> Cluster response time is slow
> -----------------------------
>
>                 Key: NIFI-4905
>                 URL: https://issues.apache.org/jira/browse/NIFI-4905
>             Project: Apache NiFi
>          Issue Type: Bug
>          Components: Core UI
>    Affects Versions: 1.5.0
>         Environment: Centos 7.3
> NiFi 1.5.0 Cluster with 8 physical nodes
>            Reporter: Josef Zahner
>            Priority: Major
>         Attachments: cpu_load_nifi.PNG, root_canvas.png
>
>
> we are actually working on a PoC with 8 nodes (HP BL460c Blades, 24 Cores, 
> 44GB RAM) in a NiFi 1.5.0 cluster. Our configuration has about 160 processors 
> and all of them are stopped. Even in the stopped state, we are constantly 
> getting the messages below for all nodes, not only for the primary node.
> {code:java}
> Response time from nifi2-07.xyz.ch:8080 was slow for each of the last 3 
> requests made. To see more information about timing, enable DEBUG logging for 
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator
> {code}
> If you are on the root canvas, you feel that it takes a few seconds until it 
> response after a refresh. We have already tuned the parameters below, but 
> without any luck. After restart of NiFi it is fine for a few minutes, but 
> then the messages return.
> {code:java}
> nifi.cluster.protocol.heartbeat.interval=15 sec
> nifi.cluster.node.protocol.threads=40
> nifi.cluster.node.protocol.max.threads=80
> nifi.cluster.node.connection.timeout=60 sec
> nifi.cluster.node.read.timeout=60 sec
> {code}
> The nodes have absolutely no load beside of NiFi. What surprised me was, that 
> when the UI refreshs (default every 30s) it produces about 20% cpu load on my 
> machine. And remember, it's a 24x2.9GHz blade server.
> !cpu_load_nifi.PNG!
> That's a picture of my root canvas:
> !root_canvas.png!
> Actually we can't work with the cluster under this circumstances because the 
> gui always gets so slow.
> Is this a bug or normal behavior? Do we have to many elements on the GUI or 
> what could cause this issue?
> Cheers
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to