Hi Folks,
I'm trying to track a very odd performance issue, this is on 1.6.0 using
S2S, would like to ask if there are any known issues like this or if my
flow configuration is broken. From point of view of the RPG, receiving
takes ~15x longer to xsfr the same 1.5gb file as a send from that RPG. I've
setup two simple flows and see this behavior consistently, also duplicated
the flows between two single node instances to verify the behavior follows
the xsfr direction versus the node, behavior follows the direction of xsfr,
ie a receive on both nodes is much slower than sending.
Flows are:
FlowA: GetFile_nodeA > OutputPort_nodeA > RPG_nodeB > PutFile_nodeB
FlowB: GetFile_nodeB > RPG_nodeB > InputPort_nodeA > PutFile_nodeA
For the same 1.5gb file, FlowA will consistently xsfr at ~3.5MB/s, FlowB
xsfrs at ~52.0MB/s, this is leaving default values for all processors,
connections and the RPG with the exception that RPG uses https (instead of
raw), the nodes are running secure. Same policy values were applied on both
nodes to both flows.
Aside from the latency diff, the xsfrs appear to work fine with no
anomalies that i can find, the file transfers correctly in both directions.
The one anomaly i do see is in the slow case, the destination node will
have cpu go to 100% for the majority of the 6 to 7 minutes it takes to
transfer the file, from a jstack on the thread that's using 99%+ of cpu, it
looks like this thread is spending a lot of time in
nifi.remote.util.SiteToSiteRestApiClient.read doing
LazyDecompressingInputStream/InflaterInputStream, which puzzles me quite a
bit because all of the ports have compression turned off, there should be
no compress/decompress activity, as far as i can tell.
Example stack for that thread:
"Timer-Driven Process Thread-6" #90 prio=5 os_prio=0 tid=0x00007f4c48002000
nid=0xdb38 runnable [0x00007f4c734f5000]
java.lang.Thread.State: RUNNABLE
at java.util.zip.Inflater.inflateBytes(Native Method)
at java.util.zip.Inflater.inflate(Inflater.java:259)
- locked <0x00007f55d891cf50> (a java.util.zip.ZStreamRef)
at
java.util.zip.InflaterInputStream.read(InflaterInputStream.java:152)
at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:117)
at
java.util.zip.InflaterInputStream.read(InflaterInputStream.java:122)
at
org.apache.http.client.entity.LazyDecompressingInputStream.read(LazyDecompressingInputStream.java:58)
at
org.apache.nifi.remote.util.SiteToSiteRestApiClient$3.read(SiteToSiteRestApiClient.java:722)
at java.io.InputStream.read(InputStream.java:179)
at
org.apache.nifi.remote.io.InterruptableInputStream.read(InterruptableInputStream.java:57)
at
org.apache.nifi.stream.io.ByteCountingInputStream.read(ByteCountingInputStream.java:51)
at java.util.zip.CheckedInputStream.read(CheckedInputStream.java:82)
at
org.apache.nifi.stream.io.LimitingInputStream.read(LimitingInputStream.java:88)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at
org.apache.nifi.stream.io.MinimumLengthInputStream.read(MinimumLengthInputStream.java:57)
at
org.apache.nifi.stream.io.MinimumLengthInputStream.read(MinimumLengthInputStream.java:53)
at
org.apache.nifi.controller.repository.io.TaskTerminationInputStream.read(TaskTerminationInputStream.java:62)
at org.apache.nifi.stream.io.StreamUtils.copy(StreamUtils.java:35)
at
org.apache.nifi.controller.repository.FileSystemRepository.importFrom(FileSystemRepository.java:744)
at
org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:2990)
at
org.apache.nifi.remote.StandardRemoteGroupPort.receiveFlowFiles(StandardRemoteGroupPort.java:419)
at
org.apache.nifi.remote.StandardRemoteGroupPort.onTrigger(StandardRemoteGroupPort.java:286)
at
org.apache.nifi.controller.AbstractPort.onTrigger(AbstractPort.java:250)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:175)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Has anyone seen this behavior or symptoms like this?
patw
