This looks like a thrashing behavior in compress/decompress, found that if i enable compression in the output port of the receiver's RPG, the issue goes away, throughput becomes just as good as for the sender's flow. Again though, i believe i have compression off for all flows and components. Only thing i can think of is if jetty's enforcing compression, and with an uncompressed stream has an issue, but not sure why only in one direction.
Could someone point me to where Nifi's embedded jetty configuration code is, or equiv controls? patw On Fri, Feb 1, 2019 at 4:13 PM Pat White <[email protected]> wrote: > Hi Folks, > > I'm trying to track a very odd performance issue, this is on 1.6.0 using > S2S, would like to ask if there are any known issues like this or if my > flow configuration is broken. From point of view of the RPG, receiving > takes ~15x longer to xsfr the same 1.5gb file as a send from that RPG. I've > setup two simple flows and see this behavior consistently, also duplicated > the flows between two single node instances to verify the behavior follows > the xsfr direction versus the node, behavior follows the direction of xsfr, > ie a receive on both nodes is much slower than sending. > > Flows are: > > FlowA: GetFile_nodeA > OutputPort_nodeA > RPG_nodeB > PutFile_nodeB > FlowB: GetFile_nodeB > RPG_nodeB > InputPort_nodeA > PutFile_nodeA > > For the same 1.5gb file, FlowA will consistently xsfr at ~3.5MB/s, FlowB > xsfrs at ~52.0MB/s, this is leaving default values for all processors, > connections and the RPG with the exception that RPG uses https (instead of > raw), the nodes are running secure. Same policy values were applied on both > nodes to both flows. > > Aside from the latency diff, the xsfrs appear to work fine with no > anomalies that i can find, the file transfers correctly in both directions. > The one anomaly i do see is in the slow case, the destination node will > have cpu go to 100% for the majority of the 6 to 7 minutes it takes to > transfer the file, from a jstack on the thread that's using 99%+ of cpu, it > looks like this thread is spending a lot of time in > nifi.remote.util.SiteToSiteRestApiClient.read doing > LazyDecompressingInputStream/InflaterInputStream, which puzzles me quite a > bit because all of the ports have compression turned off, there should be > no compress/decompress activity, as far as i can tell. > > Example stack for that thread: > "Timer-Driven Process Thread-6" #90 prio=5 os_prio=0 > tid=0x00007f4c48002000 nid=0xdb38 runnable [0x00007f4c734f5000] > java.lang.Thread.State: RUNNABLE > at java.util.zip.Inflater.inflateBytes(Native Method) > at java.util.zip.Inflater.inflate(Inflater.java:259) > - locked <0x00007f55d891cf50> (a java.util.zip.ZStreamRef) > at > java.util.zip.InflaterInputStream.read(InflaterInputStream.java:152) > at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:117) > at > java.util.zip.InflaterInputStream.read(InflaterInputStream.java:122) > at > org.apache.http.client.entity.LazyDecompressingInputStream.read(LazyDecompressingInputStream.java:58) > at > org.apache.nifi.remote.util.SiteToSiteRestApiClient$3.read(SiteToSiteRestApiClient.java:722) > at java.io.InputStream.read(InputStream.java:179) > at > org.apache.nifi.remote.io.InterruptableInputStream.read(InterruptableInputStream.java:57) > at > org.apache.nifi.stream.io.ByteCountingInputStream.read(ByteCountingInputStream.java:51) > at > java.util.zip.CheckedInputStream.read(CheckedInputStream.java:82) > at > org.apache.nifi.stream.io.LimitingInputStream.read(LimitingInputStream.java:88) > at java.io.FilterInputStream.read(FilterInputStream.java:133) > at > org.apache.nifi.stream.io.MinimumLengthInputStream.read(MinimumLengthInputStream.java:57) > at > org.apache.nifi.stream.io.MinimumLengthInputStream.read(MinimumLengthInputStream.java:53) > at > org.apache.nifi.controller.repository.io.TaskTerminationInputStream.read(TaskTerminationInputStream.java:62) > at org.apache.nifi.stream.io.StreamUtils.copy(StreamUtils.java:35) > at > org.apache.nifi.controller.repository.FileSystemRepository.importFrom(FileSystemRepository.java:744) > at > org.apache.nifi.controller.repository.StandardProcessSession.importFrom(StandardProcessSession.java:2990) > at > org.apache.nifi.remote.StandardRemoteGroupPort.receiveFlowFiles(StandardRemoteGroupPort.java:419) > at > org.apache.nifi.remote.StandardRemoteGroupPort.onTrigger(StandardRemoteGroupPort.java:286) > at > org.apache.nifi.controller.AbstractPort.onTrigger(AbstractPort.java:250) > at > org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:175) > at > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:748) > > Has anyone seen this behavior or symptoms like this? > > patw >
