Re: Asymmetric push/pull throughput with S2S, possibly related to openConnectionForReceive compression?

2019-02-15 Thread Pat White
Sounds great Koji, thank you for looking into that. I'm trying some tests with changes in GzipHandler included methods, will update if i have any useful info from that. patw On Fri, Feb 15, 2019 at 3:39 AM Koji Kawamura wrote: > Hi Pat, > > Thanks for sharing your insights. > I will try

Re: Asymmetric push/pull throughput with S2S, possibly related to openConnectionForReceive compression?

2019-02-15 Thread Koji Kawamura
Hi Pat, Thanks for sharing your insights. I will try benchmarking before and after "gzip.setExcludedPath()" that Mark has suggested if it helps improving S2S HTTP throughput. Koji On Fri, Feb 15, 2019 at 9:31 AM Pat White wrote: > > Hi Andy, > > My requirement is to use https with minimum tls

Re: Asymmetric push/pull throughput with S2S, possibly related to openConnectionForReceive compression?

2019-02-14 Thread Pat White
Hi Andy, My requirement is to use https with minimum tls v1.2, https being an approved protocol. I haven't looked at websockets though, i need to do that, thank you for the suggestion. patw On Thu, Feb 14, 2019 at 12:24 PM Andy LoPresto wrote: > Pat, > > Just to clarify, your connection

Re: Asymmetric push/pull throughput with S2S, possibly related to openConnectionForReceive compression?

2019-02-14 Thread Pat White
Thank you Bryan, i have a lot to look at using raw protocol, will be sure to keep this setting in mind. patw On Thu, Feb 14, 2019 at 12:05 PM Bryan Bende wrote: > You can use TLS with raw s2s by setting nifi.remote.input.secure=true > > On Thu, Feb 14, 2019 at 12:56 PM Pat White > wrote: > >

Re: Asymmetric push/pull throughput with S2S, possibly related to openConnectionForReceive compression?

2019-02-14 Thread Andy LoPresto
Pat, Just to clarify, your connection must be HTTPS or it just must be secure? What about Websockets over TLS (wss://)? Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 > On Feb 14, 2019, at 9:56 AM, Pat White

Re: Asymmetric push/pull throughput with S2S, possibly related to openConnectionForReceive compression?

2019-02-14 Thread Bryan Bende
You can use TLS with raw s2s by setting nifi.remote.input.secure=true On Thu, Feb 14, 2019 at 12:56 PM Pat White wrote: > > Thanks very much folks, definitely appreciate the feedback. > > Right, required to use tls/https connections for s2s, so raw is not an option > for me. > > Will look

Re: Asymmetric push/pull throughput with S2S, possibly related to openConnectionForReceive compression?

2019-02-14 Thread Pat White
Thanks very much folks, definitely appreciate the feedback. Right, required to use tls/https connections for s2s, so raw is not an option for me. Will look further at JettyServer and setIncludedMethods, thanks again. patw On Thu, Feb 14, 2019 at 11:07 AM Mark Payne wrote: > Pat, > > It

Re: Asymmetric push/pull throughput with S2S, possibly related to openConnectionForReceive compression?

2019-02-14 Thread Mark Payne
Pat, It appears to be hard-coded, in JettyServer (full path is nifi/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-jetty/src/main/java/org/apache/nifi/web/server/JettyServer.java ) Line 294 calls the gzip method, which looks like: private Handler gzip(final Handler

Re: Asymmetric push/pull throughput with S2S, possibly related to openConnectionForReceive compression?

2019-02-14 Thread Bryan Bende
I admit I haven't read this thread in detail, but is it a requirement for you to use HTTP site-to-site? I would think you could avoid this issue by using traditional raw site-to-site which is over a direct socket and not hitting Jetty. If you do want to modify Jetty's configuration you would

Re: Asymmetric push/pull throughput with S2S, possibly related to openConnectionForReceive compression?

2019-02-14 Thread Joe Witt
...interesting. I dont have an answer but will initiate some research. Hopefully someone else replies if they know off-hand. Thanks On Thu, Feb 14, 2019 at 11:43 AM Pat White wrote: > Hi Folks, > > Could someone point me at the correct way to modify Nifi's embedded jetty > configuration

Re: Asymmetric push/pull throughput with S2S, possibly related to openConnectionForReceive compression?

2019-02-14 Thread Pat White
Hi Folks, Could someone point me at the correct way to modify Nifi's embedded jetty configuration settings? Specifically i'd like to turn off jetty's automatic compression of payload. Reason for asking, think i've found my performance issue, uncompressed input to jetty is getting automatically

Re: Asymmetric push/pull throughput with S2S, possibly related to openConnectionForReceive compression?

2019-02-05 Thread Pat White
Hi Mark, thank you very much for the feedback, and the JettyServer reference, will take a look at that code. I'll update the thread if i get any more info. Very strange issue, and hard to see what's going on in the stream due to https encryption. Our usecase is fairly basic, get/put flows using

Re: Asymmetric push/pull throughput with S2S, possibly related to openConnectionForReceive compression?

2019-02-04 Thread Mark Payne
Hey Pat, I saw this thread but have not yet had a chance to look into it. So thanks for following up! The embedded server is handled in the JettyServer class [1]. I can imagine that it may automatically turn on GZIP. When pushing data, though, the client would be the one supplying the stream

Re: Asymmetric push/pull throughput with S2S, possibly related to openConnectionForReceive compression?

2019-02-04 Thread Pat White
This looks like a thrashing behavior in compress/decompress, found that if i enable compression in the output port of the receiver's RPG, the issue goes away, throughput becomes just as good as for the sender's flow. Again though, i believe i have compression off for all flows and components. Only

Asymmetric push/pull throughput with S2S, possibly related to openConnectionForReceive compression?

2019-02-01 Thread Pat White
Hi Folks, I'm trying to track a very odd performance issue, this is on 1.6.0 using S2S, would like to ask if there are any known issues like this or if my flow configuration is broken. From point of view of the RPG, receiving takes ~15x longer to xsfr the same 1.5gb file as a send from that RPG.