[go-nuts] Re: httputil.ReverseProxy adding 100+ ms of latency on localhost - any ideas?

[Tom]

The plot thickens.

My attempt to make a minimally-reproducible short example has failed - my 
demo (here ) does not reproduce the 
problem, even though it's mostly copy paste. That leaves the code in my 
original project to blame.

The server-being proxied is simply: *python -m SimpleHTTPServer*

Any recommendations for instrumenting my code to find the bottleneck? I'm 
going to have a quick look for idioms (IIRC Dave Cheney mentioned something 
about a talk he did in europe about HTTPTrace?), otherwise I'm going to 
scatter logging of the current ms.

On Wednesday, 16 November 2016 16:01:50 UTC+11, Tom wrote:
>
> Hi guys,
>
> I've been working on a HTTP reverse proxy that has an auth/authorization 
> function. It all works well, however I have noticed significant additional 
> latency - and I cannot work out why. From what I can tell, me hitting my 
> service from chrome on localhost should work just fine - I have a very 
> beefy machine! (8core 3.4Ghz Amd64)
>
> Anyone have any insights into httputil.ReverseProxy, or have any ideas 
> where to begin?
>
> The method is here: 
> https://github.com/twitchyliquid64/pushtart/blob/master/src/pushtart/webproxy/handler.go#L61
>
> Cheers,
> Tom
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: httputil.ReverseProxy adding 100+ ms of latency on localhost - any ideas?

[James Bardin]

On Thu, Nov 17, 2016 at 5:47 PM, Tom  wrote:

> I agree that creating a ReverseProxy object every time is inefficent with
> regards to transports - but surely that wouldnt cause the additional
> latency?


Depends on what exactly you're measuring and how you're testing.

A DNS query, TCP connect, or a TLS connection can all easily take 100+ms to
setup depending on latency, so losing the existing connection and starting
fresh every time could be an issue.

If you've used up the server's ephemeral ports with leaked connections, new
connection have to wait in queue for the old ones to timeout.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: httputil.ReverseProxy adding 100+ ms of latency on localhost - any ideas?

[Tom]

Thanks guys for your suggestions. I am going to try:

1. GODEBUG=netdns=go+2 or GODEBUG=netdns=cgo+2
2. Running with just IP addresses to eliminate DNS
3. Creating a minimal program to reproduce latency and post here

I agree that creating a ReverseProxy object every time is inefficent with 
regards to transports - but surely that wouldnt cause the additional 
latency?


-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: httputil.ReverseProxy adding 100+ ms of latency on localhost - any ideas?

[James Bardin]

While it might not be the initial problem, you're creating a new proxy and 
a new http.Transport on every request. At the end of that request that 
transport will probably still contain the open connection which you can no 
longer use. That's going to tie up the FD until the keepalive can kill it, 
which IIRC how Go sets the options in linux is going to be 10min with the 
30s interval you're setting.  

You always want to reuse http.Transports.


On Wednesday, November 16, 2016 at 12:01:50 AM UTC-5, Tom wrote:
>
> Hi guys,
>
> I've been working on a HTTP reverse proxy that has an auth/authorization 
> function. It all works well, however I have noticed significant additional 
> latency - and I cannot work out why. From what I can tell, me hitting my 
> service from chrome on localhost should work just fine - I have a very 
> beefy machine! (8core 3.4Ghz Amd64)
>
> Anyone have any insights into httputil.ReverseProxy, or have any ideas 
> where to begin?
>
> The method is here: 
> https://github.com/twitchyliquid64/pushtart/blob/master/src/pushtart/webproxy/handler.go#L61
>
> Cheers,
> Tom
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: httputil.ReverseProxy adding 100+ ms of latency on localhost - any ideas?

[Slawomir Pryczek]

If you're on a beefy machine with recent linux, maybe MTU of loopback 
interface is set to 65k, setting it to default 1500 would help i think...

ifconfig lo mtu 1500 up 

https://www.cyberciti.biz/faq/centos-rhel-redhat-fedora-debian-linux-mtu-size/



W dniu środa, 16 listopada 2016 06:01:50 UTC+1 użytkownik Tom napisał:
>
> Hi guys,
>
> I've been working on a HTTP reverse proxy that has an auth/authorization 
> function. It all works well, however I have noticed significant additional 
> latency - and I cannot work out why. From what I can tell, me hitting my 
> service from chrome on localhost should work just fine - I have a very 
> beefy machine! (8core 3.4Ghz Amd64)
>
> Anyone have any insights into httputil.ReverseProxy, or have any ideas 
> where to begin?
>
> The method is here: 
> https://github.com/twitchyliquid64/pushtart/blob/master/src/pushtart/webproxy/handler.go#L61
>
> Cheers,
> Tom
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.