Re: Buffering issues with nginx

2017-07-31 Thread Dan34
I've run some tests and I'm pretty sure the reason I was getting 5MB stuck on nginx side was because of RCVBUF on upstream socket uses default socket buffers and by default it ends up with 5MB RCV buffer. I added logs to check that value and even after I configured sndbuf and rcvbuf inside listen

Re: Buffering issues with nginx

2017-07-30 Thread Dan34
Yes, I tested that and it appears to be the case. However, I don't see where nginx sets rcvbuf on the upstream socket as this one cannot be inherited. Somehow even with SND/RCV buffers set to low values and buffering disabled I get around 2.5BM stuck on nginx side. With my own simple proxy I get

Re: Buffering issues with nginx

2017-07-30 Thread Valentin V. Bartenev
On Saturday 29 July 2017 22:41:25 Dan34 wrote: > In nginx docs I see sndbuf option in listen directive. > Is there something that I don't understand about it, or developers of nginx > don't understand meaning of sndbuf... but I do not see a point to set sndbuf > on a listening socket. It just does

Re: Buffering issues with nginx

2017-07-29 Thread Dan34
In nginx docs I see sndbuf option in listen directive. Is there something that I don't understand about it, or developers of nginx don't understand meaning of sndbuf... but I do not see a point to set sndbuf on a listening socket. It just does not make any sense! sndbuf/rcvbuf is needed perhaps

Re: Buffering issues with nginx

2017-07-29 Thread Dan34
It looks like for localhost buffers are bigger, but even if it's not local host I do get 5MB stuck in socket buffers. I was only able to get perfect results by writing my own proxy in c and doing some obscure nodejs code to avoid buffering. In any case, if nginx does not provide a way to control

Re: Buffering issues with nginx

2017-07-24 Thread Francis Daly
On Mon, Jul 24, 2017 at 12:24:43PM -0400, Dan34 wrote: Hi there, > I did some logs on my proxy test and compared results with wireshark trace > at some random point in time (t=511sec) > And numbers match exactly between logs and wireshark. Out of interest -- are these buffers especially big

Re: Buffering issues with nginx

2017-07-24 Thread Dan34
I did some logs on my proxy test and compared results with wireshark trace at some random point in time (t=511sec) And numbers match exactly between logs and wireshark. This is a log line from my test proxy: time: 511s, bytesSent:5571760, down:{ SND:478720 OUTQ:280480 } up:{ RCV:5109117

Re: Buffering issues with nginx

2017-07-24 Thread Dan34
I wrote my own proxy and it appears that the data is all stuck in socket buffers. If SNDBUF isn't set, then OS will resize it if you try to write more data than remote can accept. Overall, in my tests I see that this buffer grows to 2.5MB and in wireshark I see that difference grows up to 5MB. As

Re: Buffering issues with nginx

2017-07-22 Thread Dan34
> You should check tcpdump (or wireshark) to see where actually 12.5MB > of data have been stuck. Wireshark confirms my assumption. All the data is buffered by nginx. More over, I see some buggy behavior, and I've seen that happen quite often. This is localhost tcp screenshot:

Re: Buffering issues with nginx

2017-07-21 Thread Valentin V. Bartenev
On Friday 21 July 2017 13:45:51 Dan34 wrote: [..] > > In short, these could affect my case: SO_RCVBUF, SO_SNDBUF on nginx side and > whatever buffering nginx uses for handling data. I run that same test with > 25MB data and I got totally identical result: 12.5MB was buffered on nginx > side. That

Re: Buffering issues with nginx

2017-07-21 Thread Dan34
Hello Valentin, > 1. Write socket buffer in kernel on node.js side where node.js writes data. we can throw this out from equation, as I measure my end time by the event when socket is closed on nodejs side, (I use http1.0 from nginx to node to make it simple for this case). > 2. Read socket

Re: Buffering issues with nginx

2017-07-21 Thread Valentin V. Bartenev
On Friday 21 July 2017 07:02:07 Dan34 wrote: [..] > I run nginx on 8080, for testing, since it's not suitable for live use on 80 > in my case and I'm trying to figure out how to fix it. > And here's why I believe that there is a bug. > > In my case, I wrote test code on node side that serves some

Re: Buffering issues with nginx

2017-07-21 Thread Dan34
> Depending on the compromises you are willing to make, to accuracy or > convenience, you may be able to come up with something good enough. I have a more or less working solution. nginx breaks it and I'm trying to figure out how to fix it. > Yes. That is (part of) what a proxy does. Even

Re: Buffering issues with nginx

2017-07-21 Thread Dan34
> X-Accel-Buffering: no > That will disable nginx's buffering for the request. At first it looked like exactly what I was looking for (after reading nginx docs), but after trying I observed that there were no effects from that. In code that writes headers I added

Re: Buffering issues with nginx

2017-07-19 Thread Igal @ Lucee.org
my problem is that response that my node app (upstream server) generates is buffered by nginx. Generate the following header from node: X-Accel-Buffering: no That will disable nginx's buffering for the request. Igal Sapir Lucee Core Developer Lucee.org

Re: Buffering issues with nginx

2017-07-19 Thread Francis Daly
On Mon, Jul 17, 2017 at 09:47:39PM -0400, Dan34 wrote: Hi there, > > Are you concerned about the request or the response being buffered? > > my problem is that response that my node app (upstream server) generates is > buffered by nginx. So you want proxy_buffering

Re: Buffering issues with nginx

2017-07-17 Thread Francis Daly
On Mon, Jul 17, 2017 at 02:06:27AM -0400, Dan34 wrote: Hi there, > No matter what configs I try, nginx still keeps buffering my requests. proxy_request_buffering (http://nginx.org/r/proxy_request_buffering) relates to nginx buffering the request from the client. proxy_buffering