Responses inline. On 07/25/2017 02:23 AM, Aleksandar Lazic wrote: > Hi Liam, > > Liam Middlebrook wrote on 25.07.2017: > >> Hi Aleksandar, > >> Responses inline. > >> On 07/24/2017 11:57 PM, Aleksandar Lazic wrote: >>> Hi Liam, >>> >>> Liam Middlebrook wrote on 24.07.2017: >>> >>>> Hi, >>> >>>> I'm currently running HAProxy within an Openshift Origin cluster. Until >>>> a recent update of Openshift I did not experience issues with connection >>>> timeouts, the connections would last up until the specified timeout as >>>> defined by the application. >>>> >>>> After an update to Openshift I changed HAProxy settings around to give a >>>> global 600s timeout for client and server. However when I make a form >>>> upload request the connection is killed after 30 seconds. When I signal >>>> an XHR Replay in my network inspector the connection lasts longer than >>>> the 30 seconds and is able to successfully upload the file. >>> >>> This smells like this timeout. >>> >>> ### >>> ROUTER_DEFAULT_SERVER_TIMEOUT 30s >>> Length of time within which a server has to acknowledge or send data. >>> (TimeUnits) >>> ### >>> >>> https://docs.openshift.org/latest/architecture/core_concepts/routes.html#env-variables >>> >>> You can change it via. >>> >>> I assume here that you have the openshift router in the default >>> namespace and the router is deployed as "router". >>> >>> Too much routers here ;-) >>> >>> oc env -n default dc/router ROUTER_DEFAULT_SERVER_TIMEOUT=1h > >> I wish this were the case. I've changed the router's deployment config >> quite a bit. At first extending it to 5m, then I grew suspicious that >> HAProxy settings were even changing and set it to a lower 15s. > > uff that's low, imho.
Yeah, I just set it that low as a proof of concept from my suspicion that the timeout values weren't changing the upload behavior. > >> I found that when adjusting this environment variable the timeout limit >> does not change. This is especially odd since the generated >> haproxy.config file appears to have the proper changed values. >> >> sh-4.2$ grep -i timeout haproxy.config -n |head -n 10 >> 11: stats timeout 2m >> 42: timeout connect 5s >> 45: timeout client 15s >> 48: timeout server 15s >> 51: timeout http-request 10s >> 54: # Long timeout for WebSocket connections. >> 56: timeout tunnel 1h >> 218: timeout check 5000ms >> 259: timeout check 5000ms >> 300: timeout check 5000ms >> >> All the timeout values after line 300 are the same as lines 218, 259, >> and 300. > > That means the 'oc env ...' works, good. > >> With this config I am still receiving connection timeouts at 30s. > >> I'm not sure if this is important to note, but the requests that are >> timing out are HTTP file uploads. > > Yep I have seen this line in one of your previous mail. > >> Jul 24 23:51:08 proton.csh.rit.edu haproxy[127]: 67.188.94.238:43996 >> [24/Jul/2017:23:50:38.543] fe_sni~ >> be_edge_http_gallery_gallery/5792c687271726c3c4b5d54ae219aaa2 >> 85/0/1/-1/29913 -1 0 - - CHVN 1/0/0/0/0 0/0 "POST /upload HTTP/1.1" > > ### > CH The client aborted while waiting for the server to start responding. > It might be the server taking too long to respond or the client > clicking the 'Stop' button too fast. > > VN A cookie was provided by the client, none was inserted in the > response. This happens for most responses for which the client has > already got a cookie. > ### > > The timing looks also odd. > http://cbonte.github.io/haproxy-dconv/1.5/configuration.html#8.4 > > What's in your gallery log? > Does ANY Upload works? > Uploads that don't last 30 seconds appear to work fine. Uploads that are canceled due to the timeout do not appear in the application logs at all. (Normally the logs are output at the completion of the request). For the record this is a Flask application running via gunicorn. I have the gunicorn timeout set to 600s. >>>> I asked in irc with no luck. Any ideas why this may be happening> >>> Do you mean the #openshift-dev channel on Freenode? >>> >> I had asked in the #haproxy channel a few weeks ago and have just >> recently found time to re-explore this issue. > > Okay I don't know that this channel exists 8-O, every day is a learning > day ;-) > >>>> Thanks, >>>> >>>> Liam Middlebrook (loothelion) >>> >> Thanks, >> Liam Middlebrook (loothelion) >
