please check my sites

[Henry Jaxson]

*Hello *
*I am a professional Link builder. I am providing guest posting services. I
have many blogs in different niches. I am sending you some blogs.  If you
interested*

URL DA Monthly Visits Alexa rank
https://lifehacker.com/

91 9.8M 5.26K
https://www.forbes.com/

94 130.1M 597
https://www.businesstomark.com/

63 29.3K 59.3K
https://www.scoopearth.com/

59 178.0K 45.4K
https://ventsmagazine.co.uk/

51 19.5K 69.3K
https://urbanmatter.com/

54 265.2K 20.8K
https://techbullion.com/

64 413.6K 38.7K
https://programminginsider.com/

56 133.0K 336K
https://www.bignewsnetwork.com/

66 216.3K 576K
https://timebusinessnews.com/

69 314.5K 59.3K
https://filmdaily.co/

64 3.5M 12.2K
https://www.marketwatch.com/

92 56.9M 157
https://mozusa.com/

60 69.3K 519K
https://ventsmagazine.com/

62 180.5K 52.9K
https://www.houzz.com/

90 16.1M 831

*I am waiting for the response *

Re: please check

[Willy Tarreau]

On Wed, May 07, 2014 at 07:30:43AM +0200, Willy Tarreau wrote:
  The strange news: Contrary to your statement, the client connection is
  closed after the 1 second timeout. It even logs this. The only thing
  that doesn't happen properly is the absence of any response. Just
  immediate connection close.
  
  
  Before patch:
  haproxy[26318]: 127.0.0.1:51995 [06/May/2014:18:55:33.002] f1 b1/s1
  0/0/0/-1/2001 504 194 - - sH-- 0/0/0/0/0 0/0 GET / HTTP/1.1
  
  After patch:
  haproxy[27216]: 127.0.0.1:52027 [06/May/2014:18:56:34.165] f1 b1/s1
  0/0/0/-1/1002 -1 0 - - cD-- 0/0/0/0/0 0/0 GET / HTTP/1.1
 
 Interesting, for me it waited till the end. Or maybe you have
 option abortonclose ?

OK I found, it's because your server receives the abort and closes immediately
that the client timeout is enforced. In my test, the server was waiting a
predefined amount of time, thus the server timeout was enforced.

Willy

Re: please check

[Willy Tarreau]

Hi Patrick, hi Rachel,

so with these two patches applied on top of the previous one, I get the
behaviour that we discussed here.

Specifically, we differentiate client-read timeout, server-write timeouts
and server read timeouts during the data forwarding phase. Also, we disable
server read timeout until the client has sent its whole request. That way
I'm seeing the following flags in the logs :

  - cH when client does not send everything before the server starts to
respond, which is OK. Status=408 there.

  - cD when client stops sending data after the server starts to respond,
or if the client stops reading data, which in both case is a clear
client timeout. In both cases, the status is unaltered and nothing
is emitted since the beginning of the response was already transmitted ;

  - sH when the server does not respond, including if it stops reading the
message body (eg: process stuck). Then we have 504.

  - sD if the server stops reading or sending data during the data phase.

The changes were a bit tricky, so any confirmation from any of you would
make me more comfortable merging them into mainline. I'm attaching these
two extra patches, please give them a try.

Thanks,
Willy

From b9edf8fbecc9d1b5c82794735adcc367a80a4ae2 Mon Sep 17 00:00:00 2001
From: Willy Tarreau w...@1wt.eu
Date: Wed, 7 May 2014 14:24:16 +0200
Subject: BUG/MEDIUM: http: correctly report request body timeouts

This is the continuation of previous patch BUG/MEDIUM: http/session:
disable client-side expiration only after body.

This one takes care of properly reporting the client-side read timeout
when waiting for a body from the client. Since the timeout may happen
before or after the server starts to respond, we have to take care of
the situation in three different ways :
  - if the server does not read our data fast enough, we emit a 504
if we're waiting for headers, or we simply break the connection
if headers were already received. We report either sH or sD
depending on whether we've seen headers or not.

  - if the server has not yet started to respond, but has read all of
the client's data and we're still waiting for more data from the
client, we can safely emit a 408 and abort the request ;

  - if the server has already started to respond (thus it's a transfer
timeout during a bidirectional exchange), then we silently break
the connection, and only the session flags will indicate in the
logs that something went wrong with client or server side.

This bug is tagged MEDIUM because it touches very sensible areas, however
its impact is very low. It might be worth performing a careful backport
to 1.4 once it has been confirmed that everything is correct and that it
does not introduce any regression.
---
 src/proto_http.c | 76 ++--
 1 file changed, 74 insertions(+), 2 deletions(-)

diff --git a/src/proto_http.c b/src/proto_http.c
index e473228..797b3b8 100644
--- a/src/proto_http.c
+++ b/src/proto_http.c
@@ -5175,6 +5175,13 @@ int http_request_forward_body(struct session *s, struct 
channel *req, int an_bit
 */
msg-msg_state = HTTP_MSG_ERROR;
http_resync_states(s);
+
+   if (req-flags  CF_READ_TIMEOUT)
+   goto cli_timeout;
+
+   if (req-flags  CF_WRITE_TIMEOUT)
+   goto srv_timeout;
+
return 1;
}
 
@@ -5455,6 +5462,68 @@ int http_request_forward_body(struct session *s, struct 
channel *req, int an_bit
s-flags |= SN_FINST_D;
}
return 0;
+
+ cli_timeout:
+   if (!(s-flags  SN_ERR_MASK))
+   s-flags |= SN_ERR_CLITO;
+
+   if (!(s-flags  SN_FINST_MASK)) {
+   if (txn-rsp.msg_state  HTTP_MSG_ERROR)
+   s-flags |= SN_FINST_H;
+   else
+   s-flags |= SN_FINST_D;
+   }
+
+   if (txn-status  0) {
+   /* Don't send any error message if something was already sent */
+   stream_int_retnclose(req-prod, NULL);
+   }
+   else {
+   txn-status = 408;
+   stream_int_retnclose(req-prod, http_error_message(s, 
HTTP_ERR_408));
+   }
+
+   msg-msg_state = HTTP_MSG_ERROR;
+   req-analysers = 0;
+   s-rep-analysers = 0; /* we're in data phase, we want to abort both 
directions */
+
+   session_inc_http_err_ctr(s);
+   s-fe-fe_counters.failed_req++;
+   s-be-be_counters.failed_req++;
+   if (s-listener-counters)
+   s-listener-counters-failed_req++;
+   return 0;
+
+ srv_timeout:
+   if (!(s-flags  SN_ERR_MASK))
+   s-flags |= SN_ERR_SRVTO;
+
+   if (!(s-flags  SN_FINST_MASK)) {
+   if (txn-rsp.msg_state  HTTP_MSG_ERROR)
+   s-flags |= SN_FINST_H;
+   else
+   s-flags |= SN_FINST_D;
+   }
+
+ 

Re: please check

[Patrick Hemmer]

*From: *Willy Tarreau w...@1wt.eu
*Sent: * 2014-05-07 09:45:47 E
*To: *Patrick Hemmer hapr...@stormcloud9.net, Rachel Chavez
rachel.chave...@gmail.com
*CC: *haproxy@formilux.org
*Subject: *Re: please check

 Hi Patrick, hi Rachel,

 so with these two patches applied on top of the previous one, I get the
 behaviour that we discussed here.

 Specifically, we differentiate client-read timeout, server-write timeouts
 and server read timeouts during the data forwarding phase. Also, we disable
 server read timeout until the client has sent its whole request. That way
 I'm seeing the following flags in the logs :

   - cH when client does not send everything before the server starts to
 respond, which is OK. Status=408 there.

   - cD when client stops sending data after the server starts to respond,
 or if the client stops reading data, which in both case is a clear
 client timeout. In both cases, the status is unaltered and nothing
 is emitted since the beginning of the response was already transmitted ;

   - sH when the server does not respond, including if it stops reading the
 message body (eg: process stuck). Then we have 504.

   - sD if the server stops reading or sending data during the data phase.

 The changes were a bit tricky, so any confirmation from any of you would
 make me more comfortable merging them into mainline. I'm attaching these
 two extra patches, please give them a try.

 Thanks,
 Willy

Works beautifully. I had created a little test suite to test to test a
bunch of conditions around this, and they all pass.
Will see about throwing this in our development environment in the next
few days if a release doesn't come out before then.

Thank you much :-)

-Patrick

Re: please check

[Willy Tarreau]

On Wed, May 07, 2014 at 09:55:35AM -0400, Patrick Hemmer wrote:
 Works beautifully. I had created a little test suite to test to test a
 bunch of conditions around this, and they all pass.

Wow, impressed with the speed of your test! Thanks!

 Will see about throwing this in our development environment in the next
 few days if a release doesn't come out before then.

Perfect, I'm merging them now. Emeric is currently investigating the
issue with SSL on FreeBSD, and I'd like to at least get rid of the
bind-process item before the dev25 since it's supposed to be a quick
change when I don't spend my time working on other bugs :-)

Cheers,
Willy

Re: please check

[Willy Tarreau]

Hi Patrick, hi Rachel,

I might have fixed half of the issue, I'd like you to test the attached patch.
It ensures that the client-side timeout is only disabled after transmitting
the whole body and not during the transmission. It will report cD in the
flags, but does not affect the status code yet. It does not abort when the
client timeout strikes, but still when the server timeout strikes, which is
another tricky thing to do properly. That's why I would be happy if you could
at least confirm that you correctly get cD or sH (or even sD) depending on
who times out first.

Thanks,
Willy

From 3eb80eacadf8739b487765dc1d7929ca25a00d95 Mon Sep 17 00:00:00 2001
From: Willy Tarreau w...@1wt.eu
Date: Tue, 6 May 2014 22:57:53 +0200
Subject: BUG/MEDIUM: http/session: disable client-side expiration only after
 body

For a very long time, back in the early 1.3 days, we used to rely on a
trick to avoid expiring the client side while transferring a payload to
the server. The problem was that if a client was able to quickly fill
the buffers, and these buffers took some time to reach the server, the
client should not expire while not sending anything.

In order to cover this situation, the client-side timeout was disabled
once the connection to the server was OK, since it implied that we would
at least expire on the server if required.

But there is a drawback to this : if a client stops uploading data before
the end, its timeout is not enforced and we only expire on the server's
timeout, so the logs report a 504.

Since 1.4, we have message body analysers which ensure that we know whether
all the expected data was received or not (HTTP_MSG_DATA or HTTP_MSG_DONE).
So we can fix this problem by disabling the client-side or server-side
timeout at the end of the transfer for the respective side instead of
having it unconditionally in session.c during all the transfer.

With this, the logs now report the correct side for the timeout. Note that
this patch is not enough, because another issue remains : the HTTP body
forwarders do not abort upon timeout, they simply rely on the generic
handling from session.c. So for now, the session is still aborted when
reaching the server timeout, but the culprit is properly reported. A
subsequent patch will address this specific point.

This bug was tagged MEDIUM because of the changes performed. The issue
it fixes is minor however. After some cooling down, it may be backported
to 1.4.

It was reported by and discussed with Rachel Chavez and Patrick Hemmer
on the mailing list.
---
 src/proto_http.c | 11 +++
 src/session.c| 14 --
 2 files changed, 11 insertions(+), 14 deletions(-)

diff --git a/src/proto_http.c b/src/proto_http.c
index 24c7cf8..e473228 100644
--- a/src/proto_http.c
+++ b/src/proto_http.c
@@ -5341,6 +5341,11 @@ int http_request_forward_body(struct session *s, struct 
channel *req, int an_bit
channel_auto_read(req);
}
 
+   /* if we received everything, we don't want to expire 
anymore */
+   if (msg-msg_state == HTTP_MSG_DONE) {
+   req-flags |= CF_READ_NOEXP;
+   req-rex = TICK_ETERNITY;
+   }
return 0;
}
}
@@ -6529,6 +6534,12 @@ int http_response_forward_body(struct session *s, struct 
channel *res, int an_bi
}
return 1;
}
+
+   /* if we received everything, we don't want to expire 
anymore */
+   if (msg-msg_state == HTTP_MSG_DONE) {
+   res-flags |= CF_READ_NOEXP;
+   res-rex = TICK_ETERNITY;
+   }
return 0;
}
}
diff --git a/src/session.c b/src/session.c
index 0824167..48bdf7e 100644
--- a/src/session.c
+++ b/src/session.c
@@ -2411,20 +2411,6 @@ struct task *process_session(struct task *t)
s-si[0].flags = ~(SI_FL_ERR|SI_FL_EXP);
s-si[1].flags = ~(SI_FL_ERR|SI_FL_EXP);
 
-   /* Trick: if a request is being waiting for the server to 
respond,
-* and if we know the server can timeout, we don't want the 
timeout
-* to expire on the client side first, but we're still 
interested
-* in passing data from the client to the server (eg: POST). 
Thus,
-* we can cancel the client's request timeout if the server's
-* request timeout is set and the server has not yet sent a 
response.
-*/
-
-   if ((s-rep-flags  (CF_AUTO_CLOSE|CF_SHUTR)) == 0 
-   (tick_isset(s-req-wex) || tick_isset(s-rep-rex))) {
-   s-req-flags |= CF_READ_NOEXP;
-   s-req-rex = TICK_ETERNITY;
-   }
-
  

Re: please check

[Patrick Hemmer]

*From: *Willy Tarreau w...@1wt.eu
*Sent: * 2014-05-06 17:41:18 E
*To: *Patrick Hemmer hapr...@stormcloud9.net, Rachel Chavez
rachel.chave...@gmail.com
*CC: *haproxy@formilux.org
*Subject: *Re: please check

 Hi Patrick, hi Rachel,

 I might have fixed half of the issue, I'd like you to test the attached patch.
 It ensures that the client-side timeout is only disabled after transmitting
 the whole body and not during the transmission. It will report cD in the
 flags, but does not affect the status code yet. It does not abort when the
 client timeout strikes, but still when the server timeout strikes, which is
 another tricky thing to do properly. That's why I would be happy if you could
 at least confirm that you correctly get cD or sH (or even sD) depending on
 who times out first.

 Thanks,
 Willy

So good news, bad news, and strange news.

The good news: It is reporting cD-- as it should

The bad news: It's not reporting any return status at all. Before it
would log 504 and send a 504 response back. Now it logs -1 and doesn't
send anything back. It's just closing the connection.

The strange news: Contrary to your statement, the client connection is
closed after the 1 second timeout. It even logs this. The only thing
that doesn't happen properly is the absence of any response. Just
immediate connection close.


Before patch:
haproxy[26318]: 127.0.0.1:51995 [06/May/2014:18:55:33.002] f1 b1/s1
0/0/0/-1/2001 504 194 - - sH-- 0/0/0/0/0 0/0 GET / HTTP/1.1

After patch:
haproxy[27216]: 127.0.0.1:52027 [06/May/2014:18:56:34.165] f1 b1/s1
0/0/0/-1/1002 -1 0 - - cD-- 0/0/0/0/0 0/0 GET / HTTP/1.1

-Patrick

Re: please check

[Willy Tarreau]

Hi Patrick,

On Tue, May 06, 2014 at 07:38:04PM -0400, Patrick Hemmer wrote:
 *From: *Willy Tarreau w...@1wt.eu
 *Sent: * 2014-05-06 17:41:18 E
 *To: *Patrick Hemmer hapr...@stormcloud9.net, Rachel Chavez
 rachel.chave...@gmail.com
 *CC: *haproxy@formilux.org
 *Subject: *Re: please check
 
  Hi Patrick, hi Rachel,
 
  I might have fixed half of the issue, I'd like you to test the attached 
  patch.
  It ensures that the client-side timeout is only disabled after transmitting
  the whole body and not during the transmission. It will report cD in the
  flags, but does not affect the status code yet. It does not abort when the
  client timeout strikes, but still when the server timeout strikes, which is
  another tricky thing to do properly. That's why I would be happy if you 
  could
  at least confirm that you correctly get cD or sH (or even sD) depending on
  who times out first.
 
  Thanks,
  Willy
 
 So good news, bad news, and strange news.
 
 The good news: It is reporting cD-- as it should
 
 The bad news: It's not reporting any return status at all. Before it
 would log 504 and send a 504 response back. Now it logs -1 and doesn't
 send anything back. It's just closing the connection.

Till there that's expected since the timeout is not handled as an error
in the forwarding path. That's why I said it should fix half of the issue.

 The strange news: Contrary to your statement, the client connection is
 closed after the 1 second timeout. It even logs this. The only thing
 that doesn't happen properly is the absence of any response. Just
 immediate connection close.
 
 
 Before patch:
 haproxy[26318]: 127.0.0.1:51995 [06/May/2014:18:55:33.002] f1 b1/s1
 0/0/0/-1/2001 504 194 - - sH-- 0/0/0/0/0 0/0 GET / HTTP/1.1
 
 After patch:
 haproxy[27216]: 127.0.0.1:52027 [06/May/2014:18:56:34.165] f1 b1/s1
 0/0/0/-1/1002 -1 0 - - cD-- 0/0/0/0/0 0/0 GET / HTTP/1.1

Interesting, for me it waited till the end. Or maybe you have
option abortonclose ?

Next step will be to assign status 408 and try to send the response
back at this step but only if nothing was sent yet.

Best regards,
Willy

Re: please check

[Willy Tarreau]

On Thu, May 01, 2014 at 03:44:46PM -0400, Rachel Chavez wrote:
 The problem is:
 
 when client sends a request with incomplete body (it has content-length but
 no body) then haproxy returns a 5XX error when it should be a client issue.

It's a bit more complicated than that. When the request body flows from the
client to the server, at any moment the server is free to respond (either
with an error, a redirect, a timeout or whatever). So as soon as we start
to forward a request body from the client to the server, we're *really*
waiting for the server to send a verdict about that request.

 In the session.c file starting in 2404 i make sure that if I haven't
 received the entire body of the request I continue to wait for it by
 keeping AN_REQ_WAIT_HTTP as part of the request analyzers list as long as
 the client read timeout hasn't fired yet.

It's unrelated unfortunately and it cannot work. AN_REQ_WAIT_HTTP is meant
to wait for a *new* request. So if the client doesn't send a complete
request, it's both wrong and dangerous to expect a new request inside the
body. When the body is being forwarded, the request flows through
http_request_forward_body(). This one already tests for the client timeout
as you can see. I'm not seeing any error processing there though, maybe
we'd need to set some error codes there to avoid them getting the default
ones.

 In the proto_http.c file what I tried to do is avoid getting a server
 timeout when the client had timed-out already.

I agree that it's always the *first* timeout which strikes which should
indicate the faulty side, because eventhough they're generally set to the
same value, people who want to enforce a specific processing can set them
apart.

Regards,
Willy

Re: please check

[Patrick Hemmer]

*From: *Willy Tarreau w...@1wt.eu
*Sent: * 2014-05-02 02:02:11 E
*To: *Rachel Chavez rachel.chave...@gmail.com
*CC: *haproxy@formilux.org
*Subject: *Re: please check

 On Thu, May 01, 2014 at 03:44:46PM -0400, Rachel Chavez wrote:
 The problem is:

 when client sends a request with incomplete body (it has content-length but
 no body) then haproxy returns a 5XX error when it should be a client issue.
 It's a bit more complicated than that. When the request body flows from the
 client to the server, at any moment the server is free to respond (either
 with an error, a redirect, a timeout or whatever). So as soon as we start
 to forward a request body from the client to the server, we're *really*
 waiting for the server to send a verdict about that request.
At any moment the server is free to respond yes, but the server cannot
respond *properly* until it gets the complete request.
If the response depends on the request payload, the server doesn't know
whether to respond with 200 or with a 400.

RFC2616 covers this behavior in depth. See 8.2.3 Use of the 100
(Continue) Status. This section indicates that it should not be
expected for the server to respond without a request body unless the
client explicitly sends a Expect: 100-continue



 In the session.c file starting in 2404 i make sure that if I haven't
 received the entire body of the request I continue to wait for it by
 keeping AN_REQ_WAIT_HTTP as part of the request analyzers list as long as
 the client read timeout hasn't fired yet.
 It's unrelated unfortunately and it cannot work. AN_REQ_WAIT_HTTP is meant
 to wait for a *new* request. So if the client doesn't send a complete
 request, it's both wrong and dangerous to expect a new request inside the
 body. When the body is being forwarded, the request flows through
 http_request_forward_body(). This one already tests for the client timeout
 as you can see. I'm not seeing any error processing there though, maybe
 we'd need to set some error codes there to avoid them getting the default
 ones.

 In the proto_http.c file what I tried to do is avoid getting a server
 timeout when the client had timed-out already.
 I agree that it's always the *first* timeout which strikes which should
 indicate the faulty side, because eventhough they're generally set to the
 same value, people who want to enforce a specific processing can set them
 apart.

 Regards,
 Willy

Re: please check

[Willy Tarreau]

Hi Patrick,

On Fri, May 02, 2014 at 10:57:38AM -0400, Patrick Hemmer wrote:
 *From: *Willy Tarreau w...@1wt.eu
 *Sent: * 2014-05-02 02:02:11 E
 *To: *Rachel Chavez rachel.chave...@gmail.com
 *CC: *haproxy@formilux.org
 *Subject: *Re: please check
 
  On Thu, May 01, 2014 at 03:44:46PM -0400, Rachel Chavez wrote:
  The problem is:
 
  when client sends a request with incomplete body (it has content-length but
  no body) then haproxy returns a 5XX error when it should be a client issue.
  It's a bit more complicated than that. When the request body flows from the
  client to the server, at any moment the server is free to respond (either
  with an error, a redirect, a timeout or whatever). So as soon as we start
  to forward a request body from the client to the server, we're *really*
  waiting for the server to send a verdict about that request.
 At any moment the server is free to respond yes, but the server cannot
 respond *properly* until it gets the complete request.

Yes it can, redirects are the most common anticipated response, as the
result of a POST to a page with an expired cookie. And the 302 is a
clean response, it's not even an error.

 If the response depends on the request payload, the server doesn't know
 whether to respond with 200 or with a 400.

With WAFs deployed massively on server infrastructures, 403 are quite
common long before the whole data. 413 request entity too large appears
quite commonly as well. 401 and 407 can also happen when authentication
is needed.

 RFC2616 covers this behavior in depth. See 8.2.3 Use of the 100
 (Continue) Status. This section indicates that it should not be
 expected for the server to respond without a request body unless the
 client explicitly sends a Expect: 100-continue

Well, 2616 is 15-years old now and pretty obsolete, which is why the
HTTP-bis WG is working on refreshing this. New wording is clearer about
how a request body is used :

   o  A server MAY omit sending a 100 (Continue) response if it has
  already received some or all of the message body for the
  corresponding request, or if the framing indicates that there is
  no message body.

Note the some or all.

It's very tricky to find which side is responsible for a stalled upload.
I've very commonly found that frozen servers, or those with deep request
queues will stall during body transfers because they still didn't start
to consume the part of the request that's queued into network buffers.

All I mean is that it's unfortunately not *that* white and black. We
*really* need to make a careful difference between what happens on the
two sides. The (hard) goal I'm generally seeking is to do my best so
that a misbehaving user doesn't make us believe that a server is going
badly. That's not easy, considering for example the fact that the 501
message could be understood as a server error while it's triggered by
the client.

In general (unless there's something wrong with the way client timeouts
are reported in http_request_forward_body), client timeouts should be
reported as such, and same for server timeouts. It's possible that there
are corner cases, but we need to be extremely careful about them and not
try to generalize.

Best regards,
Willy

Re: please check

[Patrick Hemmer]

*From: *Willy Tarreau w...@1wt.eu
*Sent: * 2014-05-02 11:15:07 E
*To: *Patrick Hemmer hapr...@stormcloud9.net
*CC: *Rachel Chavez rachel.chave...@gmail.com, haproxy@formilux.org
*Subject: *Re: please check

 Hi Patrick,

 On Fri, May 02, 2014 at 10:57:38AM -0400, Patrick Hemmer wrote:
 *From: *Willy Tarreau w...@1wt.eu
 *Sent: * 2014-05-02 02:02:11 E
 *To: *Rachel Chavez rachel.chave...@gmail.com
 *CC: *haproxy@formilux.org
 *Subject: *Re: please check

 On Thu, May 01, 2014 at 03:44:46PM -0400, Rachel Chavez wrote:
 The problem is:

 when client sends a request with incomplete body (it has content-length but
 no body) then haproxy returns a 5XX error when it should be a client issue.
 It's a bit more complicated than that. When the request body flows from the
 client to the server, at any moment the server is free to respond (either
 with an error, a redirect, a timeout or whatever). So as soon as we start
 to forward a request body from the client to the server, we're *really*
 waiting for the server to send a verdict about that request.
 At any moment the server is free to respond yes, but the server cannot
 respond *properly* until it gets the complete request.
 Yes it can, redirects are the most common anticipated response, as the
 result of a POST to a page with an expired cookie. And the 302 is a
 clean response, it's not even an error.
I should have clarified what I meant by properly more. I didn't mean
that the server can't respond at all, as there are many cases it can,
some of which you point out. I meant that if the server is expecting a
request body, it can't respond with a 200 until it verifies that request
body.

 If the response depends on the request payload, the server doesn't know
 whether to respond with 200 or with a 400.
 With WAFs deployed massively on server infrastructures, 403 are quite
 common long before the whole data. 413 request entity too large appears
 quite commonly as well. 401 and 407 can also happen when authentication
 is needed.

 RFC2616 covers this behavior in depth. See 8.2.3 Use of the 100
 (Continue) Status. This section indicates that it should not be
 expected for the server to respond without a request body unless the
 client explicitly sends a Expect: 100-continue
 Well, 2616 is 15-years old now and pretty obsolete, which is why the
 HTTP-bis WG is working on refreshing this. New wording is clearer about
 how a request body is used :

o  A server MAY omit sending a 100 (Continue) response if it has
   already received some or all of the message body for the
   corresponding request, or if the framing indicates that there is
   no message body.

 Note the some or all.
I'm assuming you're quoting from:
http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-26#section-5.1.1

This only applies if the Expect: 100-continue was sent. Expect:
100-continue was meant to solve the issue where the client has a large
body, and wants to make sure that the server will accept the body before
sending it (and wasting bandwidth). Meaning that without sending
Expect: 100-continue, it is expected that the server will not send a
response until the body has been sent.


 It's very tricky to find which side is responsible for a stalled upload.
 I've very commonly found that frozen servers, or those with deep request
 queues will stall during body transfers because they still didn't start
 to consume the part of the request that's queued into network buffers.

 All I mean is that it's unfortunately not *that* white and black. We
 *really* need to make a careful difference between what happens on the
 two sides. The (hard) goal I'm generally seeking is to do my best so
 that a misbehaving user doesn't make us believe that a server is going
 badly. That's not easy, considering for example the fact that the 501
 message could be understood as a server error while it's triggered by
 the client.

 In general (unless there's something wrong with the way client timeouts
 are reported in http_request_forward_body), client timeouts should be
 reported as such, and same for server timeouts. It's possible that there
 are corner cases, but we need to be extremely careful about them and not
 try to generalize.
I agree, a client timeout should be reported as such, and that's what
this is all about. If the client sends half the body (or no body), and
then freezes, the client timeout should kick in and send back a 408, not
the server timeout resulting in a 504.

I think in this regards it is very clear.
* The server may respond with the HTTP response status code any time it
feels like it.
* Enable the server timeout and disable the client timeout upon any of
the following:
* The client sent Expect: 100-continue and has completed all headers
* The complete client request has been sent, including body if
Content-Length  0
* Writing to the server socket would result in a blocking write
(indicating that the remote end is not processing).
* Enable the client timeout

Re: please check

[Willy Tarreau]

On Fri, May 02, 2014 at 12:18:43PM -0400, Patrick Hemmer wrote:
  At any moment the server is free to respond yes, but the server cannot
  respond *properly* until it gets the complete request.
  Yes it can, redirects are the most common anticipated response, as the
  result of a POST to a page with an expired cookie. And the 302 is a
  clean response, it's not even an error.
 I should have clarified what I meant by properly more. I didn't mean
 that the server can't respond at all, as there are many cases it can,
 some of which you point out. I meant that if the server is expecting a
 request body, it can't respond with a 200 until it verifies that request
 body.

OK, but from a reverse-proxy point of view, all of them are equally valid,
and there's even no way to know if the server is interested in receiving
these data at all. The only differences are that some of them are considered
precious (ie those returning 200) and other ones less since they're
possibly ephemeral.

  If the response depends on the request payload, the server doesn't know
  whether to respond with 200 or with a 400.
  With WAFs deployed massively on server infrastructures, 403 are quite
  common long before the whole data. 413 request entity too large appears
  quite commonly as well. 401 and 407 can also happen when authentication
  is needed.
 
  RFC2616 covers this behavior in depth. See 8.2.3 Use of the 100
  (Continue) Status. This section indicates that it should not be
  expected for the server to respond without a request body unless the
  client explicitly sends a Expect: 100-continue
  Well, 2616 is 15-years old now and pretty obsolete, which is why the
  HTTP-bis WG is working on refreshing this. New wording is clearer about
  how a request body is used :
 
 o  A server MAY omit sending a 100 (Continue) response if it has
already received some or all of the message body for the
corresponding request, or if the framing indicates that there is
no message body.
 
  Note the some or all.
 I'm assuming you're quoting from:
 http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-26#section-5.1.1

Yes indeed. Ah in fact I found the exact part I was looking for, it's in
the same block, two points below :

   o  A server that responds with a final status code before reading the
  entire message body SHOULD indicate in that response whether it
  intends to close the connection or continue reading and discarding
  the request message (see Section 6.6 of [Part1]).

 This only applies if the Expect: 100-continue was sent. Expect:
 100-continue was meant to solve the issue where the client has a large
 body, and wants to make sure that the server will accept the body before
 sending it (and wasting bandwidth). Meaning that without sending
 Expect: 100-continue, it is expected that the server will not send a
 response until the body has been sent.

No, it is expected that it will need to consume all the data before the
connection may be reused for sending another request. That is the point
of 100. And the problem is that if the server closes the connection when
responding early (typically a 302) and doesn't drain the client's data,
there's a high risk that the TCP stack will send an RST that can arrive
before the actual response, making the client unaware of the response.
That's why the server must consume the data even if it responds before
the end.

(...)
  In general (unless there's something wrong with the way client timeouts
  are reported in http_request_forward_body), client timeouts should be
  reported as such, and same for server timeouts. It's possible that there
  are corner cases, but we need to be extremely careful about them and not
  try to generalize.
 I agree, a client timeout should be reported as such, and that's what
 this is all about. If the client sends half the body (or no body), and
 then freezes, the client timeout should kick in and send back a 408, not
 the server timeout resulting in a 504.

Yes, I agree with this description.

 I think in this regards it is very clear.
 * The server may respond with the HTTP response status code any time it
 feels like it.

OK

 * Enable the server timeout and disable the client timeout upon any of
 the following:
 * The client sent Expect: 100-continue and has completed all headers

No, this one is wrong as well, as the client is expected to start sending
if it does not see the 100-continue, for compatibility with 1.0 and pre-2616
servers, because this header was invented very late. So both sides are
responsible for acting here, and the client timeout must not be cleared.

 * The complete client request has been sent, including body if
 Content-Length  0

Yes or chunked encoding is used and all the request, body and trailers
have been received. This is already done exactly this way (unless there's
a but of course).

 * Writing to the server socket would result in a blocking write
 (indicating that the remote end is 

Re: please check

[Patrick Hemmer]

*From: *Willy Tarreau w...@1wt.eu
*Sent: * 2014-05-02 12:56:16 E
*To: *Patrick Hemmer hapr...@stormcloud9.net
*CC: *Rachel Chavez rachel.chave...@gmail.com, haproxy@formilux.org
*Subject: *Re: please check

 On Fri, May 02, 2014 at 12:18:43PM -0400, Patrick Hemmer wrote:
 At any moment the server is free to respond yes, but the server cannot
 respond *properly* until it gets the complete request.
 Yes it can, redirects are the most common anticipated response, as the
 result of a POST to a page with an expired cookie. And the 302 is a
 clean response, it's not even an error.
 I should have clarified what I meant by properly more. I didn't mean
 that the server can't respond at all, as there are many cases it can,
 some of which you point out. I meant that if the server is expecting a
 request body, it can't respond with a 200 until it verifies that request
 body.
 OK, but from a reverse-proxy point of view, all of them are equally valid,
 and there's even no way to know if the server is interested in receiving
 these data at all. The only differences are that some of them are considered
 precious (ie those returning 200) and other ones less since they're
 possibly ephemeral.

 If the response depends on the request payload, the server doesn't know
 whether to respond with 200 or with a 400.
 With WAFs deployed massively on server infrastructures, 403 are quite
 common long before the whole data. 413 request entity too large appears
 quite commonly as well. 401 and 407 can also happen when authentication
 is needed.

 RFC2616 covers this behavior in depth. See 8.2.3 Use of the 100
 (Continue) Status. This section indicates that it should not be
 expected for the server to respond without a request body unless the
 client explicitly sends a Expect: 100-continue
 Well, 2616 is 15-years old now and pretty obsolete, which is why the
 HTTP-bis WG is working on refreshing this. New wording is clearer about
 how a request body is used :

o  A server MAY omit sending a 100 (Continue) response if it has
   already received some or all of the message body for the
   corresponding request, or if the framing indicates that there is
   no message body.

 Note the some or all.
 I'm assuming you're quoting from:
 http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-26#section-5.1.1
 Yes indeed. Ah in fact I found the exact part I was looking for, it's in
 the same block, two points below :

o  A server that responds with a final status code before reading the
   entire message body SHOULD indicate in that response whether it
   intends to close the connection or continue reading and discarding
   the request message (see Section 6.6 of [Part1]).

 This only applies if the Expect: 100-continue was sent. Expect:
 100-continue was meant to solve the issue where the client has a large
 body, and wants to make sure that the server will accept the body before
 sending it (and wasting bandwidth). Meaning that without sending
 Expect: 100-continue, it is expected that the server will not send a
 response until the body has been sent.
 No, it is expected that it will need to consume all the data before the
 connection may be reused for sending another request. That is the point
 of 100. And the problem is that if the server closes the connection when
 responding early (typically a 302) and doesn't drain the client's data,
 there's a high risk that the TCP stack will send an RST that can arrive
 before the actual response, making the client unaware of the response.
 That's why the server must consume the data even if it responds before
 the end.
 A 100-continue expectation informs recipients that the client is
   about to send a (presumably large) message body in this request and
   wishes to receive a 100 (Continue) interim response if the request-
   line and header fields are not sufficient to cause an immediate
   success, redirect, or error response.  This allows the client to wait
   for an indication that it is worthwhile to send the message body
   before actually doing so, which can improve efficiency when the
   message body is huge or when the client anticipates that an error is
   likely


 (...)
 In general (unless there's something wrong with the way client timeouts
 are reported in http_request_forward_body), client timeouts should be
 reported as such, and same for server timeouts. It's possible that there
 are corner cases, but we need to be extremely careful about them and not
 try to generalize.
 I agree, a client timeout should be reported as such, and that's what
 this is all about. If the client sends half the body (or no body), and
 then freezes, the client timeout should kick in and send back a 408, not
 the server timeout resulting in a 504.
 Yes, I agree with this description.

 I think in this regards it is very clear.
 * The server may respond with the HTTP response status code any time it
 feels like it.
 OK

 * Enable the server timeout and disable the client

Re: please check

[Willy Tarreau]

On Fri, May 02, 2014 at 01:32:30PM -0400, Patrick Hemmer wrote:
  This only applies if the Expect: 100-continue was sent. Expect:
  100-continue was meant to solve the issue where the client has a large
  body, and wants to make sure that the server will accept the body before
  sending it (and wasting bandwidth). Meaning that without sending
  Expect: 100-continue, it is expected that the server will not send a
  response until the body has been sent.
  No, it is expected that it will need to consume all the data before the
  connection may be reused for sending another request. That is the point
  of 100. And the problem is that if the server closes the connection when
  responding early (typically a 302) and doesn't drain the client's data,
  there's a high risk that the TCP stack will send an RST that can arrive
  before the actual response, making the client unaware of the response.
  That's why the server must consume the data even if it responds before
  the end.
  A 100-continue expectation informs recipients that the client is
about to send a (presumably large) message body in this request and
wishes to receive a 100 (Continue) interim response if the request-
line and header fields are not sufficient to cause an immediate
success, redirect, or error response.  This allows the client to wait
for an indication that it is worthwhile to send the message body
before actually doing so, which can improve efficiency when the
message body is huge or when the client anticipates that an error is
likely

Yes exactly. Since there's no way to stop in the middle of a sent body,
when you start you need to complete and the other side needs to drain.
I think we're saying the same thing from two different angles :-)

 While I strongly disagree with your interpretation of Expect:
 100-continue, I also don't much care about 100-continue. Hardly anyone
 uses it.

100% of web services I've seen use it in order to maintain connection pools :-)
And that's stupid BTW, because they keep the connections open in order to save
a connect round trip, which is replaced with a longer roundtrip involving half
of the request in the first packet, and keeping large amounts of memory in use!

 I was just using it as documentation that the server should not
 be expected to respond before the entire request has been sent.

I know that you used it for this but I disagree with your conclusion,
based on reality in field and even on what the spec says.

 The main thing I care about is not responding with 504 if the client
 freezes while sending the body. This has been a thorn in our side for
 quite some time now, and why I am interested in this patch.

I easily understand. I've seen a place where webservices were used a lot,
and in these environments, they use 500 to return not found! Quite a
mess when you want to set up some monitoring and alerts to report servers
going sick!!!

 I've set up a test scenario, and the only time haproxy responds with 408
 is if the client times out in the middle of request headers. If the
 client has sent all headers, but no body, or partial body, it times out
 after the configured 'timeout server' value, and responds with 504.

OK that's really useful. I'll try to reproduce that case. Could you please
test again with a shorter client timeout than server timeout, just to ensure
that it's not just a sequencing issue ?

 Applying the patch solves this behavior. But my test scenario is very
 simple, and I'm not sure if it has any other consequences.

It definitely has, which is why I'm trying to find the *exact* problem in
order to fix it.

Thanks,
Willy

Re: please check

[Patrick Hemmer]

*From: *Willy Tarreau w...@1wt.eu
*Sent: * 2014-05-02 14:00:24 E
*To: *Patrick Hemmer hapr...@stormcloud9.net
*CC: *Rachel Chavez rachel.chave...@gmail.com, haproxy@formilux.org
*Subject: *Re: please check

 On Fri, May 02, 2014 at 01:32:30PM -0400, Patrick Hemmer wrote:
 I've set up a test scenario, and the only time haproxy responds with 408
 is if the client times out in the middle of request headers. If the
 client has sent all headers, but no body, or partial body, it times out
 after the configured 'timeout server' value, and responds with 504.
 OK that's really useful. I'll try to reproduce that case. Could you please
 test again with a shorter client timeout than server timeout, just to ensure
 that it's not just a sequencing issue ?
I have. In my test setup, timeout client 1000 and timeout server 2000.

With incomplete headers I get:
haproxy[8893]: 127.0.0.1:41438 [02/May/2014:14:11:26.373] f1 f1/NOSRV
-1/-1/-1/-1/1001 408 212 - - cR-- 0/0/0/0/0 0/0 BADREQ

With no body I get:
haproxy[8893]: 127.0.0.1:41439 [02/May/2014:14:11:29.576] f1 b1/s1
0/0/0/-1/2002 504 194 - - sH-- 1/1/1/1/0 0/0 GET / HTTP/1.1

With incomplete body I get:
haproxy[8893]: 127.0.0.1:41441 [02/May/2014:14:11:29.779] f1 b1/s1
0/0/0/-1/2002 504 194 - - sH-- 0/0/0/0/0 0/0 GET / HTTP/1.1




 Applying the patch solves this behavior. But my test scenario is very
 simple, and I'm not sure if it has any other consequences.
 It definitely has, which is why I'm trying to find the *exact* problem in
 order to fix it.

 Thanks,
 Willy



-Patrick

Re: please check

[Willy Tarreau]

On Fri, May 02, 2014 at 02:14:41PM -0400, Patrick Hemmer wrote:
 *From: *Willy Tarreau w...@1wt.eu
 *Sent: * 2014-05-02 14:00:24 E
 *To: *Patrick Hemmer hapr...@stormcloud9.net
 *CC: *Rachel Chavez rachel.chave...@gmail.com, haproxy@formilux.org
 *Subject: *Re: please check
 
  On Fri, May 02, 2014 at 01:32:30PM -0400, Patrick Hemmer wrote:
  I've set up a test scenario, and the only time haproxy responds with 408
  is if the client times out in the middle of request headers. If the
  client has sent all headers, but no body, or partial body, it times out
  after the configured 'timeout server' value, and responds with 504.
  OK that's really useful. I'll try to reproduce that case. Could you please
  test again with a shorter client timeout than server timeout, just to ensure
  that it's not just a sequencing issue ?
 I have. In my test setup, timeout client 1000 and timeout server 2000.
 
 With incomplete headers I get:
 haproxy[8893]: 127.0.0.1:41438 [02/May/2014:14:11:26.373] f1 f1/NOSRV
 -1/-1/-1/-1/1001 408 212 - - cR-- 0/0/0/0/0 0/0 BADREQ
 
 With no body I get:
 haproxy[8893]: 127.0.0.1:41439 [02/May/2014:14:11:29.576] f1 b1/s1
 0/0/0/-1/2002 504 194 - - sH-- 1/1/1/1/0 0/0 GET / HTTP/1.1
 
 With incomplete body I get:
 haproxy[8893]: 127.0.0.1:41441 [02/May/2014:14:11:29.779] f1 b1/s1
 0/0/0/-1/2002 504 194 - - sH-- 0/0/0/0/0 0/0 GET / HTTP/1.1

Great, thank you. I think that it tends to fuel the theory that the
response error is not set where it should be in the forwarding path.

I'll check this ASAP. BTW, it would be nice if you could check this
as well with 1.4.25, I guess it does the same.

Best regards,
Willy

Re: please check

[Patrick Hemmer]

 



*From: *Willy Tarreau w...@1wt.eu
*Sent: * 2014-05-02 15:06:13 E
*To: *Patrick Hemmer hapr...@stormcloud9.net
*CC: *Rachel Chavez rachel.chave...@gmail.com, haproxy@formilux.org
*Subject: *Re: please check

 On Fri, May 02, 2014 at 02:14:41PM -0400, Patrick Hemmer wrote:
 *From: *Willy Tarreau w...@1wt.eu
 *Sent: * 2014-05-02 14:00:24 E
 *To: *Patrick Hemmer hapr...@stormcloud9.net
 *CC: *Rachel Chavez rachel.chave...@gmail.com, haproxy@formilux.org
 *Subject: *Re: please check

 On Fri, May 02, 2014 at 01:32:30PM -0400, Patrick Hemmer wrote:
 I've set up a test scenario, and the only time haproxy responds with 408
 is if the client times out in the middle of request headers. If the
 client has sent all headers, but no body, or partial body, it times out
 after the configured 'timeout server' value, and responds with 504.
 OK that's really useful. I'll try to reproduce that case. Could you please
 test again with a shorter client timeout than server timeout, just to ensure
 that it's not just a sequencing issue ?
 I have. In my test setup, timeout client 1000 and timeout server 2000.

 With incomplete headers I get:
 haproxy[8893]: 127.0.0.1:41438 [02/May/2014:14:11:26.373] f1 f1/NOSRV
 -1/-1/-1/-1/1001 408 212 - - cR-- 0/0/0/0/0 0/0 BADREQ

 With no body I get:
 haproxy[8893]: 127.0.0.1:41439 [02/May/2014:14:11:29.576] f1 b1/s1
 0/0/0/-1/2002 504 194 - - sH-- 1/1/1/1/0 0/0 GET / HTTP/1.1

 With incomplete body I get:
 haproxy[8893]: 127.0.0.1:41441 [02/May/2014:14:11:29.779] f1 b1/s1
 0/0/0/-1/2002 504 194 - - sH-- 0/0/0/0/0 0/0 GET / HTTP/1.1
 Great, thank you. I think that it tends to fuel the theory that the
 response error is not set where it should be in the forwarding path.

 I'll check this ASAP. BTW, it would be nice if you could check this
 as well with 1.4.25, I guess it does the same.

 Best regards,
 Willy

Confirmed. Exact same behavior with 1.4.25

-Patrick

Re: please check

[Willy Tarreau]

On Fri, May 02, 2014 at 03:22:45PM -0400, Patrick Hemmer wrote:
  
 
 
 
 *From: *Willy Tarreau w...@1wt.eu
 *Sent: * 2014-05-02 15:06:13 E
 *To: *Patrick Hemmer hapr...@stormcloud9.net
 *CC: *Rachel Chavez rachel.chave...@gmail.com, haproxy@formilux.org
 *Subject: *Re: please check
 
  On Fri, May 02, 2014 at 02:14:41PM -0400, Patrick Hemmer wrote:
  *From: *Willy Tarreau w...@1wt.eu
  *Sent: * 2014-05-02 14:00:24 E
  *To: *Patrick Hemmer hapr...@stormcloud9.net
  *CC: *Rachel Chavez rachel.chave...@gmail.com, haproxy@formilux.org
  *Subject: *Re: please check
 
  On Fri, May 02, 2014 at 01:32:30PM -0400, Patrick Hemmer wrote:
  I've set up a test scenario, and the only time haproxy responds with 408
  is if the client times out in the middle of request headers. If the
  client has sent all headers, but no body, or partial body, it times out
  after the configured 'timeout server' value, and responds with 504.
  OK that's really useful. I'll try to reproduce that case. Could you please
  test again with a shorter client timeout than server timeout, just to 
  ensure
  that it's not just a sequencing issue ?
  I have. In my test setup, timeout client 1000 and timeout server 2000.
 
  With incomplete headers I get:
  haproxy[8893]: 127.0.0.1:41438 [02/May/2014:14:11:26.373] f1 f1/NOSRV
  -1/-1/-1/-1/1001 408 212 - - cR-- 0/0/0/0/0 0/0 BADREQ
 
  With no body I get:
  haproxy[8893]: 127.0.0.1:41439 [02/May/2014:14:11:29.576] f1 b1/s1
  0/0/0/-1/2002 504 194 - - sH-- 1/1/1/1/0 0/0 GET / HTTP/1.1
 
  With incomplete body I get:
  haproxy[8893]: 127.0.0.1:41441 [02/May/2014:14:11:29.779] f1 b1/s1
  0/0/0/-1/2002 504 194 - - sH-- 0/0/0/0/0 0/0 GET / HTTP/1.1
  Great, thank you. I think that it tends to fuel the theory that the
  response error is not set where it should be in the forwarding path.
 
  I'll check this ASAP. BTW, it would be nice if you could check this
  as well with 1.4.25, I guess it does the same.
 
  Best regards,
  Willy
 
 Confirmed. Exact same behavior with 1.4.25

Thank you!

Willy

please check

[Rachel Chavez]

Could someone please tell me if what I did is right?
I posted this fix a while ago and I didn't get a response.

http://marc.info/?l=haproxym=139506908430417w=2

Thank you in advance,

-- 
Rachel Chavez

Re: please check

[Willy Tarreau]

Hi,

On Thu, May 01, 2014 at 09:56:01AM -0400, Rachel Chavez wrote:
 Could someone please tell me if what I did is right?
 I posted this fix a while ago and I didn't get a response.
 
 http://marc.info/?l=haproxym=139506908430417w=2
 
 Thank you in advance,

Could you please first describe the problem you encountered ?

It's very hard to say if some code does the right thing in front
of no known problem. Moreover, your mailer has totally mangled
the patch, making it harder to understand. But at first glance
I'm seeing thing that don't look normal, such as returning 0
from the analyser in case of timeout without even setting any
error.

Thanks,
Willy

Re: please check

[Rachel Chavez]

The problem is:

when client sends a request with incomplete body (it has content-length but
no body) then haproxy returns a 5XX error when it should be a client issue.

In the session.c file starting in 2404 i make sure that if I haven't
received the entire body of the request I continue to wait for it by
keeping AN_REQ_WAIT_HTTP as part of the request analyzers list as long as
the client read timeout hasn't fired yet.
In the proto_http.c file what I tried to do is avoid getting a server
timeout when the client had timed-out already.

Thank you for your prompt answer,
Rachel

On Thu, May 1, 2014 at 3:27 PM, Willy Tarreau w...@1wt.eu wrote:

 Hi,

 On Thu, May 01, 2014 at 09:56:01AM -0400, Rachel Chavez wrote:
  Could someone please tell me if what I did is right?
  I posted this fix a while ago and I didn't get a response.
 
  http://marc.info/?l=haproxym=139506908430417w=2
 
  Thank you in advance,

 Could you please first describe the problem you encountered ?

 It's very hard to say if some code does the right thing in front
 of no known problem. Moreover, your mailer has totally mangled
 the patch, making it harder to understand. But at first glance
 I'm seeing thing that don't look normal, such as returning 0
 from the analyser in case of timeout without even setting any
 error.

 Thanks,
 Willy




-- 
Rachel Chavez