Re: Truncated response on 2.0.8

[Christopher Faulet]

Le 31/10/2019 à 11:01, Ing. Andrea Vettori a écrit :

Sorry to bother you with all these stuff. But for now, I don't know where to 
start my investigation.

Sure I’ll do as soon as possibile and thank you very much for your continued 
support.



Hi,

Sorry for the delay and thanks for the captures you sent me. In them, I noticed 
2 things that explain the truncated responses :


* Firstly, when you send your request through HAProxy, your server send a frame 
GOAWAY with the error code ERR_ENHANCE_YOUR_CALM(11) in the middle of the 
response transfer. Because, once the request is sent, HAProxy only send frames 
WINDOW_UPDATE, I guess your server has some "security" mechanism about those 
frames. Or it is a bug. I don't know. The last stream ID indicated in the frame 
GOAWAY is the one of your request. So, the response transfer should normally be 
continued. In itself, it is not really a problem. And the transfer is 
effectively still in progress because in the captures, we can see a frame DATA 
following the GOAWAY. But after that, your server aborts the transfer by closing 
the connection with a TCP reset, probably because it receives some other frames 
WINDOW_UPDATE from HAProxy.


* Secondly, the server abort is not properly handled on the client side by the 
mux h2. It is definitely a bug. We should normally send a RST_STREAM to abort 
the request. Instead, we mark the end of the response with an empty frame DATA 
with the flag END_STREAM set. Because there is no content-length specified on 
the response, the client has no way to know that the response was truncated.


So, this bug must be addressed. We're still discussing with Willy to know how to 
do it properly. But, in the end, it requires some fundamental changes on the way 
aborts are handle in HAProxy. We are rather reluctant to do a quick patch.


Once said, the problem remains on your server side, because it aborts the 
response transfer. I noticed the last WINDOW_UPDATE before the GOAWAY has a 
length of 1. This may be related.


--
Christopher Faulet

Re: Truncated response on 2.0.8

[Willy Tarreau]

On Thu, Oct 31, 2019 at 11:01:06AM +0100, Ing. Andrea Vettori wrote:
> Is it normal that the data frame with END_STREAM has zero length on haproxy
> responses while it's > 0 (1 actually) on the direct connection ?

It simply corresponds to a late shutdown. It's the normal way of
closing when you only know later that you've reached the end, typically
because you parse a zero-sized chunk. It may be an indication of something
that went wrong in this particular test, but it is totally valid and normal.

Thanks,
Willy

Re: Truncated response on 2.0.8

[Ing. Andrea Vettori]

> On 31 Oct 2019, at 10:02, Christopher Faulet  wrote:
> 
> Le 29/10/2019 à 18:48, Ing. Andrea Vettori a écrit :
>> Here is a direct session to backend server from the client machine in clear 
>> text http2 without upgrade. Note that in this case output is NOT truncated 
>> if I call the command without the ‘-n’ option. To do this test I temporary 
>> overriden the host’s name dns resolution on the client (curl has a handy 
>> option to do that but haven’t found it on nghttp).
> ...
>> The following is the same client to the haproxy server, to port 80 i.e. no 
>> “proto h2” on the front-end. Proto h2 on the backend. Note that in this case 
>> output is truncated if I call the command without the ‘-n’ option
> ...
>> Finally this is the output when port 82 is called on haproxy, i.e. the 
>> frontend with “proto h2” option. This also produces truncated output.
>> 
> ...
> 
> Ok, Thanks for these infos. So, from the H2 point of view, the responses are 
> valid and well formatted. But the responses size are radically different. If 
> the size should be the same for all the three responses, the bug is probably 
> in the communication between HAProxy and the server. Because the 2nd response 
> is ~144K lower than the 1st one. And the 3rd is ~232K lower. So, far from the 
> internal buffer size.

Yes, I also noticed that the connections that go through haproxy have data 
packets with very different lengths while the direct connection seems a lot 
more consistent with respect to size. 

> 
> To emit a flag END_STREAM on a DATA frame, internally, we need to reach the 
> end of the HTTP message. It is marked by a dedicated block of data 
> (HTX_BLK_EOM). This one is present if the flag END_STREAM was received on the 
> other side.

Is it normal that the data frame with END_STREAM has zero length on haproxy 
responses while it’s > 0 (1 actually) on the direct connection ?

> 
> So now, the next move is to provide a network capture. Sorry to ask it again 
> :) Because you use dedicated ports for your tests, I guess it will be quite 
> simple. Could you provide the result of the following command, executed on 
> HAProxy server ?
> 
>  tcpdump -i any -w h2toh2.cap  port 82 or port 8080
> 
> You can fill the capture with some requests on http://g.testhost.com:82. It 
> could be also usefull to have the capture of a successfull request.
> 
> Sorry to bother you with all these stuff. But for now, I don't know where to 
> start my investigation.


Sure I’ll do as soon as possibile and thank you very much for your continued 
support.

—
Ing. Andrea Vettori
Responsabile Sistemi Informativi

Re: Truncated response on 2.0.8

[Lukas Tribus]

On Thu, Oct 31, 2019 at 10:02 AM Christopher Faulet  wrote:
> Could you provide the result of the following command, executed on HAProxy 
> server ?
>
>tcpdump -i any -w h2toh2.cap  port 82 or port 8080

Better add -s0, otherwise you may only get 68 bytes (at least on older
tcpdump releases).


Lukas

Re: Truncated response on 2.0.8

[Christopher Faulet]

Le 29/10/2019 à 18:48, Ing. Andrea Vettori a écrit :
Here is a direct session to backend server from the client machine in clear text 
http2 without upgrade. Note that in this case output is NOT truncated if I call 
the command without the ‘-n’ option. To do this test I temporary overriden the 
host’s name dns resolution on the client (curl has a handy option to do that but 
haven’t found it on nghttp).



...



The following is the same client to the haproxy server, to port 80 i.e. no 
“proto h2” on the front-end. Proto h2 on the backend. Note that in this case 
output is truncated if I call the command without the ‘-n’ option



...


Finally this is the output when port 82 is called on haproxy, i.e. the frontend 
with “proto h2” option. This also produces truncated output.



...

Ok, Thanks for these infos. So, from the H2 point of view, the responses are 
valid and well formatted. But the responses size are radically different. If the 
size should be the same for all the three responses, the bug is probably in the 
communication between HAProxy and the server. Because the 2nd response is ~144K 
lower than the 1st one. And the 3rd is ~232K lower. So, far from the internal 
buffer size.


To emit a flag END_STREAM on a DATA frame, internally, we need to reach the end 
of the HTTP message. It is marked by a dedicated block of data (HTX_BLK_EOM). 
This one is present if the flag END_STREAM was received on the other side.


So now, the next move is to provide a network capture. Sorry to ask it again :) 
Because you use dedicated ports for your tests, I guess it will be quite simple. 
Could you provide the result of the following command, executed on HAProxy server ?


  tcpdump -i any -w h2toh2.cap  port 82 or port 8080

You can fill the capture with some requests on http://g.testhost.com:82. It 
could be also usefull to have the capture of a successfull request.


Sorry to bother you with all these stuff. But for now, I don't know where to 
start my investigation.


Thanks,
--
Christopher Faulet

Re: Truncated response on 2.0.8

[Christopher Faulet]

Le 29/10/2019 à 15:33, Ing. Andrea Vettori a écrit :


On 29 Oct 2019, at 10:20, Christopher Faulet > wrote:
Please consider to use nghttp if possible. Curl does not give enough details 
at the protocol layer. Also enable HTTP logs (option httplog in your defaults 
section and the info level on the log line in your global section instead of 
warning).



I’m collecting a few tests using nghttp and to do so I aligned the dev web 
server configuration with those in production (to compare the outputs).
While doing this haproxy emitted this error and I think it’s related to a 
connection that seemed stuck.



haproxy[20636]: A bogus STREAM [0x7efd240595f0] is spinning at 189145 calls per 
second and refuses to die, aborting now! Please report this error to developers 
[strm=0x7efd240595f0 src=ipaddr fe=dev be=dev-ssl-servers dst=webdev rqf=c40e060 
rqa=0 rpf=8000a020 rpa=0 sif=CLO,c8000 sib=CON,280191 af=(nil),0 
csf=0x7efd24027420,0 ab=(nil),0 csb=0x7efd240577b0,800 
cof=0x7efcdc05cb50,243300:H2(0x7efd24056c10)/SSL(0x7efd24057d70)/tcpv4(306) 
cob=0x7efd24021a70,202306:H2(0x7efd2403a960)/RAW((nil))/tcpv4(6) ]



I guess you hit the bug fixed by the commit 27ebcefd4 ("BUG/MAJOR: stream-int: 
Don't receive data from mux until SI_ST_EST is reached") in the 2.0. Are you 
sure your server supports non-TLS HTTP/2 (without upgrade) ? This bug was fixed 
after the 2.0.8. It could also be a problem with the H2 settings exchanged with 
your server (from the protocol point of view).


BTW, the abort is probably not responsible to your truncated responses because 
here the stream is aborted during the connection to the server. At this stage, 
nothing was yet sent to the server and thus nothing was received. But it 
explains your connection issues.


If I'm right, try to fix your platform issues first. Truncated responses may 
just be a side effect. But for now, it is hard to track a possible bug.


At this stage, a solution could be to install HAProxy 2.1 to have the H2 traces. 
It is a live debug mechanism. But it is  pretty verbose. Before that, a solution 
would also be to get a network capture on the HAProxy server. At least on the 
server side.


Finally, now that you are able to send requests using nghttp, could you share 
the verbose output of a h2 request sent to HAProxy and the same sent to your H2 
server please ?


Thanks,
--
Christopher Faulet

Re: Truncated response on 2.0.8

[Ing. Andrea Vettori]

> On 29 Oct 2019, at 10:20, Christopher Faulet  wrote:
> Please consider to use nghttp if possible. Curl does not give enough details 
> at the protocol layer. Also enable HTTP logs (option httplog in your defaults 
> section and the info level on the log line in your global section instead of 
> warning).


I’m collecting a few tests using nghttp and to do so I aligned the dev web 
server configuration with those in production (to compare the outputs).
While doing this haproxy emitted this error and I think it’s related to a 
connection that seemed stuck.


haproxy[20636]: A bogus STREAM [0x7efd240595f0] is spinning at 189145 calls per 
second and refuses to die, aborting now! Please report this error to developers 
[strm=0x7efd240595f0 src=ipaddr fe=dev be=dev-ssl-servers dst=webdev 
rqf=c40e060 rqa=0 rpf=8000a020 rpa=0 sif=CLO,c8000 sib=CON,280191 af=(nil),0 
csf=0x7efd24027420,0 ab=(nil),0 csb=0x7efd240577b0,800 
cof=0x7efcdc05cb50,243300:H2(0x7efd24056c10)/SSL(0x7efd24057d70)/tcpv4(306) 
cob=0x7efd24021a70,202306:H2(0x7efd2403a960)/RAW((nil))/tcpv4(6) ]

Re: Truncated response on 2.0.8

[Christopher Faulet]

Le 29/10/2019 à 09:54, Ing. Andrea Vettori a écrit :
On 29 Oct 2019, at 09:39, Christopher Faulet > wrote:


Le 28/10/2019 à 22:00, Christopher Faulet a écrit :

macbookpro:~ andreavettori$ curl --http2 -v http://g.testhost

[SNIP]
Here, it is the expected behavior. HAProxy does not support explicit H2 upgrade.


In Fact, for this one, I was wrong. The result should be more or less the same 
that an HTTP/1.1 request. The upgrade should be ignored and the request should 
be sent in H2 to the server without the header "Upgrade". In return, the H2 
response should be converted to HTTP/1.1 and sent to the client.


So, as for all requests on the ports 80 and 82, you have a configuration 
issue. Looking at you complete configuration (with global and defaults 
sections), it does not seem to be problem with your HAProxy configuration.

So there is something else.


I’m not sure to understand what you’re referring about ?  Wrong/old libraries 
when compiling haproxy or … ?





I don't know where is the problem. Your configuration is pretty simple but all 
requests on ports 80 and 82 failed. Connections are closed with no reply. It is 
totally unexpected of course. It may be a firewalling problem. And I don't know 
if it is related or not to you issue.


Please consider to use nghttp if possible. Curl does not give enough details at 
the protocol layer. Also enable HTTP logs (option httplog in your defaults 
section and the info level on the log line in your global section instead of 
warning).


Thanks,
--
Christopher Faulet

Re: Truncated response on 2.0.8

[Ing. Andrea Vettori]

> On 29 Oct 2019, at 09:39, Christopher Faulet  wrote:
> 
> Le 28/10/2019 à 22:00, Christopher Faulet a écrit :
>>> macbookpro:~ andreavettori$ curl --http2 -v http://g.testhost
>> [SNIP]
>> Here, it is the expected behavior. HAProxy does not support explicit H2 
>> upgrade.
> 
> In Fact, for this one, I was wrong. The result should be more or less the 
> same that an HTTP/1.1 request. The upgrade should be ignored and the request 
> should be sent in H2 to the server without the header "Upgrade". In return, 
> the H2 response should be converted to HTTP/1.1 and sent to the client.
> 
> So, as for all requests on the ports 80 and 82, you have a configuration 
> issue. Looking at you complete configuration (with global and defaults 
> sections), it does not seem to be problem with your HAProxy configuration.
> So there is something else.

I’m not sure to understand what you’re referring about ?  Wrong/old libraries 
when compiling haproxy or … ?

This is the output of haproxy -vv

HA-Proxy version 2.0.8 2019/10/23 - https://haproxy.org/
Build options :
  TARGET  = linux-glibc
  CPU = generic
  CC  = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv 
-Wno-unused-label -Wno-sign-compare -Wno-unused-parameter 
-Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered 
-Wno-missing-field-initializers -Wtype-limits
  OPTIONS = USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 USE_SYSTEMD=1

Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER +PCRE -PCRE_JIT 
-PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED -REGPARM 
-STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT 
+CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL -LUA +FUTEX +ACCEPT4 -MY_ACCEPT4 +ZLIB 
-SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD 
-OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=40).
Built with OpenSSL version : OpenSSL 1.0.2k-fips  26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k-fips  26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with network namespace support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT 
IP_FREEBIND
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity("identity"), deflate("deflate"), 
raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes

Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as  cannot be specified using 'proto' keyword)
  h2 : mode=HTXside=FE|BE mux=H2
  h2 : mode=HTTP   side=FEmux=H2
: mode=HTXside=FE|BE mux=H1
: mode=TCP|HTTP   side=FE|BE mux=PASS

Available services : none

Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace


There are no trailers in the response.

Here’s the curl log from the server where haproxy is running to the backend 
direct connection

curl --http2 -v --resolve g.testhost:8083:10.2.2.50 "http://g.testhost:8083; > 
/tmp/1
* Added g.testhost:8083:10.2.2.50 to DNS cache
* Hostname g.testhost was found in DNS cache
*   Trying 10.2.2.50:8083...
* TCP_NODELAY set
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0* 
Connected to g.testhost (10.2.2.50) port 8083 (#0)
> GET / HTTP/1.1
> Host: g.testhost:8083
> User-Agent: curl/7.66.0
> Accept: */*
> Connection: Upgrade, HTTP2-Settings
> Upgrade: h2c
> HTTP2-Settings: AAMAAABkAARAAAIA
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 101 
< Connection: Upgrade
< Upgrade: h2c
< Date: Tue, 29 Oct 2019 08:45:37 GMT
* Received 101
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 200 
< set-cookie: JSESSIONID=AD70DA604A53D7095BF5EB46FA7DA33B; Path=/; Secure; 
HttpOnly
* Added cookie agentid="TT62BGKFSXG2B577FZ9CN0TWFIHLWI1CYO371BWN" for domain 
g.testhost, path /, expire 3719822383
< set-cookie: agentid=TT62BGKFSXG2B577FZ9CN0TWFIHLWI1CYO371BWN; 
Max-Age=2147483647; Expires=Sun, 16-Nov-2087 11:59:44 GMT; Path=/
* Added cookie 

Re: Truncated response on 2.0.8

[Christopher Faulet]

Le 28/10/2019 à 22:00, Christopher Faulet a écrit :



macbookpro:~ andreavettori$ curl --http2 -v http://g.testhost

[SNIP]
Here, it is the expected behavior. HAProxy does not support explicit H2 upgrade.




In Fact, for this one, I was wrong. The result should be more or less the same 
that an HTTP/1.1 request. The upgrade should be ignored and the request should 
be sent in H2 to the server without the header "Upgrade". In return, the H2 
response should be converted to HTTP/1.1 and sent to the client.


So, as for all requests on the ports 80 and 82, you have a configuration issue. 
Looking at you complete configuration (with global and defaults sections), it 
does not seem to be problem with your HAProxy configuration. So there is 
something else.


BTW, could you also describe the response. Its size ? Is there any trailers ? If 
any, How big are they ?


--
Christopher Faulet

Re: Truncated response on 2.0.8

[Ing. Andrea Vettori]

> On 28 Oct 2019, at 22:00, Christopher Faulet  wrote:
>> macbookpro:~ andreavettori$ curl --http1.1 -v http://g.testhost
> [SNIP]
> Here there is a problem with your configuration. Don't know what, but HAProxy 
> closed abruptly the client connection. Could you provide your default and 
> global section please ?


Here are the global and defaults section.
Thanks for you help


global
daemon
maxconn 7500
log /dev/log local0 warning
description descr
tune.ssl.default-dh-param 2048
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11
ssl-default-bind-ciphers 
ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
stats socket /var/run/haproxy_nagios.sock mode 666 level user
stats timeout 2m

defaults
mode http
log global
timeout connect 15s
timeout client 50s
timeout server 50s

option forwardfor

stats enable
stats refresh 60s
stats show-desc

option redispatch

Re: Truncated response on 2.0.8

[Christopher Faulet]

Le 28/10/2019 à 15:49, Ing. Andrea Vettori a écrit :


If it is on a development platform, is there any way to have a full network 
capture ?


Thanks for your help. The server where haproxy is running is a production server 
while this specific front/backend is a development web site. I can arrange a 
network capture if needed.


It could be very helpful. The best would be to have a trace from both sides, the 
client and the server.


Finally, nghttp (https://nghttp2.org) is also a good tool to debug HTTP/2. You 
may try it to know why the response is truncated.


Thanks, I see that curl uses that library.


Yes, but the verbose mode of nghttp (the tool not the library) shows you all 
HTTP/2 frames exchanged with the server. It could be interesting to get the 
output of nghttp of a request sent to HAProxy and another one sent directly to 
your h2 server.




macbookpro:~ andreavettori$ curl --http2 -v https://g.testhost

[SNIP]
Ok, so here from the protocol point of view everything seems to be ok.



macbookpro:~ andreavettori$ curl --http1.1 -v https://g.testhost

[SNIP]
Here the response seems to be malformed. It is hard to said much more for now. 
But it could help to find what is going wrong.




macbookpro:~ andreavettori$ curl --http1.1 -v http://g.testhost

[SNIP]
Here there is a problem with your configuration. Don't know what, but HAProxy 
closed abruptly the client connection. Could you provide your default and global 
section please ?




macbookpro:~ andreavettori$ curl --http2 -v http://g.testhost

[SNIP]
Here, it is the expected behavior. HAProxy does not support explicit H2 upgrade.



macbookpro:~ andreavettori$ curl --http2-prior-knowledge -v http://g.testhost

[SNIP]
Here too there is a problem with your configuration. Same as above. It should 
work. It is an implicit upgrade to H2. When the first request received on a 
client connection is the H2 preface, we do an implicit upgrade.



Finally trying on the third bind (the one where h2 protocol is specified and no 
ssl option) gives the same results as the last two tests.


For the test with the option "--http2", it is normal and expected. But with the 
option "--http2-prior-knowledge", it is a real problem. Because "proto h2" on 
the bind line is exactly here to receive clear H2 requests.


It does not explain why you received truncated responses. But it seems you have 
a configuration problem that which disturbs you in your tests.


--
Christopher Faulet

Re: Truncated response on 2.0.8

[Ing. Andrea Vettori]

> On 28 Oct 2019, at 11:51, Christopher Faulet  wrote:
> 
> Le 26/10/2019 à 18:10, Ing. Andrea Vettori a écrit :
>> Hello,
>> I'm using haproxy 2.0.8 and ssl termination with h2 and http1.1 protocols.
>> Since today we always used http1.1 on the backends.
>> I’ve tried to use http2 on the development backend but I get truncated 
>> response (not always but very often).
>> Trying to connect from the server running haproxy to the backend server 
>> using curl with http2 I never get a truncated response.
>> Client-side I tried with two different browsers.
>> Any hint on what can cause this ?
>> Thank you!
> 
> Hi,
> 
> If it is on a development platform, is there any way to have a full network 
> capture ?

Thanks for your help. The server where haproxy is running is a production 
server while this specific front/backend is a development web site. I can 
arrange a network capture if needed.

> BTW, is there anything strange in your logs ?


No but I really don’t know how to configure haproxy logging (at the moment it 
just logs when background server goes down). I’ll try to read the documentation 
on logging later today.

> 
> Finally, nghttp (https://nghttp2.org) is also a good tool to debug HTTP/2. 
> You may try it to know why the response is truncated.

Thanks, I see that curl uses that library.

Here are the test I did today as per your suggestions, with the results. All 
connections are done from a client on a different network (so like a real 
user). Note that the client is on OSX while haproxy and backend are on linux.
This is the haproxy configuration I used for the tests

frontend dev
bind TESTIP:80
bind TESTIP:82 proto h2
bind TESTIP:443 ssl crt /etc/haproxy/ssl/wildcard1.pem crt 
/etc/haproxy/ssl/wildcard2.pem crt /etc/haproxy/ssl/wildcard2.pem alpn 
h2,http/1.1
acl local_hostname hdr(host) -m beg 10.
http-request deny if local_hostname

use_backend dev-ssl-servers if { ssl_fc }
default_backend dev-servers

backend dev-ssl-servers
server webdev 10.2.2.50:8083 maxconn 750 proto h2

backend dev-servers
server webdev 10.2.2.50:8080 maxconn 750 proto h2


This is an http2 connection with tls (third bind on the config file above)

macbookpro:~ andreavettori$ curl --http2 -v https://g.testhost
*   Trying TESTIP...
* TCP_NODELAY set
* Connected to g.testhost (TESTIP) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=IT; ST=Treviso; L=Cessalto; O=B2BIres s.r.l.; CN=*.testhost
*  start date: Oct  4 00:00:00 2018 GMT
*  expire date: Dec  2 12:00:00 2020 GMT
*  subjectAltName: host "g.testhost" matched cert's "*.testhost"testhost
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=Thawte RSA CA 2018
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fe206005400)
> GET / HTTP/2
> Host: g.testhost
> User-Agent: curl/7.64.1
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 200 
< set-cookie: JSESSIONID=722E5F380244B0C9A9577CABFB103D7C; Path=/; Secure; 
HttpOnly
< set-cookie: agentid=7RLX0GOL8GJC04N0P8K4EB5N77VTULZR7QJT3DD3; 
Max-Age=2147483647; Expires=Sat, 15-Nov-2087 16:45:45 GMT; Path=/
< set-cookie: mtt_id=7RLX0GOL8GJC04N0P8K4EB5N77VTULZR7QJT3DD3|; 
Max-Age=2147483647; Expires=Sat, 15-Nov-2087 16:45:45 GMT; Domain=testhost; 
Path=/
< cache-control: no-store, must-revalidate, max-age=0
< pragma: no-cache
< expires: Sat, 26 Jul 1997 05:00:00 GMT
< content-type: text/html;charset=UTF-8
< content-language: en-US
< date: Mon, 28 Oct 2019 13:31:38 GMT
< 

[partial content returned]

* Connection #0 to host g.testhost left intact
* Closing connection 0


This is a http1.1 client with tls (third bind on the config file above)

macbookpro:~ andreavettori$ curl --http1.1 -v https://g.testhost
*   Trying TESTIP...
* TCP_NODELAY set
* Connected to g.testhost (TESTIP) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, 

Re: Truncated response on 2.0.8

[Christopher Faulet]

Le 26/10/2019 à 18:10, Ing. Andrea Vettori a écrit :

Hello,
I'm using haproxy 2.0.8 and ssl termination with h2 and http1.1 protocols.
Since today we always used http1.1 on the backends.

I’ve tried to use http2 on the development backend but I get truncated response 
(not always but very often).


Trying to connect from the server running haproxy to the backend server using 
curl with http2 I never get a truncated response.

Client-side I tried with two different browsers.

Any hint on what can cause this ?
Thank you!

Here is the configuration I’m using for the dev frontend/backend

frontend dev
         bind :80
bind :443 ssl crt /etc/haproxy/ssl/wildcard1.pem crt 
/etc/haproxy/ssl/wildcard2.pem crt /etc/haproxy/ssl/wildcard3.pem alpn h2,http/1.1

acl local_hostname hdr(host) -m beg 10.
http-request deny if local_hostname

use_backend dev-ssl-servers if { ssl_fc }
         default_backend dev-servers

backend dev-ssl-servers
         server webdev 10.2.2.50:8083 maxconn 750 proto h2

backend dev-servers
         server webdev 10.2.2.50:8080 maxconn 750 proto h2



Hi,

If it is on a development platform, is there any way to have a full network 
capture ? I guess you send HTTP/2 requests to HAProxy, in SSL. Thus, the capture 
from the client side will be encrypted. Do you know if the responses are also 
truncated for an http/1.1 client ? If yes, try to get the network capture this 
way to have clear traffic on both sides. Another way is to use a client able to 
send clear HTTP/2 requests. If it is possible, you may add "proto h2" on your 
first bind line and try to get your capture this way.


BTW, is there anything strange in your logs ?

Finally, nghttp (https://nghttp2.org) is also a good tool to debug HTTP/2. You 
may try it to know why the response is truncated.


Thanks,
--
Christopher Faulet

Re: Truncated response on 2.0.8

[Sander Klein]

On 2019-10-26 18:10, Ing. Andrea Vettori wrote:

Hello,
I'm using haproxy 2.0.8 and ssl termination with h2 and http1.1
protocols.
Since today we always used http1.1 on the backends.

I’ve tried to use http2 on the development backend but I get
truncated response (not always but very often).

Trying to connect from the server running haproxy to the backend
server using curl with http2 I never get a truncated response.
Client-side I tried with two different browsers.

Any hint on what can cause this ?


Just to confirm. I think I have the same problem in the same setup. The 
first time I noticed this was on 2.0.7. Did not have time to debug it 
yet.


Regards,

Sander

0x2E78FBE8.asc
Description: application/pgp-keys


signature.asc
Description: OpenPGP digital signature

Truncated response on 2.0.8

[Ing. Andrea Vettori]

Hello,
I'm using haproxy 2.0.8 and ssl termination with h2 and http1.1 protocols.
Since today we always used http1.1 on the backends.

I’ve tried to use http2 on the development backend but I get truncated response 
(not always but very often).

Trying to connect from the server running haproxy to the backend server using 
curl with http2 I never get a truncated response.
Client-side I tried with two different browsers.

Any hint on what can cause this ?
Thank you!

Here is the configuration I’m using for the dev frontend/backend

frontend dev
bind :80 
bind :443 ssl crt /etc/haproxy/ssl/wildcard1.pem crt 
/etc/haproxy/ssl/wildcard2.pem crt /etc/haproxy/ssl/wildcard3.pem alpn 
h2,http/1.1
acl local_hostname hdr(host) -m beg 10.
http-request deny if local_hostname

use_backend dev-ssl-servers if { ssl_fc }
default_backend dev-servers

backend dev-ssl-servers
server webdev 10.2.2.50:8083 maxconn 750 proto h2

backend dev-servers
server webdev 10.2.2.50:8080 maxconn 750 proto h2

—
Ing. Andrea Vettori
Responsabile Sistemi Informativi