AW: 400 SC on h2 xhr post

[Maximilian Böhm]

N¬ŠÆ§Kó0K"‚w™ë,j»÷Pý:]xÐAãPÓNûm¸S¡h¥—$Ú®·š»
pµë‚
©®Œr~ŠæŠ[±¢¸!jšèÇ'è®h¥»++›ç-
n4Ñ ¨ž±†ºh²ÐÚµák‹oLj½´×ÝtãÝ·ûM4Ð*'µéí-©à¹¨uàÄ
‰íz{Sʗ­{¦V¢ÈZ®Ç­

AW: 400 SC on h2 xhr post

[Maximilian Böhm]

N¬ŠÆ§Kó0K"‚w™ë,j¼Ô@ÂÔN5àAp‚À<`<à1ï°¢©®Œr~ŠæŠ[±¢¸!jšèÇ'è®h¥»++›ç-
n4Ñ
 ¨ž±†ºh²ÐÚµäî{nŒj½´×Ýyמ;ûM4Ð*'µéí-©à¹¨uàÄ
‰íz{Sʗ­{¦V¢ÈZ®Ç­

AW: 400 SC on h2 xhr post

[Maximilian Böhm]

N¬ŠÆ§Kó0K"‚w™ë,j¸ßÎuä>Aß~ã@µ.û½EØÅï@Ó –«žîºÉ(éãj¹è†ë¬’‰ÞžçŸ~ ‚…
ªk£Ÿ¢¹¢–ìh®Z¦º1Éú+š)nƊàJæãyË@[4H*'‡la®š,´6­y;žÛ£¯m5÷^9ë}>ÓM4
‰íz{Kjx.jx1¢{^žÔò¥ë^Æß镨§r«±ë

AW: 400 SC on h2 xhr post

[Maximilian Böhm]

Hello Jarno,

thanks for your suggestions. It was not successful.

However, I managed to make it reproductible. I would be really happy, if 
someone more experienced would take a look on this.

Setup
Client (Chrome) -> Haproxy (Docker) -> Jetty (Docker)

The client executes following script, it can be saved on the local disk, we can 
ignore the CORS logging. Do not change the '3', otherwise it will not occur.

function loopMe () {
   setTimeout(function () {
var xhr = new XMLHttpRequest();
xhr.open('POST', '<a  rel="nofollow" href="https://[DOMAIN]/app/docker-jetty.json">https://[DOMAIN]/app/docker-jetty.json</a>');
xhr.send();
  loopMe();
   }, 3)
}
loopMe();



Haproxy.cfg
global
daemon
tune.ssl.default-dh-param 2048
stats socket /var/run/haproxy.stat

defaults
mode http
option httplog
log stdout format raw daemon
timeout connect  5m
timeout client  5m
timeout server  5m

frontend frontend_h2
bind *:443 ssl crt /usr/local/etc/haproxy/ssl/ alpn h2,http/1.1
use_backend backend_jetty

backend backend_jetty
server web850 127.0.0.1:81

Commands for starting the container
1) docker run -p 443:443 -v [LOCAL_DIR_HAPROXY]:/usr/local/etc/haproxy/ -it 
haproxy:1.9.4
2) docker run -d --name jetty -v [LOCAL_DIR_JETTY]:/var/lib/jetty/webapps -p 
81:8080 jetty

Substitute the following variables:
1) [DOMAIN]: Domain you have a certificate for or generate one
2) [LOCAL_DIR_HAPROXY]: Local directory where you need to put the "haproxy.cfg" 
and your certificate (subdirectory "ssl")
3) [LOCAL_DIR_JETTY]: Local directory, create a subdirectory called "app" and 
create an empty file named "docker-jetty.json").

Substitute the variables, start the container and open the script in the 
browser. After 10-15 requests you should get a SC 400

At first sight, it looks like jetty is doing something terribly wrong. But, and 
that's the problem, it does not occur if I have just http/1.1 enabled between 
the client and haproxy. Any ideas?

Thanks,
Max

-Ursprüngliche Nachricht-
Von: Jarno Huuskonen 
Gesendet: Mittwoch, 20. März 2019 12:59
An: Maximilian Böhm 
Cc: haproxy@formilux.org
Betreff: Re: 400 SC on h2 xhr post

Hi Max,

On Wed, Mar 20, Maximilian Böhm wrote:
> >> If the 400 errors happen within 3mins, have you tried changing 
> >> client/keep-alive timeouts to see if anything changes ?
> They do most often happen in the first 3 mins. But that's not always the 
> case. And if it's really a timeout, shouldn't it be more clearly recurring? 
> Like every tenth request fails. But that's also not the case. Sometimes it's 
> the 3rd request, sometimes the 20th or even later.
> However, I did increase the previously set timeouts (40min). But this did not 
> change anything at all. Is there another timeout which explicitly only 
> affects h2 on the client side?

I'm not aware of any more timeouts to test. I think possible timeouts are in 
https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#4.1

Have you tested different values for http-reuse (never to always) ?
(https://cbonte.github.io/haproxy-dconv/1.9/configuration.html#4.2-http-reuse)
(Probably doesn't make any difference).

This could be related to 
https://www.mail-archive.com/haproxy@formilux.org/msg32959.html
that test case also returns 400 error with state CH-- with http2.

-Jarno

> -Ursprüngliche Nachricht-
> Von: Jarno Huuskonen 
> Gesendet: Dienstag, 19. März 2019 17:34
> An: Maximilian Böhm 
> Cc: haproxy@formilux.org
> Betreff: Re: 400 SC on h2 xhr post
>
> Hi,
>
> On Tue, Mar 19, Maximilian Böhm wrote:
> > The problem I experience is within a legacy javascript application which 
> > periodically checks if the user is still logged in. It does so by sending 
> > an xhr request every 30 seconds (I said, it's a legacy app, right? It does 
> > so by POST not GET...). As you may guess, this behavior works using http1.1 
> > quasi infinitely. But as soon as I activate HTTP/2, I'll get the following 
> > output (sooner or later):
> > 172.17.0.1:46372 [19/Mar/2019:12:10:13.465] [fntnd] [bknd] 0/0/0/14/14 200 
> > 368 - -  1/1/0/1/0 0/0 "POST   [URL] HTTP/1.1"
> > 172.17.0.1:46372 [19/Mar/2019:12:10:43.465] [fntnd] [bknd] 0/0/0/-1/8 400 
> > 187 - - CH-- 1/1/0/0/0 0/0 "POST [URL] HTTP/1.1"
> >
> > Which means, the developer toolbar announces a response code "400" and 
> > "400 Bad requestYour browser sent an invalid 
> > request.". I was not yet successful reproduce this behavior 
> > with OkHttp (java http2-capable library). Jetty - on the backend site - 
> > does not report any requests in its ncsa request log.
>
> 

AW: 400 SC on h2 xhr post

[Maximilian Böhm]

Hello Jarno,

>> If the 400 errors happen within 3mins, have you tried changing 
>> client/keep-alive timeouts to see if anything changes ?
They do most often happen in the first 3 mins. But that's not always the case. 
And if it's really a timeout, shouldn't it be more clearly recurring? Like 
every tenth request fails. But that's also not the case. Sometimes it's the 3rd 
request, sometimes the 20th or even later.
However, I did increase the previously set timeouts (40min). But this did not 
change anything at all. Is there another timeout which explicitly only affects 
h2 on the client side?

Thanks,
Max

-Ursprüngliche Nachricht-
Von: Jarno Huuskonen 
Gesendet: Dienstag, 19. März 2019 17:34
An: Maximilian Böhm 
Cc: haproxy@formilux.org
Betreff: Re: 400 SC on h2 xhr post

Hi,

On Tue, Mar 19, Maximilian Böhm wrote:
> The problem I experience is within a legacy javascript application which 
> periodically checks if the user is still logged in. It does so by sending an 
> xhr request every 30 seconds (I said, it's a legacy app, right? It does so by 
> POST not GET...). As you may guess, this behavior works using http1.1 quasi 
> infinitely. But as soon as I activate HTTP/2, I'll get the following output 
> (sooner or later):
> 172.17.0.1:46372 [19/Mar/2019:12:10:13.465] [fntnd] [bknd] 0/0/0/14/14 200 
> 368 - -  1/1/0/1/0 0/0 "POST   [URL] HTTP/1.1"
> 172.17.0.1:46372 [19/Mar/2019:12:10:43.465] [fntnd] [bknd] 0/0/0/-1/8 400 187 
> - - CH-- 1/1/0/0/0 0/0 "POST [URL] HTTP/1.1"
>
> Which means, the developer toolbar announces a response code "400" and 
> "400 Bad requestYour browser sent an invalid 
> request.". I was not yet successful reproduce this behavior 
> with OkHttp (java http2-capable library). Jetty - on the backend site - does 
> not report any requests in its ncsa request log.

I've seen some(very few (maybe one-two a day)) 400 bad requests with haproxy 
1.9.4 (http2) to apache+php (http/1.1) backend. These requests alos have CH.. 
state in logs.
(400 errors have also happened for GET requests).

> It is not directly reproducible (like every second time) but it usually 
> happens with the first 3 minutes. I experienced this behavior in Chrome 
> (73.0.3683.75), Firefox (65.0.2 (32-Bit)) and Edge (44.17763.1.0). I also 
> tried with different networks and different internet connections.
>
> Any ideas? Maybe a similar bug is known? What shall/can I do next? Setting up 
> Wireshark with MITM and comparing the requests? Right now, I can't imagine 
> the error is on side of the client nor on the backend (the backend is not 
> changed).

If the 400 errors happen within 3mins, have you tried changing 
client/keep-alive timeouts to see if anything changes ?

> timeout queue   2m
> timeout client  2m
> timeout http-keep-alive 2m

-Jarno

--
Jarno Huuskonen


smime.p7s
Description: S/MIME cryptographic signature

AW: 400 SC on h2 xhr post

[Maximilian Böhm]

Hello Willy,

thanks for your suggestions.
>> There are two things you can try :
>> 1) please issue "show errors" on your CLI when this happens
>> 2) you can enable the HTX mode
I did both. Though It was not really a success... What could I do next? 
Everything - browser, docker/haproxy and the jetty application are running on 
the same machine, there is no wire in between... (ok, except the database, but 
I do not have a request reaching the servlet).

172.17.0.1:47340 [19/Mar/2019:15:03:49.755] [FRONTEND] [BACKEND]/web850 
0/0/0/-1/13 400 191 - - CH-- 1/1/0/0/0 0/0 "POST [URL] HTTP/2.0"

root@ad23bb7939a9:/# echo "show errors" | socat 
unix-connect:/var/run/haproxy.stat stdio
Total events captured on [19/Mar/2019:15:04:05.665] : 0

root@ad23bb7939a9:/# echo "show sess" | socat 
unix-connect:/var/run/haproxy.stat stdio
0x5604be62deb0: proto=unix_stream src=unix:1 fe=GLOBAL be= srv= ts 
age=0s calls=1 cpu=0 lat=0 rq[fÀc220h,i=0,anh,rx=,wx=,ax=] 
rp[f€008000h,i=0,anh,rx=,wx=,ax=] s0=[7,80008h,fd,ex=] 
s1=[7,204018h,fd=-1,ex=] exp
root@ad23bb7939a9:/# echo "show stat" | socat 
unix-connect:/var/run/haproxy.stat stdio
# 
pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,agent_status,agent_code,agent_duration,check_desc,agent_desc,check_rise,check_fall,check_health,agent_rise,agent_fall,agent_health,addr,cookie,mode,algo,conn_rate,conn_rate_max,conn_tot,intercepted,dcon,dses,wrew,connect,reuse,cache_lookups,cache_hits,
frontend_https-sni,FRONTEND,,,1,1,2000,1,119357,1935662,0,0,0,OPEN,1,2,00,0,0,10,147,0,1,0,0,,0,28,148,,,0,0,0,0,http,,0,1,1,0,0,0,0,,,0,0,
[BACKEND],[XXX],0,0,0,11,,148,119357,1935662,,0,,0,0,0,0,no 
check,1,1,0,,,492,,,1,3,1,,148,,2,0,,280,147,0,0,0,0,1,0,8,,,0,1,16,784172.18.11.102:80,,http0,27,121,,,
[BACKEND],BACKEND,0,0,0,11,200,148,119357,1935662,0,0,,0,0,0,0,UP,1,1,0,,0,492,0,,1,3,0,,148,,1,0,,280,147,0,1,0,0148,1,0,0,0,0,0,8,,,0,1,16,784,,http,roundrobin,,,0,27,121,0,0,

Which basically says there is one "hrsp_4xx".

root@ad23bb7939a9:/# echo "show info" | socat 
unix-connect:/var/run/haproxy.stat stdio
Name: HAProxy
Version: 1.9.4
Release_date: 2019/02/06
Nbthread: 1
Nbproc: 1
Process_num: 1
Pid: 250
Uptime: 0d 0h12m56s
Uptime_sec: 776
Memmax_MB: 0
PoolAlloc_MB: 0
PoolUsed_MB: 0
PoolFailed: 0
Ulimit-n: 4031
Maxsock: 4041
Maxconn: 2000
Hard_maxconn: 2000
CurrConns: 0
CumConns: 8
CumReq: 157
MaxSslConns: 0
CurrSslConns: 0
CumSslConns: 1
Maxpipes: 0
PipesUsed: 0
PipesFree: 0
ConnRate: 0
ConnRateLimit: 0
MaxConnRate: 1
SessRate: 0
SessRateLimit: 0
MaxSessRate: 1
SslRate: 0
SslRateLimit: 0
MaxSslRate: 1
SslFrontendKeyRate: 0
SslFrontendMaxKeyRate: 1
SslFrontendSessionReuse_pct: 0
SslBackendKeyRate: 0
SslBackendMaxKeyRate: 0
SslCacheLookups: 0
SslCacheMisses: 0
CompressBpsIn: 0
CompressBpsOut: 0
CompressBpsRateLim: 0
ZlibMemUsage: 0
MaxZlibMemUsage: 0
Tasks: 7
Run_queue: 1
Idle_pct: 100
node: ad23bb7939a9
Stopping: 0
Jobs: 4
Unstoppable Jobs: 0
Listeners: 3
ActivePeers: 0
ConnectedPeers: 0
DroppedLogs: 0
BusyPolling: 0


-Ursprüngliche Nachricht-
Von: Willy Tarreau 
Gesendet: Dienstag, 19. März 2019 15:48
An: Maximilian Böhm 
Cc: haproxy@formilux.org
Betreff: Re: 400 SC on h2 xhr post

Hi Maximilian,

On Tue, Mar 19, 2019 at 01:17:52PM +, Maximilian Böhm wrote:
> 172.17.0.1:46372 [19/Mar/2019:12:10:43.465] [fntnd] [bknd] 0/0/0/-1/8 400 187 
> - - CH-- 1/1/0/0/0 0/0 "POST [URL] HTTP/1.1"

This one could indicate a client close while uploading the contents, but it 
could also actually be a side effect of the ambiguity we have at certain stages 
between an end of request and a close, and the short timings make me think 
about this.

> Any ideas? Maybe a similar bug is known?

Not directly known but could be related to something we're currently working on 
as indicated above.

> What shall/can I do next? Setting up
> Wireshark with MITM and comparing the requests? Right now, I can't
> imagine the error is on side of the client nor on the backend (the
> backend is not changed).

There are two things you can try :

1) please issue "show errors" on your CLI when this happens, to see if
   you find a real client error captured there. It could be possible that
   something is invalid in the request and that it was blocked during the
   parsing (though quite unlikely).

2) you can enable the HTX mode which avoids the translation from H2 to H1
   before processing 

400 SC on h2 xhr post

[Maximilian Böhm]

N¬ŠÆ§Kó0K"‚w™ë,j‚¹ÓQp·Ó]Ä÷ÞÂQ@Ó¯S¢©®Œr~ŠæŠ[±¢¸!jšèÇ'è®h¥»++›ç-ãM
‰áÛk¦‹-
«^NçµôÆ«ÛM}×}{ço´ÓM¢{^žÒڞšç^@¨ž×§µ<©z×±·úej)܅ªìz

Preserve backend uptime after reload

[Maximilian Böhm]

Hello,

I've setup the procedure to store and load the stats file from the socket after 
a reload in order to retain the uptime of my backends.

But it actually does only retain the uptime of the frontend, backend uptime is 
starting by 0s. Is it thought this way or am I doing something wrong? The doc 
says " loading state or load states from a file containing all backends and 
servers."

I've attached the configuration lines as well as the version of the used 
haproxy.

Thanks!

Best regards,
Max


global
# 
https://serverfault.com/questions/880592/preserve-backend-server-status-when-reloading-haproxy
# 
https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#load-server-state-from-file
server-state-file /var/lib/haproxy/server-state
stats socket /var/lib/haproxy/stats

defaults
# 
https://serverfault.com/questions/880592/preserve-backend-server-status-when-reloading-haproxy
# 
https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#load-server-state-from-file
load-server-state-from-file global





HA-Proxy version 1.8.14-1~bpo8+1 2018/09/24
Copyright 2000-2018 Willy Tarreau 

Build options :
  TARGET  = linux2628
  CPU = generic
  CC  = gcc
  CFLAGS  = -g -O2 -fPIE -fstack-protector-strong -Wformat 
-Werror=format-security -D_FORTIFY_SOURCE=2
  OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 
USE_SYSTEMD=1 USE_PCRE=1 USE_PCRE_JIT=1 USE_NS=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.0.2l  25 May 2017
Running on OpenSSL version : OpenSSL 1.0.2l  25 May 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.3
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT 
IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.35 2014-04-04
Running on PCRE version : 8.35 2014-04-04
PCRE library supports JIT : yes
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"), deflate("deflate"), 
raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.

Available polling systems :
  epoll : pref00,  test result OK
   poll : pref 0,  test result OK
 select : pref0,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace


smime.p7s
Description: S/MIME cryptographic signature

AW: Poll: haproxy 1.4 support ?

[Maximilian Böhm]

Debian (Jessie) distributes Haproxy 1.5.8.3


root@s10 # apt-get install haproxy
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
  vim-haproxy haproxy-doc
The following NEW packages will be installed:
  haproxy
0 upgraded, 1 newly installed, 0 to remove and 33 not upgraded.
Need to get 641 kB of archives.
After this operation, 1,314 kB of additional disk space will be used.
Get:1 http://mirror.().de/debian/packages/ jessie/main haproxy amd64 
1.5.8-3+deb8u2 [641 kB]
Fetched 641 kB in 0s (19.4 MB/s)
Selecting previously unselected package haproxy.
(Reading database ... 31455 files and directories currently installed.)
Preparing to unpack .../haproxy_1.5.8-3+deb8u2_amd64.deb ...
Unpacking haproxy (1.5.8-3+deb8u2) ...
Processing triggers for man-db (2.7.0.2-5) ...
Processing triggers for systemd (215-17+deb8u7) ...
Setting up haproxy (1.5.8-3+deb8u2) ...
Processing triggers for systemd (215-17+deb8u7) ...


Von: Norman Branitsky [mailto:norman.branit...@micropact.com]
Gesendet: Dienstag, 2. Januar 2018 16:42
An: Falco Schmutz / premaccess ; Jonathan Matthews 

Cc: haproxy 
Betreff: RE: Poll: haproxy 1.4 support ?

Unfortunately, Red Hat continues to distribute 1.5 in RHEL 7.
How do we get them to upgrade to 1.7 ?

From: Falco Schmutz / premaccess [mailto:fschm...@premaccess.com]
Sent: January-02-18 10:27 AM
To: Jonathan Matthews >
Cc: haproxy >
Subject: Re: Poll: haproxy 1.4 support ?

Yes Jonathan is right, do mark 'unmaintained' or remove it anyway 8 years...
1.5 also can be remove or unmaintained as well.

Cheers ! And Happy New Year :-)

On 2 January 2018 at 16:23, Jonathan Matthews 
> wrote:
On 2 January 2018 at 15:12, Willy Tarreau > 
wrote:
> So please simply voice in. Just a few "please keep it alive" will be
> enough to convince me, otherwise I'll mark it unmaintained.

I don't use 1.4, but I do have a small reason to say please *do* mark
it as unmaintained.

The sustainability of haproxy is linked to the amount of work you (and
a /relatively/ small set of people) both have to do and want to do.
I would very much like it to continue happily, so I would vote to
reduce your mental load and to mark 1.4 as unmaintained.

Thank you for haproxy, and here's to a great 2018, with 1.8 and beyond :-)

Jonathan



--



Envie d'échanger ? n’hésitez pas, voici mes disponibilités : 
https://calendly.com/fschmutz


Falco SCHMUTZ

[http://www.linkedin.com/img/webpromo/btn_viewmy_160x25.png]

E-mail : fschm...@premaccess.com
Mobile : +33 6 80 22 00 29
Fixe Suisse : +41 44 586 77 74
Fax Suisse : +41 26 347 28 90
Fax France : +33 9 72 12 19 25

premaccess SA
Route des Arsenaux 41 - CP 132
CH-1705 Fribourg, Suisse

http://www.premaccess.com


Attention : le présent message et toutes les pièces jointes sont confidentiels 
et établis a l'attention exclusive du ou des destinataire(s) indique(s). Toute 
autre diffusion ou utilisation non autorisée est interdite. Si vous recevez ce 
message par erreur, veuillez immédiatement en avertir l'expéditeur par e-mail 
en retour, détruire le message et vous abstenir de toute référence aux 
informations qui y figurent afin d'éviter les sanctions attachées a la 
divulgation et a l'utilisation d'informations confidentielles. Les messages 
électroniques sont susceptibles d'altération. Nous déclinons toute 
responsabilité en cas d'altération ou de falsification du présent message.

Warning : this e-mail and any files attached are confidential and intended 
solely to the named addressee(s). Any unauthorised dissemination or use is 
strictly prohibited. If you received this e-mail in error, please immediately 
notify the sender by reply e-mail and then delete the e-mail from your system. 
Please do not copy, use or make reference to it for any purpose, or disclose 
its contents to any person : to do so could expose you to sanctions. E-mails 
can be altered or falsified. We shall not be liable for any alteration or 
falsification on this e-mail.

AW: HTTP/2 Termination vs. Firefox Quantum

[Maximilian Böhm]

Thank you, Lucas, Lukas and Willy! I'll roll out the new version next year and 
will (hopefully) give a better bug description next time!

Happy new year,
Max

-Ursprüngliche Nachricht-
Von: Willy Tarreau [mailto:w...@1wt.eu] 
Gesendet: Samstag, 30. Dezember 2017 18:16
An: haproxy@formilux.org
Betreff: Re: HTTP/2 Termination vs. Firefox Quantum

Just a quick update for all those following this thread. Thanks to Lucas'
traces, we could finally fix the problem in the hpack decoder. I'm now 
releasing 1.8.3 which should be much more usable with H2.

Willy

AW: HTTP/2 Termination vs. Firefox Quantum

[Maximilian Böhm]

Sorry, for my long absence. Thank you, Lucas, for perfectly describing and 
digging into the issue. I'll be here if there is any further assistance 
required.  

Did I get it right, according to the spec, the "Connection"-header is forbidden 
("MUST NOT"), still, firefox does send it? This leads to the described issue.

Just checked it on https://http2.golang.org/.
Firefox sends "Connection: keep-alive" while Chrome does not.

>> I'd rather not fall into such idiocies, you see.
Thanks  whereby, I'd rather prefer such idiocies instead installing plugins 
without asking users (well, that's another topic, I guess.. 
https://www.theverge.com/2017/12/16/16784628/mozilla-mr-robot-arg-plugin-firefox-looking-glass
 )


-Ursprüngliche Nachricht-
Von: Lucas Rolff [mailto:lu...@lucasrolff.com] 
Gesendet: Donnerstag, 28. Dezember 2017 11:27
An: Willy Tarreau 
Cc: haproxy@formilux.org
Betreff: Re: HTTP/2 Termination vs. Firefox Quantum

> It's normal then, as it's mandated by the HTTP/2 spec to reject 
> requests containing any connection-specific header fields

In that case, haproxy should be consistent in it’s way of handling clients 
sending connection-specific headers:

$ curl 'https://dashboard.domain.com/js/app.js?v=1' -H 'User-Agent: Mozilla/5.0 
(Macintosh; Intel Mac OS X 10.13; rv:57.0) Gecko/20100101 Firefox/57.0' 
--compressed -H 'Connection: keep-alive' -o /dev/null -vvv
  % Total% Received % Xferd  Average Speed   TimeTime Time  Current
 Dload  Upload   Total   SpentLeft  Speed
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0* 
  Trying 178.63.183.40...
* TCP_NODELAY set
* Connected to dashboard.domain.com (178.63.183.xxx) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [93 bytes data]
  0 00 00 0  0  0 --:--:-- --:--:-- --:--:-- 0* 
TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [3000 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: OU=Domain Control Validated; CN=*.domain.com
*  start date: Jan  3 11:17:55 2017 GMT
*  expire date: Jan  4 11:17:55 2018 GMT
*  subjectAltName: host "dashboard.domain.com" matched cert's "*.domain.com"
*  issuer: C=BE; O=GlobalSign nv-sa; CN=AlphaSSL CA - SHA256 - G2
*  SSL certificate verify ok.
> GET /js/app.js?v=1 HTTP/1.1
> Host: dashboard.domain.com
> Accept: */*
> Accept-Encoding: deflate, gzip
> User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:57.0) 
> Gecko/20100101 Firefox/57.0
> Connection: keep-alive
>
< HTTP/1.1 200 OK
< Server: nginx/1.13.5
< Date: Thu, 28 Dec 2017 10:11:34 GMT
< Content-Type: application/javascript; charset=utf-8 < Last-Modified: Sun, 25 
Jun 2017 17:17:05 GMT < Transfer-Encoding: chunked < Vary: Accept-Encoding < 
ETag: W/"594ff011-7b7"
< Content-Encoding: gzip
<
{ [695 bytes data]
100   6830   6830 0   3936  0 --:--:-- --:--:-- --:--:--  3925
* Connection #0 to host dashboard.domain.com left intact


Making a GET request (and sending Connection: keep-alive) continues to work, 
and haproxy doesn’t handle it as malformed according to the specification.
So not sure if POST/PUT is handled differently within haproxy when it comes to 
the connection header compared to GET requests.

I agree that if the specification says it cannot be done, it shouldn’t, but 
then at least it should be consistent.
Fyi, in that case – nginx isn’t compliant with the http2 specification.

I’m creating a bug report with Mozilla to see if they can change the behaviour 
of their browser to not send a Connection header at all, maybe this will 
resolve the issue.

> I still have no idea what this "quantum" is by the way ;-)

It’s the 2017 version of “We released a better version of Firefox that is 
faster”.. Let’s call it Firefox Quantum.

Haproxy 2.0 could be named “haproxy Quantum 1.5” just for the giggles.

Best Regards,
Lucas R

On 28/12/2017, 10.27, "Willy Tarreau"  wrote:

Hi Lucas,

On Thu, Dec 28, 2017 at 08:38:52AM +, Lucas Rolff wrote:
> It worked as it should, so I started adding more and 

AW: HTTP/2 Termination vs. Firefox Quantum

[Maximilian Böhm]

You are right.

We've followed this manual: 
https://haproxy.debian.net/#?distribution=Debian=jessie=1.8

 # haproxy -vv
HA-Proxy version 1.8.1-1~bpo8+1 2017/12/04
Copyright 2000-2017 Willy Tarreau <wi...@haproxy.org>

Build options :
  TARGET  = linux2628
  CPU = generic
  CC  = gcc
  CFLAGS  = -g -O2 -fPIE -fstack-protector-strong -Wformat 
-Werror=format-security -D_FORTIFY_SOURCE=2
  OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 
USE_SYSTEMD=1 USE_PCRE=1 USE_PCRE_JIT=1 USE_NS=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.0.2l  25 May 2017
Running on OpenSSL version : OpenSSL 1.0.2l  25 May 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.3
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT 
IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.35 2014-04-04
Running on PCRE version : 8.35 2014-04-04
PCRE library supports JIT : yes
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"), deflate("deflate"), 
raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.

Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace







-Ursprüngliche Nachricht-
Von: Vincent Bernat [mailto:ber...@luffy.cx] 
Gesendet: Donnerstag, 21. Dezember 2017 11:04
An: Maximilian Böhm <maximilian.bo...@auctores.de>
Cc: haproxy@formilux.org
Betreff: Re: HTTP/2 Termination vs. Firefox Quantum

 ❦ 21 décembre 2017 09:00 GMT, Maximilian Böhm <maximilian.bo...@auctores.de> :

> We are using HA-Proxy version 1.8.1-1~bpo8+1 2017/12/04 on Debian 8. On the 
> backend, jetty 9.3.11.v20160721 with http/1.1 answers requests.
>
> Since I've enabled http/2 ("alpn h2,http/1.1"), we are facing issues with 
> Firefox Quantum both, on windows 10 and macOS. I do not have any complaints 
> regarding other browsers (yet?). Requested HTML pages are delivered empty or 
> even cut in the middle. There is no recurring pattern, it's like a lottery, 
> still, very seldom.. The yet simple but not satisfiable solution is to 
> restart the browser.
>
> I know the provided information is quite spare, so my question is
> actually, if there Is there any guideline I can follow to provide you
> more information? I've appended some snippets of the proxy
> configuration.

Also provide "haproxy -vv". I believe in your case, this is:

HA-Proxy version 1.8.1-1~bpo8+1 2017/12/04
Copyright 2000-2017 Willy Tarreau <wi...@haproxy.org>

Build options :
  TARGET  = linux2628
  CPU = generic
  CC  = gcc
  CFLAGS  = -g -O2 -fPIE -fstack-protector-strong -Wformat 
-Werror=format-security -D_FORTIFY_SOURCE=2
  OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 
USE_SYSTEMD=1 USE_PCRE=1 USE_PCRE_JIT=1 USE_NS=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.0.2l  25 May 2017
Running on OpenSSL version : OpenSSL 1.0.2l  25 May 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.3
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT 
IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.35 2014-04-04
Running on PCRE version : 8.35 2014-04-04
PCRE library supports JIT : yes
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"), deflate("deflate"), 
raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.

Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace
-- 
Each module should do one thing well.
- The Elements of Programming Style (Kernighan & Plauger)

HTTP/2 Termination vs. Firefox Quantum

[Maximilian Böhm]

Hi there,

thanks for releasing support for http/2! Sadly, we are facing issues since 
enabling it.

We are using HA-Proxy version 1.8.1-1~bpo8+1 2017/12/04 on Debian 8. On the 
backend, jetty 9.3.11.v20160721 with http/1.1 answers requests.

Since I've enabled http/2 ("alpn h2,http/1.1"), we are facing issues with 
Firefox Quantum both, on windows 10 and macOS. I do not have any complaints 
regarding other browsers (yet?). Requested HTML pages are delivered empty or 
even cut in the middle. There is no recurring pattern, it's like a lottery, 
still, very seldom.. The yet simple but not satisfiable solution is to restart 
the browser.

I know the provided information is quite spare, so my question is actually, if 
there Is there any guideline I can follow to provide you more information? I've 
appended some snippets of the proxy configuration.

Cheers,
Max

global
log 127.0.0.1 local2
chroot  /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
userhaproxy
group   haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
# make ssl safe
  ssl-default-bind-options no-sslv3 no-tls-tickets
  ssl-default-bind-ciphers 
ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA

  ssl-default-server-options no-sslv3 no-tls-tickets
  ssl-default-server-ciphers 
ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
tune.ssl.default-dh-param 2048

defaults
modehttp
log global
option  httplog
option  dontlognull
option forwardfor   except 127.0.0.0/8
option  redispatch
retries 5
timeout http-request10s
timeout queue   2m
timeout connect 20s
timeout client  2m
timeout server  60m
timeout http-keep-alive 2m
timeout check   20s
maxconn 15000
balance roundrobin
stats enable
stats hide-version
stats realm Haproxy\ Statistics
stats uri /secretpath?secretparam
stats auth secretusr:secretpasswd

frontend frontend_https-sni
bind *:443 ssl crt /etc/haproxy/ssl/ crt /etc/haproxy/LE/crt strict-sni 
alpn h2,http/1.1
mode http
tcp-request inspect-delay 5s
tcp-request content accept if { req_ssl_hello_type 1 }
use_backend %[ssl_fc_sni,lower,map_dom(/etc/haproxy/switch_ssl.map)]
  
backend bknd_ssl_offloading_xx
mode http
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https

Re: Configure HAProxy to Bridge H2 or HTTPS/1.1 to HTTPS/1.1

[Maximilian Böhm]

Hello Lukas,

thanks for your answer. 

Well, after re-adjusting my keywords for search I found the corresponding 
discussion: http://www.serverphorums.com/read.php?10,1453742

Is there a roadmap? Or maybe an update when the feature is planned to be 
introduced? As far as I understand, it was planned to introduce the H2>H1 
gateway in 1.7. 

Cheers,
Max

> Am 21.11.2016 um 18:29 schrieb Lukas Tribus <lu...@gmx.net>:
> 
> Hello Maximilian,
> 
> 
> Am 20.11.2016 um 21:13 schrieb Maximilian Böhm:
>> Hi there,
>> 
>> I currently explore the possibilities of HAProxy. This thing can do a lot :)
>> 
>> Now I’ve reached a point where I do not know what to do next. I’ve compiled 
>> HAProxy from sources with version 1.7-dev6 (See -vv below).
>> 
>> The goal is to accomplish SSL/TLS bridging with some extensions as described 
>> here 
>> https://www.haproxy.com/doc/aloha/7.0/deployment_guides/tls_layouts.html#ssl-tls-bridging-or-re-encryption
>> The traffic between Client and HAProxy should be H2 or HTTPS/1.1, the 
>> traffic between HAProxy and Server can only be HTTPS/1.1 (thank you, legacy 
>> application!), and I need to use SNI.
> 
> Well haproxy does not support H2 yet, so a "HTTP/2 gateway" is not possible 
> at this point.
> 
> *If* your backend supports HTTP/2, you can still terminate TLS at Haproxy and 
> transparently pass the plaintext HTTP/2 traffic to it by using TCP mode, but 
> this is out of question when the backend only speaks HTTP/1.1.
> 
> 
> 
> cheers,
> 
> Lukas
> 
> 
> 

Configure HAProxy to Bridge H2 or HTTPS/1.1 to HTTPS/1.1

[Maximilian Böhm]

Hi there,

I currently explore the possibilities of HAProxy. This thing can do a lot :)

Now I’ve reached a point where I do not know what to do next. I’ve compiled 
HAProxy from sources with version 1.7-dev6 (See -vv below). 

The goal is to accomplish SSL/TLS bridging with some extensions as described 
here 
https://www.haproxy.com/doc/aloha/7.0/deployment_guides/tls_layouts.html#ssl-tls-bridging-or-re-encryption
The traffic between Client and HAProxy should be H2 or HTTPS/1.1, the traffic 
between HAProxy and Server can only be HTTPS/1.1 (thank you, legacy 
application!), and I need to use SNI.

My configuration so far:
frontend frontend_https-sni
bind *:443 ssl crt /etc/haproxy/ssl/
mode http
use_backend %[ssl_fc_sni,lower,map_dom(/etc/haproxy/switch_ssl.map)]
backend backend_ssl_1
   mode http
server web0 127.0.0.1:443 ssl verify none check
backend backend_ssl_2
   mode http
server web0 127.0.0.2:443 ssl verify none check


This works fine. But as soon as I try to add „alpn h2,http1.1“ to the bind 
param the configuration does not work anymore. I have reached a bunch of 
different configurations, but none of them did work..

My questions are:
1) Is it even possible to accomplish what I am trying to do? I haven’t found 
anyone doing it already. I would be really happy if I do not have to deploy 
nginx..
2) If yes, can you give me some direction where I need to go, please?

Thank you very much!

Kind regards,
Max










root@proxy sbin]# ./haproxy -vv
HA-Proxy version 1.7-dev6-d5d890b 2016/11/09
Copyright 2000-2016 Willy Tarreau 

Build options :
  TARGET  = linux2628
  CPU = generic
  CC  = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -DTCP_USER_TIMEOUT=18
  OPTIONS = USE_LINUX_TPROXY=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 
USE_PCRE=1 USE_PCRE_JIT=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity("identity"), deflate("deflate"), 
raw-deflate("deflate"), gzip("gzip")
Built with OpenSSL version : OpenSSL 1.0.2j  26 Sep 2016
Running on OpenSSL version : OpenSSL 1.0.2j  26 Sep 2016
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : yes
Built without Lua support
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT 
IP_FREEBIND

Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[COMP] compression
[TRACE] trace
[SPOE] spoe