Re: Relayd with TLS and non-TLS backends - bug
This specific Backend in my test lab is an IIS machine, but in production I have OpenBSD/HAProxy in front of IIS, Apache, Tomcat, etc. I'm not doing anything fancy either... although the certificate in the lab is signed by an internal CA. Here's the relevant output from openssl s_client: The cert verifies perfectly fine. openssl s_client -connect 192.168.42.61:443 --- New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher: ECDHE-RSA-AES256-SHA384 Session-ID: C0138246D3973B3106A378C0DB503D4BCDE02C6461AB073949027C90CDCF Session-ID-ctx: Master-Key: AACBE3A3E34406F9371B4E85D4DC82C177C641C94806562053C000FE0E019D2E1456702F69DECFB6D11C4B4A12A0D555 Start Time: 1593970747 Timeout : 7200 (sec) Verify return code: 0 (ok) --- and Netcat: nc -zv 192.168.42.61 443 Connection to 192.168.42.61 443 port [tcp/https] succeeded! On Fri, Jul 3, 2020 at 9:40 PM Daniel Jakots wrote: > > On Fri, 3 Jul 2020 19:14:17 -0400, Henry Bonath > wrote: > > > Daniel, > > > > Thanks for taking the time to test this out. > > I just reloaded a test machine from scratch with -current and > > installed the HAProxy 2.0.15-4f39279 package. > > I loaded a very basic config file, and am also seeing the same exact > > issue on this one as well. > > Very strange that you are not - > > Would you mind sharing any additional details of your config file? > > Is there anything special about the certificate you have on the > > backend server? > > > > I would love to understand what is going on here and what the > > difference is with my experience. > > What is your backend running? Can you connect from the haproxy host with > nc(1) and/or openssl(1)? > > I try to do my stuff as vanilla as possible so it's an RSA key signed > by LE.
Re: Relayd with TLS and non-TLS backends - bug
Henry Bonath writes: > I would like to chime in here and confirm that I am seeing very > similar behavior with HAProxy on OpenBSD 6.7, > I was preparing to create my own post on this issue until I saw your thread. > I too believe this is a bug. We saw the same thing after upgrading our proxy host from 6.5 -> 6.7. Since our use case was such a simple one, we tossed haproxy overboard and moved things over to relayd. --lyndon
Re: Relayd with TLS and non-TLS backends - bug
> On Jul 3, 2020, at 7:17 PM, Henry Bonath wrote: > > Daniel, > > Thanks for taking the time to test this out. > I just reloaded a test machine from scratch with -current and > installed the HAProxy 2.0.15-4f39279 package. > I loaded a very basic config file, and am also seeing the same exact > issue on this one as well. > Very strange that you are not - > Would you mind sharing any additional details of your config file? > Is there anything special about the certificate you have on the backend > server? > > I would love to understand what is going on here and what the > difference is with my experience. > >> On Thu, Jul 2, 2020 at 4:38 PM Daniel Jakots wrote: >> >> On Thu, 2 Jul 2020 14:00:48 -0400, Henry Bonath >> wrote: >> >>> Note the missing Client Hello on the 6.7 machine as it jumps to >>> Application Data straight away. >>> Configuration files for HAProxy are identical on both systems. >>> >>> I'm currently spinning up a machine on -CURRENT just to see if there >>> is any difference, >>> as there is a newer version of HAProxy in packages under Snapshots. >>> >>> I was initially going to try to reach out to the package maintainer >>> for HAProxy but if this is happening in Relayd, then this "feels >>> like" a de-facto bug. I wonder if NGINX would exhibit the same >>> behavior. >>> >>> Has anyone else experienced such behavior with Load-Balancing TLS >>> Backends since upgrading to 6.7? >> >> I don't use TLS for my backend (the only backend I use nowadays is on >> localhost) so I can't speak for 6.7 (I only use -current, and when >> -current was 6.7, I didn't test that). >> >> I just tested my -current haproxy using another -current host of mine >> running nginx as a backend with TLS and it worked fine. >> >> backend https >> option forwardfor >> server web1 ln.chown.me:443 check ssl verify none >> >> and also with "verify required ca-file /etc/ssl/cert.pem" >> >> >> Maybe some libressl fix happened on -current was not deemed critical >> enough to be backported to 6.7? >> >> Cheers, >> Daniel > This thread is conflating two issues: 1) Henry’s original relayd.conf is wrong. Notice the TLS connection attempt to port 80 in his relayd logs. This will never work. See my email regarding two relays required. 2) There was conversation about a compatibility issue with LibreSSL in 6.7 release. Check the archives.
Re: Relayd with TLS and non-TLS backends - bug
> On Jun 11, 2020, at 4:28 PM, Toyam Cox wrote:
>
> Hello Misc,
>
> Full config at end of email.
>
> I've discussed the below in #openbsd on freenode, and was told to come
> here. At present, I have a setup where I need multiple unrelated
> servers under a single IP address. I used relayd to do https
> interception, read the Host header, and make decisions.
>
> The very relevant part of my config is this:
>
> forward to port 80
> forward with tls to port 443
>
> The order here does not matter (unlike most relayd configs, I know,
> but I've tested in my configuration and it works).
>
> When I have "with tls" on that second line, I see error lines like:
> relay web, session 3 (1 active), 0, [redacted] -> 10.0.0.102:80, TLS
> handshake error: handshake failed: error:14FFF3E7:SSL
> routines:(UNKNOWN)SSL_internal:unknown failure occurred, GET:
> Undefined error: 0
>
> and, unhelpfully, relayd responds with no response. There is no
> return. Or, as curl puts it: curl: (52) Empty reply from server
>
> When I remove "with tls" then I successfully reach the http backend,
> but since the https backend requires ssl, that connection no longer
> works. So it seems that 'with tls" affects all "forward" clauses, not
> just the one to which it's attached.
>
> I believe this to be a bug.
>
> cat >/etc/relayd.conf < table { "10.0.0.101" }
> table { "10.0.0.102" }
> # obviously obfuscated some values
>
> interval 5
> timeout 1000
>
> log connection
>
> http protocol web {
> return error
>
> match header set "X-Client-IP" value "$REMOTE_ADDR:$REMOTE_PORT"
> match header set "X-Forwarded-For" value "$REMOTE_ADDR"
> match header set "X-Forwarded-By" value "$SERVER_ADDR:$SERVER_PORT"
>
> http websockets
> pass request quick header "Host" value "myhost.example.com" path
> "/Client/*" forward to
> pass request quick header "Host" value "otherhost.example.com" forward
> to
>
> block
> }
>
> relay web {
> listen on 10.0.0.100 port 443 tls
> protocol web
>
> forward to port 80 check http "/webservice.asmx" code 405
> forward with tls to port 443 check https
> "/Client/SupportedBrowsers.html" host "myhost.example.com" code 200
> }
> EOF
>
Hi Toyam,
Split http and https into two separate relay stanzas.
The “with tls” will be needed on your https relay and not the http backhaul. I
believe this gets what you want.
I do not think this is a bug, but perhaps a design choice by the developers.
Cheers,
Brian
Re: Relayd with TLS and non-TLS backends - bug
> On Jul 3, 2020, at 9:46 PM, Daniel Jakots wrote: > > On Fri, 3 Jul 2020 20:25:12 -0400, Brian Brombacher > wrote: > >> My subjective net gain is simplicity, security, performance, and >> flexibility. > > I don't think adding ipsec (or a mesh vpn) into the mix achieve that but > ymmv. > Subjective is right :) He has two hosts. IPsec from one to the other. Pre-negotiated encrypted channel. MTU 1400 or so... Four round-trip TCP packets to get the request on the backend... if the HTTP request is smaller than say 1300 bytes, to be really safe. How is that slower? -Brian
Re: Relayd with TLS and non-TLS backends - bug
On Fri, 3 Jul 2020 20:25:12 -0400, Brian Brombacher wrote: > My subjective net gain is simplicity, security, performance, and > flexibility. I don't think adding ipsec (or a mesh vpn) into the mix achieve that but ymmv.
Re: Relayd with TLS and non-TLS backends - bug
On Fri, 3 Jul 2020 19:14:17 -0400, Henry Bonath wrote: > Daniel, > > Thanks for taking the time to test this out. > I just reloaded a test machine from scratch with -current and > installed the HAProxy 2.0.15-4f39279 package. > I loaded a very basic config file, and am also seeing the same exact > issue on this one as well. > Very strange that you are not - > Would you mind sharing any additional details of your config file? > Is there anything special about the certificate you have on the > backend server? > > I would love to understand what is going on here and what the > difference is with my experience. What is your backend running? Can you connect from the haproxy host with nc(1) and/or openssl(1)? I try to do my stuff as vanilla as possible so it's an RSA key signed by LE.
Re: Relayd with TLS and non-TLS backends - bug
> On Jun 11, 2020, at 4:28 PM, Toyam Cox wrote:
>
> Hello Misc,
>
> Full config at end of email.
>
> I've discussed the below in #openbsd on freenode, and was told to come
> here. At present, I have a setup where I need multiple unrelated
> servers under a single IP address. I used relayd to do https
> interception, read the Host header, and make decisions.
>
> The very relevant part of my config is this:
>
> forward to port 80
> forward with tls to port 443
>
> The order here does not matter (unlike most relayd configs, I know,
> but I've tested in my configuration and it works).
>
> When I have "with tls" on that second line, I see error lines like:
> relay web, session 3 (1 active), 0, [redacted] -> 10.0.0.102:80, TLS
> handshake error: handshake failed: error:14FFF3E7:SSL
> routines:(UNKNOWN)SSL_internal:unknown failure occurred, GET:
> Undefined error: 0
>
> and, unhelpfully, relayd responds with no response. There is no
> return. Or, as curl puts it: curl: (52) Empty reply from server
>
> When I remove "with tls" then I successfully reach the http backend,
> but since the https backend requires ssl, that connection no longer
> works. So it seems that 'with tls" affects all "forward" clauses, not
> just the one to which it's attached.
>
> I believe this to be a bug.
>
> cat >/etc/relayd.conf < table { "10.0.0.101" }
> table { "10.0.0.102" }
> # obviously obfuscated some values
>
> interval 5
> timeout 1000
>
> log connection
>
> http protocol web {
> return error
>
> match header set "X-Client-IP" value "$REMOTE_ADDR:$REMOTE_PORT"
> match header set "X-Forwarded-For" value "$REMOTE_ADDR"
> match header set "X-Forwarded-By" value "$SERVER_ADDR:$SERVER_PORT"
>
> http websockets
> pass request quick header "Host" value "myhost.example.com" path
> "/Client/*" forward to
> pass request quick header "Host" value "otherhost.example.com" forward
> to
>
> block
> }
>
> relay web {
> listen on 10.0.0.100 port 443 tls
> protocol web
>
> forward to port 80 check http "/webservice.asmx" code 405
> forward with tls to port 443 check https
> "/Client/SupportedBrowsers.html" host "myhost.example.com" code 200
> }
> EOF
>
Not to change topics too drastically :)
Consider running the backend connection over a different encrypted transport,
such as mesh iked(8) or upcoming wg(4). It’s super easy to setup, and
compatible with the other server OS. Go further into the “SDN realm” with
everything encapsulated in vxlan(4) for even more flexibility, including
long-haul internet endpoints across varying firewall and NAT designs. Pimp out
the configs of your networking groups’ routers to de-encapsulate and decrypt
the traffic for even more performance and compatibility. Anything is possible
as a front-end relay server with OpenBSD.
Why? Well for one, you save on many rounds of TLS negotiation. Upcoming
performance enhancements to the networking stack will only help scale this
method of relaying to more and more acceptable levels compared to non-encrypted
networking. My subjective net gain is simplicity, security, performance, and
flexibility.
-Brian
Re: Relayd with TLS and non-TLS backends - bug
Daniel, Thanks for taking the time to test this out. I just reloaded a test machine from scratch with -current and installed the HAProxy 2.0.15-4f39279 package. I loaded a very basic config file, and am also seeing the same exact issue on this one as well. Very strange that you are not - Would you mind sharing any additional details of your config file? Is there anything special about the certificate you have on the backend server? I would love to understand what is going on here and what the difference is with my experience. On Thu, Jul 2, 2020 at 4:38 PM Daniel Jakots wrote: > > On Thu, 2 Jul 2020 14:00:48 -0400, Henry Bonath > wrote: > > > Note the missing Client Hello on the 6.7 machine as it jumps to > > Application Data straight away. > > Configuration files for HAProxy are identical on both systems. > > > > I'm currently spinning up a machine on -CURRENT just to see if there > > is any difference, > > as there is a newer version of HAProxy in packages under Snapshots. > > > > I was initially going to try to reach out to the package maintainer > > for HAProxy but if this is happening in Relayd, then this "feels > > like" a de-facto bug. I wonder if NGINX would exhibit the same > > behavior. > > > > Has anyone else experienced such behavior with Load-Balancing TLS > > Backends since upgrading to 6.7? > > I don't use TLS for my backend (the only backend I use nowadays is on > localhost) so I can't speak for 6.7 (I only use -current, and when > -current was 6.7, I didn't test that). > > I just tested my -current haproxy using another -current host of mine > running nginx as a backend with TLS and it worked fine. > > backend https >option forwardfor >server web1 ln.chown.me:443 check ssl verify none > > and also with "verify required ca-file /etc/ssl/cert.pem" > > > Maybe some libressl fix happened on -current was not deemed critical > enough to be backported to 6.7? > > Cheers, > Daniel
Re: Relayd with TLS and non-TLS backends - bug
On Thu, 2 Jul 2020 14:00:48 -0400, Henry Bonath wrote: > Note the missing Client Hello on the 6.7 machine as it jumps to > Application Data straight away. > Configuration files for HAProxy are identical on both systems. > > I'm currently spinning up a machine on -CURRENT just to see if there > is any difference, > as there is a newer version of HAProxy in packages under Snapshots. > > I was initially going to try to reach out to the package maintainer > for HAProxy but if this is happening in Relayd, then this "feels > like" a de-facto bug. I wonder if NGINX would exhibit the same > behavior. > > Has anyone else experienced such behavior with Load-Balancing TLS > Backends since upgrading to 6.7? I don't use TLS for my backend (the only backend I use nowadays is on localhost) so I can't speak for 6.7 (I only use -current, and when -current was 6.7, I didn't test that). I just tested my -current haproxy using another -current host of mine running nginx as a backend with TLS and it worked fine. backend https option forwardfor server web1 ln.chown.me:443 check ssl verify none and also with "verify required ca-file /etc/ssl/cert.pem" Maybe some libressl fix happened on -current was not deemed critical enough to be backported to 6.7? Cheers, Daniel
Re: Relayd with TLS and non-TLS backends - bug
I would like to chime in here and confirm that I am seeing very
similar behavior with HAProxy on OpenBSD 6.7,
I was preparing to create my own post on this issue until I saw your thread.
I too believe this is a bug.
I also get the same "Empty reply" response you get via Curl:
> GET / HTTP/1.1
> Host: haproxy.example.com
> User-Agent: curl/7.58.0
> Accept: */*
>
* TLSv1.2 (IN), TLS alert, Client hello (1):
* Empty reply from server
curl: (52) Empty reply from server
For years I have run several OpenBSD Machines with CARP Interfaces and
the HAProxy Load Balancer frontending various services.
I have always used the HAProxy version that is installed via pkg_add,
and for years this has worked out beautifully.
I have upgraded about 8 of the systems from 6.6 to 6.7 and *all* have
exhibited similar behavior when using a TLS backend.
If I switch to a non-TLS backend, it works perfectly, but this of
course breaks the end-to-end encryption that my organization requires.
I have done one further and have taken packet captures to analyze the
traffic pattern in order to try to decipher what is actually
happening,
Below is a summary of the TCP Flow:
OpenBSD 6.6 - haproxy-1.9.15:
-
OpenBSD -> Backend:443 - SYN
Backend:443 -> OpenBSD - SYN/ACK
OpenBSD -> Backend:443 - ACK
OpenBSD -> Backend:443 - TLSv1.2 Client Hello
Backend:443 -> OpenBSD - ACK
Backend:443 -> OpenBSD - ACK
OpenBSD -> Backend:443 - TLSv1.2 Server Hello
OpenBSD -> Backend:443 - ACK
OpenBSD -> Backend:443 - TLSv1.2 Client Key Exhange
<-the rest of the successful session->
OpenBSD 6.7 - haproxy-2.0.14:
-
OpenBSD -> Backend:443 - SYN
Backend:443 -> OpenBSD - SYN/ACK
OpenBSD -> Backend:443 - ACK
OpenBSD -> Backend:443 - TLSv1.2 Application Data
Backend:443 -> OpenBSD - RST/ACK
Note the missing Client Hello on the 6.7 machine as it jumps to
Application Data straight away.
Configuration files for HAProxy are identical on both systems.
I'm currently spinning up a machine on -CURRENT just to see if there
is any difference,
as there is a newer version of HAProxy in packages under Snapshots.
I was initially going to try to reach out to the package maintainer for HAProxy
but if this is happening in Relayd, then this "feels like" a de-facto bug.
I wonder if NGINX would exhibit the same behavior.
Has anyone else experienced such behavior with Load-Balancing TLS
Backends since upgrading to 6.7?
On Thu, Jun 11, 2020 at 4:27 PM Toyam Cox wrote:
>
> Hello Misc,
>
> Full config at end of email.
>
> I've discussed the below in #openbsd on freenode, and was told to come
> here. At present, I have a setup where I need multiple unrelated
> servers under a single IP address. I used relayd to do https
> interception, read the Host header, and make decisions.
>
> The very relevant part of my config is this:
>
> forward to port 80
> forward with tls to port 443
>
> The order here does not matter (unlike most relayd configs, I know,
> but I've tested in my configuration and it works).
>
> When I have "with tls" on that second line, I see error lines like:
> relay web, session 3 (1 active), 0, [redacted] -> 10.0.0.102:80, TLS
> handshake error: handshake failed: error:14FFF3E7:SSL
> routines:(UNKNOWN)SSL_internal:unknown failure occurred, GET:
> Undefined error: 0
>
> and, unhelpfully, relayd responds with no response. There is no
> return. Or, as curl puts it: curl: (52) Empty reply from server
>
> When I remove "with tls" then I successfully reach the http backend,
> but since the https backend requires ssl, that connection no longer
> works. So it seems that 'with tls" affects all "forward" clauses, not
> just the one to which it's attached.
>
> I believe this to be a bug.
>
> cat >/etc/relayd.conf < table { "10.0.0.101" }
> table { "10.0.0.102" }
> # obviously obfuscated some values
>
> interval 5
> timeout 1000
>
> log connection
>
> http protocol web {
> return error
>
> match header set "X-Client-IP" value "$REMOTE_ADDR:$REMOTE_PORT"
> match header set "X-Forwarded-For" value "$REMOTE_ADDR"
> match header set "X-Forwarded-By" value "$SERVER_ADDR:$SERVER_PORT"
>
> http websockets
> pass request quick header "Host" value "myhost.example.com" path
> "/Client/*" forward to
> pass request quick header "Host" value "otherhost.example.com" forward
> to
>
> block
> }
>
> relay web {
> listen on 10.0.0.100 port 443 tls
> protocol web
>
> forward to port 80 check http "/webservice.asmx" code 405
> forward with tls to port 443 check https
> "/Client/SupportedBrowsers.html" host "myhost.example.com" code 200
> }
> EOF
>
