>Synopsis: relayd applies 'with tls' to all relay backends if any have it
>Category: system
>Environment:
System : OpenBSD 6.6
Details : OpenBSD 6.6 (GENERIC.MP) #372: Sat Oct 12 10:56:27 MDT
2019
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
Architecture: OpenBSD.amd64
Machine : amd64
>Description:
I put relayd in front of httpd and some webapps (as in my previous bug
report:
https://marc.info/?l=openbsd-bugs&m=158130991528086&w=2) and discovered
connections
to the webapps breaking when (the backend) httpd uses TLS.
The trouble turned out to be that relayd is applying my `with tls` on
the httpd relay
to the webapps as well, even though I'm not telling it to do any such
thing.
>How-To-Repeat:
# Make an empty test folder
```
$ mkdir relayd-with-tls; cd relayd-with-tls
```
## Make a cert for TLS
This makes a self-signed cert so we'll have to `curl --cacert
$(hostname).pem` (or just `curl -k`) to use it;
if you have a live server you can experiment on, use acme-client to get
a "real" cert, or reuse any cert you have.
```
$ openssl req -x509 -newkey rsa:4096 -subj '/CN='"localhost" -nodes
-keyout localhost.key -out localhost.pem -days 3
Generating a 4096 bit RSA private key
...........++++
.....................................................................................................................................................................................................................................................................................................................++++
writing new private key to 'localhost.key'
-----
```
## Set up httpd and a "webapp"
```
$ mkdir -p site
$ mkdir -p logs # httpd insists on this
$ echo "Static Site" > site/index.html
$ cat > httpd.conf <<EOF
chroot "."
server "site" {
listen on localhost tls port 8443
tls {
certificate "localhost.pem"
key "localhost.key"
}
root "site"
directory auto index
}
EOF
$ doas httpd -f httpd.conf
$ curl --cacert localhost.pem https://localhost:8443 # test httpd came
up right
Static Site
```
In a parallel terminal, run the "webapp":
```
$ nc -l -k -v 8082 | hexdump -C # our 'webapp'; leave this running.
Listening on 0.0.0.0 8082
```
## Set up relayd
relayd needs certs to be global:
```
$ doas cp localhost.pem /etc/ssl/localhost.crt
$ doas cp localhost.key /etc/ssl/private/localhost.key
```
```
$ cat > relayd.conf <<EOF
table <web> { "127.0.0.1" }
table <app> { "127.0.0.1" }
http protocol web {
# Return HTTP/HTML error pages to the client
return error
tls keypair localhost
tls ca file "localhost.pem"
match request path "/app/*" forward to <app>
}
relay https_proxy {
listen on 0.0.0.0 port 443 tls
protocol web
forward with tls to <web> port 8443
forward to <app> port 8082
}
EOF
$ doas relayd -f relayd.conf
$ curl --cacert localhost.pem https://localhost # test that relayd and
its `forward to <web>` relay came up
Static Site
```
## Tests
But the problem is when trying to relay to the "webapp":
```
$ curl --cacert localhost.pem https://localhost/app/
```
the app receives encrypted traffic:
```
$ nc -l -k -v 8082 | hexdump -C # our 'webapp' (that we left running)
Listening on 0.0.0.0 8082
Connection received on localhost 37169
00000000 16 03 01 00 b0 01 00 00 ac 03 03 4e bf d5 f5 1a
|...........N....|
00000010 e3 16 73 40 c5 2f d6 00 76 8d 2d 35 c9 55 50 34
|..s@./..v.-5.UP4|
00000020 84 b1 eb 84 5f 31 cc b4 eb 8f 21 00 00 4c c0 30
|...._1....!..L.0|
00000030 c0 2c c0 28 c0 24 c0 14 c0 0a 00 9f 00 6b 00 39
|.,.(.$.......k.9|
00000040 cc a9 cc a8 cc aa ff 85 00 c4 00 88 00 81 00 9d
|................|
00000050 00 3d 00 35 00 c0 00 84 c0 2f c0 2b c0 27 c0 23
|.=.5...../.+.'.#|
00000060 c0 13 c0 09 00 9e 00 67 00 33 00 be 00 45 00 9c
|.......g.3...E..|
00000070 00 3c 00 2f 00 ba 00 41 00 ff 01 00 00 37 00 05
|.<./...A.....7..|
00000080 00 05 01 00 00 00 00 00 0b 00 02 01 00 00 0a 00
|................|
00000090 08 00 06 00 1d 00 17 00 18 00 0d 00 18 00 16 08
|................|
000000a0 06 06 01 06 03 08 05 05 01 05 03 08 04 04 01 04
|................|
```
It doesn't work any better if we switch `nc` for an actual http server:
```
$ nc -l -k -v 8082 | hexdump -C
...
^C
$ python -m http.server 8082
Serving HTTP on 0.0.0.0 port 8082 (http://0.0.0.0:8082/) ...
```
Then test again:
```
$ curl --cacert localhost.pem https://localhost/app/
curl: (52) Empty reply from server
```
```
$ python -m http.server 8082
Serving HTTP on 0.0.0.0 port 8082 (http://0.0.0.0:8082/) ...
127.0.0.1 - - [10/Feb/2020 15:27:49] code 400, message Bad HTTP/0.9
request type
('\x16\x03\x01\x00°\x01\x00\x00¬\x03\x03åÌH\x00ù°fõ÷3t\x9c\x99r5Íþ\x14¼±')
127.0.0.1 - - [10/Feb/2020 15:27:49]
"°¬åÌHù°fõ÷3tr5Íþ¼±Lñ¥+_qÁåLÀ0À,À(À$ÀÀ" 400 -
```
>Fix:
The quickest workaround is to disable the backend TLS. Since relayd is
running on the
same server as httpd encrypting the second hop doesn't give much (any?)
security. But
if that's not possible -- say, if the backend servers are distributed
or you don't
want to rely entirely on a VPN for encryption -- then you would have to
go the other
way and make sure every webapp has its own internally valid cert.
That is, do this:
```
$ doas pkill httpd relayd
$ cat > httpd.conf <<EOF
chroot "."
server "site" {
listen on localhost tls port 8443
listen on localhost port 8080
tls {
certificate "localhost.pem"
key "localhost.key"
}
root "site"
directory auto index
}
EOF
$ cat > relayd.conf <<EOF
table <web> { "127.0.0.1" }
table <app> { "127.0.0.1" }
http protocol web {
# Return HTTP/HTML error pages to the client
return error
tls keypair localhost
tls ca file "localhost.pem"
match request path "/app/*" forward to <app>
}
relay https_proxy {
listen on 0.0.0.0 port 443 tls
protocol web
#forward with tls to <web> port 8443
forward to <web> port 8080
forward to <app> port 8082
}
EOF
$ doas httpd -f httpd.conf
$ doas relayd -f relayd.conf
```
Now
```
$ curl --cacert localhost.pem https://localhost/
Static Site
$ curl --cacert localhost.pem https://localhost/app/
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
"http://www.w3.org/TR/html4/strict.dtd";>
<html>
<head>
<meta http-equiv="Content-Type"
content="text/html;charset=utf-8">
<title>Error response</title>
</head>
<body>
<h1>Error response</h1>
<p>Error code: 404</p>
<p>Message: File not found.</p>
<p>Error code explanation: HTTPStatus.NOT_FOUND - Nothing
matches the given URI.</p>
</body>
</html>
```
and the "webapp"'s log:
```
$ python -m http.server 8082
Serving HTTP on 0.0.0.0 port 8082 (http://0.0.0.0:8082/) ...
127.0.0.1 - - [10/Feb/2020 18:28:47] code 404, message File not found
127.0.0.1 - - [10/Feb/2020 18:28:47] "GET /app/ HTTP/1.1" 404 -
```
