Re: Feature Proposal: Add ssltermination parameter for SNI-based LoadBalancing
Hey, whats the state of this? Can we start working out a plan to remove the inital SSLRequest from the connection protocol or is there any reason to keep it? I would start by removing the need of the SSLRequest in the psql-server if its started with a special parameter(ssl-only or so). Simultaneously I would add a parameter to this disable the SSLRequest in the client as well. Later we could make this behaviour default for psql-server with ssl-enabled and clients and some far time ahead we could remove the implementation of SSLRequest in both server and client. What are your thaughts about this? Best Regards
Re: Feature Proposal: Add ssltermination parameter for SNI-based LoadBalancing
I liked the idea with separate ports for ssl and non ssl requests and
tried it with haproxy.
The psql-client connects with haproxy and receives the fixed 'S' byte
response. After that he tried to continue on the same connection and
doens't open a new one. This crashes the connection because haproxy
expects a new tcp connection.
psqlClient: opens connection (ARP: SYN)
haproxy: accepts connection (ARP: SYN ACK)
psqlClient: confirmes the connection (ARP: ACK)
psqlClient: sends SSLRequest
haproxy: sends confirmation (ARP: ACK)
haproxy: sends fixed byte response ('S')
haproxy: closes connection (ARP: FIN, ACK)
psqlclient: confirmed fixed byte response (ARP: ACK)
psqlclient: sends ssl hello request --> error connection already
closed("psql: error: SSL SYSCALL error: No error (0x/0))
In my eyes the problem lies in upgrading the connection rather then
opening a new one.
Am 14-Dec-20 um 14:50 schrieb Heikki Linnakangas:
On 12/12/2020 13:52, Lukas Meisegeier wrote:
Thanks for the provided ideas :)
I use HaProxy for my load-balancing and unfortunately I can't define
that I want to listen on a port for both ssl and non ssl requests.
Could you configure HaProxy to listen on separate ports for SSL and
non-SSL connections, then? And forward both to the same Postgres server.
That means if I try to return a fixed response 'S' on the SSLRequest it
fails with an SSL-Handshake failure cause the server expects a ssl
message.
That doesn't sound right to me, or perhaps I have misunderstood what you
mean. If you don't send the SSLRequest to the Postgres server, but "eat"
it in the proxy, the Postgres server will not try to do an SSL handshake.
I have to say the psql ssl handshake procedure is really unique and
challenging :D
Yeah. IMAP and SMTP can use "STARTTLS" to switch an unencrypted
connection to encrypted, though. That's pretty similar to the
'SSLRequest' message used in the postgres protocol.
- Heikki
Re: Feature Proposal: Add ssltermination parameter for SNI-based LoadBalancing
On 12/12/2020 13:52, Lukas Meisegeier wrote: Thanks for the provided ideas :) I use HaProxy for my load-balancing and unfortunately I can't define that I want to listen on a port for both ssl and non ssl requests. Could you configure HaProxy to listen on separate ports for SSL and non-SSL connections, then? And forward both to the same Postgres server. That means if I try to return a fixed response 'S' on the SSLRequest it fails with an SSL-Handshake failure cause the server expects a ssl message. That doesn't sound right to me, or perhaps I have misunderstood what you mean. If you don't send the SSLRequest to the Postgres server, but "eat" it in the proxy, the Postgres server will not try to do an SSL handshake. I have to say the psql ssl handshake procedure is really unique and challenging :D Yeah. IMAP and SMTP can use "STARTTLS" to switch an unencrypted connection to encrypted, though. That's pretty similar to the 'SSLRequest' message used in the postgres protocol. - Heikki
Re: Feature Proposal: Add ssltermination parameter for SNI-based LoadBalancing
Thanks for the provided ideas :) I use HaProxy for my load-balancing and unfortunately I can't define that I want to listen on a port for both ssl and non ssl requests. That means if I try to return a fixed response 'S' on the SSLRequest it fails with an SSL-Handshake failure cause the server expects a ssl message. I searched for some way to forward to a default backend on ssl failure but this seems to be no use-case for haproxy and isn't supported. I also didn't find any other tcp-loadbalancer, which could handle this type of ssl-failure fallback. My only option would therefore be to write a custom loadbalancer for this usecase, which is not really feasible given the amount of features of haproxy I plan to use. I have to say the psql ssl handshake procedure is really unique and challenging :D The tool stunnel is capable of this protocol but I can't do sni-based loadbalancing with it so this is kind of a dead end here. Lukas Am 11-Dec-20 um 16:44 schrieb Heikki Linnakangas: On 11/12/2020 16:46, Lukas Meisegeier wrote: Hey Heikki, thanks for providing feedback :) The traffic between proxy and psql-server is unencrypted thats why I don't need to patch the server. Ok. I tried returning a fixed response on the first plain SSLRequest forwarding it to a psql-server with ssl enabled an tried to switch then on the ssl connection startup but that didn't work out. I guess its because the psql-server won't accept an ssl connection if its not requested via SSLRequest. Your proxy could receive the client's SSLRequest message, and respond with a single byte 'S'. You don't need to forward that to the real PostgreSQL server, since the connection to the PostgreSQL server is unencrypted. Then perform the TLS handshake, and forward all traffic to the real server only after that. Client: -> SSLRequest Proxy: <- 'S' Client: -> TLS ClientHello Proxy: [finish TLS handshake] - Heikki
Re: Feature Proposal: Add ssltermination parameter for SNI-based LoadBalancing
Hey Heikki, thanks for providing feedback :) The traffic between proxy and psql-server is unencrypted thats why I don't need to patch the server. I tried returning a fixed response on the first plain SSLRequest forwarding it to a psql-server with ssl enabled an tried to switch then on the ssl connection startup but that didn't work out. I guess its because the psql-server won't accept an ssl connection if its not requested via SSLRequest. I would definitly appreciate if the psql-server could accept the TLS-client hello directly but we would still need to set the tls-sni-extension correctly. Perhaps we could rename the parameter to "sslplainrequest(yes/no)" and start with making the plain SSLRequest optional in the psql-server. Best Regards Lukas Am 11-Dec-20 um 14:26 schrieb Heikki Linnakangas: On 10/12/2020 17:49, Lukas Meisegeier wrote: I try to host multiple postgresql-servers on the same ip and the same port through SNI-based load-balancing. Currently this is not possible because of two issues: 1. The psql client won't set the tls-sni-extension correctly (https://www.postgresql.org/message-id/20181211145240.GL20222%40redhat.com) 2. The psql connection protocol implements a SSLRequest in plain text before actually opening a connection. The first issue is easily solvable by calling `SSL_set_tlsext_host_name(conn->ssl, conn->connhost[conn->whichhost].host)` before opening the connection. The second issue is also solvable through a new parameter "ssltermination" which if set to "proxy" will skip the initial SSLRequest and connects directly through ssl. The default value would be "server" which changes nothing on the existing behaviour. Don't you need backend changes as well? The backend will still expect the client to send an SSLRequest. Or is the connection from the proxy to the actual server unencrypted? It's not very nice that the client needs to set special options, depending on whether the server is behind a proxy or not. Could you teach the proxy to deal with the SSLRequest message? Perhaps we should teach the backend to accept a TLS ClientHello directly, without the SSLRequest message. That way, the client could send the ClientHello without SSLRequest, and the proxy wouldn't need to care about SSLRequest. It would also eliminate one round-trip from the protocol handshake, which would be nice. A long deprecation/transition period would be needed before we could make that the default behavior, but that's ok. - Heikki
Re: Feature Proposal: Add ssltermination parameter for SNI-based LoadBalancing
On 11/12/2020 16:46, Lukas Meisegeier wrote: Hey Heikki, thanks for providing feedback :) The traffic between proxy and psql-server is unencrypted thats why I don't need to patch the server. Ok. I tried returning a fixed response on the first plain SSLRequest forwarding it to a psql-server with ssl enabled an tried to switch then on the ssl connection startup but that didn't work out. I guess its because the psql-server won't accept an ssl connection if its not requested via SSLRequest. Your proxy could receive the client's SSLRequest message, and respond with a single byte 'S'. You don't need to forward that to the real PostgreSQL server, since the connection to the PostgreSQL server is unencrypted. Then perform the TLS handshake, and forward all traffic to the real server only after that. Client: -> SSLRequest Proxy: <- 'S' Client: -> TLS ClientHello Proxy: [finish TLS handshake] - Heikki
Re: Feature Proposal: Add ssltermination parameter for SNI-based LoadBalancing
On 10/12/2020 17:49, Lukas Meisegeier wrote: I try to host multiple postgresql-servers on the same ip and the same port through SNI-based load-balancing. Currently this is not possible because of two issues: 1. The psql client won't set the tls-sni-extension correctly (https://www.postgresql.org/message-id/20181211145240.GL20222%40redhat.com) 2. The psql connection protocol implements a SSLRequest in plain text before actually opening a connection. The first issue is easily solvable by calling `SSL_set_tlsext_host_name(conn->ssl, conn->connhost[conn->whichhost].host)` before opening the connection. The second issue is also solvable through a new parameter "ssltermination" which if set to "proxy" will skip the initial SSLRequest and connects directly through ssl. The default value would be "server" which changes nothing on the existing behaviour. Don't you need backend changes as well? The backend will still expect the client to send an SSLRequest. Or is the connection from the proxy to the actual server unencrypted? It's not very nice that the client needs to set special options, depending on whether the server is behind a proxy or not. Could you teach the proxy to deal with the SSLRequest message? Perhaps we should teach the backend to accept a TLS ClientHello directly, without the SSLRequest message. That way, the client could send the ClientHello without SSLRequest, and the proxy wouldn't need to care about SSLRequest. It would also eliminate one round-trip from the protocol handshake, which would be nice. A long deprecation/transition period would be needed before we could make that the default behavior, but that's ok. - Heikki
Feature Proposal: Add ssltermination parameter for SNI-based LoadBalancing
Hi,
I try to host multiple postgresql-servers on the same ip and the same
port through SNI-based load-balancing.
Currently this is not possible because of two issues:
1. The psql client won't set the tls-sni-extension correctly
(https://www.postgresql.org/message-id/20181211145240.GL20222%40redhat.com)
2. The psql connection protocol implements a SSLRequest in plain text
before actually opening a connection.
The first issue is easily solvable by calling
`SSL_set_tlsext_host_name(conn->ssl,
conn->connhost[conn->whichhost].host)` before opening the connection.
The second issue is also solvable through a new parameter
"ssltermination" which if set to "proxy" will skip the initial
SSLRequest and connects directly through ssl.
The default value would be "server" which changes nothing on the
existing behaviour.
I compiled the psql-client with these changes and was able to connect to
2 different databases through the same ip and port just by changing the
hostname.
This fix is important to allow multiple postgres instances on one ip
without having to add a port number.
I implemented this change on a fork of the postgres mirror on github:
https://github.com/klg71/mayope_postgres
The affected files are:
- src/interfaces/libpq/fe-connect.c (added ssltermination parameter)
- src/interfaces/libpq/libpq-int.h (added ssltermination parameter)
- src/interfaces/libpq/fe-secure-openssl.c (added tls-sni-extension)
I appended the relevant diff.
Best Regards
Lukas
diff --git a/src/interfaces/libpq/fe-connect.c
b/src/interfaces/libpq/fe-connect.c
index 7d04d3664e..43fcfc2274 100644
--- a/src/interfaces/libpq/fe-connect.c
+++ b/src/interfaces/libpq/fe-connect.c
@@ -131,6 +131,7 @@ static int ldapServiceLookup(const char *purl,
PQconninfoOption *options,
#define DefaultTargetSessionAttrs "any"
#ifdef USE_SSL
#define DefaultSSLMode "prefer"
+#define DefaultSSLTermination "server"
#else
#define DefaultSSLMode "disable"
#endif
@@ -293,6 +294,11 @@ static const internalPQconninfoOption PQconninfoOptions[]
= {
"SSL-Mode", "", 12, /* sizeof("verify-full") == 12
*/
offsetof(struct pg_conn, sslmode)},
+ {"ssltermination", "PGSSLTERMINATION", DefaultSSLMode, NULL,
+ "SSL-Termination-Mode", "", 6, /* sizeof("server") ==
6 */
+ offsetof(struct pg_conn, ssltermination)},
+
+
{"sslcompression", "PGSSLCOMPRESSION", "0", NULL,
"SSL-Compression", "", 1,
offsetof(struct pg_conn, sslcompression)},
@@ -1278,6 +1284,16 @@ connectOptions2(PGconn *conn)
return false;
}
+ if (strcmp(conn->ssltermination, "server") != 0
+ && strcmp(conn->ssltermination, "proxy") != 0)
+ {
+ conn->status = CONNECTION_BAD;
+ printfPQExpBuffer(>errorMessage,
+
libpq_gettext("invalid %s value: \"%s\"\n"),
+ "ssltermination",
conn->ssltermination);
+ return false;
+ }
+
#ifndef USE_SSL
switch (conn->sslmode[0])
{
@@ -2915,6 +2931,13 @@ keep_going:
/* We will come back to here until there is
if (conn->allow_ssl_try && !conn->wait_ssl_try
&&
!conn->ssl_in_use)
{
+ /*
+ * SSL termination is handled by
proxy/load-balancer, no need to send SSLRequest
+ */
+ if(conn->ssltermination[0]=='p'){
+ conn->status =
CONNECTION_SSL_STARTUP;
+ return PGRES_POLLING_WRITING;
+ }
ProtocolVersion pv;
/*
@@ -2995,77 +3018,89 @@ keep_going:
/* We will come back to here until there is
if (!conn->ssl_in_use)
{
/*
-* We use pqReadData here since it has
the logic to
-* distinguish no-data-yet from
connection closure. Since
-* conn->ssl isn't set, a plain recv()
will occur.
+* Skip SSLRequest package and
initialize ssl directly with proxy
*/
- charSSLok;
- int rdresult;
-
-
