Re: how make backend applications aware about tls-offloading

[Reindl Harald]

Am 08.01.2017 um 11:01 schrieb Stefan Eissing:

There is the reverse situation which is called opportunistic encryption, namely 
the transfer of a http: request over a TLS connection.

Both cases are tricky on HTTP/1.x because the URI scheme is not transported in requests 
(commonly. the spec would allow it but no one does it, so no one is prepared to 
honor/handle it). HTTP/2, with opportunistic encryption in mind, added the 
":scheme" header for this. But implementation is also tricky.

So, mod_http2 has similar problems to this ATS setup: convincing the request 
processing parts in the server that a request has a certain scheme, 
*independent* of mod_ssl's presence. I think it would be nice if we could fix 
this.

One approach that comes to mind:
 * add the uri scheme to request_rec->scheme
 * set it by:
   1. parse the request uri
   2. if not set, fix it in very late read hooks to "http:"
   3. have mod_ssl install an earlier hook that sets "https:" if not present
 * check that URI host and Host: header are an allowed combination
 * check that r->scheme and r->server are an allowed combination

ATS would then be configured to forward requests to the httpd backend by using 
absolute request uris (so they carry the scheme) or HTTP/2. httpd would be 
configured to accept https: uris from ATS remote ip.

And while there are numerous parts/applications where the wheels fall off in 
such a setup, it is not the default setup. No one initially needs to be able to 
get it right. But in a concrete deployment, the configuration can be made and 
the application code fixed where necessary.


yeah - something more or less standard instead
a) change this in tttpd config
b) change this in ATS config, dunno how handle it with a different proxy
c) change this in your application

when there is something you can detect in the application code when 
proxy / backend play in a more or less defined way together other 
proxies and backend servers could follow



Am 07.01.2017 um 09:30 schrieb Reindl Harald :

* Apache Trafficserver in front
* ATS configured for TLS-offloading
* connection to backend-httpd on the LAN unencrypted
* mod_remoteip correctly configured on backend httpd

is there any way to make the backend php application aware that in fact 
$_SERVER['HTTPS'] and $_SERVER['REQUEST_SCHEME'] should be 'on' / https:// in 
case of generate absolute URLs like for emails

in a perfect world this would be handeled like the transparent translation of the client 
IP with https://httpd.apache.org/docs/current/mod/mod_remoteip.html and it's 
RemoteIPInternalProxy and a header like "X-Forwarded-TLS"

something like below where "X-TLS-Offloading" is only evaluated from 
"RemoteIPInternalProxy" pyhsical addressess

RemoteIPHeader X-Forwarded-For
RemoteTLSHeaderX-TLS-Offloading
RemoteIPInternalProxy  192.168.196.1

Re: how make backend applications aware about tls-offloading

[Stefan Eissing]

There is the reverse situation which is called opportunistic encryption, namely 
the transfer of a http: request over a TLS connection.

Both cases are tricky on HTTP/1.x because the URI scheme is not transported in 
requests (commonly. the spec would allow it but no one does it, so no one is 
prepared to honor/handle it). HTTP/2, with opportunistic encryption in mind, 
added the ":scheme" header for this. But implementation is also tricky.

So, mod_http2 has similar problems to this ATS setup: convincing the request 
processing parts in the server that a request has a certain scheme, 
*independent* of mod_ssl's presence. I think it would be nice if we could fix 
this.

One approach that comes to mind:
 * add the uri scheme to request_rec->scheme
 * set it by:
   1. parse the request uri
   2. if not set, fix it in very late read hooks to "http:"
   3. have mod_ssl install an earlier hook that sets "https:" if not present
 * check that URI host and Host: header are an allowed combination
 * check that r->scheme and r->server are an allowed combination

ATS would then be configured to forward requests to the httpd backend by using 
absolute request uris (so they carry the scheme) or HTTP/2. httpd would be 
configured to accept https: uris from ATS remote ip.

And while there are numerous parts/applications where the wheels fall off in 
such a setup, it is not the default setup. No one initially needs to be able to 
get it right. But in a concrete deployment, the configuration can be made and 
the application code fixed where necessary.

Cheers,
Stefan

> Am 07.01.2017 um 09:30 schrieb Reindl Harald :
> 
> * Apache Trafficserver in front
> * ATS configured for TLS-offloading
> * connection to backend-httpd on the LAN unencrypted
> * mod_remoteip correctly configured on backend httpd
> 
> is there any way to make the backend php application aware that in fact 
> $_SERVER['HTTPS'] and $_SERVER['REQUEST_SCHEME'] should be 'on' / https:// in 
> case of generate absolute URLs like for emails
> 
> in a perfect world this would be handeled like the transparent translation of 
> the client IP with 
> https://httpd.apache.org/docs/current/mod/mod_remoteip.html and it's 
> RemoteIPInternalProxy and a header like "X-Forwarded-TLS"
> 
> something like below where "X-TLS-Offloading" is only evaluated from 
> "RemoteIPInternalProxy" pyhsical addressess
> 
> RemoteIPHeader X-Forwarded-For
> RemoteTLSHeaderX-TLS-Offloading
> RemoteIPInternalProxy  192.168.196.1
> 

Stefan Eissing

bytes GmbH
Hafenstrasse 16
48155 Münster
www.greenbytes.de

Re: how make backend applications aware about tls-offloading

[Leif Hedstrom]

> On Jan 7, 2017, at 3:25 PM, Reindl Harald  wrote:
> 
> 
> 
> Am 07.01.2017 um 22:53 schrieb Yann Ylavic:
>> On Sat, Jan 7, 2017 at 9:30 AM, Reindl Harald  wrote:
>>> 
>>> something like below where "X-TLS-Offloading" is only evaluated from
>>> "RemoteIPInternalProxy" pyhsical addressess
>>> 
>>> RemoteIPHeader X-Forwarded-For
>>> RemoteTLSHeaderX-TLS-Offloading
>>> RemoteIPInternalProxy  192.168.196.1
>> 
>> Wouldn't something like this work?
>> 
>> RewriteRule on
>> RewriteCond %{ENV:remoteip-proxy-ip-list} .
>> RewriteCond %{HTTP:X-TLS-Offloading} ^true$
>> RewriteRule ^ - [E=HTTPS:on,E=REQUEST_SCHEME:https]
>> 
>> Given that remoteip-proxy-ip-list is filled by mod_remoteip if (and
>> only if) RemoteIPInternalProxy matches
> 
> currently not because nothing provides "X-TLS-Offloading" which is the reason 
> for add both parties to this conversation


Hmmm, maybe I’m not understanding all the details, but in ATS, why not use 
header_rewrite to add whatever headers you wish to send upstream (to HTTPD)? 
That’s what I do for a particular app, which needs to know that ATS did TLS 
offloading in front of it (the app being Drupal / PHP).

— Leif

cond %{SEND_REQUEST_HDR_HOOK}
set-header X-Forwarded-Proto https



> 
> such global rewrite rules are not very appealing while the intention of get 
> this handeled by mod_remoteip is that for the admin this would be the central 
> place to deal with backendsservers with a proxy in front
> 
> it is handeled perfectly for the REMOTE_ADDR where for every access(deny 
> rules, loggings, mod_security-rules and within applications you can trust 
> it's the clients IP and not one from own infrastructure
> 
> end-to-end-encryption (one argunmet which came against it) is something one 
> needs to consider anyways if TLS-offloading come into the mix and the 
> connection between proxy and backend needs to be 100% trusted, but it's a 
> great way to spread load of generate dynamic content and encryption to 
> different machines and should be 100% transparent to the application
> 
> generate links for emails and secure-flag for cookies are the first things 
> coming to my mind but likely not all hidden cases where missing awarness the 
> the client connection is encrypted may lead to undesired behavior

Re: how make backend applications aware about tls-offloading

[Yann Ylavic]

On Sun, Jan 8, 2017 at 12:39 AM, Reindl Harald  wrote:
>
> Am 08.01.2017 um 00:31 schrieb Yann Ylavic:
>>
>> On Sun, Jan 8, 2017 at 12:22 AM, Reindl Harald 
>> wrote:
>>>
>>>
>>> ok, so we need to continue the code below and set the option in every
>>> tls-offloaded application - intention of this thread was maybe get this
>>> transparent which seems not to be possible
>>
>>
>> It is "technically" possible, but not wise IMHO.
>> Making every httpd module/CGI/app think the local connection is https
>> could lead to things like "; Secure" cookies sent on the (clear) wire,
>> and that option would be accompanied with so much warnings ("unless
>> you're really on the same switch, but even that...") that it'd be hard
>> to defend (endlessly?).
>
> excatly *that* would be the desired result if configured that way because
> the "clear wire" is controlled and trusted in that context and you *want*
> the secure flag sent for cookies between the tls-offloading server and the
> enduser to not get them back unencrypted over the "real clear wire"

I'm pretty sure the RFC does not allow for "secure" cookies to go in
clear, but that'd be an admin choice after all.
An RFC (as-much-as-it-can-)compliant server can hardly feature it, though.

>
> the whole purpose of *tls offloading* is run the application on a virtual
> machine with a preforked httpd and encryption on the reverse-proxy running
> multithreaded with keep-alive
>
> another secuity gain here is that the amchine which runs application code
> never has a change to see the private ssl key while a breach on the proxy
> with no application code is less likely

That's your architecture choice (constraint?), but you could achieve
the same by proxying to php-fpm for example, and possibly have more
response rewrite/reverse options too...

Re: how make backend applications aware about tls-offloading

[Reindl Harald]

Am 08.01.2017 um 00:31 schrieb Yann Ylavic:

On Sun, Jan 8, 2017 at 12:22 AM, Reindl Harald  wrote:


ok, so we need to continue the code below and set the option in every
tls-offloaded application - intention of this thread was maybe get this
transparent which seems not to be possible


It is "technically" possible, but not wise IMHO.
Making every httpd module/CGI/app think the local connection is https
could lead to things like "; Secure" cookies sent on the (clear) wire,
and that option would be accompanied with so much warnings ("unless
you're really on the same switch, but even that...") that it'd be hard
to defend (endlessly?).


excatly *that* would be the desired result if configured that way 
because the "clear wire" is controlled and trusted in that context and 
you *want* the secure flag sent for cookies between the tls-offloading 
server and the enduser to not get them back unencrypted over the "real 
clear wire"


the whole purpose of *tls offloading* is run the application on a 
virtual machine with a preforked httpd and encryption on the 
reverse-proxy running multithreaded with keep-alive


another secuity gain here is that the amchine which runs application 
code never has a change to see the private ssl key while a breach on the 
proxy with no application code is less likely



if(!empty($cms_tls_offload))
{
 $_SERVER['REQUEST_SCHEME'] = 'https';
 $_SERVER['HTTPS']  = 'on';
}


Your choice ;)

Re: how make backend applications aware about tls-offloading

[Yann Ylavic]

On Sun, Jan 8, 2017 at 12:22 AM, Reindl Harald  wrote:
>
> ok, so we need to continue the code below and set the option in every
> tls-offloaded application - intention of this thread was maybe get this
> transparent which seems not to be possible

It is "technically" possible, but not wise IMHO.
Making every httpd module/CGI/app think the local connection is https
could lead to things like "; Secure" cookies sent on the (clear) wire,
and that option would be accompanied with so much warnings ("unless
you're really on the same switch, but even that...") that it'd be hard
to defend (endlessly?).

>
> if(!empty($cms_tls_offload))
> {
>  $_SERVER['REQUEST_SCHEME'] = 'https';
>  $_SERVER['HTTPS']  = 'on';
> }

Your choice ;)

Re: how make backend applications aware about tls-offloading

[Reindl Harald]

Am 07.01.2017 um 23:53 schrieb Yann Ylavic:

On Sat, Jan 7, 2017 at 11:25 PM, Reindl Harald  wrote:

Am 07.01.2017 um 22:53 schrieb Yann Ylavic:


Wouldn't something like this work?

RewriteRule on
RewriteCond %{ENV:remoteip-proxy-ip-list} .
RewriteCond %{HTTP:X-TLS-Offloading} ^true$
RewriteRule ^ - [E=HTTPS:on,E=REQUEST_SCHEME:https]


That wouldn't work anyway, both variables will be overridden later
when the env is constructed.


Given that remoteip-proxy-ip-list is filled by mod_remoteip if (and
only if) RemoteIPInternalProxy matches


currently not because nothing provides "X-TLS-Offloading" which is the
reason for add both parties to this conversation


OK, that's a prerequisite in any case..


such global rewrite rules are not very appealing while the intention of get
this handeled by mod_remoteip is that for the admin this would be the
central place to deal with backendsservers with a proxy in front


Admittedly.


it is handeled perfectly for the REMOTE_ADDR where for every access(deny
rules, loggings, mod_security-rules and within applications you can trust
it's the clients IP and not one from own infrastructure


Right, but HTTPS and REQUEST_SCHEME have a meaning for the httpd
server, and they refer to its *local* configuration, so overriding
them is very misleading (and does not work as mentioned above).

Thus RemoteTLSHeader cannot be something that overrides them, and the
best it could do is to unset the header if not trusted.


end-to-end-encryption (one argunmet which came against it) is something one
needs to consider anyways if TLS-offloading come into the mix and the
connection between proxy and backend needs to be 100% trusted, but it's a
great way to spread load of generate dynamic content and encryption to
different machines and should be 100% transparent to the application


From the above, the app would have to rely on the (un)defined
RemoteTLSHeader instead of HTTPS/REQUEST_SCHEME, so it can't be as
transparent you'd like...

A new mod_remoteip feature for what you could do with mod_rewrite or
mod_headers is less appealing then


ok, so we need to continue the code below and set the option in every 
tls-offloaded application - intention of this thread was maybe get this 
transparent which seems not to be possible


if(!empty($cms_tls_offload))
{
 $_SERVER['REQUEST_SCHEME'] = 'https';
 $_SERVER['HTTPS']  = 'on';
}

Re: how make backend applications aware about tls-offloading

[Yann Ylavic]

On Sat, Jan 7, 2017 at 11:25 PM, Reindl Harald  wrote:
>
> Am 07.01.2017 um 22:53 schrieb Yann Ylavic:
>>
>> Wouldn't something like this work?
>>
>> RewriteRule on
>> RewriteCond %{ENV:remoteip-proxy-ip-list} .
>> RewriteCond %{HTTP:X-TLS-Offloading} ^true$
>> RewriteRule ^ - [E=HTTPS:on,E=REQUEST_SCHEME:https]

That wouldn't work anyway, both variables will be overridden later
when the env is constructed.

>>
>> Given that remoteip-proxy-ip-list is filled by mod_remoteip if (and
>> only if) RemoteIPInternalProxy matches
>
> currently not because nothing provides "X-TLS-Offloading" which is the
> reason for add both parties to this conversation

OK, that's a prerequisite in any case..

>
> such global rewrite rules are not very appealing while the intention of get
> this handeled by mod_remoteip is that for the admin this would be the
> central place to deal with backendsservers with a proxy in front

Admittedly.

>
> it is handeled perfectly for the REMOTE_ADDR where for every access(deny
> rules, loggings, mod_security-rules and within applications you can trust
> it's the clients IP and not one from own infrastructure

Right, but HTTPS and REQUEST_SCHEME have a meaning for the httpd
server, and they refer to its *local* configuration, so overriding
them is very misleading (and does not work as mentioned above).

Thus RemoteTLSHeader cannot be something that overrides them, and the
best it could do is to unset the header if not trusted.

>
> end-to-end-encryption (one argunmet which came against it) is something one
> needs to consider anyways if TLS-offloading come into the mix and the
> connection between proxy and backend needs to be 100% trusted, but it's a
> great way to spread load of generate dynamic content and encryption to
> different machines and should be 100% transparent to the application

>From the above, the app would have to rely on the (un)defined
RemoteTLSHeader instead of HTTPS/REQUEST_SCHEME, so it can't be as
transparent you'd like...

A new mod_remoteip feature for what you could do with mod_rewrite or
mod_headers is less appealing then.

Re: how make backend applications aware about tls-offloading

[Reindl Harald]

Am 07.01.2017 um 22:53 schrieb Yann Ylavic:

On Sat, Jan 7, 2017 at 9:30 AM, Reindl Harald  wrote:


something like below where "X-TLS-Offloading" is only evaluated from
"RemoteIPInternalProxy" pyhsical addressess

RemoteIPHeader X-Forwarded-For
RemoteTLSHeaderX-TLS-Offloading
RemoteIPInternalProxy  192.168.196.1


Wouldn't something like this work?

RewriteRule on
RewriteCond %{ENV:remoteip-proxy-ip-list} .
RewriteCond %{HTTP:X-TLS-Offloading} ^true$
RewriteRule ^ - [E=HTTPS:on,E=REQUEST_SCHEME:https]

Given that remoteip-proxy-ip-list is filled by mod_remoteip if (and
only if) RemoteIPInternalProxy matches


currently not because nothing provides "X-TLS-Offloading" which is the 
reason for add both parties to this conversation


such global rewrite rules are not very appealing while the intention of 
get this handeled by mod_remoteip is that for the admin this would be 
the central place to deal with backendsservers with a proxy in front


it is handeled perfectly for the REMOTE_ADDR where for every access(deny 
rules, loggings, mod_security-rules and within applications you can 
trust it's the clients IP and not one from own infrastructure


end-to-end-encryption (one argunmet which came against it) is something 
one needs to consider anyways if TLS-offloading come into the mix and 
the connection between proxy and backend needs to be 100% trusted, but 
it's a great way to spread load of generate dynamic content and 
encryption to different machines and should be 100% transparent to the 
application


generate links for emails and secure-flag for cookies are the first 
things coming to my mind but likely not all hidden cases where missing 
awarness the the client connection is encrypted may lead to undesired 
behavior

Re: how make backend applications aware about tls-offloading

[Yann Ylavic]

On Sat, Jan 7, 2017 at 9:30 AM, Reindl Harald  wrote:
>
> something like below where "X-TLS-Offloading" is only evaluated from
> "RemoteIPInternalProxy" pyhsical addressess
>
> RemoteIPHeader X-Forwarded-For
> RemoteTLSHeaderX-TLS-Offloading
> RemoteIPInternalProxy  192.168.196.1
>

Wouldn't something like this work?

RewriteRule on
RewriteCond %{ENV:remoteip-proxy-ip-list} .
RewriteCond %{HTTP:X-TLS-Offloading} ^true$
RewriteRule ^ - [E=HTTPS:on,E=REQUEST_SCHEME:https]

Given that remoteip-proxy-ip-list is filled by mod_remoteip if (and
only if) RemoteIPInternalProxy matches.

Re: how make backend applications aware about tls-offloading

[Reindl Harald]

Am 07.01.2017 um 17:04 schrieb Jered Floyd:

Does the "sslheaders" experimental plugin meet your needs?

https://docs.trafficserver.apache.org/en/latest/admin-guide/plugins/sslheaders.en.html


not really beause it's not transparent to the application and so i can 
continue fake the $_SERVER vars based on application configs - it also 
needs to make sure that this headers never ever are passed through from 
untrusted amchines in front fo the own proxy or faked by clients 
pointing directly to the origin - the way mod_remoteip works takes cae 
of such things


"end-to-end" don't matter when both ATS and httpd are on the same switch 
or even running on the same vritualization host - what matters much more 
is that your applications is aware about the https fact and set the 
*encryption flag for cookies* as example


the fake-by-configuration hacking makes things just more complex because 
you have one more place to care besides DNS, ATS and httpd and the 
Magento hacks placing $_SERVER['xyz'] into 'index.php' are anything but 
not beautiful


well, and for sites which should be reachable with https *and* http you 
can forget this entirely when don't have any clue



- On Jan 7, 2017, at 3:30 AM, Reindl Harald h.rei...@thelounge.net wrote:


* Apache Trafficserver in front
* ATS configured for TLS-offloading
* connection to backend-httpd on the LAN unencrypted
* mod_remoteip correctly configured on backend httpd

is there any way to make the backend php application aware that in fact
$_SERVER['HTTPS'] and $_SERVER['REQUEST_SCHEME'] should be 'on' /
https:// in case of generate absolute URLs like for emails

in a perfect world this would be handeled like the transparent
translation of the client IP with
https://httpd.apache.org/docs/current/mod/mod_remoteip.html and it's
RemoteIPInternalProxy and a header like "X-Forwarded-TLS"

something like below where "X-TLS-Offloading" is only evaluated from
"RemoteIPInternalProxy" pyhsical addressess

RemoteIPHeader X-Forwarded-For
RemoteTLSHeaderX-TLS-Offloading
RemoteIPInternalProxy  192.168.196.1

Re: how make backend applications aware about tls-offloading

[William A Rowe Jr]

On Sat, Jan 7, 2017 at 2:30 AM, Reindl Harald  wrote:
> * Apache Trafficserver in front
> * ATS configured for TLS-offloading
> * connection to backend-httpd on the LAN unencrypted
> * mod_remoteip correctly configured on backend httpd
>
> is there any way to make the backend php application aware that in fact
> $_SERVER['HTTPS'] and $_SERVER['REQUEST_SCHEME'] should be 'on' / https://
> in case of generate absolute URLs like for emails
>
> in a perfect world this would be handeled like the transparent translation
> of the client IP with
> https://httpd.apache.org/docs/current/mod/mod_remoteip.html and it's
> RemoteIPInternalProxy and a header like "X-Forwarded-TLS"
>
> something like below where "X-TLS-Offloading" is only evaluated from
> "RemoteIPInternalProxy" pyhsical addressess
>
> RemoteIPHeader X-Forwarded-For
> RemoteTLSHeaderX-TLS-Offloading
> RemoteIPInternalProxy  192.168.196.1


The presence of TLS is a rather strict contract, in this example it
doesn't exist
end-to-end, so there is a wire segment that is sniffable (and yes you can also
argue that a physical box, given access, is sniffable). This particular example
would require all RemoteIPInternalProxy gateways to correctly de-taint any
previous X-TLS-Offloading value. In other words, many ways to get this wrong
in a day where end-to-end encryption is required by many policies.

I'm hesitant to introduce this into mod_remoteip... the configuration
of absolute
URL's in text/html output would probably follow the same setup as any other
situation where the app server is not physically on the wire, but hiding behind
a gateway. mod_proxy_html or mod_sed in httpd easily handle this for apps
that can't be reconfigured, I'd presume ATS offers a similar feature.

You can float this as a patch to mod_remoteip in bugzilla. It might not be
accepted, but that would be a place for interested users to find such an
example hack and perhaps adopt it.